How To Manage Your Servers With Rex - Best Practice

(R)?ex is a server orchestration and configuration management tool. With (R)?ex you can manage all your boxes from a central point through the complete process of configuration management and software deployment.

In short words, Rex is like Make. There is a central Rexfile where you can define tasks. These tasks gets executed on remote machines over ssh. The tasks are written in plain perl.

You can get (R)?ex from its website http://rexify.org/.

Preface

In this guide I will use Subversion to manage all the Tasks. You can use any other SCM system as well, as long it supports something similar to Subversion's external directive. I'm using Ubuntu 12.04 but you can use other distributions as well.

I won't append "sudo" on every command, please use "sudo" where it is applicable.

In this guide I will create two example projects. One project is named 'website' and an other called 'database' because in larger companies there is often a split between system administrators and database administrators. Both projects will use the "common tasks" that can be managed for example from a central operations team.

I will use multiple servers:

• Subversion server, svn01
• Database Server, db01
• Web Server, web01
• Workstation, wks01

Set Up Code Repositories

First your have to install all the needed packages. Execute this command on the Subversion server.

svn01# apt-get install libapache2-svn subversion apache2-mpm-prefork

Now edit the file /etc/apache2/mods-enabled/dav_svn.conf and paste the following code into it (replacing existing content).

<Location /svn>
  DAV svn
  SVNParentPath /var/lib/svn
  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile /etc/apache2/dav_svn.passwd
  <LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
  </LimitExcept>
</Location>

Now create the directory /var/lib/svn and all the needed repositories. I will describe the contents of the repositories later.

svn01# mkdir /var/lib/svn
svn01# cd /var/lib/svn
svn01 /var/lib/svn# svnadmin create common
svn01 /var/lib/svn# svnadmin create service
svn01 /var/lib/svn# svnadmin create database
svn01 /var/lib/svn# svnadmin create website
svn01 /var/lib/svn# chown -R www-data: .

After we've created the repositories we need to set up the authentication for apache.

svn01# htpasswd -c /etc/apache2/dav_svn.passwd your-user-name

Now it is time to restart apache.

svn01# service apache2 restart

Congratulations. Your Subversion Server is now ready. Lets head over to your Workstation and checkout the repositories.

Writing The Tasks

At your workstation you can now checkout the repositories.

wks01# svn co http://svn01/svn/common Common
wks01# svn co http://svn01/svn/service Service
wks01# svn co http://svn01/svn/database
wks01# svn co http://svn01/svn/website

First we will add a common task to set up NTP. Here you can add other common tasks later. So change to the Common directory and create a file named NTP.pm.

wks01# cd Common

# Common/NTP.pm
package Common::NTP;
use Rex -base;
task prepare => sub {
   install "ntp";
   file "/etc/ntp.conf",
      source => "files/ntp.conf",
      on_change => sub {
         service ntp => "restart";
      };
};
1;

This creates a task called "prepare". This task gets registered in the "Namespace" NTP. The task will install the package "ntp" if it is not already installed and upload the configuration file to the server. If the content of the file changes it will restart the ntp service. Now, you need to create the ntp.conf file.

bash Common# mkdir files

Paste this to the file files/ntp.conf. This is a simple default ntp.conf file. Of course you can change this to suite your needs :)

# /etc/ntp.conf, managed with rex
driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
server 0.ubuntu.pool.ntp.org
server 1.ubuntu.pool.ntp.org
server 2.ubuntu.pool.ntp.org
server 3.ubuntu.pool.ntp.org
# Use Ubuntu's ntp server as a fallback.
server ntp.ubuntu.com
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

For the beginning this is enough. So now you can add the new files to the repository.

wks01 Common# svn add NTP.pm files
wks01 Common# svn ci -m "added NTP task"

Now we will add something to the service repository.

wks01 Common# cd ../Service
wks01 Service# touch Apache.pm MySQL.pm

Now paste the following code to the Apache.pm Module.

package Service::Apache;
use Rex -base;
task prepare => sub {
   install "apache2";
};
task configure => sub {
   my $param = shift;
   file "/etc/apache2/apache2.conf",
      owner => "root",
      mode => 644,
      content => template("templates/apache2/apache2.conf.tpl", %{ $param });
   file "/etc/apache2/conf.d/security",
      owner => "root",
      mode => 644,
      content => template("templates/apache2/conf.d/security.tpl", %{ $param });
};
1;

And create the templates used by this module.

wks01 Service# mkdir -p templates/apache2/conf.d

The contents of templates/apache2/apache2.conf.tpl. I've stripped all comments.

LockFile /var/run/apache2/accept.lock
PidFile /var/run/apache2.pid
Timeout <%= is_defined($::timeout, "300") %>
KeepAlive <%= is_defined($::keepalive, "On") %>
MaxKeepAliveRequests <%= is_defined($::max_keepalive_requests, "100") %>
KeepAliveTimeout <%= is_defined($::keepalive_timeout, "5") %>
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_event_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
User <%= is_defined($::user, "www-data") %>
Group <%= is_defined($::group, "www-data") %>

AccessFileName .htaccess
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy all
</Files>
DefaultType None

HostnameLookups <%= is_defined($::hostname_lookups, "Off") %>
ErrorLog <%= is_defined($::error_log, "/var/log/apache2/error.log") %>
LogLevel <%= is_defined($::log_level, "warn") %>
Include mods-enabled/*.load
Include mods-enabled/*.conf
Include httpd.conf
Include ports.conf
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Include conf.d/
Include sites-enabled/

The content of templates/apache2/conf.d/security.tpl. I've stripped all comments.

ServerTokens <%= is_defined($::server_tokens, "Prod") %>
ServerSignature <%= is_defined($::server_signature, "Off") %>
TraceEnable <%= is_defined($::trace_enable, "Off") %>

Now we will continue with the MySQL module. Open the file MySQL.pm and add the following content to it.

package Service::MySQL;
use Rex -base;
task prepare => sub {
   install "mysql-server";
};
task configure => sub {
   my $param = shift;
   file "/etc/mysql/my.cnf",
      owner => "root",
      mode => 644,
      content => template("templates/mysql/my.cnf.tpl", %{ $param });
};
1;

And create the file templates/mysql/my.cnf.tpl.

[mysqld]
user     = <%= is_defined($::user, "mysql") %>
pid-file = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port     = <%= is_defined($::port, "3306") %>
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /tmp
lc-messages-dir   = /usr/share/mysql
skip-external-locking
bind-address      = <%= $::eth0_ip %>
key_buffer     = <%= is_defined($::key_buffer, "16M") %>
max_allowed_packet   = <%= is_defined($::max_allowed_packet, "16M") %>
thread_stack      = <%= is_defined($::thread_stack, "192K") %>
thread_cache_size       = <%= is_defined($::thread_cache_size, "8") %>
myisam-recover         = BACKUP
query_cache_limit = <%= is_defined($::query_cache_limit, "1M") %>
query_cache_size        = <%= is_defined($::query_cache_size, "16M") %>
expire_logs_days  = <%= is_defined($::expire_logs_days, "10") %>
max_binlog_size         = <%= is_defined($::max_binlog_size, "100M") %>

[mysqldump]
quick
quote-names
max_allowed_packet   = <%= is_defined($::max_allowed_packet, "16M") %>
[mysql]
[isamchk]
key_buffer     = <%= is_defined($::key_buffer, "16M") %>
!includedir /etc/mysql/conf.d/

Now add all the files to the repository.

wks01 Service# svn add *
wks01 Service# svn ci -m "inital commit of apache and mysql service"

Okay, now you have your first common modules. Now it is time to create the tasks for the database project.