The Perfect Load-Balanced & High-Availability Web Cluster With 2 Servers Running Xen On Ubuntu 8.04 Hardy Heron - Page 8

Want to support HowtoForge? Become a subscriber!
 
Submitted by marchost (Contact Author) (Forums) on Fri, 2008-10-03 09:17. ::

14. Mirroring web & mail files with rsync (web1, web2)

This is the tricky part. Depending on the type of website(s) you will put on the cluster, the technique used to mirror web files might be different.

We will consider web1.example.com as the master, web2.example.com will sync to web1.example.com

This can create problems in some cases. If you have a website were users can upload files, you dont want them to be uploaded on web2.example.com because the two servers will not be in sync.

In my case, I redirect all page were users can upload data to HTTP port 81 which is only forwarded to one server, web1.example.com. If the master server fails, then all port 81 traffic is forwarded to the fallback server, web2.example.com and mirroring is temporary stopped.

Redirect can be done with in apache vhost.conf :

[...]
Redirect /admin http://www.example.com:81/webmail
[...]

If you have a lot of file upload done by users, you might consider using a load balance NFS :

http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat

 

14.1 Install rsync

apt-get install rsync

 

14.2 Create The Keys On web2.example.com

Now we create the private/public key pair on web2.example.com:

mkdir /root/rsync
ssh-keygen -t dsa -b 1024 -f /root/rsync/mirror-rsync-key

You will see something like this:

Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase): [press enter here]
Enter same passphrase again: [press enter here]
Your identification has been saved in /root/rsync/mirror-rsync-key.
Your public key has been saved in /root/rsync/mirror-rsync-key.pub.
The key fingerprint is:
68:95:35:44:91:f1:45:a4:af:3f:69:2a:ea:c5:4e:d7 root@web2

It is important that you do not enter a passphrase otherwise the mirroring will not work without human interaction so simply hit enter!

Next, we copy our public key to web1.example.com:

scp /root/rsync/mirror-rsync-key.pub vmail@web1.example.com:/home/vmail/

The public key mirror-rsync-key.pub should now be available in /home/vmail on web1.example.com.

 

14.3 Configure web1.example.com

web1.example.com

Now log in through SSH as vmail (not root!) and do this:

login vmail

mkdir ~/.ssh
chmod 700 ~/.ssh
mv ~/mirror-rsync-key.pub ~/.ssh/
cd ~/.ssh
touch authorized_keys
chmod 600 authorized_keys
cat mirror-rsync-key.pub >> authorized_keys

By doing this, we have appended the contents of mirror-rsync-key.pub to the file /home/vmail/.ssh/authorized_keys. /home/vmail/.ssh/authorized_keys should look similar to this:

(Still as webmaster! on web1.example.com)

vi ~/.ssh/authorized_keys

ssh-dss AAAAB3NzaC1kc3MAAA[...]lSUom root@web2

Now we want to allow connections only from web2.example.com, and the connecting user should be allowed to use only rsync, so we add

command="/home/vmail/rsync/checkrsync",from="192.168.0.105",no-port-forwarding,no-X11-forwarding,no-pty

right at the beginning of /home/vmail/.ssh/authorized_keys:

command="/home/vmail/rsync/checkrsync",from="192.168.0.105",no-port-forwarding,no-X11-forwarding,no-pty ssh-dss AAAAB3NzaC1kc3MAAA[...]lSUom root@
web2

Now we create the script /home/vmail/rsync/checkrsync that rejects all commands except rsync on web1.example.com.

(We still do this as vmail!)

mkdir ~/rsync
vi ~/rsync/checkrsync

#!/bin/sh





case "$SSH_ORIGINAL_COMMAND" in
        *\&*)
                echo "Rejected"
                ;;
        *\(*)
                echo "Rejected"
                ;;
        *\{*)
                echo "Rejected"
                ;;
        *\;*)
                echo "Rejected"
                ;;
        *\<*)
                echo "Rejected"
                ;;
        *\`*)
                echo "Rejected"
                ;;
        rsync\ --server*)
                $SSH_ORIGINAL_COMMAND
                ;;
        *)
                echo "Rejected"
                ;;
esac

chmod 700 ~/rsync/checkrsync

 

14.4 Test automated rsync

web2.example.com

First we have to tell rsync which file we dont want to sync (such as munit graph!)

vi /root/exclude_www.txt

and make it look like this :

 /example/web/monit/
 /example/web/monitoring/
 /example/ssl/
 /yoursite/ssl/ 

and now for mail :

vi /root/exclude_mail.txt

/rsync/
/.ssh/
#if you use my monitoring script, see next page
server1_was_down/
.bash_history/
.bash_logout/
.bashrc/
.mysql_history/
.profile/
.viminfo/

Now we must test on web2.example.com if we can mirror web1.example.com without being prompted for vmail's password. We do this:

(We do this as root!)

web2.example.com

rsync -avz --delete --ignore-errors --exclude-from '/root/exclude_www.txt' -e "ssh -i /root/rsync/mirror-rsync-key" vmail@192.168.0.104:/var/www/ /var/www/

followed by :

rsync -avz --delete --ignore-errors --exclude-from '/root/exclude_mail.txt' -e "ssh -i /root/rsync/mirror-rsync-key" vmail@192.168.0.104:/home/vmail/ /home/vmail/

You should now see that the mirroring takes place without being prompted for a password :

receiving file list ... done
sent 71 bytes  received 643 bytes  476.00 bytes/sec
total size is 64657  speedup is 90.56

 

14.5 Create a cron job

First we will create a script that will make sure that if for some reason web1.example.com went down, rsync will be stopped and that root as to restart manually the process.

The reason why we have to do this is because web2.example.com will take over web1.example.com for mail if it went down. New mail will then be received on web2.example.com during that period, so let say web1.example.com comes back online one hour after, all new mail during that period of time would be deleted on web2.example.com because of rsync --delete command. We will see below how to sync back both servers it that happend.

For now lets write that script.

On web2.example.com

vi /root/rsync_web1

#!/bin/bash
# Script to rsync web data and mail between 2 load balanced server
# Copyright (c) 2008 blogama.org
# This script is licensed under GNU GPL version 2.0 or above
# ---------------------------------------------------------------------
### This script has 2 purpose ###
### 1) Check connection on port 25 to master server and then rsync mail ###
### 2) Check connection on port 80 to master server and then rsync www data ###
### To be modified ###
MASTERSERVERIP="192.168.0.104"
WEBPORT="80"
SMTPPORT="25"
SSHPORT="22"
EMAIL="admin@example.com"
WWWRSYNCUSER="vmail"
WWWDIR="/var/www/"
MAILRSYNCUSER="vmail"
MAILDIR="/home/vmail/"

###### Do not make modifications below ######
### Binaries ###
MAIL=$(which mail)
TELNET=$(which telnet)
RSYNC=$(which rsync)
SSH=$(which ssh)
### To restore to original when problem fixed ###
if [ $1 ]; then
  if [ $1=="fix" ]; then
    if [ -f smtp_down.txt ]; then
      rm /root/smtp_down.txt
    fi
    if [ -f www_down.txt ]; then
      rm /root/www_down.txt
    fi
  fi
fi
####################SMTP####################
### If already notified for SMTP problem exit ###
cd /root
if [ -f smtp_down.txt ]; then
  exit 1;
fi
### Check if server 1 is responding on SMTP. If yes rsync mail ###
### If server 1 was down, cannot rsync --delete server 2 with 1 ###
### Must resync server1 with 2 (no delete) and run /root/sync fix ###
(
echo "quit"
) | $TELNET $MASTERSERVERIP $SMTPPORT | grep Connected > /dev/null 2>&1
if [ "$?" -ne "1" ]; then
  $RSYNC -avz --delete --ignore-errors --exclude-from '/root/exclude_mail.txt' -e "ssh -p $SSHPORT -i /root/rsync/mirror-rsync-key" $MAILRSYNCUSER@$MASTERSERVERIP:$MAILDIR $MAILDIR
else
  echo "Server 1 down. Mail sync is not working anymore" > /root/smtp_down.txt
  $MAIL -s "Server 1 down port 25" $EMAIL < /root/smtp_down.txt
fi
####################HTTP####################
### If already notified for HTTP problem exit ###
cd /root
if [ -f http_down.txt ]; then
  exit 1;
fi
### Check if server 1 is responding on HTTP. If yes rsync www data ###
(
echo "quit"
) | $TELNET $MASTERSERVERIP $WEBPORT | grep Connected > /dev/null 2>&1
if [ "$?" -ne "1" ]; then
  $RSYNC -avz --delete --ignore-errors --exclude-from '/root/exclude_www.txt' -e "ssh -p $SSHPORT -i /root/rsync/mirror-rsync-key" $WWWRSYNCUSER@$MASTERSERVERIP:$WWWDIR $WWWDIR
else
  echo "Server 1 down. WWW sync is not working anymore" > /root/http_down.txt
  $MAIL -s "Server 1 down port 80" $EMAIL < /root/http_down.txt
fi

And make this file executable :

chmod +x /root/rsync_web1

Of course you can put whatever you want in those files ;) If you dont put slash it will ignore the keyword recursively (ex : *.gz), if you put slash it is for a precise location (ex : /example/ssl/).

Now we add the script to our crontab by doing the following :

crontab -e

and add this line :

[...]
*/5 * * * * /root/rsync_web1  >/dev/null 2>&1
[...]

This will run it every 5 minutes, of course you can change that to your needs.

 

14.6 What if web1.example.com was down

Generally speaking, if web1.example.com is down, you then have to :

1) rsync WEB1 TO WEB2 without --delete command :

web1.example.com

rsync -avz vmail@192.168.0.104:/var/www/ /var/www/
rsync -avz vmail@192.168.0.104:/home/vmail/ /home/vmail/


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.