The Perfect Load-Balanced & High-Availability Web Cluster With 2 Servers Running Xen On Ubuntu 8.04 Hardy Heron - Page 8
14. Mirroring web & mail files with rsync (web1, web2)
This is the tricky part. Depending on the type of website(s) you will put on the cluster, the technique used to mirror web files might be different.
We will consider web1.example.com as the master, web2.example.com will sync to web1.example.com
This can create problems in some cases. If you have a website were users can upload files, you dont want them to be uploaded on web2.example.com because the two servers will not be in sync.
In my case, I redirect all page were users can upload data to HTTP port 81 which is only forwarded to one server, web1.example.com. If the master server fails, then all port 81 traffic is forwarded to the fallback server, web2.example.com and mirroring is temporary stopped.
Redirect can be done with in apache vhost.conf :
[...] Redirect /admin http://www.example.com:81/webmail [...]
If you have a lot of file upload done by users, you might consider using a load balance NFS :
https://www.howtoforge.com/high_availability_nfs_drbd_heartbeat
14.1 Install rsync
apt-get install rsync
14.2 Create The Keys On web2.example.com
Now we create the private/public key pair on web2.example.com:
mkdir /root/rsync
ssh-keygen -t dsa -b 1024 -f /root/rsync/mirror-rsync-key
You will see something like this:
Generating public/private dsa key pair. Enter passphrase (empty for no passphrase): [press enter here] Enter same passphrase again: [press enter here] Your identification has been saved in /root/rsync/mirror-rsync-key. Your public key has been saved in /root/rsync/mirror-rsync-key.pub. The key fingerprint is: 68:95:35:44:91:f1:45:a4:af:3f:69:2a:ea:c5:4e:d7 root@web2
It is important that you do not enter a passphrase otherwise the mirroring will not work without human interaction so simply hit enter!
Next, we copy our public key to web1.example.com:
scp /root/rsync/mirror-rsync-key.pub [email protected]:/home/vmail/
The public key mirror-rsync-key.pub should now be available in /home/vmail on web1.example.com.
14.3 Configure web1.example.com
web1.example.com
Now log in through SSH as vmail (not root!) and do this:
login vmail
mkdir ~/.ssh
chmod 700 ~/.ssh
mv ~/mirror-rsync-key.pub ~/.ssh/
cd ~/.ssh
touch authorized_keys
chmod 600 authorized_keys
cat mirror-rsync-key.pub >> authorized_keys
By doing this, we have appended the contents of mirror-rsync-key.pub to the file /home/vmail/.ssh/authorized_keys. /home/vmail/.ssh/authorized_keys should look similar to this:
(Still as webmaster! on web1.example.com)
vi ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAA[...]lSUom root@web2
Now we want to allow connections only from web2.example.com, and the connecting user should be allowed to use only rsync, so we add
command="/home/vmail/rsync/checkrsync",from="192.168.0.105",no-port-forwarding,no-X11-forwarding,no-pty
right at the beginning of /home/vmail/.ssh/authorized_keys:
command="/home/vmail/rsync/checkrsync",from="192.168.0.105",no-port-forwarding,no-X11-forwarding,no-pty ssh-dss AAAAB3NzaC1kc3MAAA[...]lSUom root@ web2
Now we create the script /home/vmail/rsync/checkrsync that rejects all commands except rsync on web1.example.com.
(We still do this as vmail!)
mkdir ~/rsync
vi ~/rsync/checkrsync
#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac
chmod 700 ~/rsync/checkrsync
14.4 Test automated rsync
web2.example.com
First we have to tell rsync which file we dont want to sync (such as munit graph!)
vi /root/exclude_www.txt
and make it look like this :
/example/web/monit/ /example/web/monitoring/ /example/ssl/ /yoursite/ssl/
and now for mail :
vi /root/exclude_mail.txt
/rsync/ /.ssh/ #if you use my monitoring script, see next page server1_was_down/ .bash_history/ .bash_logout/ .bashrc/ .mysql_history/ .profile/ .viminfo/
Now we must test on web2.example.com if we can mirror web1.example.com without being prompted for vmail's password. We do this:
(We do this as root!)
web2.example.com
rsync -avz --delete --ignore-errors --exclude-from '/root/exclude_www.txt' -e "ssh -i /root/rsync/mirror-rsync-key" [email protected]:/var/www/ /var/www/
followed by :
rsync -avz --delete --ignore-errors --exclude-from '/root/exclude_mail.txt' -e "ssh -i /root/rsync/mirror-rsync-key" [email protected]:/home/vmail/ /home/vmail/
You should now see that the mirroring takes place without being prompted for a password :
receiving file list ... done sent 71 bytes received 643 bytes 476.00 bytes/sec total size is 64657 speedup is 90.56
14.5 Create a cron job
First we will create a script that will make sure that if for some reason web1.example.com went down, rsync will be stopped and that root as to restart manually the process.
The reason why we have to do this is because web2.example.com will take over web1.example.com for mail if it went down. New mail will then be received on web2.example.com during that period, so let say web1.example.com comes back online one hour after, all new mail during that period of time would be deleted on web2.example.com because of rsync --delete command. We will see below how to sync back both servers it that happend.
For now lets write that script.
On web2.example.com
vi /root/rsync_web1
#!/bin/bash # Script to rsync web data and mail between 2 load balanced server # Copyright (c) 2008 blogama.org # This script is licensed under GNU GPL version 2.0 or above # --------------------------------------------------------------------- ### This script has 2 purpose ### ### 1) Check connection on port 25 to master server and then rsync mail ### ### 2) Check connection on port 80 to master server and then rsync www data ### ### To be modified ### MASTERSERVERIP="192.168.0.104" WEBPORT="80" SMTPPORT="25" SSHPORT="22" EMAIL="[email protected]" WWWRSYNCUSER="vmail" WWWDIR="/var/www/" MAILRSYNCUSER="vmail" MAILDIR="/home/vmail/" ###### Do not make modifications below ###### ### Binaries ### MAIL=$(which mail) TELNET=$(which telnet) RSYNC=$(which rsync) SSH=$(which ssh) ### To restore to original when problem fixed ### if [ $1 ]; then if [ $1=="fix" ]; then if [ -f smtp_down.txt ]; then rm /root/smtp_down.txt fi if [ -f www_down.txt ]; then rm /root/www_down.txt fi fi fi ####################SMTP#################### ### If already notified for SMTP problem exit ### cd /root if [ -f smtp_down.txt ]; then exit 1; fi ### Check if server 1 is responding on SMTP. If yes rsync mail ### ### If server 1 was down, cannot rsync --delete server 2 with 1 ### ### Must resync server1 with 2 (no delete) and run /root/sync fix ### ( echo "quit" ) | $TELNET $MASTERSERVERIP $SMTPPORT | grep Connected > /dev/null 2>&1 if [ "$?" -ne "1" ]; then $RSYNC -avz --delete --ignore-errors --exclude-from '/root/exclude_mail.txt' -e "ssh -p $SSHPORT -i /root/rsync/mirror-rsync-key" $MAILRSYNCUSER@$MASTERSERVERIP:$MAILDIR $MAILDIR else echo "Server 1 down. Mail sync is not working anymore" > /root/smtp_down.txt $MAIL -s "Server 1 down port 25" $EMAIL < /root/smtp_down.txt fi ####################HTTP#################### ### If already notified for HTTP problem exit ### cd /root if [ -f http_down.txt ]; then exit 1; fi ### Check if server 1 is responding on HTTP. If yes rsync www data ### ( echo "quit" ) | $TELNET $MASTERSERVERIP $WEBPORT | grep Connected > /dev/null 2>&1 if [ "$?" -ne "1" ]; then $RSYNC -avz --delete --ignore-errors --exclude-from '/root/exclude_www.txt' -e "ssh -p $SSHPORT -i /root/rsync/mirror-rsync-key" $WWWRSYNCUSER@$MASTERSERVERIP:$WWWDIR $WWWDIR else echo "Server 1 down. WWW sync is not working anymore" > /root/http_down.txt $MAIL -s "Server 1 down port 80" $EMAIL < /root/http_down.txt fi
And make this file executable :
chmod +x /root/rsync_web1
Of course you can put whatever you want in those files ;) If you dont put slash it will ignore the keyword recursively (ex : *.gz), if you put slash it is for a precise location (ex : /example/ssl/).
Now we add the script to our crontab by doing the following :
crontab -e
and add this line :
[...] */5 * * * * /root/rsync_web1 >/dev/null 2>&1 [...]
This will run it every 5 minutes, of course you can change that to your needs.
14.6 What if web1.example.com was down
Generally speaking, if web1.example.com is down, you then have to :
1) rsync WEB1 TO WEB2 without --delete command :
web1.example.com
rsync -avz [email protected]:/var/www/ /var/www/
rsync -avz [email protected]:/home/vmail/ /home/vmail/
