Mirror Your Web Site With rsync - Page 2

Falko Timme

About Falko Timme

Falko Timme is an experienced Linux administrator and founder of Timme Hosting, a leading nginx business hosting company in Germany. He is one of the most active authors on HowtoForge since 2005 and one of the core developers of ISPConfig since 2000. He has also contributed to the O'Reilly book "Linux System Administration".

Share this page:

Suggested articles

34 Comment(s)

Add comment

Comments

By: Strong at: 2009-05-13 05:36:09

Hi..thanks for the tutorial.. I have a question. How if SSH use another port instead of 22 on both server ? e.g port 10000 What is the exact command to run rsync with port 10000 ? Thanks for all Strong

By: Anonymous at: 2009-07-28 12:54:28

use -p option.

eg:

rsync --delete-during -zaOH -e 'ssh -2 -i <PrivateKey> -p 10000' /source_folder/ <destination ip>:/<destination_folder>/
 

By: at: 2011-11-20 16:59:34

Hi, 

 but you run this command on the mirror server?

 on the mirror server this command not work:

 rsync -avz -e 'ssh -p 49150' --delete "ssh -i /root/rsync/mirror-rsync-key" [email protected]:/var/www/ /var/www/

 error:

Unexpected remote arg: [email protected]:/var/www/

rsync error: syntax or usage error (code 1) at main.c(1222) [sender=3.0.7]

 You see the sintax error?

 Thanks-you in advance for the attention.

 Regards.

 

By: Anonymous at: 2006-04-21 19:03:50

Three things:

1.) rsync does only transfer changed files but not even the whole file... but more than just the changes. rSync has some algorithm that splits up the file in multiple sections and then creates a checksum for it and compares it and only the changed parts will then be transmitted.
For example I had a mysql dump of 270 MB. I deleted it and made a new dump with a few changes. Now rsync noted that the file was changed but it didn't transmit the whole 270 MB again but only 25 MB.

2.) Instead of using --exclude I would rather use --exclude-from="/path/to/file" because I think it's much simpler to add there exclusions. Just add one pattern per line. I have for example this here:

/backup/
/bin/
/dev/
/initrd/
/lib/
/lost+found/
/mnt/
/opt/
/proc/
/sbin/
/sys/
/tmp/
/usr/
/var/log/
/var/cache/
/var/spool/
/var/lib/mysql/

I know, I still need to fine-tune that a bit.
3.) I would also add --delete-excluded for the simple reason that when you exclude something from being backuped then you don't want have older versions on the backup server any longer. This switch takes care of that.

By: Anonymous at: 2006-05-22 17:38:47

Nice post and there is another way to do this also. I just implemented the same functionality for our main website by using rdist. The good thing about this is that it checks timestamp automatically and if the files have been modified it will replicate it accordingly. Put it in cron and you don't have to worry about anything else and let it email you stating which files have changed.

By: at: 2008-04-28 01:46:34

If you use the flag -W rsync will copy the entire file not just the blocks it thinks are modified. http://www.oreillynet.com/linux/cmd/cmd.csp?path=r/rsync

By: Anonymous at: 2006-04-23 23:09:59

According to the suggested setup if something goes wrong to the contents of the server1.example.com:/var/www/ the mess would be propagated to the mirror server too. On the other hand, as you say you are offering just a plain rsync over ssh mirroring solution not a bullet proof backup solution..

By: Anonymous at: 2006-04-27 08:07:50

Well, if you run rsync like that then doing incremental backups isn't all that difficult again. This here was the base for my altered script:
http://www.mikerubel.org/computers/rsync_snapshots/
It uses hardlinks. Well, I run the thing as root because I want to keep permissions. Here's my backup.sh
#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH

# suggestion from H. Milz: avoid accidental use of $PATH


# Make MySQL Backups
#!/bin/bash
# Remove old files
rm -f /mysql_backup/*

#Dump new files
USER=root
PASSWORD=************
HOST=localhost

for i in $(echo 'SHOW DATABASES;' | mysql -u$USER -p$PASSWORD -h$HOST|grep -v '^Database$'); do
mysqldump \
-u$USER -p$PASSWORD -h$HOST \
-Q -c -C --add-drop-table --add-locks --quick --lock-tables \
$i > /mysql_backup/$i.sql;
done;


# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;
SSH=/usr/bin/ssh
KEY=/root/.ssh/id_rsa

# ------------- file locations -----------------------------------------

SNAPSHOT_RW=/backup/backup;
EXCLUDES=/backup/backup_exclude;

# ------------- the script itself --------------------------------------

# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/hourly.3 ] ; then\
$RM -Rf $SNAPSHOT_RW//hourly.3 ;\
fi;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/hourly.2 ] ;then\
$MV $SNAPSHOT_RW/hourly.2 $SNAPSHOT_RW/hourly.3 ;\
fi;

if [ -d $SNAPSHOT_RW/hourly.1 ] ; then\
$MV $SNAPSHOT_RW/hourly.1 $SNAPSHOT_RW/hourly.2 ;\
fi;

# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/hourly.0 ] ; then\
$CP -al $SNAPSHOT_RW/hourly.0 $SNAPSHOT_RW/hourly.1 ;\
fi;

# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first. If it were not so, this would copy over the other
# snapshot(s) too!

$RSYNC\
-avz --delete --delete-excluded \
--exclude-from="$EXCLUDES"\
-e "$SSH -i $KEY" \
[email protected]:/ $SNAPSHOT_RW/hourly.0 ;

# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/hourly.0 ;


I run this script 4 times daily through cron. Then I have another script which makes daily snapshots for 7 days (backup_daily.sh):

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH

# suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;
SSH=/usr/bin/ssh
KEY=/root/.ssh/id_rsa

# ------------- file locations -----------------------------------------

SNAPSHOT_RW=/backup/backup;
EXCLUDES=/backup/backup_exclude;

# ------------- the script itself --------------------------------------

# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/daily.6 ] ; then\
$RM -Rf $SNAPSHOT_RW/daily.6 ;\
fi;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/daily.5 ] ; then\
$MV $SNAPSHOT_RW/daily.5 $SNAPSHOT_RW/daily.6 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.4 ] ; then\
$MV $SNAPSHOT_RW/daily.4 $SNAPSHOT_RW/daily.5 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.3 ] ; then\
$MV $SNAPSHOT_RW/daily.3 $SNAPSHOT_RW/daily.4 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.2 ] ; then\
$MV $SNAPSHOT_RW/daily.2 $SNAPSHOT_RW/daily.3 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.1 ] ; then\
$MV $SNAPSHOT_RW/daily.1 $SNAPSHOT_RW/daily.2 ;\
fi;

if [ -d $SNAPSHOT_RW/daily.0 ] ; then\
$MV $SNAPSHOT_RW/daily.0 $SNAPSHOT_RW/daily.1 ;\
fi;


# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/hourly.3 ] ; then\
$CP -al $SNAPSHOT_RW/hourly.3 $SNAPSHOT_RW/daily.0 ;\
fi;

# step 4: update the mtime of daily.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/daily.0 ;


And finally I have a weekly script that makes weekly snapshots during a 4-week period (backup_weekly.sh):

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH

# suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;
SSH=/usr/bin/ssh
KEY=/root/.ssh/id_rsa

# ------------- file locations -----------------------------------------

SNAPSHOT_RW=/backup/backup;
EXCLUDES=/backup/backup_exclude;

# ------------- the script itself --------------------------------------

# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/weekly.3 ] ; then\
$RM -Rf $SNAPSHOT_RW/weekly.3 ;\
fi;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/weekly.2 ] ; then\
$MV $SNAPSHOT_RW/weekly.2 $SNAPSHOT_RW/weekly.3 ;\
fi;

if [ -d $SNAPSHOT_RW/weekly.1 ] ; then\
$MV $SNAPSHOT_RW/weekly.1 $SNAPSHOT_RW/weekly.2 ;\
fi;

if [ -d $SNAPSHOT_RW/weekly.0 ] ; then\
$MV $SNAPSHOT_RW/weekly.0 $SNAPSHOT_RW/weekly.1 ;\
fi;


# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/daily.6 ] ; then\
$CP -al $SNAPSHOT_RW/daily.6 $SNAPSHOT_RW/weekly.0 ;\
fi;

# step 4: update the mtime of weekly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/weekly.0 ;



And of course you also need to run crons ^^ # Make Backups
0 0 * * Sun sh /backup/backup_weekly.sh
15 0 * * * sh /backup/backup_daily.sh
45 0,6,12,18 * * * sh /backup/backup.sh



With these scripts there is just a small issue. You need to create first a manual daily.0 and weekly.0 folder :)

By: Anonymous at: 2006-05-22 21:31:56

I understand Rsync has a windows plugin. Does anyone know how to rsync a /Inetpub/ directory to a linux server /backup/ folder?

By: Anonymous at: 2006-06-13 01:32:44

If things work up until the last rsync using the public/private key pair, and you're having problems, use the ssh -v switch:


rsync -avz --delete --exclude=**/stats --exclude=**/error --exclude=**/files/pictures -e "ssh -v -i /root/rsync/mirror-rsync-key" [email protected]:/var/www/ /var/www/

By: Anonymous at: 2010-08-17 22:41:27

ssh -v shows that public key fails, even though it is recognized!!! Any ideas?? [email protected] [/home/chlngday]# /usr/bin/rsync -aqzu --exclude 'video/' --exclude 'access-logs/' -e "ssh -v -i /root/rsync/mirror-rsync-key" [email protected]:/home/chlngday/public_html/ /home/chlngday/public_html/ OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to server.spameater.com [72.44.80.21] port 22. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/rsync/mirror-rsync-key type 2 debug1: Remote protocol version 1.99, remote software version OpenSSH_3.9p1 debug1: match: OpenSSH_3.9p1 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_3.9p1 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-cbc hmac-md5 none debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host 'server.spameater.com' is known and matches the RSA host key. debug1: Found key in /root/.ssh/known_hosts:3 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug1: Next authentication method: gssapi-with-mic debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug1: Next authentication method: publickey debug1: Offering public key: /root/rsync/mirror-rsync-key debug1: Authentications that can continue: publickey,gssapi-with-mic,password debug1: Next authentication method: password [email protected]'s password:

By: Anonymous at: 2009-05-15 13:42:11

thks for the details

By: at: 2007-08-28 14:24:50

I am not sure how important this is but you are using archive mode which preserves file ownership.

You should have the same accounts on both servers if you have some directories on the source server that are owned by different accounts.  I have a file repository that only one specific user has write access to (not apache).  I have to create that owner on the target machine before running this script.  One thing I am not sure about is if the UID has to be identical

 

 

By: Anonymous at: 2008-12-04 14:21:29

the --numeric-ids parameter to rsync is for this purpose. No need to create accounts.

By: at: 2008-04-16 18:14:34

i have been spending several hours to make this thing automatic but it keeps asking for password.

but finally i found the reason of my failure. I forgot a COMA!

"... no-port-forwarding,no-X11-forwarding,no-pty, ssh-dss AAAAB3NzaC1kc3MAAAEBALGZJ34a5QwC2 .... " 

 please update your tutorial for the sake of other newbies out there...

anyway, this howto is very helpful. thanks 

By: at: 2008-08-04 16:42:01

What finally worked for me was a new-line instead of a comma.  Also, if you use nano - be careful to switch off long-line wrapping (M-L).

By: xLinux4Lifex at: 2009-08-26 03:52:01

Same here.  I also needed a new-line instead of comma.  My OS is SLES11.

By: Anonymous at: 2011-06-23 17:41:44

I have tried everything to get rid of the password.  I have tried comma, new line, every iteration above.  Still no luck.  Below is the output from the verbose command.  Any ideas?

[email protected]:~$ sudo rsync -avz --delete --exclude=**/stats --exclude=**/error --exclude=**/files/pictures -e "ssh -v -i /root/rsync/mirror-rsync-key" [email protected]:/var/www/ /var/www/
OpenSSH_5.3p1 Debian-3ubuntu6, OpenSSL 0.9.8k 25 Mar 2009
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to LS1 [192.168.0.103] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/rsync/mirror-rsync-key type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.7p1 Debian-8ubuntu1.2
debug1: match: OpenSSH_4.7p1 Debian-8ubuntu1.2 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3p1 Debian-3ubuntu6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'ls1' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:3
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /root/rsync/mirror-rsync-key
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: password
[email protected]'s password:
debug1: Authentication succeeded (password).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.utf8
debug1: Sending command: rsync --server --sender -vlogDtprze.iLsf . /var/www/
receiving file list ... done

sent 71 bytes  received 86 bytes  13.65 bytes/sec
total size is 45  speedup is 0.29
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2272, received 2248 bytes, in 0.0 seconds
Bytes per second: sent 254907.1, received 252214.4
debug1: Exit status 0

By: Anonymous at: 2009-03-24 16:26:32

I have added the comma and I still have the issue with it requesting a password.  What am I doing wrong?

By: Diego at: 2008-09-11 14:34:48

Newbie question: How about if the main server goes down, how could I make them switch over automatically to the mirror server?

By: Anonymous at: 2008-09-15 05:52:10

You can do that easily by creating a high-availability load balancer that uses haproxy/hearbeat. I've set up my web server cluster using these instructions:

http://www.howtoforge.com/high-availability-load-balancer-haproxy-heartbeat-debian-etch

It's pretty straightforward and works like a charm.

By: jed at: 2009-01-13 15:19:27

server_A: Red hat

server_B: Window 2003 (cygwin)

i was on server_B running this command: ssh -i /root/rsync/mirror-rsync-key [email protected]_A:/var/www/ /var/www/

all files on server_A to be transferred on server_B /var/www/

I got this error: Connection closed by server_A rsync: connection unexpectedly closed (0 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/io.c(632) [receiver=3.0.4]

I've already setup ssh on cygwin http://ist.uwaterloo.ca/~kscully/CygwinSSHD_W2K3.html.

Thanks in advance.

By: Anonymous at: 2009-01-14 05:38:06

I after few hours of tracing the file mirroring is already working. I found out that the error occurs when rsync encounters a unpermitted folders, files etc. that can't be transferred.

By: Michael Potter at: 2009-03-02 15:26:57

Thanks for the great how-to!

By: Anonymous at: 2009-07-28 13:00:09

I think you typo in the instruction step 5 (configure server1.example.com) ==> I think this is meant to be configure mirror.example.com as you want to configure remote authorized files

By: Anthony at: 2010-01-15 21:55:37

You can also avoid generating a rule structure by using shorewall for Linux. You'd just setup a rule to allow rsync requests by specific target IP addresses. Personally, it made it a whole lot easier for my setup.

By: Gell at: 2010-04-08 04:49:10

Okay Guys,,First i want to thank for such a great efforts. I spent the whole night googling and eventually i figured out this is the best tutorial ever though i ignored it when i first saw it.

Okay I just want you to be careful about one thing that really wa a pain for me until i figured it out. As you ssh from mirror into the server please use the domain name you identified first and not the ip. e.g i was doing like this on the mirror "ssh [email protected]" where 10.10.10.1 when i try to follow the steps above ssh-ing from mirror into server1 which is not correct. You should do "ssh [email protected]" to copy the right key on the mirror based on the domain name.

 

One more thing to add i used a very fiirst new line and not comma (i am using Centos 5.4)

Regards

 

By: Anonymous at: 2010-06-25 13:29:22

Hi,

What if "/var/www" size is in GBs. will cron in every 5 minute would  still work fine.

 Thanks,

 Pankaj Sain.

By: Pascal Stetenfeld at: 2010-10-23 20:39:10

wonderfull it's work very nice

thanks

By: Chan Myay Aung at: 2011-03-09 08:29:32

I had the problem with rsa key. I  finished all steps above but server is being prompt for me. I tried to restart ssh with sudo /etc/init.d/ssh restart and it was going well... Thank for amazing post.


By: Harry at: 2011-07-06 08:52:46

Excellent work, thanks for a really easy to follow article. Worked first pop !!!

By: burak at: 2011-10-06 06:10:39

That is really great. Thanks a lot...

By: Anonymous at: 2012-11-06 15:20:53

And what about to run rsync as a daemon? In this way, you can set up a daemon and let the clients to connect in read-only mode. More safety...

By: Azmi at: 2014-04-04 01:01:56

I like your tutorial. Easy to followed