Comments on Backing Up With rsync And Managing Previous Versions/History

I have directory called /home/dinkar which i have to backup everyday. I backup this directory in /mnt/dinkar. When I run rsync, it removes or overwrite  deleted or modified files. I created another directory called /mnt/dinkar_old. In this directory i store all removed or updated files. I use following command to achieve this

 rsync -vax --perms --progress --numeric-ids --delete --delete-excluded --exclude '*~' --backup  --backup-dir=/mnt/dinkar_old/ --suffix=.`date +%y%m%d_%H%M` /home/dinkar/ /mnt/dinkar

 This command will move deleted or updated files from /mnt/dinkar and put them in /mnt/dinkar_old and add suffix to file name as date and time.

 I use ubuntu. Not sure if this will work in other distros.

 Hope this information helps somebody.

 Dinkar

Your script is working fine for me. The only problem I am facing is the time stamp is added after the file extension (i.e. test.docx.16_01_16_1923) I am unable to open the file without chaging the extension manually. Can you help me out?

My Code is: rsync -vax -rsa=ssh --perms --progress --numeric-ids --exclude '*~' --backup  --backup-dir='/mnt/Mirror/Rsync_Deleted_files/' --suffix=.`date +%y_%m_%d_%H%M` /media/bhavamayananda/GENSEC\ Backup/ [email protected]:'/mnt/Mirror/GENSEC Backup'

 

First, rsync rocks! It is fantastic when you want to make mirror copies of directory systems. Combined with file system hard linking methods, it is possible to create differential backups.

Or you could just use rdiff-backup. http://rdiff-backup.nongnu.org/ The main options are very similar to rsync. It handles differential backups just fine and stores the latest backup as a mirror and all the prior backups in gzip'd differential files. It also supports remote backups over ssh. There is a win32 version, but I've never been too impressed. I think it works best with samba mounts due to the extra dependences for ssh on Windows.

For example, to backup a $HOME directory to another box,

EXCLUDE=“—exclude-symbolic-links —exclude /.gnupg
TARGET=“romulus::media/Lap-Backup/xubuntu/${LOGNAME}”
/usr/bin/rdiff-backup $EXCLUDE ${HOME}  “$TARGET”
/usr/bin/rdiff-backup —remove-older-than 90D —force “$TARGET”

I prefer to keep only 90 days worth of backups, so any older than 90 days is removed. Perhaps I'm missing something, but this seems much easier to me. In the real script, I have about 20 more "excludes." File permissions are retained in metadata which is also differentially maintained.

The only caveat I have is with very large files (like virtual machine .IMG files). rdiff-backup isn't very good at handling those, IMHO. But, I do use it to backup about 10 VMs nightly. Each entire VM takes about 2 minutes to backup all files differentially. Using rsync, it was taking over 45 minutes, but honestly it wasn't with the differentials.

To restore

/usr/bin/rdiff-backup -r now “$TARGET”   ${HOME}

After we do rdiff-backups, we rsync the resulting backup areas off-site as part of our DR plan. We've had to test the restores a few times - ooops. Everything worked perfectly, with no surprises.

I know rdiff-backup. Also very good, but I missed somehow the

ability to configure the way it's maintaining the history. That's why I've decided to write this very simple tool.

I can make this tool do everything, backup previous version in the way  I want it.

Stef

 

 

rsnapshot could be worth looking at … http://rsnapshot.org/

It is unclear to me if you are getting more from your system than rsnapshot provides, but it does versioned snaphot backups.

Managing a specific number of backups is easy in rdiff-backup:

/usr/bin/rdiff-backup --remove-older-than 90D --force "$TARGET"

 Just to be clear, rdiff and rdiff-backup ARE different tools.

as i have use, all rdif option is rsync option, is it right ?

Good info,

 Im doing something simliar to backup my linux and windows machines, however im probably doing it the worst possible way. Im in the process of reviewing it atm.

 

But basicly I rsync to a "full" backup folder where changes are over written then I create hardlinks of those files to another "snapshot" location. using the bellow command

cp -al source destination

 

It works well but I have to manage my own backup intervals etc.. and the disk IO doing the hardlink snapshot is quite high, esp for the 15 servers im using to backup with it. But it doesn mean I get nice incramental backups.

 

Just gave rdiff a go but it seems it doesnt talk to the rsync servers already setup on my windows machines so I may have to re-evaluate it.

Another option to rdiff is RIBS.  It uses php and rsync to to create incremental backups at the intervals the user configures.

 http://www.rustyparts.com/ribs.php

You have some cool ideas here. But I wonder why you elected not to use hard links with the --link-dest option?

e.g., http://www.backupcentral.com/components/com_mambowiki/index.php/Rsync_snapshots

That method seems to better leverage functionality already built into rsync for incremental backups. Just asking ... I like your ideas too.

Thanks for the awk and sed extraction commandlines for itemized info.

I use rsync to Solaris zfs filesystem.  Then do a zfs snapshot.  All rsync has to do is get the changed files over.  The rsyncd.conf option "post-xfer exec" is perfect for taking the snapshot when rsync is all done movin the changes.

 PEACE

Have you tried rsnapshot - pretty much does this with a simple configuration file.

I believe you have a small typo -- a double quote instead of a single quote:

 grep --invert-match "^.f+++++++++" "/srv/backupsimple/log/localhost/$timestamp.onlyfiles" | 
     awk -F '|' '{print $2 }' | sed 's@^/@@" >>
     "/srv/backupsimple/log/localhost/$timestamp.changed"

There is a double-quote at the end of the sed command, but I believe it should be a single quote, to match the single after the word sed

 

Hello,I am  impressed about this post. Due to my bad English skills I am a little lost.

My plan is to mirroring a folder to a backup folder.

Additionally should the script copy changed and or deleted files to a versions folder.

But i want to keep only 5 or n versions of a changed file. Deleted ones can be kept

 

Which steps do I have to bring together of your writeup? Maybe you can help me a little, please.