Backup script ( no question, just script )

Discussion in 'Tips/Tricks/Mods' started by almere, Mar 18, 2013.

  1. almere

    almere HowtoForge Supporter

    Hey guys.

    I have made a backup script for my machines. Maybe you will like it :)

    It's making a .gz file for each directory and puts all .gz files to *.tar archive. Than it will transfer the *.tar file to a FTP server.

    I have 3 servers, each is something, like 1.5 TB. For each server it takes 1.5 hours to make full backup and transfer it. I also limited tar and gzip commands for 20% CPU usage, because it loaded my CPU for 99% :D

    As security for my hosting is at first place, i backup almost everything. If you are not SOOOO paranoid, as i am, you should disable directories, like: root, sys, boot, lib, lib64, bin... etc.

    Here it is:
    Code:
    
    #!/bin/sh
    ### System Setup ###
    ### backup directory for temp. file sotrage.
    BACKUP=/backups/tempbackup
    
    ### FTP ###
    ### your FTP server
    FTPD="/"
    FTPU="user"
    FTPP="password"
    FTPS="host.name"
    
    ### Binaries ###
    TAR="$(which tar)"
    GZIP="$(which gzip)"
    FTP="$(which ftp)"
    
    ## Today + hour in 24h format ###
    NOW=$(date +%Y%m%d)
    
    
    ### Create tmp dir ###
    mkdir $BACKUP/$NOW
    ### you can add or delete directories. They will be set to *.gz file
    
    $TAR -cf $BACKUP/$NOW/home.tar /home
    $TAR -cf $BACKUP/$NOW/var.tar /var
    $TAR -cf $BACKUP/$NOW/etc.tar /etc
    $TAR -cf $BACKUP/$NOW/root.tar /root
    $TAR -cf $BACKUP/$NOW/boot.tar /boot
    $TAR -cf $BACKUP/$NOW/opt.tar /opt
    $TAR -cf $BACKUP/$NOW/usr.tar /usr
    $TAR -cf $BACKUP/$NOW/sys.tar /sys
    $TAR -cf $BACKUP/$NOW/sbin.tar /sbin
    $TAR -cf $BACKUP/$NOW/lib.tar /lib
    $TAR -cf $BACKUP/$NOW/lib64.tar /lib64
    $TAR -cf $BACKUP/$NOW/bin.tar /bin
    
    ARCHIVE=$BACKUP/server01-files-$NOW.tar.gz
    ARCHIVED=$BACKUP/$NOW
    
    $TAR -zcvf $ARCHIVE $ARCHIVED
    
    ### ftp ###
    cd $BACKUP
    DUMPFILE=server01-files-$NOW.tar.gz
    $FTP -in $FTPS <<END_SCRIPT
    quote USER $FTPU
    quote PASS $FTPP
    cd $FTPD
    mput $DUMPFILE
    bye
    END_SCRIPT
    
    ### deleting temp files ###
    rm -rf $ARCHIVED
    rm -rf $DUMPFILE
    echo "Backup finished and transferred"
    exit
    
    
    Have fun.

    UPD: see the post below, for full and complete backup structure.
     
    Last edited: Mar 19, 2013
  2. almere

    almere HowtoForge Supporter

    So, small update... Now i also want to delete files, that are older than 7 days. Cuz my 8 TB backup server is full :D :D
    Let's start:

    Open server.php (/usr/local/ispconfig/server/server.php)
    find this line
    Code:
    $server_config = $app->getconf->get_server_config($conf['server_id'], 'server');
    (something like 745)

    add befor that line paste the following code (but of course with replacing some variables to your own, like FTP values, count of days etc.)

    Code:
    //delete server backups
    /**
    * Function to delete old backups from a remote FTP server
    * obj $connection - FTP connection object
    * int $ttl - count days to delete backups. If older than $ttl, will be removed
    * $source - directory , where all backups are stored
    * be sure, that you have server id in $conf variable
    */
    function deleteOldFiles($connect, $ttl, $source){
        $files = ftp_nlist($connect, '/');
        if (substr($source, -1, 1) == '/') $source = substr($source, -1);
        
        $files = ftp_nlist($connect, $source);
    
        foreach ($files as $f) {
        	$server_id = $conf['server_id'];
        	if($server_id < 10){$server_id = '0'.$conf['server_id'];}
            if (preg_match('#server'.$server_id.'-files-(.*?).tar.gz#', $f, $m)) {
                $t = $m[2];
                if (date('Ymd') - $t > $ttl) {
                    ftp_delete($connect, $source.'/'.$f);
                }
            }
        }
    }
    
    $server = 'hostname';
    $ftp_user_name = 'username';
    $ftp_user_pass = 'password';
    $mode = FTP_BINARY;//do not touch it
    
    $connection = ftp_connect($server);
    if(!$connection) {@ftp_close($connection); $app->log('Connection attempt failed for the remote FTP backup server!', LOGLEVEL_ERROR); }
    
    ftp_pasv($connection, true); //enable passive mode
    
    $login = ftp_login($connection, $ftp_user_name, $ftp_user_pass);
    if (!$login) { ftp_close($connection); $app->log('Login attempt failed for the remote FTP backup server!', LOGLEVEL_ERROR); }
    
    ftp_pasv($connection, true); //enable passive mode
    
    deleteOldFiles($connection, 7, '/'); // files that are older 7 days
    
    ftp_close($connection);
    $app->log('Deleting old backups from the remote FTP backup server is done!', LOGLEVEL_DEBUG);
    
    Than go to your ssh and run:
    Code:
    crontab -e
    than enter something like:
    Code:
    03 00 * * * /backups/fullserver.sh > /dev/null 2>> /var/log/ispconfig/cron.log
    /backups/fullserver.sh - you should replace with directory and file, where you have the FTP backup script from the previous post.


    Done. Now your server will be fully backup to a remote FTP server and all files, older than N days will be deleted.

    Why do i use bash and php ?
    - I do not know bash that good. For regular expressions etc. Would be grate, if somebody could translate my PHP to bash.


    Good luck & have fun ;)
     
  3. florian030

    florian030 HowtoForge Supporter

    I prefere numbered-backups instead of adding the date to each backup.

    Instead of server01-files-$NOW.tar.gz i use something like server01-files-$DOY.tar.gz

    Code:
    DOY1=`date +%j`
    DOY=`expr $DOY1 % 10` # for 10 Backups (0..9).
    
    No need to delete old files, they are just overwritten.
     
  4. almere

    almere HowtoForge Supporter

    Thank you for your post :p

    In my case i need to store backups every day and to delete it if it's older, than 7 days. I think it's more readable for the system and for user to have it like i did. Then you have server ID, and a day, when the backup was made. If you will get a problem with your server after an update, i will be easier to restore files with date in the file name , than to guess, from what date was the backup made ( yeah, i know about date in the file info, but it's allways possible that the fil will be rewrited or changed... You never know. ).
     
  5. florian030

    florian030 HowtoForge Supporter

    You´re welcome. ;)

    I`m running my script each day, too and have never more than 10 backups in the backup-space.

    Personally I don´t like too much code in a simple backup-script. Instead of delete-and-upload i just like upload-with-overwrite. :)
     
  6. tahunasky

    tahunasky New Member

    I do backups every weekday, and then a weekly backup for 4 weeks, that way i only ever have 8 backup files, but my backups go back a month, and i can even do a monthly backup if needed. Everything gets over-written, so no need to delete anything. and the weekly backups are stored off-site.

    eg:
    backup-daily-monday.tar to backup-daily-thursday.tar, backup-weekly-01.tar to backup-weekly-04.tar

    I also run a script on a backup server, which has a wake-on-bios function. Every morning at 1am the server turns on, runs through the list of other servers to check if the servers are running, if so syncs with it/them (emails/websites/cloud storage area, and copies over the backup file from that night, and turns itself off... if it finds a server that is down, it will configure itself to be the server that is down (changing IP/hostname etc) then emails me to say what has happened. Then i just have to send my mum or sister an email to swap out the downed server with a pre-configured spare or they can swap out a faulty HD with pre-configured spares. I can then start the replacement server (with wake-on-lan) when i am ready and configure remotely, and restore files etc from the backup server which does the syncing.

    By doing this i have only had to do one restore from backup in the last couple of years, and is particularly useful for me as i spend alot of the year traveling and living in other countries, and can be away for home for 9 months or more at a time.
     
  7. Fulike

    Fulike New Member

    incremental backup

    I have downloaded Ioannis Sannos's backup srcipt, after I rewrote.
    My version is weekly incremental for website, and e-mail account,

    My script is in hungarian language :)
    (backup.zip is a gzipped tar)
     

    Attached Files:

Share This Page