Backup script ( no question, just script )

Discussion in 'Tips/Tricks/Mods' started by almere, Mar 18, 2013.

  1. almere

    almere ISPConfig Developer ISPConfig Developer

    Hey guys.

    I have made a backup script for my machines. Maybe you will like it :)

    It's making a .gz file for each directory and puts all .gz files to *.tar archive. Than it will transfer the *.tar file to a FTP server.

    I have 3 servers, each is something, like 1.5 TB. For each server it takes 1.5 hours to make full backup and transfer it. I also limited tar and gzip commands for 20% CPU usage, because it loaded my CPU for 99% :D

    As security for my hosting is at first place, i backup almost everything. If you are not SOOOO paranoid, as i am, you should disable directories, like: root, sys, boot, lib, lib64, bin... etc.

    Here it is:
    Code:
    
    #!/bin/sh
    ### System Setup ###
    ### backup directory for temp. file sotrage.
    BACKUP=/backups/tempbackup
    
    ### FTP ###
    ### your FTP server
    FTPD="/"
    FTPU="user"
    FTPP="password"
    FTPS="host.name"
    
    ### Binaries ###
    TAR="$(which tar)"
    GZIP="$(which gzip)"
    FTP="$(which ftp)"
    
    ## Today + hour in 24h format ###
    NOW=$(date +%Y%m%d)
    
    
    ### Create tmp dir ###
    mkdir $BACKUP/$NOW
    ### you can add or delete directories. They will be set to *.gz file
    
    $TAR -cf $BACKUP/$NOW/home.tar /home
    $TAR -cf $BACKUP/$NOW/var.tar /var
    $TAR -cf $BACKUP/$NOW/etc.tar /etc
    $TAR -cf $BACKUP/$NOW/root.tar /root
    $TAR -cf $BACKUP/$NOW/boot.tar /boot
    $TAR -cf $BACKUP/$NOW/opt.tar /opt
    $TAR -cf $BACKUP/$NOW/usr.tar /usr
    $TAR -cf $BACKUP/$NOW/sys.tar /sys
    $TAR -cf $BACKUP/$NOW/sbin.tar /sbin
    $TAR -cf $BACKUP/$NOW/lib.tar /lib
    $TAR -cf $BACKUP/$NOW/lib64.tar /lib64
    $TAR -cf $BACKUP/$NOW/bin.tar /bin
    
    ARCHIVE=$BACKUP/server01-files-$NOW.tar.gz
    ARCHIVED=$BACKUP/$NOW
    
    $TAR -zcvf $ARCHIVE $ARCHIVED
    
    ### ftp ###
    cd $BACKUP
    DUMPFILE=server01-files-$NOW.tar.gz
    $FTP -in $FTPS <<END_SCRIPT
    quote USER $FTPU
    quote PASS $FTPP
    cd $FTPD
    mput $DUMPFILE
    bye
    END_SCRIPT
    
    ### deleting temp files ###
    rm -rf $ARCHIVED
    rm -rf $DUMPFILE
    echo "Backup finished and transferred"
    exit
    
    
    Have fun.

    UPD: see the post below, for full and complete backup structure.
     
    Last edited: Mar 19, 2013
  2. almere

    almere ISPConfig Developer ISPConfig Developer

    So, small update... Now i also want to delete files, that are older than 7 days. Cuz my 8 TB backup server is full :D :D
    Let's start:

    Open server.php (/usr/local/ispconfig/server/server.php)
    find this line
    Code:
    $server_config = $app->getconf->get_server_config($conf['server_id'], 'server');
    (something like 745)

    add befor that line paste the following code (but of course with replacing some variables to your own, like FTP values, count of days etc.)

    Code:
    //delete server backups
    /**
    * Function to delete old backups from a remote FTP server
    * obj $connection - FTP connection object
    * int $ttl - count days to delete backups. If older than $ttl, will be removed
    * $source - directory , where all backups are stored
    * be sure, that you have server id in $conf variable
    */
    function deleteOldFiles($connect, $ttl, $source){
        $files = ftp_nlist($connect, '/');
        if (substr($source, -1, 1) == '/') $source = substr($source, -1);
        
        $files = ftp_nlist($connect, $source);
    
        foreach ($files as $f) {
        	$server_id = $conf['server_id'];
        	if($server_id < 10){$server_id = '0'.$conf['server_id'];}
            if (preg_match('#server'.$server_id.'-files-(.*?).tar.gz#', $f, $m)) {
                $t = $m[2];
                if (date('Ymd') - $t > $ttl) {
                    ftp_delete($connect, $source.'/'.$f);
                }
            }
        }
    }
    
    $server = 'hostname';
    $ftp_user_name = 'username';
    $ftp_user_pass = 'password';
    $mode = FTP_BINARY;//do not touch it
    
    $connection = ftp_connect($server);
    if(!$connection) {@ftp_close($connection); $app->log('Connection attempt failed for the remote FTP backup server!', LOGLEVEL_ERROR); }
    
    ftp_pasv($connection, true); //enable passive mode
    
    $login = ftp_login($connection, $ftp_user_name, $ftp_user_pass);
    if (!$login) { ftp_close($connection); $app->log('Login attempt failed for the remote FTP backup server!', LOGLEVEL_ERROR); }
    
    ftp_pasv($connection, true); //enable passive mode
    
    deleteOldFiles($connection, 7, '/'); // files that are older 7 days
    
    ftp_close($connection);
    $app->log('Deleting old backups from the remote FTP backup server is done!', LOGLEVEL_DEBUG);
    
    Than go to your ssh and run:
    Code:
    crontab -e
    than enter something like:
    Code:
    03 00 * * * /backups/fullserver.sh > /dev/null 2>> /var/log/ispconfig/cron.log
    /backups/fullserver.sh - you should replace with directory and file, where you have the FTP backup script from the previous post.


    Done. Now your server will be fully backup to a remote FTP server and all files, older than N days will be deleted.

    Why do i use bash and php ?
    - I do not know bash that good. For regular expressions etc. Would be grate, if somebody could translate my PHP to bash.


    Good luck & have fun ;)
     
  3. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    I prefere numbered-backups instead of adding the date to each backup.

    Instead of server01-files-$NOW.tar.gz i use something like server01-files-$DOY.tar.gz

    Code:
    DOY1=`date +%j`
    DOY=`expr $DOY1 % 10` # for 10 Backups (0..9).
    
    No need to delete old files, they are just overwritten.
     
  4. almere

    almere ISPConfig Developer ISPConfig Developer

    Thank you for your post :p

    In my case i need to store backups every day and to delete it if it's older, than 7 days. I think it's more readable for the system and for user to have it like i did. Then you have server ID, and a day, when the backup was made. If you will get a problem with your server after an update, i will be easier to restore files with date in the file name , than to guess, from what date was the backup made ( yeah, i know about date in the file info, but it's allways possible that the fil will be rewrited or changed... You never know. ).
     
  5. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    You´re welcome. ;)

    I`m running my script each day, too and have never more than 10 backups in the backup-space.

    Personally I don´t like too much code in a simple backup-script. Instead of delete-and-upload i just like upload-with-overwrite. :)
     
  6. tahunasky

    tahunasky New Member

    I do backups every weekday, and then a weekly backup for 4 weeks, that way i only ever have 8 backup files, but my backups go back a month, and i can even do a monthly backup if needed. Everything gets over-written, so no need to delete anything. and the weekly backups are stored off-site.

    eg:
    backup-daily-monday.tar to backup-daily-thursday.tar, backup-weekly-01.tar to backup-weekly-04.tar

    I also run a script on a backup server, which has a wake-on-bios function. Every morning at 1am the server turns on, runs through the list of other servers to check if the servers are running, if so syncs with it/them (emails/websites/cloud storage area, and copies over the backup file from that night, and turns itself off... if it finds a server that is down, it will configure itself to be the server that is down (changing IP/hostname etc) then emails me to say what has happened. Then i just have to send my mum or sister an email to swap out the downed server with a pre-configured spare or they can swap out a faulty HD with pre-configured spares. I can then start the replacement server (with wake-on-lan) when i am ready and configure remotely, and restore files etc from the backup server which does the syncing.

    By doing this i have only had to do one restore from backup in the last couple of years, and is particularly useful for me as i spend alot of the year traveling and living in other countries, and can be away for home for 9 months or more at a time.
     
  7. Fulike

    Fulike New Member

    incremental backup

    I have downloaded Ioannis Sannos's backup srcipt, after I rewrote.
    My version is weekly incremental for website, and e-mail account,

    My script is in hungarian language :)
    (backup.zip is a gzipped tar)
     

    Attached Files:

Share This Page