ISPConfig 3 - back-res a Backup and Restore script

Discussion in 'Plugins/Modules/Addons' started by go0ogl3, Dec 6, 2009.

  1. Mannekino

    Mannekino New Member

    Few errors while executing the back-res script

    Hello,

    I'm trying to use the back-res script. I'm getting a few errors and it doesn't work for me. I'm using ISPConfig 3 on Ubuntu.

    First I got this:

    Code:
    mysqldump: Got error: 1044: Access denied for user 'root'@'localhost' to database 'information_schema' when using LOCK TABLES
    I checked the rights and the root user has all rights globally, also for LOCK TABLES. I added "--single-transaction" to the mysqldump link in the script. But I find the error message strange because all the necessary rights are provided. I also added LOCK TABLES specifically for the information_schema database but I got the same message.

    After adding the single transaction line the script would continue but then I got this message.

    Code:
    /bin/tar: /root.: Cannot stat: No such file or directory
    /bin/tar: Exiting with failure status due to previous errors
    Any thoughts? It's a pretty basic ISPConfig setup. Only a few domains for private use. I used the HowtoForge guido for installing ISPConfig 3 on Ubuntu.
     
  2. clttech

    clttech HowtoForge Supporter

    update

    I was able to get the backup-res to restore the directories. I had to mkdir /tmp/tmpbck I am running the script as root. Don't know why it will not create the /tmp directories.
    Now when I try to run as root ./backup-res db all 2011-04-01 this is the error I get

    date: invalid date `11/'
    ./backup-res: line 448: [: : integer expression expected

    here is line 448
    if [ "$RD" -gt "$ID" ] ; then

    Restoring all mysql databases from date 2011-04-01 to local server:
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    mysql: unknown option '-x'
    All restore jobs done!
    Database all restored to date 2011-04-01!
    ./backup-res: line 573: /tmp/tmpbck/maildata: No such file or directory
    The data bases are not restored.

    Thanks so much for you help and patient with me and the others.:)
     
  3. go0ogl3

    go0ogl3 New Member

    Some errors

    Hello,

    - for the first error please check your mysql version, use latest available for Ubuntu. Maybe this can help: http://bugs.mysql.com/bug.php?id=21527
    For what I know there is no need to modify the database dumps lines in the script... I will make some checks later this weekend if I'll have some spare time.

    - for the second error please check your dirs you want to backup, you have some dot "." in the message and the script complains about not finding the "/root." directory, not the "/root"...

    If you have more problems, or you can figure out a way to get rid of the mysql error please return here so your message helps others too.


     
  4. go0ogl3

    go0ogl3 New Member

    Date problems

    Hello,

    I think there is a date formatting problem on your system...

    Can you run the next commands and post the output here:

    Code:
    date +%d 
    date +%F
    date +%Y-%m
    The output should be for example:
    Code:
    user@main:~$ date +%d
    15
    user@main:~$ date +%F
    2011-04-15
    user@main:~$ date +%Y-%m
    2011-04
    gabi@main:~$
    The last mysql errors are because of the date problems.

    With the tempdir, I don't know why you get the error, there is the "check_tempdir" function which should create the tempdir if it's not there already and it's only failling if the /tmp dir does not exist (it's not a dir). I can put the "-p" option to mkdir but I don't know a linux computer whitout /tmp... so it's like a security option... I'm open to suggestions :)

     
  5. clttech

    clttech HowtoForge Supporter

    Posting the date code


    Here is the results
    root@c3po:~# date +%d
    15
    root@c3po:~# date +%F
    2011-04-15
    root@c3po:~# date +%Y-%m
    2011-04

    Here is the result of the ls -l of /tmp

    drwxrwxrwx 6 root root 4096 2011-04-15 14:35 tmp
     
  6. go0ogl3

    go0ogl3 New Member

    Hello,

    The date results seems ok, but your temp dir does not have the sticky bit set (permissions like drwxrwxrwxt). I'm not sure this can influence the script so much...

    On a second review of your errors...
    - do you have any dir with "-" in the name, excepting the date? This can be a problem because the date of the backup archives is extracted from their names. (Please read the detailed description)
    - for the mysql error - do you have a line in your script identical with the one below?
    Code:
    mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i
    If you do then the error is because $TAR variable is empty, which means the script can't find tar on your system (which tar)
    If you don't have the line identical with the one above please get the latest version of the script.


     
  7. clttech

    clttech HowtoForge Supporter

    More info on error

    I changed the sticky bit this morning
    drwxrwxrwt 5 root root 4096 2011-04-17 08:01
    still get the error

    touch: cannot touch `/tmp/tmpbck/datestart': No such file or directory
    touch: cannot touch `/tmp/tmpbck/dateend': No such file or directory
    find: `/tmp/tmpbck/datestart': No such file or directory
    find: `/tmp/tmpbck/datestart': No such file or directory
    find: `/tmp/tmpbck/dateend': No such file or directory

    When I tried to do the ./backup-res db all 2011-04-01
    I get this
    date: invalid date `11/'
    ./backup-res: line 448: [: : integer expression expected
    this is line 448
    if [ "$RD" -gt "$ID" ] ; then

    I am not sure what the `11 is all about
    I will post on some of the other sites about the making and deleting of the /tmp/tmpbck and see what I can find

    Here is the line in my backup-res
    mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i

    here is the only part of the script I made changes to and I do have the latest

    ###############################
    ### Begin variables section ###
    ###############################

    # Change the variables below to fit your computer/backup

    COMPUTER=`cat /etc/hostname | awk 'NR==1{print $1}'` # name of this computer
    DIRECTORIES="/var/lib/mysql /var/www
    /var/vmail /etc" # directories to backup
    EXCLUDED="/bck /tmp /dev /proc /sys /srv /media
    /var/adm /var/cache /var/run /var/lock /lib/init/rw /var/tmp
    /var/log/verlihub /var/lib/amavis /var/amavis /var/spool/postfix/p*
    /var/spool/postfix/var *.pid *.lock *.lck" # exclude those dir's and files
    BACKUPDIR="/bck/$COMPUTER" # where to store the backups
    dbuser="root" # database user
    dbpassword="xxxxxxxxxxxx" # database password
    email="myemail@my.com" # mail for the responsible person
    TAR=`which tar` # name and location of tar
    ARG="-cjpSPf" #sparse # tar arguments P = removed /.
    EARG="-xjpf" # tar extract arguments P = removed /
    tmpdir="/tmp/tmpbck" # temp dir for database dump and other stuff
    del_en="yes" # Enable delete of files if used space percent > than $maxp (yes or anything else)
    maxp="85" # Max percent of used space before start of delete
    hm="2359" # last minute of the day = last minute of the restored backup of the day restored

    ###################################
    ### End user editable variables ###
    ###################################

    as for the "-" in the file names, in /var/lib/mysql there are not dir or files with that in it they all have "_". In the /var/www folder there are folders with that "-" in it, but those files restored. in the /var/www there are php-cgi-scripts and php-fcgi-scripts that I gues were made by ISPconfig 3 install...

    Thanks again for your help...is there anything else you can think of I can get that may help to solve the issue.
     
  8. go0ogl3

    go0ogl3 New Member

    Dirty fixes

    Hello,

    a quick fix for the tmpdir is to change the line:
    Code:
    mkdir $tmpdir
    in
    Code:
    mkdir -p $tmpdir
    If the database names contains the "-" or the "_" characters is another thing wich can cause problems. If the date from the filename for the databases is not in form 2011-04-17 there is another point of failure.

    But still the error from the 448 line is because $RD it's not an integer. RD should be the first backup date calculated in epoch,
    Code:
    date --date "$resdate" +%s
    So if all of the date parts are ok (previous commands you run), why RD is not? Maybe $redate is not what it should be... And $resdate
    is obtained from those lines:
    Code:
    year=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 2`
    md=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 3`
    day=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 4 | cut -d "." -f 1`
    resdate=$year$md$day
    
    So the problem it's because something strage happens when we extract the date parts from existing files.
    I will list here the "debug procedure", hope we'll be able to solve this issue now:
    - We can start with the $BACKUPDIR which is composed from the /etc/hostname file.
    - If that is correct we can move to the next step,
    Code:
    ls -ctF $BACKUPDIR
    command which you can run in the comand line, like
    Code:
    ls -ctF /bck/your_hostname
    where your_hostname is the one from the /etc/hostname
    - If that list seems like:
    Code:
    2011-04/                     full_var-2011-04-15.tar.bz2  full_sbin-2011-04-15.tar.bz2   full_lib64-2011-04-15.tar.bz2  full_home-2011-04-15.tar.bz2  full_boot-2011-04-15.tar.bz2  log/
    full_www-2011-04-15.tar.bz2  full_usr-2011-04-15.tar.bz2  full_root.-2011-04-15.tar.bz2  full_lib-2011-04-15.tar.bz2    full_etc-2011-04-15.tar.bz2   full_bin-2011-04-15.tar.bz2
    
    we can move to the next step:
    - the first pipe
    Code:
    ls -ctF /bck/your_hostname | grep -v ^log/
    the result should be like
    Code:
    2011-04/
    full_www-2011-04-15.tar.bz2
    full_var-2011-04-15.tar.bz2
    full_usr-2011-04-15.tar.bz2
    full_sbin-2011-04-15.tar.bz2
    full_root.-2011-04-15.tar.bz2
    full_lib64-2011-04-15.tar.bz2
    full_lib-2011-04-15.tar.bz2
    full_home-2011-04-15.tar.bz2
    full_etc-2011-04-15.tar.bz2
    full_boot-2011-04-15.tar.bz2
    full_bin-2011-04-15.tar.bz2
    
    - then we add the next pipe
    Code:
    ls -ctF /bck/your_hostname  | grep -v ^log/ | tail -n 1
    and the result should be the first file, like:
    Code:
    full_bin-2011-04-15.tar.bz2
    - we then proceed to the last pipe,
    Code:
    ls -ctF /bck/your_hostname  | grep -v ^log/ | tail -n 1 | cut -d "-" -f 2
    which should give us the year like
    Code:
    2011
    Hope you can follow those and post where the output is different so we can see where the problem is.

     
  9. erosbk

    erosbk New Member

    Awesome script, really.

    I read a post here about some user asking for "remote backup". Ok, it should be possible using sshfs...

    ______________________________________________________

    In debian (using "test" user and remote.domain.com as fqdn of the remote computer as example)

    apt-get -y install sshfs
    sudo -i
    mkdir /localmountdir
    adduser test fuse

    sshfs test@remote.domain.com:/remotefoldertomount /localmountdir -p 22

    next, you must specify "/localmountdir/" as backup dir in go0ogl3's script.
    _____________________________________________________

    I can access the mounted fs with this, but I can't write on it (permission denied)...

    Maybe it is not working because I don't allow access in ssh for "root user". I have to make a few tests using root as user to login in ssh... if someone of you could help me with this, maybe we can use this script to make a backup on a remote computer.

    Thanks to all,

    edit: ok, as root it works perfect (you must add root to fuse group, don't use sudo, login as root in ssh and sshfs, it works for perfect).
    I need to make it work using another user, because I don't want to allow root login in ssh/sshfs, please help! :p)

    edit2: I made it work without root access. You must chown the remote folder firts (in the server) and next you must mount it on the "local" server. There is no way to chown files on mounter remote folder, but it is not a problem for this scripts, because you don't need any special owner for the tar.gz files... remember to "restore" the files in a folder OUT of the mounter fs, in order to keep original ownership of the files.

    Regards
     
    Last edited: Apr 24, 2011
  10. cpaun

    cpaun New Member

    Backp-res can be use for ISPConfig 2 too?

    I read the thread and seems that everybody accept that it is a very nice script and working well.
    Can be that script used on ISPConfig 2?
    I'm looking to have a production server ( ubuntu 10.10 ) and a test server, same configuration with same ISPconfig version and every nigt the script to make a full backup( incremental ) , send it to test server and other script to restore the backup . If anything hapent to the production server I have only to change the ip adreses .
    Anybody have that scenario working?
     
  11. erosbk

    erosbk New Member

    I think that it should work, but maybe rsync is better for what you want (but you will not have an incremental backup)
     
  12. Croydon

    Croydon HowtoForge Supporter

    Hi, great script.
    I would suggest using the --hex-blob option and the field and database name quoting when using mysqldump.
     
  13. cpaun

    cpaun New Member

    ISPC 2 backup and restore

    Thank you for suggestion, do you have any HOWTO steps for that. I'm sure that it is many users will like to have an automatic backup and restore module.


    Thanks,
     
  14. Croydon

    Croydon HowtoForge Supporter

    I just meant the back-res script uses mysqldump without the --hex-blob and --quote-names (-Q) command line options.

    So i would change
    Code:
        ### Starting database dumps
        for i in `mysql -u$dbuser -p$dbpassword -Bse 'show databases'`; do
    	log "Starting mysqldump $i"
    	`mysqldump -u$dbuser -p$dbpassword $i --allow-keywords --comments=false --add-drop-table > $tmpdir/db-$i-$FDATE.sql`
    	$TAR $ARG $BACKUPDIR/$MDATE/db-$i-$FDATE.tar.bz2 -C $tmpdir db-$i-$FDATE.sql
    	rm -rf $tmpdir/db-$i-$FDATE.sql
    	log "Dump OK. $i database saved OK!"
        done
    
    to

    Code:
        ### Starting database dumps
        for i in `mysql -u$dbuser -p$dbpassword -Bse 'show databases'`; do
    	log "Starting mysqldump $i"
    	`mysqldump -u$dbuser -p$dbpassword $i --allow-keywords --comments=false --add-drop-table --hex-blob --quote-names > $tmpdir/db-$i-$FDATE.sql`
    	$TAR $ARG $BACKUPDIR/$MDATE/db-$i-$FDATE.tar.bz2 -C $tmpdir db-$i-$FDATE.sql
    	rm -rf $tmpdir/db-$i-$FDATE.sql
    	log "Dump OK. $i database saved OK!"
        done
    
     
  15. Croydon

    Croydon HowtoForge Supporter

    I had a closer look at how the incremental directory backup is done here.

    Code:
    	    # If it's not the first day of the month we make incremental backup
    	    if [ ! -e $tmpdir/full-backup$XX.lck ] ; then
    		log "Starting daily backup for: $YX"
    		NEWER="--newer $FDATE"
    		$TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
    		log "Daily backup for $YX done."
    	    else
    	    log "Lock file for $YX full backup exists!"
    	    fi
    
    I think this could lead to problems if (for any reason) one incremental run did not work. You could be missing files in the archives then, as the --newer flag is date-based.

    Wouldn't it be a good idea to use the incremental flag of tar?

    So instead of using
    Code:
    $TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
    you could use
    Code:
    $TAR $ARG -g $BACKUPDIR/$MDATE/backup$XX.state $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
    Of course you would have to add the -g $BACKUPDIR/$MDATE/backup$XX.state to the full backup command, too.
     
    Last edited: May 14, 2011
  16. go0ogl3

    go0ogl3 New Member

    tar minuses



    Hello,

    In my case, the tar incremental backup is not a very good solution. In the case you loose a incremental backup with my script you can restore the others. See below what's happening with tar incremental backups.

    With tar "if you plan to create more ‘level 1’ backups, it is necessary to create a working copy of the snapshot file before running tar". With the script this is not the case.
    The second minus from tar is "Incremental dumps depend crucially on time stamps, so the results are unreliable if you modify a file's time stamps during dumping". My script does not suffer from this and it was created partially because of this tar problem.

    You can read here more: http://www.gnu.org/software/automake/manual/tar/Incremental-Dumps.html
    The quoted examples are from the link above.
     
  17. go0ogl3

    go0ogl3 New Member

    to be changed

    Hello,

    In the next version of the script perhaps I might fix this and some other small issues.
     
  18. erosbk

    erosbk New Member

    Hello!

    I have a problem with this line... all is working ok but... in this line I get an error when trying to restore...

    Code:
    		mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i
    the $TAR variable is ok (echo $TAR shows its value), but, if I run:

    echo mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i

    I get: mysql -uuser -ppassword notremember < -xvjp /mibackupdir/date/blabla

    It is like mysql were eating the value of $TAR!!!

    I write ANOTHER $TAR in the sentence... (... < $TAR $TAR ... ) and the echo shows it correctly:

    mysql -uuser -ppassword notremember < /TARPATH -xvjp /mibackupdir/date/blabla

    but I get an error =(

    I am using Debian, using bash as shell... could you help me? I think that a work around could to untar in a tmp directory, and next do the dump... (this workaround would be great... I didn't have time to modify it...)

    Thanks for this perfect and usefull script.
     
  19. go0ogl3

    go0ogl3 New Member

    ?

    I've have this script working in multiple debian machines, different stable, but never had any issues...
    Right now I can't imagine what is happening in your particular case, but please if you find out what was wrong, or if you find a solution to your case, or maybe how can I reproduce the error, please write here.

    By the way what's your Debian version? What packages are installed on your computer? Did you try to duplicate the install onto a virtual machine (virtualbox or vmware) to check if the problem persist?

    Thank you for your time!
     
  20. StefanoVanzo

    StefanoVanzo New Member

    I think cyrus1997 is right. Can you please check.
    For me in the script there is a dot (.) after /root and this prevents from backing up the /root directory.
    If I am wrong, please tell me why.

    working with IspConfig3 on Debian Squeeze 64bit
    thank you for your work and you time
     
    Last edited: May 22, 2011

Share This Page