ISPConfig 3 - back-res a Backup and Restore script

[il.manuel]

hi google and thx for your great job!

i have two problem related on server rather than script.

my scenario: a VPS and a ftp backup server on remote machine with a 10GB quota on it.

i have mounted the remote ftp server using curlftpfs (http://curlftpfs.sourceforge.net/) and everything seems to be ok.

first problem: df show me the entire disk space on remote server (7,5TB !) and not my quota (10GB). max percentage calculation have no way to run well and the ftp space will end without a check possibility.

second problem: i thgink that curlftpfs not support append to file ('>>'). i got a lot of this error

./backupres: line 236: /backup/<<HOSTNAME>>/log/backup.log: Operation not supported

when he try to update logfile.

I think for an alternative, but i can't find a solution (manually deleting old backups!!!)

third problem

he not send me an email at the end of script. /tmp/tmpbck/maildata exist and he contains every log entry.

[flyingscubakev]

Just a quick question from a relative newbie.
I've followed the perfect server - ispconfig3 on opesuse and everything is working, I've now loaded this backup script and it seems to work OK, my question is since I followed the perfect server install, it has me disable mail, I was wondering if there was a way to mail the logs to me using what ever the server is configured with (postfix,dovecot,???)?

Thanks

[erosbk]

            # If it's not the first day of the month we make incremental backup
            if [ ! -e $tmpdir/full-backup$XX.lck ] ; then
                log "Starting daily backup for: $YX"
                NEWER="--newer $FDATE"
                $TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
                log "Daily backup for $YX done."
            else
            log "Lock file for $YX full backup exists!"
            fi

Hi, I am running backup at 1 35 each night, so today, it will run at 25/08/2011 01:35 AM.

NEWER="--newer $FDATE"

with that line, it will check if a file is newer than 25/08/2011, and, if a file was created/modified between 24/08/2011 1:35 and 24/08/2011 23:59, it will be ignored.

I think that the solution should be to work with yesterday date and not with the actual date. It will copy twice files created between 25/08/2011 00:00 and 25/08/2011 1:35 (at next backup), but right now, incremental backup is doing almost nothing.

Is this correct or I forget to check something?

I will try this night with line value:
NEWER="--newer `date --date=yesterday +%F`"

instead of:
NEWER="--newer $FDATE"

But, I don't know is --newer checks only creation date or modified date too... we will see this night :P

Best regards

Edit: working ok, right now inc backup working, I think that this should be a modification for the scripts (or a custom value to modify depending on script running time.

[go0ogl3]

Originally Posted by flyingscubakev

Just a quick question from a relative newbie.
I've followed the perfect server - ispconfig3 on opesuse and everything is working, I've now loaded this backup script and it seems to work OK, my question is since I followed the perfect server install, it has me disable mail, I was wondering if there was a way to mail the logs to me using what ever the server is configured with (postfix,dovecot,???)?

Thanks

If you have a smtp mail account you can configure one of the multiple mail programs (mail, mutt, etc.) to send mail at the end of the script using that account.

[go0ogl3]

Originally Posted by il.manuel

hi google and thx for your great job!

i have two problem related on server rather than script.

my scenario: a VPS and a ftp backup server on remote machine with a 10GB quota on it.

i have mounted the remote ftp server using curlftpfs (http://curlftpfs.sourceforge.net/) and everything seems to be ok.

first problem: df show me the entire disk space on remote server (7,5TB !) and not my quota (10GB). max percentage calculation have no way to run well and the ftp space will end without a check possibility.

second problem: i thgink that curlftpfs not support append to file ('>>'). i got a lot of this error

./backupres: line 236: /backup/<<HOSTNAME>>/log/backup.log: Operation not supported

when he try to update logfile.

I think for an alternative, but i can't find a solution (manually deleting old backups!!!)

third problem

he not send me an email at the end of script. /tmp/tmpbck/maildata exist and he contains every log entry.

Hello,

the <<HOSTNAME>> is changed by you? If not that's the problem.

curlftpfs does not support append to the files. The solution is to move all the appended files away (log file, there is another?)
The alternative for the free space function is to get the free space with another method (repquota?) and use that value in the script...

The mail problem:
You need to have some mail server running (sendmail, postfix, dovecot, etc.) on your linux host to send mail with the back-res script...
I'll try to post a little script to send mail using SMTP AUTH and your ISP mail server (or another) and I'll try to modify the back-res script later this month.

[go0ogl3]

Originally Posted by erosbk

            # If it's not the first day of the month we make incremental backup
            if [ ! -e $tmpdir/full-backup$XX.lck ] ; then
                log "Starting daily backup for: $YX"
                NEWER="--newer $FDATE"
                $TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
                log "Daily backup for $YX done."
            else
            log "Lock file for $YX full backup exists!"
            fi

Hi, I am running backup at 1 35 each night, so today, it will run at 25/08/2011 01:35 AM.

NEWER="--newer $FDATE"

with that line, it will check if a file is newer than 25/08/2011, and, if a file was created/modified between 24/08/2011 1:35 and 24/08/2011 23:59, it will be ignored.

I think that the solution should be to work with yesterday date and not with the actual date. It will copy twice files created between 25/08/2011 00:00 and 25/08/2011 1:35 (at next backup), but right now, incremental backup is doing almost nothing.

Is this correct or I forget to check something?

I will try this night with line value:
NEWER="--newer `date --date=yesterday +%F`"

instead of:
NEWER="--newer $FDATE"

But, I don't know is --newer checks only creation date or modified date too... we will see this night :P

Best regards

Edit: working ok, right now inc backup working, I think that this should be a modification for the scripts (or a custom value to modify depending on script running time.

Hello,

From the tar manual seen on http://www.gnu.org/software/tar/manual/tar.html#SEC114:

‘--newer=date’
‘-N date’
Only store files newer than date.

Acts on files only if their data modification or status change times are later than date. Use in conjunction with any operation.

The FDATE from the script is the actual date like

2011-09-12

which means 2011-09-12 time 00:00
In my tests I run the script at 00:01 on 8 september for example (first minute of the 8 day) and the script archives some files modified on

Sep  7 23:25

and on

Sep  8 00:00

so I can say it's doing his job....

How you use the script the backed up files are a day older, so you'll end up archiving more files on the incremental backup.

Use a simple test:
create a file in cron (touch) just before the actual run of the script... or use the tar command from the command line with the required arguments.

[go0ogl3]

Originally Posted by fordwrench

I have a server that has debian lenny and ispconfig via the perfect server howto.

I am trying to migrate to a new server I have created with debian squeeze viat the perfect server howto.

I make my backup and when I try to restore I get the following:



oot@srv1:~/scripts# sh /root/scripts/backres.sh db all 2011-07-16
date: invalid date `07/'
/root/scripts/backres.sh: line 448: [: : integer expression expected

You want to restore db all to date 2011-07-16.

Please input "yes" if the above is ok with you and press [ENTER]: yes

Restoring all mysql databases from date 2011-07-16 to local server:
touch: cannot touch `/tmp/tmpbck/datestart': No such file or directory
touch: cannot touch `/tmp/tmpbck/dateend': No such file or directory
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
All restore jobs done!
Database all restored to date 2011-07-16!
/root/scripts/backres.sh: line 573: /tmp/tmpbck/maildata: No such file or directory


Can you please give me some insight as to what the problem might be.

Sorry for seen your post so late, but I'll respond maybe others will need to restore too.

You can restore all of your files unpacking all the full_ archives, then all the incremental archives from that month OVER the full_ archives until the date needed. This way you'll end up with all your files, including the ones deleted from one incremental backup to another. You can use find --newer to delete unwanted files.

Hope this helps others too.

In your case it's possible that you have a problem with the date command and your /tmp dir is not writable or something.

The way I do the restore is using an empty partition or drive or a loop device mounted in a running system, I'm adding the needed /proc and others then make it bootable (grub, lilo, etc.), then remove the drive and run the restored system.

[go0ogl3]

Hello

here is a simple code for a mail function or script for the systems with mail server disabled:

#!/bin/bash
#
mailuser=`echo -n $email| base64`
mailpass=`echo -n $password| base64`

maildata=`$tmpdir/maildata`

nc mail.server.com 25 << EOF
EHLO test
auth login
$mailuser
$mailpass
mail from: $email
rcpt to: $email
DATA
Subject: test mail telnet

$maildata
.
quit
EOF

I'll try to modify the back-res script to add an option for this function. The code works only for mail servers which support smtp auth login but I hope will be enough...

[dar_ek]

I noticed that if you store backup on a NFS disks when DELETE old backups NOT WORKS

I have standard maxp="85" but disks was full after some days.
Problem is in "function check_space"
Line:

  pfs=`df -h $BACKUPDIR | awk 'NR==2{print $5}' | cut -d% -f 1`

In common disks it works cause all infos from df are in one line but in NFS mounted disks is IN TWO LINES, look:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              14G  1,7G   12G  13% /
10.246.120.212:/home/BACKUP/POCZTA
                       65G   65G     0 100% /home/BACKUP

in this case you can user "P" switch in df, that tell df to dont do this (2 lines output for one disk).
And fix will look that:

 pfs=`df -Ph $BACKUPDIR | awk 'NR==2{print $5}' | cut -d% -f 1`

please check that "df -Ph" will be works on other linux distros, and if its woks to change that.

regards

[go0ogl3]

Originally Posted by dar_ek

I noticed that if you store backup on a NFS disks when DELETE old backups NOT WORKS

I have standard maxp="85" but disks was full after some days.
Problem is in "function check_space"
Line:

  pfs=`df -h $BACKUPDIR | awk 'NR==2{print $5}' | cut -d% -f 1`

In common disks it works cause all infos from df are in one line but in NFS mounted disks is IN TWO LINES, look:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              14G  1,7G   12G  13% /
10.246.120.212:/home/BACKUP/POCZTA
                       65G   65G     0 100% /home/BACKUP

in this case you can user "P" switch in df, that tell df to dont do this (2 lines output for one disk).
And fix will look that:

 pfs=`df -Ph $BACKUPDIR | awk 'NR==2{print $5}' | cut -d% -f 1`

please check that "df -Ph" will be works on other linux distros, and if its woks to change that.

regards

Thank you for help, this is because your NFS path is long but is a small fix wich I'll include in the next version, sometime in december, if my timing is ok.