Extending Perfect Server - Debian Squeeze [ISPConfig 3] - Page 6

10. Clients' BackUps

The contents of the script changed on 29-03-2011. Please update

This script WILL NOT work correctly for ISPConfig v. 3.0.5 and above. You have to do a lot of changes and is not recommended. Please use the ISPConfig new way of backing up.

The following script, is an easy way to backup your clients data and your clients databases in their website folder. As you may know, in ISPConfig3 each client has a folder in the form /var/www/clients/clientXY, in which there are all his web sites. The script will back up all his websites in each web folder, in companion with his databases and the client will be able to download them. In case, a client has more than one database, then all the databases will be backed up in his first (based on webID) site. The script will also keep the last 3 days of those files and the last 3 Sundays for admin usage in a directory of your choice (the default is /var/backup/sites).

This is a very simple script. For a more advanced solution look at this post.

Create the script, make it executable and edit it:

cd /root/scripts/
touch mybackup.sh
chmod 0700 mybackup.sh
nano mybackup.sh

The contents must be the following:(Change the variables ispUSER, ispPASS, ispHOST, DEST and SITES to fit your needs):

#!/bin/bash
# Shell script to backup MySql database and clients websites
#
# Last updated: March - 2011
# --------------------------------------------------------------------
# This is a free shell script under GNU GPL version 2.0 or above
# Copyright (C) 2011 iopen.gr
# Feedback/comment/suggestions : http://iopen.gr
# --------------------------------------------------------------------
#
# INTENDED for the ISPConfig 3.0.x and above
#
# This script will back up every web folder (web, stats, cgi e.t.c
# of every client in companion with all the client's DBs
# The backups will be placed in the website client's folder
# The scipt will keep the current and the 2 previous backup
# It will also keep the last 3 sundays
# --------------------------------------------------------------------

# Database credentials. Use a DB user with full read access or use the root user
ispUSER="root" # DB user
ispPASS="---yourpass---" # user's password
ispHOST="localhost" # Hostname

CURDIR="$(pwd)"
# Variables with full path to binaries
MYSQL="$(which mysql)"
MYSQLDUMP="$(which mysqldump)"
CHOWN="$(which chown)"
CHMOD="$(which chmod)"
GZIP="$(which gzip)"
TAR="$(which tar)"

# Your Server's Main Backup Directory
DEST="/var/backup"

# Sites (ONLY) backup directory in your Main Backup Directory
SITES="$DEST/sites"


# Variables for Dates in yymmdd format
TODAY=`date +%Y%0m%0d`
YESTERDAY=`date -d '1 day ago' +%Y%0m%0d`
BACK2=`date -d '2 day ago' +%Y%0m%0d`
BACK3=`date -d '3 day ago' +%Y%0m%0d`
BACK22=`date -d '22 day ago' +%Y%0m%0d`

[ ! -d $SITES ] && mkdir -p $SITES || :

# Give Only root access to backups in this scripts folders
$CHOWN 0.0 -R $SITES
$CHMOD 0600 $SITES

# --------------------------------------------------------------------------
# Remove previous (current) backups of the client directory
# The backups are in the form :
# *BU*gz
# -------- CAUTION ---------
# Do not store any other file in this form in the clients directory
# --------------------------------------------------------------------------
echo "-------------------------------------------------------------"

QRY="use dbispconfig; SELECT web_domain.system_user, web_domain.system_group, \
web_domain.document_root, web_domain.domain FROM web_domain WHERE \
web_domain.type!='alias' AND web_domain.system_user IS NOT NULL AND (LENGTH(web_domain.redirect_path)<5 OR web_domain.redirect_path IS NULL) ;"

echo $QRY | mysql -u $ispUSER -h $ispHOST -p$ispPASS | while read -r line
do # ${col[0]}=domain user / folder name / system user, ${col[1]}=clientID / system group ,
while read -a col # ${col[2]}=path to website, ${col[3]}= domain name
do
echo " CLEANING OLD BACKUPS in ${col[2]} folder "
for delfile in ${col[2]}/*BU*gz ;
do [ -f $delfile ] && rm $delfile;
done
done
done

# --------------------------------------------------------------------------
# Remove anything that is 22 days old and have the form :
# *[date 22 days old]*gz
# from server's $SITES directory
# --------------------------------------------------------------------------
echo "-------------------------------------------------------------"
echo " CLEANING OLD BACKUPS in SITES folder "
for delfile in $SITES/*$BACK22*gz ;
do [ -f $delfile ] && rm $delfile;
done
echo "-------------------------------------------------------------"
echo " "
echo " "

# --------------------------------------------------------------------------
# For each client, backup his database in his website folder
# For client with multiple sites backup all dbs in his first site
# Furthermore copy today's backup in the server's $SITES directory
# Remove the backup that is older than 3 days from server's $SITES directory
# Keep the last 3 Sundays
# --------------------------------------------------------------------------

QRY="use dbispconfig; SELECT web_database.database_name , web_database.database_user ,\
 min(web_domain.system_user) as muser, web_domain.system_group, min(web_domain.document_root) as mpath, \
web_domain.domain FROM web_database, web_domain WHERE web_database.sys_userid=web_domain.sys_userid \
AND web_database.sys_groupid=web_domain.sys_groupid AND web_domain.type='vhost' \
AND web_domain.system_user IS NOT NULL AND (LENGTH(web_domain.redirect_path)<5 OR web_domain.redirect_path IS NULL) \
GROUP BY web_database.database_name , web_database.database_user, web_domain.system_group;"

echo $QRY | mysql -u $ispUSER -h $ispHOST -p$ispPASS | while read -r line
do # ${col[0]} = dbname, ${col[1]}=dbuser , ${col[2]}=domain user / folder name / system user,
while read -a col #${col[3]}=clientID / system group , ${col[4]}=path to website
do
echo " DB: "${col[0]}
echo "-------------------------------------------------------------"
echo "Backing Up DB:" ${col[0]} "in :" ${col[4]}/${col[0]}BU.gz
$MYSQLDUMP -u $ispUSER -h $ispHOST -p$ispPASS -c --add-drop-table --add-locks \
--quick --lock-tables ${col[0]} | $GZIP -9 > ${col[4]}/${col[0]}BU.gz
cp ${col[4]}/${col[0]}BU.gz $SITES/${col[0]}.$TODAY.gz
if [ `date -d '3 day ago' +%u` -ne 7 ] # if 3 days ago is not Sunday
then #remove the backup
[ -f $SITES/${col[0]}.$BACK3.gz ] && rm $SITES/${col[0]}.$BACK3.gz
fi
$CHOWN ${col[2]}:${col[3]} ${col[4]}/${col[0]}BU.gz
$CHMOD 0660 ${col[4]}/${col[0]}BU.gz
echo "-------------------------------------------------------------"
echo " "

done
done

# --------------------------------------------------------------------------
# For each client, backup his sites in his website folder
# Furthermore copy today's backup in the server's $SITES directory
# Remove the backup that is older than 3 days from server's $SITES diriectory
# Keep the last 3 Sundays
# --------------------------------------------------------------------------



QRY="use dbispconfig; SELECT web_domain.system_user, web_domain.system_group,\
 web_domain.document_root, web_domain.domain FROM web_domain WHERE \
web_domain.type!='alias' AND web_domain.system_user \
IS NOT NULL AND (LENGTH(web_domain.redirect_path)<5 OR web_domain.redirect_path IS NULL) ;"

echo $QRY | mysql -u $ispUSER -h $ispHOST -p$ispPASS | while read -r line
do # ${col[0]}=domain user / folder name / system user, ${col[1]}=clientID / system group ,
while read -a col # ${col[2]}=path to website, ${col[3]}= domain name
do
echo " "
echo " Site:" ${col[3]}
echo "-------------------------------------------------------------"
echo "Backing Up site: " ${col[2]}/ "in :" ${col[2]}/${col[3]}BU.tar.gz
cd ${col[2]}
sudo -u ${col[0]} $TAR -czf ${col[2]}/${col[3]}BU.tar.gz .
cp ${col[2]}/${col[3]}BU.tar.gz $SITES/${col[3]}.$TODAY.tar.gz
if [ `date -d '3 day ago' +%u` -ne 7 ] # if 3 days ago is not Sunday
then #remove the backup
[ -f $SITES/${col[3]}.$BACK3.tar.gz ] && rm $SITES/${col[3]}.$BACK3.tar.gz
fi
$CHOWN ${col[0]}:${col[1]} ${col[2]}/${col[3]}BU.tar.gz
$CHMOD 0660 ${col[2]}/${col[3]}BU.tar.gz

echo "-------------------------------------------------------------"
echo " "
done
done
cd $CURDIR

You can run the backup script by executing:

/root/scripts/mybackup.sh

or you can add it as a cron job (e.g. every day at 22:30):

crontab -e

and append the following line:

30 22 * * * /root/scripts/mybackup.sh > /dev/null 2>> /var/log/backup.log

 

Final Notes

Please, feel free to comment anything for this tutorial in an appropriate threat in HowtoForge forums. Useful comments will be included in a future updated version.

This is the first version and as careful as I was, the tutorial may contain errors. Please let me know of them, so as to correct them, as soon as possible.

Share this page:

32 Comment(s)

Add comment

Comments

From: at: 2011-03-22 19:50:03

There is a small typo in download of the webmin

 

cd /tmp wget http://prdownloads.sourceforge.net/webadmin/webmin_1.530_all.deb

 

should actually be 

 cd /tmp

wget http://prdownloads.sourceforge.net/webadmin/webmin_1.530_all.deb

From: teddy at: 2011-08-19 18:28:47

Hi, I've followed the perfect debian server with ispconfig3 tutorial, then the SSL post on faqforge, and everything went ok on my virtualized server and on the online one. Once it gets to changing default port for webmin, activating it on ispconfig firewall, restarting webmin and apache, and no way, the page I get on the browser (url is https://x.x.x.x:1888) is (Chrome in this case, but the timeout is consistent for all browsers.

 Error 118 (net::ERR_CONNECTION_TIMED_OUT)

 I've checked with a netstat, but the port is listening

 tcp        0      0 0.0.0.0:18888           0.0.0.0:*               LISTEN      32521/perl

udp        0      0 0.0.0.0:18888           0.0.0.0:*                           32521/perl

an iptables list gives this response
 
  <code>

 Chain INPUT (policy DROP)

target     prot opt source               destination

DROP       tcp  --  anywhere             loopback/8

ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED

ACCEPT     all  --  anywhere             anywhere

DROP       all  --  base-address.mcast.net/4  anywhere

PUB_IN     all  --  anywhere             anywhere

PUB_IN     all  --  anywhere             anywhere

PUB_IN     all  --  anywhere             anywhere

PUB_IN     all  --  anywhere             anywhere

DROP       all  --  anywhere             anywhere


Chain FORWARD (policy DROP)

target     prot opt source               destination

ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED

DROP       all  --  anywhere             anywhere


Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination

PUB_OUT    all  --  anywhere             anywhere

PUB_OUT    all  --  anywhere             anywhere

PUB_OUT    all  --  anywhere             anywhere

PUB_OUT    all  --  anywhere             anywhere


Chain INT_IN (0 references)

target     prot opt source               destination

ACCEPT     icmp --  anywhere             anywhere

DROP       all  --  anywhere             anywhere


Chain INT_OUT (0 references)

target     prot opt source               destination

ACCEPT     icmp --  anywhere             anywhere

ACCEPT     all  --  anywhere             anywhere


Chain PAROLE (15 references)

target     prot opt source               destination

ACCEPT     all  --  anywhere             anywhere


Chain PUB_IN (4 references)

target     prot opt source               destination

ACCEPT     icmp --  anywhere             anywhere            icmp destination-unreachable

ACCEPT     icmp --  anywhere             anywhere            icmp echo-reply

ACCEPT     icmp --  anywhere             anywhere            icmp time-exceeded

ACCEPT     icmp --  anywhere             anywhere            icmp echo-request

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:ftp-data

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:ftp

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:ssh

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:smtp

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:domain

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:www

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:pop3

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:imap2

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:https

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:mysql

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:http-alt

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:tproxy

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:8000

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:18888

PAROLE     tcp  --  anywhere             anywhere            tcp dpt:webmin

ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain

ACCEPT     udp  --  anywhere             anywhere            udp dpt:mysql

DROP       icmp --  anywhere             anywhere

DROP       all  --  anywhere             anywhere

...

 </code>

 So, everything seems in order, what could it be stopping me from accessing webmin from alternative port? Please consider that default port works, and all other services work, since I've followed letter by letter you instructions...

 Did I forget anything? Where could I look?

 

From: at: 2011-12-17 14:30:14

Good tutorial, I just want to thank you for taking the time to write it.

From: Frank at: 2012-05-31 10:56:40

hi,
it is possible to coexist squirrelmail and roundcube?

thanks!

From: Frederik at: 2011-06-01 14:07:54

Hy!

 

There is a typo in 4. FAIL2BAN.

 nano /etc/fail2ban/jail.local -> nano /etc/fail2ban/jail.conf

 

 

 

my regards

From: at: 2011-06-26 17:16:23

This is not a typo.

The jain.local will not be overwritten in a future update.

From: Maurizio Marini at: 2011-08-07 15:30:40

file /var/lib/roundcube/config/main.inc.php line 60 is already:

$rcmail_config['auto_create_user'] = TRUE;

i think that your

auto_create_user = TRUE;

is relative to older versions ...

or should we change:

$rcmail_config['auto_create_user'] = TRUE;

with

auto_create_user = TRUE;

?

From: at: 2011-08-04 15:13:00

Hi,

I found that if I added 0.05 to the sleep command I would end up with errors in the fail2ban.log that I also found here http://oschgan.com/drupal/node/52 and when I changed it to 0.1 it worked perfectly.

Regards,

Steve

 

From: Anonymous at: 2012-01-13 02:01:27

I'm using BIND + Dovecot and extending with roundcube and following line

failregex = FAILED login for .*. from <host>
 in /etc/fail2ban/filter.d/roundcube.conf give me an error:
fail2ban.filter : ERROR  No 'host' group in 'FAILED login for .*. from <host>'
So i think it's only a typo, if i write <HOST> with uppercase, then all seems to be great.
 THX

From: michael at: 2012-04-02 09:16:59

Hi, first, thanks for the useful info!

 In step 8:

 

"If you want to install Drupal (or other cms) you will propably need uploadprogress and json. To accomplish their installation, do:

apt-get install php5-dev php-services-json
pecl install uploadprogress
touch /etc/php5/apache2/conf.d/uploadprogress.ini
nano /etc/php5/apache2/conf.d/uploadprogress.ini"

The sites gave 500-errors all around. I ran apt-get remove php5-dev and the sites came up again. Will that have an effect on the other things or is there a work-around for the 500-errors? 

From: Benoit Lallemand at: 2011-04-21 15:53:22

I think the parameter -p in command, here after, is wrong !

#!/bin/sh
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin
/usr/bin/php -p /root/scripts/mycron.php

From: Web Worlds at: 2011-05-05 08:56:45

I agree, also with me it was not working with the '-p' in the PHP command.

From: lenz at: 2011-06-07 09:49:06

there is a typo

wget http://mysqltuner.com/mysqltuner.pl

From: at: 2012-05-19 20:27:36

I really like this tutorial, so that is why i wish to improve it by posting this error that i have found with the email the cron job mycron.php sends.

I have gone through this tutorial twice, both starting from the perfect server here.

Both times i keep getting this error email from mycron.php.

HERE IS THE ERROR:

Email Subject:    Cron <root@node1> test -x /usr/sbin/tigercron && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; nice -n$NICETIGER /usr/sbin/tigercron -q ; } (failed)

Email Message:  /usr/lib/tiger/config: line 360: /config: No such file or directory


How can i fix this error. I followed the tutorial to a 'T' so i dont think it was an error on my part.

From: at: 2012-09-05 11:07:11

Did you install tigercron?

(Please use the forums for questions)

From: at: 2011-04-22 15:56:50

unloadfw and loadfw are exactly the same... it is an error, please can be those lines corrected?

Thanks in advance, awesome tutorial ;)

From: Frederik at: 2011-06-01 14:25:40

No they are different.

The one script calls the flag -I to init a rule, the other one -D to delete a rule.

 

:)

From: Benoit Lallemand at: 2011-04-21 19:01:14

On the top of this pages : 

mkdir /root/scripts
touch /root/scripts/loadfw
touch /root/scripts/unloadfw
touch /root/scripts/IPs

touch /root/scripts/fwrules  --> fwrules with "s"

touch /root/scripts/reloadfail2ban
cd /root/scripts
nano loadfw

 and in the script, here after, you read "fwrule" without "s"

# Simple iptables IP/subnet unload script
# ---------------------------------------------------------


cd /root/scripts/

IPT=/sbin/iptables
DROPMSG="fwBLOCKED "
BADIPS=$(egrep -v -E "^#|^$" /root/scripts/IPs)

while read fwrule
do
$IPT -D INPUT $fwrule

done < /root/scripts/fwrules

From: at: 2011-06-26 17:32:08

fwrule is the variable.

At the end of the while is the file (/root/scripts/fwrules):

 while read fwrule
do
  $IPT -D INPUT $fwrule

done < /root/scripts/fwrules

 

From: Frederik at: 2011-06-01 14:59:17

Hy, I applyed this tutorial to my Debian 6 Server.

 When i execute /root/scripts/reloadfail2ban my console tells me:

 

  • root@lvpsXXXXXXXXX:~/scripts# /root/scripts/reloadfail2ban
  • Restarting authentication failure monitor: fail2ban.
  • iptables: Bad rule (does a matching rule exist in that chain?).
  • iptables: No chain/target/match by that name.
  • iptables: Bad rule (does a matching rule exist in that chain?).
  • [...] some more stuff
  • iptables: No chain/target/match by that name.
  • iptables: Bad rule (does a matching rule exist in that chain?).
  • iptables: No chain/target/match by that name.
  • iptables: No chain/target/match by that name.
  • iptables: No chain/target/match by that name.
  • [...] some more stuff
  • iptables: No chain/target/match by that name.
  • iptables: No chain/target/match by that name.
  • iptables: No chain/target/match by that name.
  • iptables: No chain/target/match by that name.
  • iptables: No chain/target/match by that name.

 

 

 Is this okay or is this an error?

 

 

my regards.

From: at: 2011-06-26 17:39:53

It's OK.

You probably changed the rules before you unload the previous rules.

So the script tries to unload the new rules, but the old ones are still loaded.

Please first unload everything, after this make your changes and finally reload the rules.

The above include the part of IP addresses

To be sure that everything is ok:

Do a full restart. Unload the rules, Make the changes. Reload the rules.

Keep in mind that this is not a fully featured firewall. It's just a script with basic rules.

From: at: 2011-12-12 20:44:31

hey i get this email from cron:

 /usr/lib/tiger/config: line 360: /config: No such file or directory

 how can i fix this?

 

From: at: 2013-03-30 22:29:51

Copy the default config from


/usr/lib/tiger/systems/default/config

or

/usr/lib/tiger/systems/Linux/2/config

to

/usr/lib/tiger/

From: Stefkom at: 2013-08-25 11:11:24

very simple :)

1. sudo ln -s /usr/lib/tiger/systems/Linux/2 /usr/lib/tiger/systems/Linux/3

or

2. sudo cp -a /usr/lib/tiger/systems/Linux/2 /usr/lib/tiger/systems/Linux/3

From: at: 2011-12-12 20:51:04

just wanted to add the subject line for the cron email:

 test -x /usr/sbin/tigercron && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; nice -n$NICETIGER /usr/sbin/tigercron -q ; } (failed)

From: at: 2011-04-14 13:07:54

I followed the client backup tutorial, however I'm getting a error after running it as root.

The backups seems to run fine but i receive te following error and this keeps repeating in the console:

 /bin/tar: ./tmp/sess_5jrh2r2d5m8lhtaq7sf04mo3n7: Cannot open: Permission denied


From: at: 2011-05-05 05:17:55

After downloading the file, you must:

chmod +x /usr/share/roundcube/plugins/fail2ban.php
touch /var/log/roundcube/userlogins
chown www-data:www.data /var/log/roundcube/userlogins

 

I don't know if chmod +x is necesary for fail2ban.php (i was traying to make it work, I am too tired to test it lol xD) but, you MUST chown userlogins, if not, apache2 (or roundcube through apache2) will not be able to write inside the file (access denied).

 Regards

 

PD: Perfect howto xD

From: at: 2011-06-26 18:04:25

You just need the following command:

 chown www-data:www-data /var/log/roundcube/userlogins

I updated the tutorial. Thx

From: webmaster eddie at: 2011-12-17 06:50:16

I tried following your instructions - just to harden the server using ipTables and the ddos - and the backup scripts - and I could no longer ftp - i only have a dynamic ip wifi public connection to the net and it was blocking me disconnecting me from ftping files after 1 second it seems... so I reversed every single thing I did following your instructions, and now cannot ftp with any program at all - I can connect but not a single file is allowed to be transferred - i get a permission denied 553 error. Can you help me ? I have checked everything - the ports in IPSCongif 3 panel are fine, etc.

 

Also I never got the backup scripts to work at all - so I removed them. I do thank you for the 2 mysql tuning scripts which work and seem to help

From: at: 2012-01-21 01:19:17

At backup script mysqldump '--all' option is deprecated and restor from backup won't work ! Use '--create-options' instead.

From: Jonas Lateur at: 2012-04-23 14:51:53

when i run /root/scripts/mybackup.sh, i get follow error


       Site: server.ttb-ltd.eu
-------------------------------------------------------------
Backing Up site:  /var/www/clients/client1/web1/ in : /var/www/clients/client1/web1/server.ttb-ltd.euBU.tar.gz
tar (child): /var/www/clients/client1/web1/server.ttb-ltd.euBU.tar.gz: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
/bin/tar: /var/www/clients/client1/web1/server.ttb-ltd.euBU.tar.gz: Cannot write: Broken pipe
/bin/tar: Error is not recoverable: exiting now
cp: cannot stat `/var/www/clients/client1/web1/server.ttb-ltd.euBU.tar.gz': No such file or directory
/bin/chown: cannot access `/var/www/clients/client1/web1/server.ttb-ltd.euBU.tar.gz': No such file or directory
/bin/chmod: cannot access `/var/www/clients/client1/web1/server.ttb-ltd.euBU.tar.gz': No such file or directory
-------------------------------------------------------------

From: at: 2013-04-07 14:35:47

Hi,

at first: thnx. for this wonderfull tutorial!!!

The only problem I've got at this point is that when I execute the mybackup.sh file I get this error:

ERROR 1054 (42S22) at line 1: Unknown column 'web_database.database_user' in 'field list'

I looked into the database and there really is missing the database_user field... Is this a never version of ispConfig3 I'm using? Can you adapt the script to the newer version?

Thnx. again for your good work.

Best regards, Ingmar