Installing A Web, Email And MySQL Database Cluster (Mirror) On Debian 5.0 With ISPConfig 3

Version 1.0
Author: Till Brehm <t [dot] brehm [at] projektfarm [dot] com>
Last edited 08/04/2010

This tutorial describes the installation of a clustered Web, Email, Database and DNS server to be used for redundancy, high availability and load balancing on Debian 5 with the ISPConfig 3 control panel. GlusterFS will be used to mirror the data between the servers and ISPConfig for mirroring the configuration files. I will use a setup of two servers here for demonstration purposes but the setup can scale to a higher number of servers with only minor modifications in the GlusterFS configuration files.

There is currently one limitation in the MySQL cluster setup. The MySQL daemon has locking problems during the initial innodb check when the second server gets started. The current workaround that I use here is to start MySQL with myisam only. I've found several reports of successfully running MySQL servers with innodb on GlusterFS, so it must be possible with some finetuning of the GlusterFS and / or MySQL configuration file to use innodb as well. I will try to find a solution for the locking issues and update this tutorial. If someone knows a solution for innodb on GlusterFS, please contact me. If you want to use the second server only as hot standby system, then you should be able to use innodb as long as you start MySQL on the second server only when the first server is disconnected.

This is currently a proof of concept setup, so there is no experience how this setup scales in production systems yet. The only part that might cause problems is the shared MySQL data directory. Another solution for accessing MySQL databases from several servers simultaneously is to use a MySQL-cluster setup (http://www.mysql.com/products/database/cluster/) or MySQL master / master replication (http://www.howtoforge.com/mysql_master_master_replication).

 

1 Setting Up The Two Base Systems

In this setup there will be one master server (which runs the ISPConfig control panel interface) and one slave server which mirrors the web (apache), email (postfix and dovecot) and database (MySQL) services of the master server.

To install the clustered setup, we need two servers with a Debian 5.0 minimal install. The base setup is described in the following tutorial in the steps 1 - 6:

http://www.howtoforge.com/perfect-server-debian-lenny-ispconfig3

Install only steps 1 - 6 of the perfect server tutorial and not the other steps as they differ for a clustered setup!

In my example I use the following hostnames and IP addresses for the two servers:

Master Server

Hostname: server1.example.tld
IP address: 192.168.0.105

Slave server

Hostname: server2.example.tld
IP address: 192.168.0.106

Whereever these hostnames or IP addresses occur in the next installation steps you will have to change them to match the IPs and hostnames of your servers.

 

2 Installing The Two Servers

The following steps have to be executed on the master and on the slave server. If a specific step is only for the master or slave, then I've added a note in the description in red.

vi /etc/hosts

127.0.0.1       localhost
192.168.0.105   server1.example.tld
192.168.0.106   server2.example.tld
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Set the hostname of the server:

echo server1.example.tld > /etc/hostname
/etc/init.d/hostname.sh start

User server1.example.tld on the first server and server2.example.tld on the second server.

Edit the sources.list file...

vi /etc/apt/sources.list

... and ensure that it contains the following two lines. The first one is for the debian volatile repository to get updated pacakges for the ClamAV antivirus software and SpamAssassin and the second one is for the backports repository which contains current GlusterFS packages.

deb http://volatile.debian.org/debian-volatile lenny/volatile main contrib non-free
deb http://www.backports.org/debian/ lenny-backports main contrib non-free

Run...

apt-get install debian-backports-keyring
apt-get update

... to install the backports repository key and update the apt package database; then run ...

apt-get upgrade

... to install the latest updates (if there are any).

It is a good idea to synchronize the system clock with an NTP (network time protocol) server over the Internet. Simply run...

apt-get -y install ntp ntpdate

... and your system time will always be in sync.

Install postfix, dovecot and mysql with one single command:

apt-get -y install postfix postfix-mysql postfix-doc mysql-client mysql-server openssl getmail4 rkhunter binutils dovecot-imapd dovecot-pop3d sudo

Enter the new password for mysql when requested by the installer and answer the next questions as decsribed below:

Create directories for web-based administration ? <-- No
General type of configuration?
<-- Internet site
Mail name?
<-- server1.mydomain.tld
SSL certificate required
<-- Ok

We want MySQL to listen on all interfaces, not just localhost, therefore we edit /etc/mysql/my.cnf and comment out the line bind-address = 127.0.0.1:

vi /etc/mysql/my.cnf

[...]  
# Instead of skip-networking the default is now to listen only on  
# localhost which is more compatible and is not less secure.  
#bind-address           = 127.0.0.1  
[...]

Then restart MySQL:

/etc/init.d/mysql restart

To install amavisd-new, SpamAssassin, and ClamAV, we run:

apt-get -y install amavisd-new spamassassin clamav clamav-daemon zoo unzip bzip2 arj nomarch lzop cabextract apt-listchanges libnet-ldap-perl libauthen-sasl-perl clamav-docs daemon libio-string-perl libio-socket-ssl-perl libnet-ident-perl zip libnet-dns-perl

Then install Apache2, PHP5, phpMyAdmin, FCGI, suExec, Pear, and mcrypt as follows:

apt-get -y install apache2 apache2.2-common apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert libapache2-mod-php5 php5 php5-common php5-gd php5-mysql php5-imap phpmyadmin php5-cli php5-cgi libapache2-mod-fcgid apache2-suexec php-pear php-auth php5-mcrypt mcrypt php5-imagick imagemagick libapache2-mod-suphp libopenssl-ruby libapache2-mod-ruby sudo

You will see the following question:

Web server to reconfigure automatically: <-- apache2

Then run the following command to enable the Apache modules suexec, rewrite, ssl, actions, and include:

a2enmod suexec rewrite ssl actions include ruby dav_fs dav auth_digest

PureFTPd and quota can be installed with the following command:

apt-get -y install pure-ftpd-common pure-ftpd-mysql quota quotatool

Edit /etc/fstab. Mine looks like this (I added ,usrquota,grpquota to the partition with the mount point /):

vi /etc/fstab

# /etc/fstab: static file system information.  
#  
# <file system> <mount point>   <type>  <options>       <dump>  <pass>  
proc            /proc           proc    defaults        0       0  
/dev/sda1       /               ext3    errors=remount-ro,usrquota,grpquota 0       1  
/dev/sda5       none            swap    sw              0       0  /dev/hda        
/media/cdrom0   udf,iso9660 user,noauto     0       0  
/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

To enable quota, run these commands:

touch /quota.user /quota.group
chmod 600 /quota.*
mount -o remount /

quotacheck -avugm
quotaon -avug

Install BIND DNS Server:

apt-get -y install bind9 dnsutils

Install vlogger and webalizer and awstats:

apt-get -y install vlogger webalizer awstats

Install Jailkit: Jailkit is needed only if you want to chroot SSH users. It can be installed as follows (important: Jailkit must be installed before ISPConfig - it cannot be installed afterwards!):

apt-get -y install build-essential autoconf automake1.9 libtool flex bison

cd /tmp
wget http://olivier.sessink.nl/jailkit/jailkit-2.11.tar.gz
tar xvfz jailkit-2.11.tar.gz
cd jailkit-2.11
./configure
make
make install
cd ..
rm -rf jailkit-2.11*

Install fail2ban: This is optional but recommended, because the ISPConfig monitor tries to show the log:

apt-get install fail2ban

Install GlusterFS and Fuse:

apt-get -y --force-yes -t lenny-backports install fuse-utils
apt-get -y --force-yes install glusterfs-server glusterfs-client

Remove the GlusterFS example configuration files:

rm -f /etc/glusterfs/*.vol

Create the data directories for the GlusterFS volumes:

mkdir /data/
mkdir /data/export-mysql
mkdir /data/export-mysql-ns
mkdir /data/export-vmail
mkdir /data/export-vmail-ns
mkdir /data/export-www
mkdir /data/export-www-ns

Create the GlusterFS server configuration file:

vi /etc/glusterfs/glusterfsd.vol

# Configuration for the mysql server volume
volume posix-mysql
  type storage/posix
  option directory /data/export-mysql
  option background-unlink yes
end-volume
volume locks-mysql
  type features/locks
  option mandatory-locks on
  subvolumes posix-mysql
end-volume
volume brick-mysql
  type performance/io-threads
  option thread-count 8
  subvolumes locks-mysql
end-volume
# Configuration for the vmail server volume
volume posix-vmail
  type storage/posix
  option directory /data/export-vmail
end-volume
volume locks-vmail
  type features/locks
  subvolumes posix-vmail
end-volume
volume brick-vmail
  type performance/io-threads
  option thread-count 8
  subvolumes locks-vmail
end-volume
# Configuration for the www server volume
volume posix-www
  type storage/posix
  option directory /data/export-www
end-volume
volume locks-www
  type features/locks
  subvolumes posix-www
end-volume
volume brick-www
  type performance/io-threads
  option thread-count 8
  subvolumes locks-www
end-volume
# export all volumes
volume server
  type protocol/server
  option transport-type tcp
  subvolumes brick-mysql brick-vmail brick-www
  
  # authentication options for the mysql volume
  option auth.addr.brick-mysql.allow 192.168.0.105,192.168.0.106
  option auth.login.brick-mysql.allow user-mysql
  option auth.login.user-mysql.password 7wQav7ExkFg6eW
  
  # Authentification options for the vmail volume
  option auth.addr.brick-vmail.allow 192.168.0.105,192.168.0.106
  option auth.login.brick-vmail.allow user-vmail
  option auth.login.user-vmail.password 7wQav7ExkFg6eW
  
  # authentification options for www
  option auth.addr.brick-www.allow 192.168.0.105,192.168.0.106
  option auth.login.brick-www.allow user-www
  option auth.login.user-www.password 7wQav7ExkFg6eW
  
end-volume

Replace the IP addresses with the IPs from your servers and replace the password 7wQav7ExkFg6eW with a password of your choice.

Start the GlusterFS server:

/etc/init.d/glusterfs-server start

Now we create the three client volume files that we need to mount the GlusterFS filesystems.

vi /etc/glusterfs/glusterfs-mysql.vol

volume remote1-mysql
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.0.105
  option remote-subvolume brick-mysql
  option username user-mysql
  option password 7wQav7ExkFg6eW
end-volume
volume remote2-mysql
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.0.106
  option remote-subvolume brick-mysql
  option username user-mysql
  option password 7wQav7ExkFg6eW
end-volume
volume replicate-mysql
  type cluster/replicate
  subvolumes remote1-mysql remote2-mysql
end-volume
volume cache-mysql
  type performance/io-cache
  option cache-size 25MB
  subvolumes replicate-mysql
end-volume

vi /etc/glusterfs/glusterfs-vmail.vol

volume remote1-vmail
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.0.105
  option remote-subvolume brick-vmail
  option username user-vmail
  option password 7wQav7ExkFg6eW
end-volume
volume remote2-vmail
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.0.106
  option remote-subvolume brick-vmail
  option username user-vmail
  option password 7wQav7ExkFg6eW
end-volume
volume replicate-vmail
  type cluster/replicate
  subvolumes remote1-vmail remote2-vmail
end-volume
volume writebehind-vmail
  type performance/write-behind
  option window-size 1MB
  subvolumes replicate-vmail
end-volume
volume cache-vmail
  type performance/io-cache
  option cache-size 256MB
  subvolumes writebehind-vmail
end-volume

vi /etc/glusterfs/glusterfs-www.vol

volume remote1-www
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.0.105
  option remote-subvolume brick-www
  option username user-www
  option password 7wQav7ExkFg6eW
end-volume
volume remote2-www
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.0.106
  option remote-subvolume brick-www
  option username user-www
  option password 7wQav7ExkFg6eW
end-volume
volume replicate-www
  type cluster/replicate
  subvolumes remote1-www remote2-www
end-volume
volume writebehind-www
  type performance/write-behind
  option window-size 1MB
  subvolumes replicate-www
end-volume
volume cache-www
  type performance/io-cache
  option cache-size 256MB
  subvolumes writebehind-www
end-volume

Before we mount our volumes to the directories, we will have to stop some services and back up current data:

/etc/init.d/mysql stop
/etc/init.d/apache2 stop
/etc/init.d/postfix stop
/etc/init.d/dovecot stop

mv /var/lib/mysql /var/lib/mysql_bak
mv /var/www /var/www_bak

mkdir /var/lib/mysql
mkdir /var/www
mkdir /var/vmail

vi /etc/fstab

Add the lines:

/etc/glusterfs/glusterfs-mysql.vol  /var/lib/mysql  glusterfs  defaults  0  0
/etc/glusterfs/glusterfs-vmail.vol  /var/vmail  glusterfs  defaults  0  0
/etc/glusterfs/glusterfs-www.vol  /var/www  glusterfs  defaults  0  0

Now mount the drives by running:

mount -a

Chown the mysql directory to user and group mysql:

chown mysql:mysql /var/lib/mysql

Copy the mysql and www data back.

cp -prf /var/lib/mysql_bak/* /var/lib/mysql/
cp -prf /var/www_bak/* /var/www/

Copy back the data (only on the master server! Skip this step on the slave!).

Switch off innodb as it causes locking problems when MySQL starts.

vi /etc/mysql/my.cnf

Change the line:

#skip-innodb

to:

skip-innodb   

and add the line

innodb_file_per_table  

in the [mysqld] section of the my.cnf file.

When you set up the slave server, copy the file /etc/mysql/debian.cnf file from the master server to the slave server before you start MySQL again!

Now start the services again:

/etc/init.d/mysql start
/etc/init.d/apache2 start
/etc/init.d/postfix start
/etc/init.d/dovecot start

Change the GlusterFS boot order to ensure that the GlusterFS server gets always started before MySQL.

mv /etc/rc2.d/S20glusterfs-server /etc/rc2.d/S19glusterfs-server
mv /etc/rc3.d/S20glusterfs-server /etc/rc3.d/S16glusterfs-server
mv /etc/rc4.d/S20glusterfs-server /etc/rc4.d/S16glusterfs-server
mv /etc/rc5.d/S20glusterfs-server /etc/rc5.d/S16glusterfs-server

Share this page:

24 Comment(s)

Add comment

Comments

From: at: 2010-11-09 15:39:20

All,


After I followed this post for creating mirror ispconfig, the glusterfs-server will not work normally to mount the volume.  I added following line to fstab and run " mount -a". It is fine to mount the volume. After I restart the server, the volumes will not be mount and show "transport endpoint is not connected gluster" with df command show.


/etc/glusterfs/glusterfs-mysql.vol  /var/lib/mysql  glusterfs  defaults  0  0
/etc/glusterfs/glusterfs-vmail.vol  /var/vmail  glusterfs  defaults  0  0
/etc/glusterfs/glusterfs-www.vol  /var/www  glusterfs  defaults  0  0


Anyone can help to solve the problem.

From: Anonymous at: 2010-11-05 13:56:03

I second that motion.


Too bad I didnt see your comment before I figured it out :)

From: Shafeek at: 2010-10-15 07:11:59

Hi Till


Thanks for this great tutorial. 


Regarding the Boot order of Glusterfs, I think that the line 


 mv /etc/rc2.d/S20glusterfs-server /etc/rc2.d/S19glusterfs-server


should have been 


 mv /etc/rc2.d/S20glusterfs-server /etc/rc2.d/S16glusterfs-server


thanks

From: crisstm at: 2011-05-06 18:59:11

apt-get clean

 and the solved problem to me :)

 

From: Rene at: 2011-04-19 19:54:39

you have found a solution. as I feel the same

From: saverserver at: 2011-08-22 10:36:40

nano /etc/apt/sources.list


Aktuell:


deb http://volatile.debian.org/debian-volatile lenny/volatile main contrib non-free
deb http://backports.debian.org/debian-backports lenny-backports main

From: at: 2011-09-01 14:55:42

In the file /etc/fstab it should be included _netdev 

Eg

 /etc/glusterfs/glusterfs-mysql.vol  /var/lib/mysql  glusterfs  defaults,_netdev  0  0

/etc/glusterfs/glusterfs-vmail.vol  /var/vmail  glusterfs  defaults,_netdev  0  0

/etc/glusterfs/glusterfs-www.vol  /var/www  glusterfs  defaults,_netdev  0  0

This is necessary because otherwise the volumes would not start automatically when the server restarts.

From: AlexGer83 at: 2011-09-15 21:19:03

Hi and thank's for this howto.

I have a Problem where i have to start the Slave-server's MySQL Server!



I've modified the /etc/mysql/debian.cnf file on the Slave-Server

-> replaced the original Content with the Master-Server's content




Now start the services again:

/etc/init.d/mysql start

...





root@L-SRV-2:/tmp# /etc/init.d/mysql start

Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!




nano /var/log/syslog says:


Sep 15 23:03:40 L-SRV-2 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql

Sep 15 23:03:40 L-SRV-2 mysqld: 110915 23:03:40 [Note] Plugin 'FEDERATED' is disabled.

Sep 15 23:03:40 L-SRV-2 mysqld: 110915 23:03:40 [Note] Plugin 'InnoDB' is disabled.

Sep 15 23:03:40 L-SRV-2 mysqld: /usr/sbin/mysqld: Table 'mysql.plugin' doesn't exist

Sep 15 23:03:40 L-SRV-2 mysqld: 110915 23:03:40 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.

Sep 15 23:03:40 L-SRV-2 mysqld: 110915 23:03:40 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist

Sep 15 23:03:40 L-SRV-2 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

Sep 15 23:03:54 L-SRV-2 /etc/init.d/mysql[27916]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in

Sep 15 23:03:54 L-SRV-2 /etc/init.d/mysql[27916]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed

Sep 15 23:03:54 L-SRV-2 /etc/init.d/mysql[27916]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'

Sep 15 23:03:54 L-SRV-2 /etc/init.d/mysql[27916]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!





Where is the Problem?

I googled around and couldn't find any helping posts.



Thank's for y'all help.

AlexGer83

From: at: 2011-10-21 19:32:54

I had a similar problem, it would be two tips.
1: use Wheezy packages. I found bugs in Squeeze.
2: permissions - for example, the MySQL UID/GID are differents on the 2 servers.

From: Crackerjack at: 2012-02-13 17:05:28

As Fuse is not launched at boot time, gluster can't mount FS. To make it work, just add line fuse in /etc/modules.

cf. : http://asaclue.wordpress.com/2011/03/26/mount-glusterfs-volume-at-boot-time-in-ubuntu-lucid-10-04/

From: Joe Julian at: 2012-06-04 22:27:01

Please do not follow this tutorial for setting up GlusterFS.


The instructions here are for version 2.0. At the time of this comment, 3.3 has been released and does not require the editing of any .vol files.

From: admin at: 2012-06-05 07:40:07

This tutorial is for Debian Lenny, and Lenny comes (came) with GlusterFS 2. Therefore these instructions are fine.

From: ICNET at: 2010-08-10 17:45:07

Excellent guide!


Is having a mirror server supposed to provide automatic redundancy? as in, if server 1 goes down, will server 2 automatically take over? If so, how is this possible when it comes to DNS? since the 2 servers have different IP addresses and hostnames, the bind server is not exactly the same so would people have to use additional name servers in their domain names so that they can have true redundancy?


Thank you.

From: Liquid at: 2010-08-22 14:13:10

Hi, great tutorial.


 But i have problem et the end. After i install second server, then i cannoct login to the ispconfig management interface on port 8080.


Any idea what is wrong???

From: fiwal61 at: 2010-09-03 14:29:21

Hi, this turorial is realy great. But I think there is an error in step 4. If you do the rm -f /var/www/ispconfig then you delete the ipsconfig at all since the /var/www is clustered using glusterfs. Is this right?

From: cps at: 2010-11-20 20:15:07

same here,, trying to tail /var/log/syslog the problem directed to /var/lib/mysql/mysql/*.HYM ls -l /var/lib/mysql/ drwxr-xr-x 2 ntp ntp 4096 2010-11-20 23:46 mysql


the owner of mysql directory is ntp,, not the mysql himself.. trying to chown -R mysql:mysql /var/lib/mysql/mysql wow,, the slave server is running!!! ok,, check the master server.. errrr... we've got many error folks..


ERROR 1017 (HY000) at line 1: Can't find file: './mysql/columns_priv.frm' (errno: 13) ERROR 1017 (HY000) at line 1: Can't find file: './mysql/db.frm' (errno: 13) ERROR 1017 (HY000) at line 1: Can't find file: './mysql/func.frm' (errno: 13) ERROR 1017 (HY000) at line 1: Can't find file: './mysql/help_category.frm' (errno: 13) ERROR 1017 (HY000) at line 1: Can't find file: './mysql/help_keyword.frm' (errno: 13)


now i'm trying to figure it out,, did anybody can resolve this problem!?

From: Anonymous at: 2010-09-21 09:29:10

Has anyone tried MySQL in a Gridlock high availability cluster?

From: Anonymous at: 2010-11-05 18:14:05

I get the same issue after adding the second server.


When I try to log into the master control panel, I get a 'access denied for user ispconfig@localhost' message.


I have not changed anything and it worked before adding the second server.


Have you managed to resolve the issue?

From: Anonymous at: 2010-11-08 15:58:11

I followed the step, worked during configuration. After I reboot the server,  it show following :


df: `/var/lib/mysql': Transport endpoint is not connected


How Can I fix it.

From: ihamouda at: 2010-11-09 00:10:10

Did you find a solution, I'm facing the same problem

From: at: 2010-11-24 07:18:41

This does not delete ispconfig, as /var/www/ispconfig is not the ispconfig installation directory, it is just a symlink.

From: at: 2010-11-24 07:19:31

The interface is only available on the master server. You can not login to the interface on the slave.

From: at: 2011-01-01 04:31:41

Before you install the control panel on the second server login to phpmyadmin on the primary server and export the mysql  database.  Install the second server as per the instructions.  When completed it will have replace your ispconfig@localhost password on the primary server.  Replace this password with the one from the mysql dump (Just this password) and then reload your privileges

From: Ron Nitzsche at: 2014-06-17 03:04:50


Hello,



load balancing should do the trick.



Greets