#1  
Old 25th August 2011, 19:58
nayr nayr is offline
Junior Member
 
Join Date: Aug 2011
Posts: 17
Thanks: 2
Thanked 11 Times in 6 Posts
Default Mirroring Questions & Info (DRBD+OCFS2 instead of GlusterFS)

Ive got 2 identical servers I am trying to do a master-master cluster with.

I have mysql running in master-master replication successfully across both servers and using DRBD + OCFS2 I also have a shared filesystem for www/vmail.

I installed them both identically, websvs1 & websvs2; and configured websvs2 to be a mirror of websvs1 in ISPConfig3.

My first test was to add a client & site, websvs1 created the directory structure and its visible from websvs2.. however the system users accounts were not created on websvs2 so the UID/GID of the files are unknown from second server... did I mess something up or is this expected? Should i just start syncing the passwd/shadow/group files using something else? (csync2 perhaps). I kinda expected the system user accounts to mirror each-other.

Regards,
Ryan
__________________
HostAP.net - ISPConfig3 powered WebHosting & Dedicated Servers

Last edited by nayr; 2nd September 2011 at 04:03.
Reply With Quote
Sponsored Links
  #2  
Old 25th August 2011, 20:14
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,717
Thanks: 820
Thanked 5,322 Times in 4,175 Posts
Default

Please ensure that the mysql master-master replication excludes the "mysql" and the ispconfig databases on both servers. ISPConfig is syncing the contents of these databases internally and ispconfig will fail if these databases are mirrored otherwise.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #3  
Old 26th August 2011, 05:34
nayr nayr is offline
Junior Member
 
Join Date: Aug 2011
Posts: 17
Thanks: 2
Thanked 11 Times in 6 Posts
Default

The mysql table is not replicated and I have both servers configured to use seperate databases (dbispconfig1 & dbispconfig2)..

I even have a 3rd server mirroring the first that I am hoping to use for shells/ftp thats not a member of the mysql-cluster and gets its filesys via NFS off the cluster.. That server is configured pretty close to the first two and runs in a VM, its not getting any updates to the passwd/shadow/group files so far..

I havent done much testing past simply creating a user and site, I had no problems mirroring DNS services on external servers when I tested it..

Only thing in the cron-logs on websvs2 is:
Code:
PHP Warning: Invalid argument supplied for foreach() in /usr/local/ispconfig/server/mods-available/monitor_core_module.inc.php on line 1118
/bin/sh: /usr/local/bin/run-getmail.sh: Permission denied
repquota: Operation not supported for filesystems with hidden quota files!
repquota: Operation not supported for filesystems with hidden quota files!
Reply With Quote
  #4  
Old 26th August 2011, 09:56
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,717
Thanks: 820
Thanked 5,322 Times in 4,175 Posts
Default

Quote:
/usr/local/bin/run-getmail.sh
Seems as if the getmail user ahs no permissions to run that script on your server.

Quote:
repquota: Operation not supported for filesystems with hidden quota files!
repquota: Operation not supported for filesystems with hidden quota files!
There must be a config problem with your Linux file system quota. Please check with:

repquota -avug

Regarding the config replication problem, most likely there is a problem with mysql permissions, so that the slave can not connect to the mysql master database. Please enable loglevel debug for the salve server in ispconfig master controlpanel and then run this command on the slave as root user and post the output:

/usr/local/ispconfig/server/server.sh
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #5  
Old 26th August 2011, 19:29
nayr nayr is offline
Junior Member
 
Join Date: Aug 2011
Posts: 17
Thanks: 2
Thanked 11 Times in 6 Posts
Default

I fixed the getmail.sh, was being denied by Trusted Path Execution (TPE) in GrSecurity, had to tweak permissions but its working now.

The quota error is apparently an issue with OCFS2, it does not support repquota.. I dunno what to do bout it other than look for a upstream fix of somesort.

I'll check out debug logs here shortly and see if that sheds any light.
Reply With Quote
  #6  
Old 26th August 2011, 23:25
nayr nayr is offline
Junior Member
 
Join Date: Aug 2011
Posts: 17
Thanks: 2
Thanked 11 Times in 6 Posts
Default

websvs1:
Quote:
26.08.2011-15:19 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:19 - DEBUG - Found 4 changes, starting update process.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Processed datalog_id 23
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Processed datalog_id 24
26.08.2011-15:19 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_delete'.
26.08.2011-15:19 - DEBUG - Calling function 'delete' from plugin 'apache2_plugin' raised by event 'web_domain_delete'.
26.08.2011-15:19 - DEBUG - Removing symlink: /etc/apache2/sites-enabled/nayr.com.vhost =>
26.08.2011-15:19 - DEBUG - Removing vhost file: /etc/apache2/sites-available/nayr.com.vhost
26.08.2011-15:19 - DEBUG - Removing website: /var/www/clients/client1/web3
26.08.2011-15:19 - DEBUG - Removing symlink: /var/www/nayr.com
26.08.2011-15:19 - DEBUG - Removing symlink: /var/www/clients/client1/nayr.com
26.08.2011-15:19 - DEBUG - Removing website logfile directory: /var/log/ispconfig/httpd/nayr.com
26.08.2011-15:19 - DEBUG - Processed datalog_id 27
26.08.2011-15:19 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_insert'.
26.08.2011-15:19 - DEBUG - Calling function 'insert' from plugin 'apache2_plugin' raised by event 'web_domain_insert'.
26.08.2011-15:19 - DEBUG - Creating Symlink: ln -s /var/log/ispconfig/httpd/nayr.com /var/www/clients/client0/web4/log
26.08.2011-15:19 - DEBUG - Creating Symlink: ln -s /var/www/clients/client0/web4/ /var/www/nayr.com
26.08.2011-15:19 - DEBUG - Creating Symlink: ln -s /var/www/clients/client0/web4/ /var/www/clients/client0/nayr.com
26.08.2011-15:19 - DEBUG - Adding the group: client0
26.08.2011-15:19 - DEBUG - Adding the user: web4
26.08.2011-15:19 - DEBUG - exec: chown -R web4:client0 /var/www/clients/client0/web4
26.08.2011-15:19 - DEBUG - exec: chown web4:client0 /var/www/clients/client0/web4
26.08.2011-15:19 - DEBUG - exec: chmod 751 /var/www/clients/client0/web4/
26.08.2011-15:19 - DEBUG - exec: chmod 751 /var/www/clients/client0/web4/*
26.08.2011-15:19 - DEBUG - exec: chmod 710 /var/www/clients/client0/web4/web
26.08.2011-15:19 - DEBUG - exec: chmod 777 /var/www/clients/client0/web4/tmp
26.08.2011-15:19 - DEBUG - exec: chmod 755 /var/www/clients/client0/web4/log
26.08.2011-15:19 - DEBUG - exec: usermod --groups sshusers web4
26.08.2011-15:19 - DEBUG - exec: chown web4:client0 /var/www/clients/client0/web4
26.08.2011-15:19 - DEBUG - exec: chown web4:client0 /var/www/clients/client0/web4/log/error.log
26.08.2011-15:19 - DEBUG - Disable SSL for:
26.08.2011-15:19 - DEBUG - Creating fastcgi starter script directory: /var/www/php-fcgi-scripts/web4/
26.08.2011-15:19 - DEBUG - Creating fastcgi starter script: /var/www/php-fcgi-scripts/web4/.php-fcgi-starter
26.08.2011-15:19 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/nayr.com.vhost
26.08.2011-15:19 - DEBUG - Creating the symlink: /etc/apache2/sites-enabled/nayr.com.vhost => /etc/apache2/sites-available/nayr.com.vhost
26.08.2011-15:19 - DEBUG - Apache status is: 1
26.08.2011-15:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
26.08.2011-15:19 - DEBUG - Apache online status after restart is: 1
26.08.2011-15:19 - DEBUG - Processed datalog_id 28
26.08.2011-15:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
26.08.2011-15:19 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:20 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:20 - DEBUG - No Updated records found, starting only the core.
26.08.2011-15:20 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:21 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:21 - DEBUG - No Updated records found, starting only the core.
26.08.2011-15:21 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:22 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:22 - DEBUG - No Updated records found, starting only the core.
26.08.2011-15:22 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
websvs2:
Code:
26.08.2011-15:19 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:19 - DEBUG - Found 8 changes, starting update process.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Processed datalog_id 21
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Processed datalog_id 22
26.08.2011-15:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
26.08.2011-15:19 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:20 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:20 - DEBUG - Found 4 changes, starting update process.
26.08.2011-15:20 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:21 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:21 - DEBUG - Found 4 changes, starting update process.
26.08.2011-15:21 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
Reply With Quote
  #7  
Old 27th August 2011, 01:28
nayr nayr is offline
Junior Member
 
Join Date: Aug 2011
Posts: 17
Thanks: 2
Thanked 11 Times in 6 Posts
Default

so I must have messed something up, uninstalled and cleaned up the databases then set it all up again... seems to be working so far thanks.

BTW DRBD+OCFS2 is performing way better than GlusterFS in my tests, plus it supports all kernel platforms.. not just x86. (Sparc64 is what Im running on).. The problem may be in getting Quota's to work properly... if your setting up a Primary/Secondary setup then you can use DRBD+ext3/ext4 and it'll probably work just fine... OCFS2 lets both servers write to the same FS

*edit*
upstream repquota works with OCFS2, uninstalled quotatool in debian and built the latest version off sourceforge and quotas work just fine.

Last edited by nayr; 27th August 2011 at 09:03.
Reply With Quote
  #8  
Old 29th August 2011, 13:18
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,717
Thanks: 820
Thanked 5,322 Times in 4,175 Posts
Default

Quote:
BTW DRBD+OCFS2 is performing way better than GlusterFS in my tests, plus it supports all kernel platforms.. not just x86. (Sparc64 is what Im running on).. The problem may be in getting Quota's to work properly... if your setting up a Primary/Secondary setup then you can use DRBD+ext3/ext4 and it'll probably work just fine... OCFS2 lets both servers write to the same FS
I've considered this setup as well for the next version of the guide as I was not satisfied with the glusterfs performance. May you post the commands and config files used in your setup so that we can include as option it in the multiserver guide?
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #9  
Old 29th August 2011, 21:02
nayr nayr is offline
Junior Member
 
Join Date: Aug 2011
Posts: 17
Thanks: 2
Thanked 11 Times in 6 Posts
 
Default

Sure, here are the issues ive found using Debian Squeeze.

Upgrade Kernel your self, I had issues w/OCFS2 in the 2.6.32 version so I updated kernel to latest version and its been smooth.. (After a while quotas stoped working and I had issues setting permissions on files, I upgraded kernel and reformatted and it hasent happened again.. yet) Take this time to install the GrSecurity patch, configure it for "high security" and disable/dont use TPE as it'll deny a few of ISPConfig3's scripts.. The GrSecurity patch will greatly harden your system, it comes with alot of additional security to chroots... use make-kpkg so it wraps your kernel into a .deb

after rebooting into new kernel I built the latest DRBD module against my kernel using:
http://oss.linbit.com/drbd/8.4/drbd-latest.tar.gz
Quote:
./configure --with-km --prefix=/usr
make
make install
uninstall quotatools debian package, download and build the latest off SF: http://sourceforge.net/projects/linu...s/quota-tools/ (I installed 4.0)
Code:
apt-get remove quotatools
apt-get builddep quotatools
Link your servers together directly, you dont want your servers becoming split-brain if a switch goes down. You should be using Gigabit Ethernet and they will auto-crossover with a straight-thru cable.. In my case bonded 3 Gigabit connections and enabled jumbo-frames:

/etc/network/interfaces
Quote:
auto bond0
iface bond0 inet static
address 192.168.253.2
netmask 255.255.255.0
network 192.168.253.0
slaves eth1 eth2 eth3
bond-mode balance-xor
bond-miimon 100
bond-downdelay 200
bond-updelay 200
post-up ifconfig bond0 mtu 9000
Configure your hosts file on each server so they will talk through the direct-link.

Fdisk drives on both servers so they have a common size, also create a partition for DRBD Meta-Disk. In my case I made a 1GB Meta Partition and a 1460GB File Partition on both servers. (if you can put the meta partition on separate disk for best performance)

create a /etc/drbd.d/r0.res file:
Quote:
resource r0 {
# startup {
# become-primary-on both;
# }
net {
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
rr-conflict disconnect;
}
on websvs1 {
device /dev/drbd0;
disk /dev/sda2;
address 192.168.253.1:7788;
meta-disk /dev/sda1;
}

on websvs2 {
device /dev/drbd0;
disk /dev/sdf2;
address 192.168.253.2:7788;
meta-disk /dev/sdf1;
}
}
Notice the startup section is commented out, you will want to uncomment this once everything is up and running.. but until then you dont want dual primaries until after we initialize DRBD.. (note starting it will error the first time, we just want to load modules)

primary:
Code:
/etc/init.d/drbd start
drbdadm create-md r0
drbdadm connect r0
drbdadm primary r0
on the slave:
Code:
/etc/init.d/drbd start
drbdadm create-md r0
drbdadm -- --clear-bitmap new-current-uuid r0 
drbdadm connect r0
drbdadm primary r0
This should start up DRBD and skip the inital sync..
/etc/init.d/drbd status
Code:
drbd driver loaded OK; device status:
version: 8.4.0 (api:1/proto:86-100)
GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by root@websvs1, 2011-08-23 02:44:22
m:res  cs         ro               ds                 p  mounted  fstype
0:r0   Connected  Primary/Primary  UpToDate/UpToDate  C
Now for OCFS2, configure backports on both servers
/etc/apt/sources.list
Code:
deb http://backports.debian.org/debian-backports squeeze-backports main
and install ocfs2-tools
Code:
apt-get update;apt-get install -t squeeze-backports ocfs2-tools
Configure OCFS2 cluster on both servers
/etc/ocfs2/cluster.conf
Code:
node:
        ip_port = 7777
        ip_address = 192.168.253.1
        number = 0
        name = websvs1
        cluster = websvs

node:
        ip_port = 7777
        ip_address = 192.168.253.2
        number = 1
        name = websvs2
        cluster = websvs

cluster:
        node_count = 2
        name = websvs
Then start OCFS2 with:
Code:
/etc/init.d/o2cb restart
on the primary lets format the DRBD raid using OCFS2:
Quote:
mkfs -t ocfs2 -N 2 -L ocfs2_drbd0 --fs-features=usrquota,grpquota /dev/drbd0
and then you should be able to mount it on both servers:
Code:
/sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv
check ocfs2 status:
Code:
/etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster websvs: Online
Heartbeat dead threshold = 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Active
then simply move /var/www to /srv and link it back, do the same for /var/vmail like the guide says too.

Now dont add /srv to fstab like you may be tempted, debain will try to fsck it at boot which will fail and force you to interact to get the system to boot..

I added a few lines to /etc/rc.local to set up the FileSystems.
Code:
/etc/init.d/ocfs2 start
/sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv
/etc/init.d/apache2 start
exit 0
Think that should cover it, I'll throw it into production soon and let you know if there is anything else that needs addressed.

Using DRBD+OCFS2 on 2 identical disks gives nearly native read speeds and about half the write speeds of a native disk.. uses very little resources compared to GlusterFS and the performance is leaps and bounds better.. I dont recommend putting your DB on it tho, use MySQL Replication for that and put your DB's on a local raid-0/10 or SSD if you can.

additional thoughts, if your going to be processing a ton of mail it might be beneficial performance wise to create a separate drbd resource for vmail storage and format it using a smaller block size.. now would be the perfect time to set that up.

feature request: It'd be great if ISPConfig3 monitored the DRBD & OCFS status like it does soft-raids..

Cheers,
-R

Last edited by nayr; 29th August 2011 at 22:09.
Reply With Quote
The Following 2 Users Say Thank You to nayr For This Useful Post:
falko (30th August 2011), till (29th August 2011)
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
DNS Mirroring Questions necrogami Installation/Configuration 4 2nd February 2011 03:26
lots of questions from noob papokergod Installation/Configuration 5 21st January 2009 13:19
Some basic pre-installation questions jeffw85 Installation/Configuration 1 26th November 2007 08:25
Hostname & DNS questions... SurfingAlien Installation/Configuration 6 16th October 2007 16:32
Questions and more questions... ellisgl General 1 21st July 2007 14:52


All times are GMT +2. The time now is 00:09.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.