Mirroring Questions

Discussion in 'Tips/Tricks/Mods' started by nayr, Aug 25, 2011.

  1. nayr

    nayr New Member

    Mirroring Questions & Info (DRBD+OCFS2 instead of GlusterFS)

    Ive got 2 identical servers I am trying to do a master-master cluster with.

    I have mysql running in master-master replication successfully across both servers and using DRBD + OCFS2 I also have a shared filesystem for www/vmail.

    I installed them both identically, websvs1 & websvs2; and configured websvs2 to be a mirror of websvs1 in ISPConfig3.

    My first test was to add a client & site, websvs1 created the directory structure and its visible from websvs2.. however the system users accounts were not created on websvs2 so the UID/GID of the files are unknown from second server... did I mess something up or is this expected? Should i just start syncing the passwd/shadow/group files using something else? (csync2 perhaps). I kinda expected the system user accounts to mirror each-other.

    Last edited: Sep 2, 2011
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Please ensure that the mysql master-master replication excludes the "mysql" and the ispconfig databases on both servers. ISPConfig is syncing the contents of these databases internally and ispconfig will fail if these databases are mirrored otherwise.
  3. nayr

    nayr New Member

    The mysql table is not replicated and I have both servers configured to use seperate databases (dbispconfig1 & dbispconfig2)..

    I even have a 3rd server mirroring the first that I am hoping to use for shells/ftp thats not a member of the mysql-cluster and gets its filesys via NFS off the cluster.. That server is configured pretty close to the first two and runs in a VM, its not getting any updates to the passwd/shadow/group files so far..

    I havent done much testing past simply creating a user and site, I had no problems mirroring DNS services on external servers when I tested it..

    Only thing in the cron-logs on websvs2 is:
    PHP Warning: Invalid argument supplied for foreach() in /usr/local/ispconfig/server/mods-available/monitor_core_module.inc.php on line 1118
    /bin/sh: /usr/local/bin/run-getmail.sh: Permission denied
    repquota: Operation not supported for filesystems with hidden quota files!
    repquota: Operation not supported for filesystems with hidden quota files!
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Seems as if the getmail user ahs no permissions to run that script on your server.

    There must be a config problem with your Linux file system quota. Please check with:

    repquota -avug

    Regarding the config replication problem, most likely there is a problem with mysql permissions, so that the slave can not connect to the mysql master database. Please enable loglevel debug for the salve server in ispconfig master controlpanel and then run this command on the slave as root user and post the output:

  5. nayr

    nayr New Member

    I fixed the getmail.sh, was being denied by Trusted Path Execution (TPE) in GrSecurity, had to tweak permissions but its working now.

    The quota error is apparently an issue with OCFS2, it does not support repquota.. I dunno what to do bout it other than look for a upstream fix of somesort.

    I'll check out debug logs here shortly and see if that sheds any light.
  6. nayr

    nayr New Member

    26.08.2011-15:19 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.08.2011-15:19 - DEBUG - Found 8 changes, starting update process.
    26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
    26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
    26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
    26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
    26.08.2011-15:19 - DEBUG - Processed datalog_id 21
    26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
    26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
    26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
    26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
    26.08.2011-15:19 - DEBUG - Processed datalog_id 22
    26.08.2011-15:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    26.08.2011-15:19 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.08.2011-15:20 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.08.2011-15:20 - DEBUG - Found 4 changes, starting update process.
    26.08.2011-15:20 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.08.2011-15:21 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    26.08.2011-15:21 - DEBUG - Found 4 changes, starting update process.
    26.08.2011-15:21 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
  7. nayr

    nayr New Member

    so I must have messed something up, uninstalled and cleaned up the databases then set it all up again... seems to be working so far thanks.

    BTW DRBD+OCFS2 is performing way better than GlusterFS in my tests, plus it supports all kernel platforms.. not just x86. (Sparc64 is what Im running on).. The problem may be in getting Quota's to work properly... if your setting up a Primary/Secondary setup then you can use DRBD+ext3/ext4 and it'll probably work just fine... OCFS2 lets both servers write to the same FS

    upstream repquota works with OCFS2, uninstalled quotatool in debian and built the latest version off sourceforge and quotas work just fine.
    Last edited: Aug 27, 2011
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    I've considered this setup as well for the next version of the guide as I was not satisfied with the glusterfs performance. May you post the commands and config files used in your setup so that we can include as option it in the multiserver guide?
  9. nayr

    nayr New Member

    Sure, here are the issues ive found using Debian Squeeze.

    Upgrade Kernel your self, I had issues w/OCFS2 in the 2.6.32 version so I updated kernel to latest version and its been smooth.. (After a while quotas stoped working and I had issues setting permissions on files, I upgraded kernel and reformatted and it hasent happened again.. yet) Take this time to install the GrSecurity patch, configure it for "high security" and disable/dont use TPE as it'll deny a few of ISPConfig3's scripts.. The GrSecurity patch will greatly harden your system, it comes with alot of additional security to chroots... use make-kpkg so it wraps your kernel into a .deb

    after rebooting into new kernel I built the latest DRBD module against my kernel using:
    uninstall quotatools debian package, download and build the latest off SF: http://sourceforge.net/projects/linuxquota/files/quota-tools/ (I installed 4.0)
    apt-get remove quotatools
    apt-get builddep quotatools
    Link your servers together directly, you dont want your servers becoming split-brain if a switch goes down. You should be using Gigabit Ethernet and they will auto-crossover with a straight-thru cable.. In my case bonded 3 Gigabit connections and enabled jumbo-frames:

    Configure your hosts file on each server so they will talk through the direct-link.

    Fdisk drives on both servers so they have a common size, also create a partition for DRBD Meta-Disk. In my case I made a 1GB Meta Partition and a 1460GB File Partition on both servers. (if you can put the meta partition on separate disk for best performance)

    create a /etc/drbd.d/r0.res file:
    Notice the startup section is commented out, you will want to uncomment this once everything is up and running.. but until then you dont want dual primaries until after we initialize DRBD.. (note starting it will error the first time, we just want to load modules)

    /etc/init.d/drbd start
    drbdadm create-md r0
    drbdadm connect r0
    drbdadm primary r0
    on the slave:
    /etc/init.d/drbd start
    drbdadm create-md r0
    drbdadm -- --clear-bitmap new-current-uuid r0 
    drbdadm connect r0
    drbdadm primary r0
    This should start up DRBD and skip the inital sync..
    /etc/init.d/drbd status
    drbd driver loaded OK; device status:
    version: 8.4.0 (api:1/proto:86-100)
    GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by root@websvs1, 2011-08-23 02:44:22
    m:res  cs         ro               ds                 p  mounted  fstype
    0:r0   Connected  Primary/Primary  UpToDate/UpToDate  C
    Now for OCFS2, configure backports on both servers
    deb http://backports.debian.org/debian-backports squeeze-backports main
    and install ocfs2-tools
    apt-get update;apt-get install -t squeeze-backports ocfs2-tools
    Configure OCFS2 cluster on both servers
            ip_port = 7777
            ip_address =
            number = 0
            name = websvs1
            cluster = websvs
            ip_port = 7777
            ip_address =
            number = 1
            name = websvs2
            cluster = websvs
            node_count = 2
            name = websvs
    Then start OCFS2 with:
    /etc/init.d/o2cb restart
    on the primary lets format the DRBD raid using OCFS2:
    and then you should be able to mount it on both servers:
    /sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv
    check ocfs2 status:
    /etc/init.d/o2cb status
    Driver for "configfs": Loaded
    Filesystem "configfs": Mounted
    Stack glue driver: Loaded
    Stack plugin "o2cb": Loaded
    Driver for "ocfs2_dlmfs": Loaded
    Filesystem "ocfs2_dlmfs": Mounted
    Checking O2CB cluster websvs: Online
    Heartbeat dead threshold = 31
      Network idle timeout: 30000
      Network keepalive delay: 2000
      Network reconnect delay: 2000
    Checking O2CB heartbeat: Active
    then simply move /var/www to /srv and link it back, do the same for /var/vmail like the guide says too.

    Now dont add /srv to fstab like you may be tempted, debain will try to fsck it at boot which will fail and force you to interact to get the system to boot..

    I added a few lines to /etc/rc.local to set up the FileSystems.
    /etc/init.d/ocfs2 start
    /sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv
    /etc/init.d/apache2 start
    exit 0
    Think that should cover it, I'll throw it into production soon and let you know if there is anything else that needs addressed.

    Using DRBD+OCFS2 on 2 identical disks gives nearly native read speeds and about half the write speeds of a native disk.. uses very little resources compared to GlusterFS and the performance is leaps and bounds better.. I dont recommend putting your DB on it tho, use MySQL Replication for that and put your DB's on a local raid-0/10 or SSD if you can.

    additional thoughts, if your going to be processing a ton of mail it might be beneficial performance wise to create a separate drbd resource for vmail storage and format it using a smaller block size.. now would be the perfect time to set that up.

    feature request: It'd be great if ISPConfig3 monitored the DRBD & OCFS status like it does soft-raids..

    Last edited: Aug 29, 2011

Share This Page