[SOLVED] Multiserver Sync Issues

Discussion in 'Installation/Configuration' started by mccyberix, Mar 26, 2019.

  1. mccyberix

    mccyberix Member

    First I want to say THANK YOU for this amazing piece of software!!!

    Since more than 10 Years I’m running ISPConfig in a multiserver setup and had never major issues with it but since I’ve decided to upgrade my Cluster to Debian 9 I’m facing a totally strange behaviour which gives me headaches. The installation went fine and when finished I can login into ISPC and everything is working as it should but when I configure the nodes as a mirror of the master server, odd things start to happen, first I will give you an overview of my configuration:

    I have 3 nodes which are all configured the same (2 of them are running on an ESXi 6.5 and one is installed on a VPS) The ESXi Servers are HP ProLiants DL360 G7 with more than enough resources.

    The config is based on the https://www.howtoforge.com/tutorial...-9-stretch-apache-bind-dovecot-ispconfig-3-1/ and https://www.howtoforge.com/tutorial...abase-cluster-on-debian-8.4-with-ispconfig-3/

    Except of the Percona Xtradb everything is configured according to the above tuts.

    OS: Debian 9.8
    Server-Typ: Percona Server
    Server-Version: 5.6.43-84.3-56 - Percona XtraDB Cluster (GPL), Release rel84.3, Revision e2908fe, WSREP version 28.32, wsrep_28.32
    Protokoll-Version: 10

    I have to mention that I had a fine working cluster for years with Percona Xtradb 5.6 on Debian 8, therefore I know that Percona is compatible with ISPC.

    The issue I’m facing is that when I add the nodes as mirror servers in ISPConfig3 it breaks all three systems, even it is a fresh install without users, websites or mailboxes. The strange thing is when I go to “System->Server Config” and click on the “Web” tab ISPConfig3 starts to sync and this is rendering all my systems inaccessible. The sync is triggered only by switching the tabs without changing anything (I know that the multipage forms are AutoSaving when switching the pages and this is probably triggering the sync. ) but when the sync is triggered ISPConfig and Phpmyadmin are not able to connect to the DB anymore and the network interfaces go down on each node, only the master is still available over the network. I have reinstalled the cluster meanwhile 20x from scratch, I have tried first with Percona 5.7 but there are some incompatibilities and during ISPConfig setup only 37 tables are created and there are no sys_ tables present, therefore I’ve decided to stay with the Percona 5.6 Version which I know that is working with ISPConfig3. I have also several times checked the credentials in various config files, all users and passwords are correct and tested.

    The only thing I can do is to bring the cluster down, re-bootstrap the mysql-cluster and then ISPConfig3 is working fine until the first sync where everything breaks again.

    I would appreciate some advice where to look or what to try because I’m trying to bring the cluster up since more than 5 weeks and I’m at the end of my wisdom!´

    Thank You!!!

    Hoewer here is the output of the test script:

    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] ISPConfig is installed.
    ##### ISPCONFIG #####
    ISPConfig version is 3.1.13p1
    ##### VERSION CHECK #####
    [INFO] php (cli) version is 7.0.33-0+deb9u3
    ##### PORT CHECK #####
    ##### MAIL SERVER CHECK #####
    [INFO] I found the following web server(s):
             Apache 2 (PID 14853)
    [INFO] I found the following mail server(s):
            Postfix (PID 3787)
    [INFO] I found the following pop3 server(s):
            Dovecot (PID 3925)
    [INFO] I found the following imap server(s):
            Dovecot (PID 3925)
    [INFO] I found the following ftp server(s):
            PureFTP (PID 4417)
    ##### LISTENING PORTS #####
    (only           ()
    Local           (Address)
    [anywhere]:995          (3925/dovecot)
    [localhost]:10023               (1057/postgrey)
    [localhost]:10024               (3823/amavisd-new)
    [anywhere]:10248                (3787/master)
    [localhost]:10025               (3787/master)
    [localhost]:10026               (3823/amavisd-new)
    [localhost]:10027               (3787/master)
    [anywhere]:587          (3787/master)
    [localhost]:11211               (553/memcached)
    [anywhere]:110          (3925/dovecot)
    [anywhere]:143          (3925/dovecot)
    [anywhere]:10000                (1317/perl)
    [anywhere]:465          (3787/master)
    ***.***.***.***:53              (4431/named)
    [localhost]:53          (4431/named)
    [anywhere]:21           (4417/pure-ftpd)
    [anywhere]:22           (648/sshd)
    [anywhere]:4567         (2171/mysqld)
    [localhost]:953         (4431/named)
    [anywhere]:25           (3787/master)
    [anywhere]:26           (3787/master)
    [anywhere]:993          (3925/dovecot)
    *:*:*:*::*:995          (3925/dovecot)
    *:*:*:*::*:10023                (1057/postgrey)
    *:*:*:*::*:10024                (3823/amavisd-new)
    [localhost]0248         (3787/master)
    *:*:*:*::*:10026                (3823/amavisd-new)
    *:*:*:*::*:3306         (2171/mysqld)
    *:*:*:*::*:587          (3787/master)
    [localhost]10           (3925/dovecot)
    [localhost]43           (3925/dovecot)
    [localhost]0000         (1317/perl)
    *:*:*:*::*:80           (14853/apache2)
    *:*:*:*::*:8080         (14853/apache2)
    *:*:*:*::*:465          (3787/master)
    *:*:*:*::*:8081         (14853/apache2)
    *:*:*:*::*:53           (4431/named)
    *:*:*:*::*:21           (4417/pure-ftpd)
    *:*:*:*::*:22           (648/sshd)
    *:*:*:*::*:953          (4431/named)
    *:*:*:*::*:25           (3787/master)
    *:*:*:*::*:26           (3787/master)
    *:*:*:*::*:443          (14853/apache2)
    *:*:*:*::*:993          (3925/dovecot)
    ##### IPTABLES #####
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    f2b-postfix-sasl  tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 25
    f2b-dovecot  tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 110,995,143,993,587,465,4190
    f2b-pure-ftpd  tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 21
    f2b-sshd   tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 22
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    Chain f2b-dovecot (1 references)
    target     prot opt source               destination
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    Chain f2b-postfix-sasl (1 references)
    target     prot opt source               destination
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    Chain f2b-pure-ftpd (1 references)
    target     prot opt source               destination
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    Chain f2b-sshd (1 references)
    target     prot opt source               destination
    REJECT     all  --  ***.***.***.***       [anywhere]/0            reject-with icmp-port-unreachable
    REJECT     all  --  ***.***.***.***       [anywhere]/0            reject-with icmp-port-unreachable
    RETURN     all  --  [anywhere]/0            [anywhere]/0
  2. mccyberix

    mccyberix Member

    after some more research I have at least managed not to end up with a totally broken ISPConfig instances after resync. I have added ispconfig users with all hosts option for testing and this seems to prevent the ISPConfig instance to get destroyed when the strange sync issue appears. Now it seems like only the network connection is droped on the nodes when ISPConfig is syncing, bringing the interface up (ifup ens192) brings the instance back and everything works well untill some syncing brings the network interfaces down again.

    Now the big question is what causes the interfaces to go down, it's defenitelly an ISPConfig task which is responsible for this, I have tested it now several times and the nodes stay up and are flawlessly working until I change something in ISPConfig.

    Is there a way to track what is going on when ISPConfig is syncing or has anybody an idea what could bring the interface down?

    thanks in advance!!!
  3. mccyberix

    mccyberix Member

    just a tought, is it maybe possible that ISPConfig has the "eth0" NIC name somewhere hardcoded and because of the new naming "ens3" it causes incompatibilities?
  4. mccyberix

    mccyberix Member

    I think that was the solution, I have renamed the nic's to "eth0" and the headache from the last 5 weeks is gone :D
    it looks like everything is working now as it should, I will report it if I'm wrong...

    however, the problem was caused by the NIC naming, probably ISPConfig has the name "eth0" hardcoded somwhere and therefore it's not compatible with Debian 9x where the NIC's are named "ens(x)". This problem does not ocurre on single-server systems so if you face similar issues rename the NIC to "eth0" and you are out of trouble!!!
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    I use Debian 9 on my multiserver system and I don't have any sync problems. The sync happens trough mysql, the name of the NIC is nowhere set therefore in ISPConfig.
  6. mccyberix

    mccyberix Member

    Hello Till,
    I can't say if my asumption is right, but the line: "Do not enable this option if your network interface is not eth0" in ISPConfig brought me to the idea to rename the NIC, despite I didn't check the box ;). All I know is that I have struggled for weeks setting up a three node cluster (which I have done successfully really a lot of times) and when I renamed the NIC to "eth0" all the odd troubles with network dropouts went away.
    Before I have renamed the NICS to "eth0" I was not able to sync over ISPConfig without loosing network conectivity on the slave nodes, I have reinstalled all three systems from scratch several times with the same results.
    Of course its also possible that some other software involved in the sync process has caused the connection dropout because of the NIC name which was "ens3".

    However my Cluster is now up and running stable!

    Anyway thank you for the amazing software and the migration tool is also awesome which spared me a lot of time migrating the users, sites, DB's and mailboxes!!!
    Keep doing the good work and have a nice day!!!

Share This Page