ISPConfig 3.1.15p2 loses configuration of web server, db server, mail server, and firewall server

Discussion in 'Installation/Configuration' started by intljs, Dec 30, 2019.

  1. intljs

    intljs New Member

    My old server uses 3.1.13 and does not exhibit this issue. Ubuntu 18.04 nginx mysql My new server exhibits the issue. Ubuntu 18.04 nginx mariadb

    On my new server, I used the perfect server guide for Ubuntu 18.04, nginx, and ispconfig 3. Then I took a snapshot of the old server and migrated it over - client directories/sites, ISPConfig files/configurations, and other configurations. I used this site's step as a guideline, but made alterations along the way (mostly what got backed up and what got restored)
    www
    zvonimir-buzanic.
    from.
    hr/
    migration-of-ispconfig-3-to-new-server


    I also migrated from MySQL to Mariadb during the move, by not making backups of the Mysql schema and config and reinstalling that portion from scratch instead.

    I then upgraded to the latest version of ispconfig 3.1.15p2 on the new server.

    I've noticed very frequently (around once a day) I can no longer log into the ISPConfig control panel. Running through all the checks I've found on various howtoforge forum threads, none of them appeared to work. So I reinstall ISPConfig using the update.php script. Here is the output:

    [email protected]:/tmp/update_stable.sh.CmXozREuP2/ispconfig3_install/install# php -q update.php
    --------------------------------------------------------------------------------
    _____ ___________ _____ __ _ ____
    |_ _/ ___| ___ \ / __ \ / _(_) /__ \
    | | \ `--.| |_/ / | / \/ ___ _ __ | |_ _ __ _ _/ /
    | | `--. \ __/ | | / _ \| '_ \| _| |/ _` | |_ |
    _| |_/\__/ / | | \__/\ (_) | | | | | | | (_| | ___\ \
    \___/\____/\_| \____/\___/|_| |_|_| |_|\__, | \____/
    __/ |
    |___/
    --------------------------------------------------------------------------------
    >> Update
    Operating System: Ubuntu 18.04.3 LTS (Bionic Beaver)
    This application will update ISPConfig 3 on your server.
    Shall the script create a ISPConfig backup in /var/backup/ now? (yes,no) [yes]:
    Creating backup of "/usr/local/ispconfig" directory...
    Creating backup of "/etc" directory...
    mysqldump: Got error: 1153: "Got a packet bigger than 'max_allowed_packet' bytes" when using LOCK TABLES
    Checking ISPConfig database .. OK
    Starting incremental database update.
    Loading SQL patch file: /tmp/update_stable.sh.CmXozREuP2/ispconfig3_install/install/sql/incremental/upd_dev_collection.sql
    Reconfigure Permissions in master database? (yes,no) [no]:
    Service 'mail_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
    Service 'web_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
    Service 'firewall_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
    Service 'db_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
    Reconfigure Services? (yes,no,selected) [yes]:
    Configuring Postfix
    Configuring Dovecot
    Configuring Spamassassin
    Configuring Amavisd
    Configuring Getmail
    Configuring Pureftpd
    Configuring nginx
    Skipping config of Apps vhost
    Configuring Jailkit
    Configuring Ubuntu Firewall
    Configuring Database
    Updating ISPConfig
    ISPConfig Port [8080]:
    Create new ISPConfig SSL certificate (yes,no) [no]:

    Reconfigure Crontab? (yes,no) [yes]:
    Updating Crontab
    Restarting services ...
    Update finished.
    [email protected]:/tmp/update_stable.sh.CmXozREuP2/ispconfig3_install/install# uptime 16:23:42 up 11 days, 22:30, 1 user, load average: 0.80, 1.03, 0.95
    As you can see, the web_server, db_server, firewall_server, and mail_server appeared to lose their configuration - as they're showing disabled in the update. Once I run this, they're enabled and I can log in for another day... and then I have to do the update process again in order to access the control panel.

    Can anyone shed some light into why these settings are being lost? I am happy to provide any logs, etc that are needed to troubleshoot, I'm just not sure which logs are relevant for the issue. When I view nginx error log and mysql's log, nothing shows out of the ordinary - looks like the usual behavior from the server.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The output says that the services are disabled under system > server services in ISPConfig, it does not say that any settings or configurations are lost. As you encounter login problems as well, the cause of all your probloems might be that mariadb rejects logins or fails which would explain why the updater can't read the system state from mariadb and why the login fails too which requires a database connection as well.
     
  3. intljs

    intljs New Member

    What's interesting is that after I run the update script, everything looks normal within ISPConfig - I can log in just fine, edit / add clients/websites etc... works like normal. But if I come back to it tomorrow, no longer able to login and all those server services show disabled in the update script.

    I can confirm when its borked, that if I attempt to login the sessions table is updated for dbispconfig to show the login session was authorized and allowed, so it doesn't seem to be a db connectivity problem (at least for that small portion).

    Is it possible to do a fresh reinstall of ispconfig but without losing the clients/websites that are configured and their mapping? That might be the best way forward for me to be honest. Is it as simple as exporting those tables from the db, backing up the folder paths/symlinks... then reinstall ISPConfig, restore those tables in the db, and restore the folders?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Check the dbispconfig database with phpmyadmin, the relevant database table is 'server' and the colums are the *_server columns to see if anything in there changes between working and non-working state.

    Which exact error do you get when the login is denied? Did you try to use your web browser in anonymous mode to ensure that you don't have any cached items from your old ISPConfig installation?

    No. And a reinstall without replacing the config is what the update script has done already.
     
  5. intljs

    intljs New Member

    Thanks. Here's the output (after the recent run of update.php)
    `mail_server`, `web_server`, `dns_server`, `file_server`, `db_server`, `vserver_server`, `proxy_server`, `firewall_server`, `xmpp_server`
    1, 1, 0, 1, 1, 0, 0, 0, 0,

    As you can see, currently mail_server, web_server, db_server are enabled - firewall still shows disabled for some reason, but I'm less concerned about that as I did see some issues in the tracker about it. More importantly is the web/mail/db servers for this specific box.

    I will wait for the issue to show up again and get you the new values in a separate post in this thread.

    There is no error present on the GUI, and there is no error in the logs either - it's as if the login was actually approved (proven in auth log) , but looping back to the login screen. I thought at first maybe it was a client-side cookie issue, so I have tried in several different browsers (safari, firefox, chrome) both in normal and incognito/anonymous mode, and all browsers exhibit the same behavior - unable to login, no error message provided. I have also tried logging in with incorrect credentials and I get the appropriate "username or password incorrect" message.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    In addition, when the issue occurs again, try to run a optimize and repair on all tables in dbispconfig with phpmyadmin to see if any of them is broken.
     
  7. intljs

    intljs New Member

    OK, the issue just occurred again. Unable to log in, no error messages at the login screen, auth log says successful login, sys_sessions table shows the session added there w/ the same session ID as in the auth log.

    Nothing was done on the server from the last message, only the 2 cron jobs that run every minute for ISPConfig were run (server.sh & cron.sh)

    Here's the relevant output of the server table:
    `mail_server`, `web_server`, `dns_server`, `file_server`, `db_server`, `vserver_server`, `proxy_server`, `firewall_server`, `xmpp_server`
    1, 1, 0, 1, 1, 0, 0, 0, 0,
    You can see it is unchanged.

    Running optimize got me "OK" status for all 80 tables except for dns_ssl_ca, which offers a note that says the Table does not support optimize, doing recreate +... (message is truncated)

    Similarly, doing a repair also got me "OK" status for all 80 tables except for dns_ssl_ca, which says the storage engine for the table doesn't support it (also truncated)

    After performing these two operations, I attempted to log in again and was greeted w/ the same behavior - login page with no errors, auth log says successful login, sys_sessions table reflects the same session ID referenced in auth log.

    After doing all that, I ran the update.php script again... here's the output

    After the update script, I can log in again.
    Any other thoughts on where the issue might be?
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    When it happens the next time, don't update, instead, try to restart apache (or nginx) and then mariadb and try to re-login after each one. also check the syslog file for errors of the database system.
     
  9. Jesse Norell

    Jesse Norell ISPConfig Developer ISPConfig Developer

    Just a guess, but try increasing max_allowed_packet and see what happens. Maybe something fails during the cron job and leaves a table locked or such? (Though I don't know how the update script would fix that offhand, hence it's just a guess...)
     

Share This Page