Help needed after upgrade from Debian 8

Discussion in 'Installation/Configuration' started by Nebhead 2, Oct 28, 2021.

  1. Nebhead 2

    Nebhead 2 New Member

    Evening all,

    Having finally had some time/headspace to look after my own projects after being swamped at work the last 6 months, I realised I had a server running Debian 8 that was well overdue an upgrade.

    I followed the instructions at How to Upgrade Debian 8 (Jessie) to 9 (Stretch) safely (howtoforge.com) and it mostly went fairly smoothly. The only points of note were that:
    • At some point I ran into a mysql error (SHOW FUNCTION STATUS WHERE Db = ‘name’: Cannot load from mysql.proc. The table is probably corrupted) that necessitated me running mysql_upgrade in order for apt-get dist-upgrade to continue
    • When first running ispconfig_update.sh I encountered an error about "the storage engine not supporting repair" and to set "sql-mode = NO_ENGINE_SUBSTITUTIONS" in the mysql-config, which I duly did
    • After restarting mysql, I encountered an issue around "mysql.serviceFailed to restart mysql.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files" which I resolved by installing policykit-1
    • Note that all the above resolutions were solutions I found online - my sysadmin skills are not all that great and I wouldn't have the first clue on how to start debugging these errors myself without reference to anything else

    However, after (re-)running ispconfig_update.sh I encountered a few further errors:

    Code:
    Creating backup of "/usr/local/ispconfig" directory...
    Creating backup of "/etc" directory...
    Checking ISPConfig database ..
    Some tables where not 'OK'. Please check the list below.
    
    dbispconfig.dns_ssl_ca
    note     : The storage engine for the table doesn't support repair
    
    Press enter to continue or CTRL-C to cancel the installation ..
    Starting incremental database update.
    Loading SQL patch file: /tmp/ispconfig3_install/install/sql/incremental/upd_dev_collection.sql
    Reconfigure Permissions in master database? (yes,no) [no]:
    
    and
    Code:
    Create new ISPConfig SSL certificate (yes,no) [no]:
    PHP Warning:  fopen(/var/www/php-fcgi-scripts/ispconfig/.php-fcgi-starter): failed to open stream: Operation not permitted in /tmp/ispconfig3_install/install/lib/install.lib.php on line 455
    PHP Warning:  fwrite() expects parameter 1 to be resource, boolean given in /tmp/ispconfig3_install/install/lib/install.lib.php on line 458
    PHP Warning:  fclose() expects parameter 1 to be resource, boolean given in /tmp/ispconfig3_install/install/lib/install.lib.php on line 459
    chmod: changing permissions of '/var/www/php-fcgi-scripts/ispconfig/.php-fcgi-starter': Operation not permitted
    chown: changing ownership of '/var/www/php-fcgi-scripts/ispconfig/.php-fcgi-starter': Operation not permitted
    Reconfigure Crontab? (yes,no) [yes]:
    
    The upshot is that services seem to be running, websites are up and available and I can send email, but:
    • the ISPConfig admin page is blank, loading the header/footer branding, search bar and logout button but nothing else
    • I can't seem to receive email currently

    Some additional info:
    Test Script output
    Code:
    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] OS version is Debian GNU/Linux 9.13 (stretch)
    
    [INFO] uptime:  18:34:04 up  1:37,  2 users,  load average: 0.01, 0.03, 0.00
    
    [INFO] memory:
                  total        used        free      shared  buff/cache   available
    Mem:           1.9G        1.5G         77M         18M        370M        279M
    Swap:          255M        173M         82M
    
    [INFO] ISPConfig is installed.
    [WARN] /usr/local/ispconfig/server/lib/config.inc.php is missing.
    
    ##### VERSION CHECK #####
    
    [INFO] php (cli) version is 7.0.33-0+deb9u12
    [INFO] php-cgi (used for cgi php in default vhost!) is version 5.6.40
    [WARN] You are using an outdated php version.
    
    ##### PORT CHECK #####
    
    
    ##### MAIL SERVER CHECK #####
    
    
    ##### RUNNING SERVER PROCESSES #####
    
    [WARN] I could not determine which web server is running.
    [WARN] I could not determine which mail server is running.
    [WARN] I could not determine which pop3 server is running.
    [WARN] I could not determine which imap server is running.
    [WARN] I could not determine which ftp server is running.
    
    ##### LISTENING PORTS #####
    (only           ()
    Local           (Address)
    [anywhere]:993          (-)
    [anywhere]:995          (-)
    [localhost]:10024               (-)
    [localhost]:10025               (-)
    [localhost]:10026               (-)
    [localhost]:10027               (-)
    [anywhere]:587          (-)
    [localhost]:11211               (-)
    [anywhere]:110          (-)
    [anywhere]:143          (-)
    [anywhere]:111          (-)
    [anywhere]:465          (-)
    [anywhere]:21           (-)
    [anywhere]:22           (-)
    [anywhere]:25           (-)
    *:*:*:*::*:993          (-)
    *:*:*:*::*:995          (-)
    *:*:*:*::*:10024                (-)
    *:*:*:*::*:10026                (-)
    *:*:*:*::*:3306         (-)
    *:*:*:*::*:587          (-)
    [localhost]10           (-)
    [localhost]43           (-)
    [localhost]11           (-)
    *:*:*:*::*:80           (-)
    *:*:*:*::*:8080         (-)
    *:*:*:*::*:465          (-)
    *:*:*:*::*:8081         (-)
    *:*:*:*::*:21           (-)
    *:*:*:*::*:22           (-)
    *:*:*:*::*:25           (-)
    *:*:*:*::*:443          (-)
    
    
    
    
    ##### IPTABLES #####
    
    
    
    
    ##### LET'S ENCRYPT #####
    Certbot is installed in /opt/eff.org/certbot/venv/bin/certbot
    
    OS
    Code:
    Distributor ID: Debian
    Description:    Debian GNU/Linux 9.13 (stretch)
    Release:        9.13
    Codename:       stretch
    PHP
    Code:
    PHP 7.0.33-0+deb9u12 (cli) (built: Oct 26 2021 17:51:39) ( NTS )
    Copyright (c) 1997-2017 The PHP Group
    Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
        with Zend OPcache v7.0.33-0+deb9u12, Copyright (c) 1999-2017, by Zend Technologies
    
     
  2. Nebhead 2

    Nebhead 2 New Member

    I re-ran ispconfig_update.sh and it completed successfully this time. I'm also now receiving email again which is great.
    However, the admin page now gives a 500 error, with the following in the error log:
    Code:
    [Thu Oct 28 19:51:08.518775 2021] [fcgid:warn] [pid 24166] (104)Connection reset by peer: [client 77.81.139.164:63814] mod_fcgid: error reading data from FastCGI server
    [Thu Oct 28 19:51:08.518839 2021] [core:error] [pid 24166] [client 77.81.139.164:63814] End of script output before headers: index.php
    
     
  3. Nebhead 2

    Nebhead 2 New Member

    I installed php7.0-cgi, restarted apache2, and the ispconfig admin page is now up and running.

    This *seems* to be all my issues resolved - is there any test or sanity check that can be run to ensure there are no more loose ends left over from the upgrade to Debian 9, and that everything is still locked up tight and secure?
     
  4. Nebhead 2

    Nebhead 2 New Member

    I suppose the other question is, whether it's advisable to now continue on with an upgrade to Debian 10!
     
  5. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    You should run through the perfect server guide for debian 9 and install all packages. Also make sure your php settings are correct in server config.

    Yes, I would upgrade to Debian 10 now, then again to 11 (though there is no perfect server guide for 11 yet, so wait a bit on that if you're more comfortable doing so).
     
  6. Nebhead 2

    Nebhead 2 New Member

    Thanks Jesse.

    I'm running through that guide now and hit an issue running mysql_secure_installation, as the root mysql password I have noted wasn't being accepted.

    I ran through the standard process to reset the root mysql password but it didn't work - after restarting mysql, I was still getting the following error:
    Code:
    [email protected]:/var$ mysql -u root -p
    Enter password:
    ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
    
    Code:
    [email protected]:~$ sudo mysql
    [sudo] password for nebhead:
    ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
    
    I thought it might be related to Can't reset MySQL (MariaDB) root password - Super User but:
    Code:
    MariaDB [mysql]> select host, user, plugin from user where user="root";
    +-----------+------+--------+
    | host      | user | plugin |
    +-----------+------+--------+
    | localhost | root |        |
    | janus     | root |        |
    | 127.0.0.1 | root |        |
    | ::1       | root |        |
    +-----------+------+--------+
    4 rows in set (0.00 sec)
    
    I'm at a loss at this point, no idea how to move forward
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    You can skip that step as you have MySQL secured in the past. What#s important is that you run all the commands that install software to get the right new software versions and add additional software which might be required now. The MySQL root password can be found in the file /usr/local/ispconfig/server/lib/mysql_clientdb.conf.
     
  8. Nebhead 2

    Nebhead 2 New Member

    Grand, thanks - can login with the root password in that file, which is great!

    Tracking a few more updates I've made here just for a sanity check:
    • Updated nano /etc/mysql/debian.cnf to replace the passwords in this file with the root password from /usr/local/ispconfig/server/lib/mysql_clientdb.conf
    • Added limits to /etc/security/limits.conf
    • Created /etc/systemd/system/mysql.service.d/limits.conf

    I'm now struggling to restart mysql, with the following error:
    Code:
    [email protected]:~$ sudo service mysql restart
    Job for mariadb.service failed because the control process exited with error code.
    See "systemctl status mariadb.service" and "journalctl -xe" for details.
    
    It looks like mysql didn't full shut down?
    Code:
    [email protected]:~$ ps -aux | grep mysql
    root      3916  0.0  0.1  49260  3248 pts/0    S    09:35   0:00 sudo mysqld_safe --skip-grant-tables
    root      3917  0.0  0.1  11460  3200 pts/0    S    09:35   0:00 /bin/bash /usr/bin/mysqld_safe --skip-grant-tables
    mysql     4067  0.1  4.7 657532 95472 pts/0    Sl   09:35   0:06 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/x86_64-linux-gnu/mariadb18/plugin --user=mysql --skip-grant-tables --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
    nebhead  11880  0.0  0.0  12788   972 pts/0    S+   10:50   0:00 grep mysql
    
     
  9. Nebhead 2

    Nebhead 2 New Member

    Okay, killed those and started, all seems well now:
    Code:
    [email protected]:~$ sudo service mysql restart
    Job for mariadb.service failed because the control process exited with error code.
    See "systemctl status mariadb.service" and "journalctl -xe" for details.
    [email protected]:~$ ps -aux | grep mysql
    root      3916  0.0  0.1  49260  3248 pts/0    S    09:35   0:00 sudo mysqld_safe --skip-grant-tables
    root      3917  0.0  0.1  11460  3200 pts/0    S    09:35   0:00 /bin/bash /usr/bin/mysqld_safe --skip-grant-tables
    mysql     4067  0.1  4.7 657532 95472 pts/0    Sl   09:35   0:06 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/x86_64-linux-gnu/mariadb18/plugin --user=mysql --skip-grant-tables --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
    nebhead  11880  0.0  0.0  12788   972 pts/0    S+   10:50   0:00 grep mysql
    [email protected]:~$ sudo kill 3916 3917 4067
    [email protected]:~$ ps -aux | grep mysql
    nebhead  12331  0.0  0.0  12788   952 pts/0    S+   10:55   0:00 grep mysql
    [1]+  Done                    sudo mysqld_safe --skip-grant-tables
    [email protected]:~$ sudo service mysql start
    [email protected]:~$ ps -aux | grep mysql
    mysql    12442  0.9  3.8 654016 76416 ?        Ssl  10:55   0:00 /usr/sbin/mysqld
    nebhead  12518  0.0  0.0  12788   952 pts/0    S+   10:56   0:00 grep mysql
    [email protected]:~$ netstat -tap | grep mysql
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    tcp       35      0 localhost.localdo:37382 localhost.localdo:mysql CLOSE_WAIT  -
    tcp       35      0 localhost.localdo:37410 localhost.localdo:mysql CLOSE_WAIT  -
    tcp6       0      0 [::]:mysql              [::]:*                  LISTEN      -
    ^[^C
    [email protected]:~$ sudo netstat -tap | grep mysql
    tcp       35      0 localhost.localdo:37382 localhost.localdo:mysql CLOSE_WAIT  1135/amavisd-new (c
    tcp       35      0 localhost.localdo:37410 localhost.localdo:mysql CLOSE_WAIT  1136/amavisd-new (c
    tcp6       0      0 [::]:mysql              [::]:*                  LISTEN      12442/mysqld
    tcp6       0      0 janus.nebhead.com:mysql 143.92.59.187:https     SYN_RECV    -
    q^C
    [email protected]:~$ sudo service mysql restart
    [email protected]:~$
    
    Onwards...

    • Installed (missing) Amavisd-new, SpamAssassin, and ClamAV packages and disabled spamassassin service
    • Skipped XMPP
    • Installed (missing) Apache2, PHP5, phpMyAdmin, FCGI, suExec, Pear, and mcrypt packages
    • Created HTTPOXY conf, enabled module & restarted apache
    • Skipped lets encrypt as all seems well already
    • Installed PHP-FPM and enabled modules
    • Skipped mailman/pureftpd/quota/bind
    • Webalizer/AWStats/Jailkit all good
    • Needed to slightly tweak /etc/fail2ban/jail.local to
      • replace 'pureftpd' with 'pure-ftpd'
      • replace 'dovecot-pop2imap' with 'dovecot' and remove 'action = iptables-multiport[name=dovecot-pop3imap, port="pop3,pop3s,imap,imaps", protocol=tcp]'
    • Install UFW firewall
    • Failed to install roundcube
    • Skipped install of ISPConfig, obviously
     
  10. Nebhead 2

    Nebhead 2 New Member

    So, roundcube:

    Packages are installed and config changes made, but I think something went wrong when setting up the database as I get the following error on accessing the URL:

    Code:
    DATABASE ERROR: CONNECTION FAILED!
    Unable to connect to the database!
    Please contact your server-administrator.
     
  11. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    You should stop using the service command. It is systemctl now, so
    Code:
    systemctl restart mysql
     
  12. Steini86

    Steini86 Active Member

    Are you trying to use mysql and mariadb in parallel?

    For roundcube: How did you install? You can postpone this until you upgraded to Debian 11, it is again in the repo then. Just use "apt install roundcube" on Debian 11.
    You might also get trouble with your php versions. The cleanest solution would be to use the migration tool to save your server and migrate to a clean new Debian 11.
     

Share This Page