After apt-get upgrade MariaDB 10.3.22 on Debian Buster won't start anymore

Discussion in 'ISPConfig 3 Priority Support' started by schmidtedv, Feb 13, 2020.

  1. schmidtedv

    schmidtedv Member HowtoForge Supporter

    I hope the subject describes my problem:
    Yesterday a normal apt-get update /upgrade came up with quite a few packages but all went through ok. Finally I rebooted the server and today (it's not a productive, but a project server at the moment) I noticed, that most Web, FTP, etc. are not working because they cannot connect to MySQL anymore:
    Code:
    ps aux | grep mysql
    root     13318  0.0  0.0   6420   828 pts/1    S+   13:19   0:00 grep mysql
    
    Code:
    /etc/init.d/mysql status
    mariadb.service - MariaDB 10.3.22 database server
    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Thu 2020-02-13 12:39:22 CET; 2min 57s ago
    ...
    Process: 10565 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
    Main PID: 10565 (code=exited, status=1/FAILURE)
    Status: "MariaDB server is down"
    
    Code:
    Feb 13 12:39:21 mail systemd[1]: Starting MariaDB 10.3.22 database server...
    Feb 13 12:39:22 mail mysqld[10565]: 2020-02-13 12:39:22 0 [Note] /usr/sbin/mysqld (mysqld 10.3.22-MariaDB-0+deb10u1-log) starting as process 10565 ...
    Feb 13 12:39:22 mail mysqld[10565]: 2020-02-13 12:39:22 0 [ERROR] Aborting
    Feb 13 12:39:22 mail systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Feb 13 12:39:22 mail systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Feb 13 12:39:22 mail systemd[1]: Failed to start MariaDB 10.3.22 database server.
    
    Some hints, that might be usefull:

    The server uses a letsencrypt-certificate for ISPConfig, Dovecot, Pure-FTPD like described in this threat:
    https://www.howtoforge.com/tutorial/securing-ispconfig-3-with-a-free-lets-encrypt-ssl-certificate/
    df -h and df -i are looking normal (nothing is using more than 2%)

    Anybody any idea, what might have gone wrong or are there any commands that I can try to find out more about this?
    The apt-log list the following changes (only yesterday should be relevant):
    Code:
    Start-Date: 2020-02-07  00:01:20
    Commandline: apt-get dist-upgrade
    Install: php7.4-cli:amd64 (7.4.2-5+0~20200202.10+debian10~1.gbp3a5fde, automatic), php7.4-json:amd64 (7.4.2-5+0~20200202.10+debian10~1.gbp3a5fde, automatic), php7.4-opcache:amd64 (7.4.2-5+0~20200202.10+debian10~1.gbp3a5fde, automatic), php7.4-xml:amd64 (7.4.2-5+0~20200202.10+debian10~1.gbp3a5fde, automatic), php7.4-readline:amd64 (7.4.2-5+0~20200202.10+debian10~1.gbp3a5fde, automatic), libpcre2-posix2:amd64 (10.34-7+0~20191220.5+debian10~1.gbp555bb9, automatic), php7.4-common:amd64 (7.4.2-5+0~20200202.10+debian10~1.gbp3a5fde, automatic), libhyperscan5:amd64 (5.1.0-1, automatic), libapache2-mod-php7.4:amd64 (7.4.2-5+0~20200202.10+debian10~1.gbp3a5fde, automatic)
    Upgrade: php-xml:amd64 (2:7.3+70+0~20191118.18+debian10~1.gbp66b4ed, 2:7.4+72+0~20200122.20+debian10~1.gbpcd96c7), libapache2-mod-php:amd64 (2:7.3+70+0~20191118.18+debian10~1.gbp66b4ed, 2:7.4+72+0~20200122.20+debian10~1.gbpcd96c7), libpcre2-32-0:amd64 (10.33-1+0~20190710.4+debian10~1.gbpa344f0, 10.34-7+0~20191220.5+debian10~1.gbp555bb9), rspamd:amd64 (2.2-1~buster, 2.3-1~buster), libpcre2-dev:amd64 (10.33-1+0~20190710.4+debian10~1.gbpa344f0, 10.34-7+0~20191220.5+debian10~1.gbp555bb9), libpcre2-8-0:amd64 (10.33-1+0~20190710.4+debian10~1.gbpa344f0, 10.34-7+0~20191220.5+debian10~1.gbp555bb9), libpcre2-16-0:amd64 (10.33-1+0~20190710.4+debian10~1.gbpa344f0, 10.34-7+0~20191220.5+debian10~1.gbp555bb9)
    Remove: libpcre2-posix0:amd64 (10.33-1+0~20190710.4+debian10~1.gbpa344f0)
    End-Date: 2020-02-07  00:01:41
    
    Start-Date: 2020-02-12  11:08:53
    Commandline: apt-get upgrade
    Upgrade: libpython3.7-minimal:amd64 (3.7.3-2, 3.7.3-2+deb10u1), libopenjp2-7:amd64 (2.3.0-2, 2.3.0-2+deb10u1), comerr-dev:amd64 (2.1-1.44.5-1+deb10u2, 2.1-1.44.5-1+deb10u3), postfix:amd64 (3.4.7-0+deb10u1, 3.4.8-0+10debu1), libcom-err2:amd64 (1.44.5-1+deb10u2, 1.44.5-1+deb10u3), libcups2:amd64 (2.2.10-6+deb10u1, 2.2.10-6+deb10u2), linux-libc-dev:amd64 (4.19.67-2+deb10u2, 4.19.98-1), postfix-mysql:amd64 (3.4.7-0+deb10u1, 3.4.8-0+10debu1), unhide:amd64 (20130526-3, 20130526-3+deb10u1), libsystemd0:amd64 (241-7~deb10u2, 241-7~deb10u3), libglapi-mesa:amd64 (18.3.6-2, 18.3.6-2+deb10u1), mariadb-common:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), e2fsprogs:amd64 (1.44.5-1+deb10u2, 1.44.5-1+deb10u3), sudo:amd64 (1.8.27-1+deb10u1, 1.8.27-1+deb10u2), mariadb-server-core-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), libpython3.7:amd64 (3.7.3-2, 3.7.3-2+deb10u1), python3.7:amd64 (3.7.3-2, 3.7.3-2+deb10u1), openssh-sftp-server:amd64 (1:7.9p1-10+deb10u1, 1:7.9p1-10+deb10u2), udev:amd64 (241-7~deb10u2, 241-7~deb10u3), libpq5:amd64 (11.5-1+deb10u1, 11.6-0+deb10u1), libpython3.7-stdlib:amd64 (3.7.3-2, 3.7.3-2+deb10u1), postfix-doc:amd64 (3.4.7-0+deb10u1, 3.4.8-0+10debu1), libudev1:amd64 (241-7~deb10u2, 241-7~deb10u3), python3.7-minimal:amd64 (3.7.3-2, 3.7.3-2+deb10u1), mariadb-server-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), libpq-dev:amd64 (11.5-1+deb10u1, 11.6-0+deb10u1), libss2:amd64 (1.44.5-1+deb10u2, 1.44.5-1+deb10u3), libext2fs2:amd64 (1.44.5-1+deb10u2, 1.44.5-1+deb10u3), clamav-docs:amd64 (0.101.4+dfsg-0+deb10u1, 0.102.1+dfsg-0+deb10u2), libboost-iostreams1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1), libboost-chrono1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1), ssh:amd64 (1:7.9p1-10+deb10u1, 1:7.9p1-10+deb10u2), systemd-sysv:amd64 (241-7~deb10u2, 241-7~deb10u3), libpam-systemd:amd64 (241-7~deb10u2, 241-7~deb10u3), clamav:amd64 (0.101.4+dfsg-0+deb10u1, 0.102.1+dfsg-0+deb10u2), systemd:amd64 (241-7~deb10u2, 241-7~deb10u3), clamav-daemon:amd64 (0.101.4+dfsg-0+deb10u1, 0.102.1+dfsg-0+deb10u2), clamdscan:amd64 (0.101.4+dfsg-0+deb10u1, 0.102.1+dfsg-0+deb10u2), clamav-freshclam:amd64 (0.101.4+dfsg-0+deb10u1, 0.102.1+dfsg-0+deb10u2), openssh-server:amd64 (1:7.9p1-10+deb10u1, 1:7.9p1-10+deb10u2), libgl1-mesa-dri:amd64 (18.3.6-2, 18.3.6-2+deb10u1), libboost-atomic1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1), openssh-client:amd64 (1:7.9p1-10+deb10u1, 1:7.9p1-10+deb10u2), mariadb-client-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), libmysofa0:amd64 (0.6~dfsg0-3, 0.6~dfsg0-3+deb10u1), mariadb-server:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), mariadb-client-core-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), mariadb-client:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), libboost-locale1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1), libtimedate-perl:amd64 (2.3000-2, 2.3000-2+deb10u1), libmariadb3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), libgnutls30:amd64 (3.6.7-4, 3.6.7-4+deb10u2), clamav-base:amd64 (0.101.4+dfsg-0+deb10u1, 0.102.1+dfsg-0+deb10u2), libboost-filesystem1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1), mesa-vdpau-drivers:amd64 (18.3.6-2, 18.3.6-2+deb10u1), libclamav9:amd64 (0.101.4+dfsg-0+deb10u1, 0.102.1+dfsg-0+deb10u2), libcupsimage2:amd64 (2.2.10-6+deb10u1, 2.2.10-6+deb10u2), libboost-date-time1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1), libboost-thread1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1), mesa-va-drivers:amd64 (18.3.6-2, 18.3.6-2+deb10u1), base-files:amd64 (10.3+deb10u2, 10.3+deb10u3), libglx-mesa0:amd64 (18.3.6-2, 18.3.6-2+deb10u1), libboost-system1.67.0:amd64 (1.67.0-13, 1.67.0-13+deb10u1)
    End-Date: 2020-02-12  11:09:49
    
    I'm not shure about the dependencies, but as far as I would guess, there might be an incompatibility somewhere with one of these new packages (?):

    Code:
    mariadb-common:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1)
    mariadb-server-core-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1), 0debu1)
    mariadb-server-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1)
    mariadb-client-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1)
    mariadb-server:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1)
    mariadb-client-core-10.3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1)
    mariadb-client:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1)
    libmariadb3:amd64 (1:10.3.18-0+deb10u1, 1:10.3.22-0+deb10u1)
    
     
    Last edited: Feb 13, 2020
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The MariaDB package that you use is the one from Debian buster?
    Do you have any third-party package repositories active on that system?
    Any errors in the mariadb / mysql log files or in the global syslog file?

    and a note, Debian 10 systems don't sue init scripts anymore, use systemd systemctl command to start mariadb and systemctl will suggest a command to get more details on the error in case it fails to start.
     
  3. schmidtedv

    schmidtedv Member HowtoForge Supporter

    Hi Till,
    thanks for taking time in this!
    As far as I can see, sources.list should be unchanged:
    Code:
    deb http://mirror.hetzner.de/debian/packages buster main contrib non-free
    deb http://mirror.hetzner.de/debian/security buster/updates main contrib non-free
    deb http://mirror.hetzner.de/debian/packages buster-updates main contrib non-free
    deb http://deb.debian.org/debian/ buster main non-free contrib
    deb-src http://deb.debian.org/debian/ buster main non-free contrib
    deb http://security.debian.org/debian-security buster/updates main contrib non-free
    deb-src http://security.debian.org/debian-security buster/updates main contrib non-free
    deb http://deb.debian.org/debian/ buster-updates main contrib non-free
    
    syslog only lists information about "connection lost" and quite a lot UFW Blocks:
    Code:
    Feb 13 18:02:04 mail dovecot: auth-worker(31101): Error: mysql(localhost): Connect failed to database (db_ispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) - waiting for 125 seconds before retry
    Feb 13 18:07:27 mail kernel: [22169.291207] [UFW BLOCK] IN=enp4s0 OUT= MAC=54:04:a6:67:5c:7d:64:64:9b:6a:77:c2:08:00 SRC=194.26.29.130 DST=136.243.22.115 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=38309 PROTO=TCP SPT=8080 DPT=3316 WINDOW=1024 RES=0x00 SYN URGP=0
    
    mysql- and mariabd-logs under /var/log/mysql are empty and apache2-error.log comes with this (when loading ISPConfig, for instance):
    Code:
    [Thu Feb 13 18:11:59.580253 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 85
    [Thu Feb 13 18:11:59.580301 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580327 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580334 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580340 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580347 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580353 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 85
    [Thu Feb 13 18:11:59.580359 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580366 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580372 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580378 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580385 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Warning:  mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    [Thu Feb 13 18:11:59.580391 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: PHP Fatal error:  Uncaught Error: Call to a member function queryOneRecord() on bool in /usr/local/ispconfig/interface/lib/app.inc.php:166
    [Thu Feb 13 18:11:59.580397 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: Stack trace:
    [Thu Feb 13 18:11:59.580403 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: #0 /usr/local/ispconfig/interface/lib/app.inc.php(93): app->conf('interface', 'session_timeout')
    [Thu Feb 13 18:11:59.580409 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: #1 /usr/local/ispconfig/interface/lib/app.inc.php(381): app->initialize_session()
    [Thu Feb 13 18:11:59.580415 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: #2 /usr/local/ispconfig/interface/web/index.php(32): require_once('/usr/local/ispc...')
    [Thu Feb 13 18:11:59.580421 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr: #3 {main}
    [Thu Feb 13 18:11:59.580427 2020] [fcgid:warn] [pid 2943] [client 87.122.75.141:63046] mod_fcgid: stderr:   thrown in /usr/local/ispconfig/interface/lib/app.inc.php on line 166
    
     
  4. schmidtedv

    schmidtedv Member HowtoForge Supporter

    Code:
    [email protected] ~ # systemctl start mariadb
    Job for mariadb.service failed because the control process exited with error code.
    See "systemctl status mariadb.service" and "journalctl -xe" for details.
    
    Code:
    [email protected] ~ # systemctl status mariadb.service
    ● mariadb.service - MariaDB 10.3.22 database server
       Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/mysql.service.d
               └─limits.conf
       Active: failed (Result: exit-code) since Thu 2020-02-13 18:33:37 CET; 1min 13s ago
         Docs: man:mysqld(8)
               https://mariadb.com/kb/en/library/systemd/
      Process: 810 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
      Process: 811 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 812 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_PO
      Process: 977 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
     Main PID: 977 (code=exited, status=1/FAILURE)
       Status: "MariaDB server is down"
    
    Feb 13 18:33:37 mail systemd[1]: Starting MariaDB 10.3.22 database server...
    Feb 13 18:33:37 mail mysqld[977]: 2020-02-13 18:33:37 0 [Note] /usr/sbin/mysqld (mysqld 10.3.22-MariaDB-0+deb10u1-log) starting as process 977 ...
    Feb 13 18:33:37 mail mysqld[977]: 2020-02-13 18:33:37 0 [ERROR] Aborting
    Feb 13 18:33:37 mail systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Feb 13 18:33:37 mail systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Feb 13 18:33:37 mail systemd[1]: Failed to start MariaDB 10.3.22 database server.
    ...skipping...
    ● mariadb.service - MariaDB 10.3.22 database server
       Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/mysql.service.d
               └─limits.conf
       Active: failed (Result: exit-code) since Thu 2020-02-13 18:33:37 CET; 1min 13s ago
         Docs: man:mysqld(8)
               https://mariadb.com/kb/en/library/systemd/
      Process: 810 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
      Process: 811 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 812 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_PO
      Process: 977 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
     Main PID: 977 (code=exited, status=1/FAILURE)
       Status: "MariaDB server is down"
    
    Feb 13 18:33:37 mail systemd[1]: Starting MariaDB 10.3.22 database server...
    Feb 13 18:33:37 mail mysqld[977]: 2020-02-13 18:33:37 0 [Note] /usr/sbin/mysqld (mysqld 10.3.22-MariaDB-0+deb10u1-log) starting as process 977 ...
    Feb 13 18:33:37 mail mysqld[977]: 2020-02-13 18:33:37 0 [ERROR] Aborting
    Feb 13 18:33:37 mail systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Feb 13 18:33:37 mail systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Feb 13 18:33:37 mail systemd[1]: Failed to start MariaDB 10.3.22 database server.
    
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    Please take a look into the /var/log/syslog file right after you restarted mariadb with systemctl to see if there is any additional info on the cause of the error in there. and check that your harddisk is not full with e.g. 'df -h' command.

    and regarding the error in the apache log, the php mysqli package is missing.
     
  6. schmidtedv

    schmidtedv Member HowtoForge Supporter

    The system is installed (and already runs) with the tutorial:
    https://www.howtoforge.com/perfect-...onfig-3-1/#-install-apache-web-server-and-php

    Code:
    apt-get install php7.3-mysql
    Paketlisten werden gelesen... Fertig
    Abhängigkeitsbaum wird aufgebaut.
    Statusinformationen werden eingelesen.... Fertig
    php7.3-mysql ist schon die neueste Version (7.3.14-5+0~20200202.52+debian10~1.gbpa71879).
    
    So, all modules are or still should be installed (and even more php-versions get updated automatically).

    Code:
    [email protected] ~ # systemctl start mariadb
    
    ...comes up with:

    Code:
    Feb 13 20:04:38 mail systemd[1]: Starting MariaDB 10.3.22 database server...
    Feb 13 20:04:39 mail mysqld[6739]: 2020-02-13 20:04:39 0 [Note] /usr/sbin/mysqld (mysqld 10.3.22-MariaDB-0+deb10u1-log) starting as process 6739 ...
    Feb 13 20:04:39 mail mysqld[6739]: 2020-02-13 20:04:39 0 [ERROR] Aborting
    Feb 13 20:04:39 mail systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Feb 13 20:04:39 mail systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Feb 13 20:04:39 mail systemd[1]: Failed to start MariaDB 10.3.22 database server.
    
    df -h looks ok to me:

    Code:
    [email protected] ~ # df -h
    Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
    udev             16G       0   16G    0% /dev
    tmpfs           3,2G    712K  3,2G    1% /run
    /dev/sda3       2,0T     18G  1,9T    1% /
    tmpfs            16G       0   16G    0% /dev/shm
    tmpfs           5,0M       0  5,0M    0% /run/lock
    tmpfs            16G       0   16G    0% /sys/fs/cgroup
    /dev/sda2       488M     91M  372M   20% /boot
    /dev/sda4       1,7T     77M  1,6T    1% /home
    tmpfs           3,2G       0  3,2G    0% /run/user/0
    
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    So this is all from syslog?

    And the php mysql package is definitely missing according to the log. Maybe you installed additional PHP versions and accidentally replaced the default PHP version?
     
  8. schmidtedv

    schmidtedv Member HowtoForge Supporter

    Yes, thats all from syslog and, evenything should or has already been installed and working fine. The additonal php-versions are from this howto: https://www.howtoforge.com/tutorial...fig-3-from-debian-packages-on-debian-8-and-9/ but that has been done already, so there hasn't been any negative influence on the system with that before.

    And this one come up with everything already installed:

    Code:
    apt-get -y install apache2 apache2-doc apache2-utils libapache2-mod-php php7.3 php7.3-common php7.3-gd php7.3-mysql php7.3-imap php7.3-cli php7.3-cgi libapache2-mod-fcgid apache2-suexec-pristine php-pear mcrypt  imagemagick libruby libapache2-mod-python php7.3-curl php7.3-intl php7.3-pspell php7.3-recode php7.3-sqlite3 php7.3-tidy php7.3-xmlrpc php7.3-xsl memcached php-memcache php-imagick php-gettext php7.3-zip php7.3-mbstring memcached libapache2-mod-passenger php7.3-soap php7.3-fpm php7.3-opcache php-apcu
    
    Code:
    php7.3-mysql ist schon die neueste Version (7.3.14-5+0~20200202.52+debian10~1.gbpa71879)
    
    ...and mysqli is acitvated under /apache/conf.d
     
    Last edited: Feb 13, 2020
  9. schmidtedv

    schmidtedv Member HowtoForge Supporter

    ...any chance maybe the package-maintainers did something wrong?

    Here a longer information after using "journalctl -xe":

    Code:
    Feb 13 21:00:27 mail systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    -- Subject: Unit process exited
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- An ExecStart= process belonging to unit mariadb.service has exited.
    --
    -- The process' exit code is 'exited' and its exit status is 1.
    Feb 13 21:00:27 mail systemd[1]: mariadb.service: Failed with result 'exit-code'.
    -- Subject: Unit failed
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- The unit mariadb.service has entered the 'failed' state with result 'exit-code'.
    Feb 13 21:00:27 mail systemd[1]: Failed to start MariaDB 10.3.22 database server.
    -- Subject: A start job for unit mariadb.service has failed
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- A start job for unit mariadb.service has finished with a failure.
    --
    -- The job identifier is 824 and the job result is failed.
    
     
    Last edited: Feb 13, 2020
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, and you set the defaults back to the right php version as mentioned towards the end of the article? My guess is that you have php 7.4 installed now without MySQL support.

    You have to check cgi and fpm, not apache.

    I guess we would have seen many more posts if MariaDB would be broken in Debian 10 by an ordinary update, so I still think it's something specific to your system and not a general MariaDB issue.
     
  11. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Have you checked the logs in /var/log/mysql ?
     
  12. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Did you check, that mysql is not already runnig? ps -ef|grep mysql
     
  13. schmidtedv

    schmidtedv Member HowtoForge Supporter

    2 Things that have changed or that I haven't noticed:

    1. I used this howto: https://www.howtoforge.com/tutorial...n-packages-on-debian-8-and-9/#-install-php--5 without installing php7.4 ! Right now there have been parts of php7.4 installed automatically (?) and - as Till thought - this was set as default. I installed all missing parts of php7.4 now and changed default back to php7.3 manually. However, a reboot didn't fix that.

    2. As Florian suggested:

    Code:
    roo[email protected] ~ # ps -ef|grep mysql
    root      1544  1538  0 07:45 pts/1    00:00:00 grep mysql
    
    Not shure, if that's the failure (but it's different to my other systems with older debian or Ubuntu):
    Under /etc/systemd/system I have 2 simlinks (mysql.service and mysqld.service), both of them pointing to /lib/systemd/system/mariadb.service. I don't have these on other systems, but the simlinks seems to be old (2019-09-14), so I don't know, should they be there or not.
     
  14. schmidtedv

    schmidtedv Member HowtoForge Supporter

    ...the last log including chars :))) is about a week old, so I haven't thought, my problem might be as old, but maybe I was stupid enough not to realize that:

    Code:
    2020-02-07  0:05:32 0 [Note] InnoDB: 10.3.18 started; log sequence number 8065985942; transaction id 8649564
    2020-02-07  0:05:32 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
    2020-02-07  0:05:32 0 [Note] Plugin 'FEEDBACK' is disabled.
    2020-02-07  0:05:32 0 [Note] Server socket created on IP: '::'.
    2020-02-07  0:05:33 6 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Tabelle 'mysql.gtid_slave_pos' existiert nicht
    2020-02-07  0:05:33 0 [Note] Reading of all Master_info entries succeeded
    2020-02-07  0:05:33 0 [Note] Added new Master_info '' to hash table
    2020-02-07  0:05:33 0 [Note] /usr/sbin/mysqld: bereit für Verbindungen.
    Version: '10.3.18-MariaDB-0+deb10u1-log'  Socket: '/run/mysqld/mysqld.sock'  Port: 3306  Debian 10
    2020-02-07  0:05:35 13 [ERROR] InnoDB: Operating system error number 17 in a file operation.
    2020-02-07  0:05:35 13 [ERROR] InnoDB: Error number 17 means 'File exists'
    2020-02-07  0:05:35 13 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
    2020-02-07  0:05:35 13 [Note] InnoDB: The file './mysql/gtid_slave_pos.ibd' already exists though the corresponding table did not exist in the InnoDB data dictionary. You can resolve the problem by removing the file.
    2020-02-07  0:05:35 13 [ERROR] InnoDB: Cannot create file './mysql/gtid_slave_pos.ibd'
    2020-02-07  0:05:36 0 [Note] InnoDB: Buffer pool(s) load completed at 200207  0:05:36
    2020-02-07  0:06:12 41 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.
    
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    Do you have a backup which includes the /var/lib/mysql folder so you can try if it starts with the files from backup?
     
  16. schmidtedv

    schmidtedv Member HowtoForge Supporter

    Well, I got it fixed, or, as I have to say, I fixed my own goofiness. Right from the beginning I (mis-)tuned some innodb-options in 50-server.cnf. One of these options is not compatible anymore with the updated mariadb! However, I don't know, what...so right now I switched back to the original configuration and it started right away. Maybe anyone can tell me, what I've done wrong in this config?
    Code:
    [server]
    [mysqld]
    user                     = mysql
    pid-file                 = /run/mysqld/mysqld.pid
    socket                   = /run/mysqld/mysqld.sock
    # port                   = 3306
    basedir                  = /usr
    datadir                  = /var/lib/mysql
    tmpdir                   = /tmp
    lc-messages-dir          = /usr/share/mysql
    lc-messages              = de_DE
    expire_logs_days         = 7
    slow_query_log           = 1
    slow_query_log_file      = /var/log/mysql/mariadb-slow.log
    long_query_time          = 3
    log_slow_verbosity       = query_plan
    log-queries-not-using-indexes
    log_error                = /var/log/mysql/error.log
    skip-external-locking
    # skip-name-resolve nicht aktivieren, sonst kann MySQL nur noch mit localhost, nicht mehr mit 127.0.0.1 angesprochen werden
    # skip-name-resolve
    open_files_limit         = 1024000
    performance_schema       = 1
    # bind-address           = 127.0.0.1
    myisam_recover_options   = BACKUP
    key_buffer_size          = 512M
    join_buffer_size         = 4M
    sort_buffer_size         = 2M
    read_buffer_size         = 2M
    read_rnd_buffer_size     = 8M
    myisam_sort_buffer_size  = 512M
    tmp_table_size           = 128M
    max_heap_table_size      = 128M
    max_binlog_size          = 100M
    max_allowed_packet       = 64M
    max_connections          = 400
    # thread_stack           = 192K
    thread_cache_size        = 24
    # table_cache            = 64
    table_open_cache         = 1024
    # thread_concurrency     = 10
    query_cache_limit        = 2M
    query_cache_size         = 64M
    query_cache_type         = 1
    query_cache_min_res_unit = 2k
    
    innodb_buffer_pool_size  = 3G
    innodb_log_file_size     = 384M
    innodb_log_buffer_size   = 32M
    innodb_file_per_table    = 1
    innodb_open_files        = 400
    innodb_io_capacity       = 4000
    innodb_flush_method      = O_DIRECT
    innodb_max_dirty_pages_pct     = 90
    innodb_buffer_pool_instances   = 3
    innodb_flush_log_at_trx_commit = 2
    character-set-server     = utf8mb4
    collation-server         = utf8mb4_general_ci
    [embedded]
    [mariadb]
    [mariadb-10.3]
    
     
  17. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    If you want help with "what is wrong in this config", show diff between what is working and the wrong configuration.
     
  18. schmidtedv

    schmidtedv Member HowtoForge Supporter

    Ok, sorry. The next 2 options are vice versa:

    Code:
    # skip-external-locking (im Original genau umgekehrt)
    skip-name-resolve (im Original genau umgekehrt)
    
    All other/next options are not used in the original .cnf:

    Code:
    open_files_limit         = 1024000
    performance_schema       = 1
    myisam_recover_options   = BACKUP
    key_buffer_size          = 512M
    join_buffer_size         = 4M
    sort_buffer_size         = 2M
    read_buffer_size         = 2M
    read_rnd_buffer_size     = 8M
    myisam_sort_buffer_size  = 512M
    tmp_table_size           = 128M
    max_heap_table_size      = 128M
    max_binlog_size          = 100M
    max_allowed_packet       = 64M
    max_connections          = 400
    thread_cache_size        = 24
    table_open_cache         = 1024
    query_cache_limit        = 2M
    query_cache_type         = 1
    query_cache_min_res_unit = 2k
    
    innodb_buffer_pool_size  = 3G
    innodb_log_file_size     = 384M
    innodb_log_buffer_size   = 32M
    innodb_file_per_table    = 1
    innodb_open_files        = 400
    innodb_io_capacity       = 4000
    innodb_flush_method      = O_DIRECT
    innodb_max_dirty_pages_pct     = 90
    innodb_buffer_pool_instances   = 3
    innodb_flush_log_at_trx_commit = 2
    
     

Share This Page