Mysql down after restarting server

Discussion in 'General' started by deividmen, Aug 23, 2019.

  1. deividmen

    deividmen Member

    I have an Ubuntu 16.04 server with ISPConfig 3.1.

    For some reason the server was working too much on a process with high load, so I powered off and restarted the machine. After that, Mysql stopped working and couldn't be restarted anymore.

    I checked the error log at /var/log/mysql/error.log but it's empty.

    When I tried to restart the service and I got the following messages:

    Code:
    Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
    systemctl status mysql.service:

    Code:
        ● mysql.service - LSB: Start and stop the mysql database server daemon
           Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
           Active: failed (Result: exit-code) since jue 2019-08-22 18:58:05 -04; 41s ago
            Docs: man:systemd-sysv-generator(8)
          Process: 12020 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
          Process: 12056 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
    journalctl -xe:

    Code:
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x955d0c]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x93b08e]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x93cd7f]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0xa1dd9d]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0xa670b6]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x9ba375]
        ago 22 18:58:32 servidor1 mysqld[12230]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f8ba90256ba]
        ago 22 18:58:32 servidor1 mysqld[12230]: x86_64/clone.S:111(clone)[0x7f8ba84cc41d]
        ago 22 18:58:32 servidor1 mysqld[12230]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
        ago 22 18:58:32 servidor1 mysqld[12230]: information that should help you find out what is causing the crash.
    
    
        ago 22 18:58:52 servidor1 dovecot[904]: auth-worker(12400): Error: sql([email protected],152.172.252.195): Password query failed: Not connected to database
        ago 22 18:58:52 servidor1 dovecot[904]: auth: Error: auth worker: Aborted PASSV request for [email protected]: Lookup timed out
        ago 22 18:58:52 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:58:53 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:58:54 servidor1 dovecot[904]: imap-login: Aborted login (auth failed, 1 attempts in 62 secs): user=<[email protected]>, method=PLAIN, rip=152.172.252.19
        ago 22 18:58:58 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:01 servidor1 CRON[12687]: pam_unix(cron:session): session opened for user root by (uid=0)
        ago 22 18:59:01 servidor1 CRON[12686]: pam_unix(cron:session): session opened for user root by (uid=0)
        ago 22 18:59:01 servidor1 CRON[12688]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log
        ago 22 18:59:01 servidor1 CRON[12689]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.l
        ago 22 18:59:01 servidor1 CRON[12686]: pam_unix(cron:session): session closed for user root
        ago 22 18:59:23 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:25 servidor1 su[12004]: pam_unix(su:session): session closed for user root
        ago 22 18:59:25 servidor1 sudo[12003]: pam_unix(sudo:session): session closed for user root
        ago 22 18:59:27 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:28 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:33 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:34 servidor1 sudo[12790]: surempresa : TTY=pts/20 ; PWD=/home/surempresa ; USER=root ; COMMAND=/bin/su
        ago 22 18:59:34 servidor1 sudo[12790]: pam_unix(sudo:session): session opened for user root by (uid=0)
        ago 22 18:59:34 servidor1 su[12809]: Successful su for root by root
        ago 22 18:59:34 servidor1 su[12809]: + /dev/pts/20 root:root
        ago 22 18:59:34 servidor1 su[12809]: pam_unix(su:session): session opened for user root by (uid=0)
        ago 22 18:59:34 servidor1 su[12809]: pam_systemd(su:session): Cannot create session: Already running in a session
    However, I managed to restart the service after I tried the solution from this post:

    https://askubuntu.com/questions/1008298/cant-start-mysql-server-database-corruption

    So I set:

    Code:
        [mysqld]
        innodb_force_recovery = 9
    But, it doesn't work well since some database tables are now read only.

    What should I do instead? Thank you.
     
    Last edited: Aug 26, 2019
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Did you do the power off with the power button? Do not touch the power button on a running server. Log in and do shutdown from root terminal session.
    If you now check those systemctl and journalctl commands, do they still show errors?
     
  3. deividmen

    deividmen Member

    Yes I did the power off with the power button. I guess that was the problem.

    This is what I got from those commands, with innodb_force_recovery = 9:

    systemctl status mysql.service:

    Code:
    mysql.service - LSB: Start and stop the mysql database server daemon
       Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
       Active: active (running) since jue 2019-08-22 20:41:44 -04; 15h ago
         Docs: man:systemd-sysv-generator(8)
      Process: 1098 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
       CGroup: /system.slice/mysql.service
               ├─1212 /bin/bash /usr/bin/mysqld_safe
               ├─1366 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/my
               └─1367 logger -t mysqld -p daemon error
    
    ago 23 12:07:04 servidor1 mysqld[1367]: 190823 12:07:04 [ERROR] InnoDB: Failed to find tablespace for table '"phpmyadmin"."pma__export_templates"' in the cache. Attempti
    Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

    journalctl -xe:

    Code:
    lines 991-1013/1013 (END)
    ago 23 12:31:14 servidor1 postfix/smtpd[24059]: disconnect from mail-eopbgr680124.outbound.protection.outlook.com[40.107.68.124] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 q
    ago 23 12:31:20 servidor1 postfix/smtpd[28032]: connect from localhost[127.0.0.1]
    ago 23 12:31:20 servidor1 postfix/smtpd[28032]: A4AAFA6DCE8: client=localhost[127.0.0.1]
    ago 23 12:31:20 servidor1 postfix/cleanup[28002]: A4AAFA6DCE8: message-id=<[email protected]od.outlook.com>
    ago 23 12:31:20 servidor1 postfix/qmgr[1773]: A4AAFA6DCE8: from=<[email protected]>, size=28575, nrcpt=1 (queue active)
    ago 23 12:31:20 servidor1 postfix/smtpd[28032]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
    ago 23 12:31:20 servidor1 amavis[22471]: (22471-06) Passed CLEAN {RelayedOpenRelay}, [40.107.68.124]:38726 [40.107.68.124] <[email protected]> -> <[email protected]
    ago 23 12:31:20 servidor1 postfix/smtp[28015]: 655B1A6DCE7: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10024, delay=8.3, delays=1.6/0.35/0.45/5.8, ds
    ago 23 12:31:20 servidor1 postfix/qmgr[1773]: 655B1A6DCE7: removed
    ago 23 12:31:21 servidor1 postfix/smtpd[26126]: warning: unknown[80.82.77.18]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    ago 23 12:31:21 servidor1 dovecot[28034]: lda([email protected]): sieve: msgid=? <[email protected]od.outlook.
    ago 23 12:31:21 servidor1 postfix/pipe[28033]: A4AAFA6DCE8: to=<[email protected]>, relay=dovecot, delay=0.59, delays=0.1/0.05/0/0.44, dsn=2.0.0, status=sent
    ago 23 12:31:21 servidor1 postfix/qmgr[1773]: A4AAFA6DCE8: removed
    ago 23 12:31:22 servidor1 postfix/smtpd[26126]: disconnect from unknown[80.82.77.18] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
    ago 23 12:31:26 servidor1 sudo[28035]: surempresa : TTY=pts/5 ; PWD=/home/surempresa ; USER=root ; COMMAND=/bin/su
    ago 23 12:31:26 servidor1 sudo[28035]: pam_unix(sudo:session): session opened for user root by (uid=0)
    ago 23 12:31:26 servidor1 su[28037]: Successful su for root by root
    ago 23 12:31:26 servidor1 su[28037]: + /dev/pts/5 root:root
    ago 23 12:31:26 servidor1 su[28037]: pam_unix(su:session): session opened for user root by (uid=0)
    ago 23 12:31:26 servidor1 su[28037]: pam_systemd(su:session): Cannot create session: Already running in a session
    ago 23 12:31:27 servidor1 dovecot[894]: pop3-login: Login: user=<[email protected]>, method=PLAIN, rip=209.85.167.4, lip=192.168.0.11, mpid=28049, session=<kQV7V
    ago 23 12:31:28 servidor1 dovecot[894]: pop3([email protected]): Disconnected: Logged out top=0/0, retr=0/0, del=0/8, size=267944
    ago 23 12:31:51 servidor1 postfix/smtpd[24059]: connect from unknown[80.82.77.18]
     
    Last edited: Aug 26, 2019
  4. concept21

    concept21 Active Member HowtoForge Supporter

    Are you a serious system administrator?!? :D

    Your data is severely corrupted! Restore it to an earlier healthy copy then! It is the only way!
    The best way is to restore it to the latest system image! :p
     

Share This Page