Mysql down after restarting server

Discussion in 'General' started by deividmen, Aug 23, 2019.

  1. deividmen

    deividmen New Member

    I have an Ubuntu 16.04 server with ISPConfig 3.1.

    For some reason the server was working too much on a process with high load, so I powered off and restarted the machine. After that, Mysql stopped working and couldn't be restarted anymore.

    I checked the error log at /var/log/mysql/error.log but it's empty.

    When I tried to restart the service and I got the following messages:

    Code:
    Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
    systemctl status mysql.service:

    Code:
        ● mysql.service - LSB: Start and stop the mysql database server daemon
           Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
           Active: failed (Result: exit-code) since jue 2019-08-22 18:58:05 -04; 41s ago
            Docs: man:systemd-sysv-generator(8)
          Process: 12020 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
          Process: 12056 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
    journalctl -xe:

    Code:
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x955d0c]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x93b08e]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x93cd7f]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0xa1dd9d]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0xa670b6]
        ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x9ba375]
        ago 22 18:58:32 servidor1 mysqld[12230]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f8ba90256ba]
        ago 22 18:58:32 servidor1 mysqld[12230]: x86_64/clone.S:111(clone)[0x7f8ba84cc41d]
        ago 22 18:58:32 servidor1 mysqld[12230]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
        ago 22 18:58:32 servidor1 mysqld[12230]: information that should help you find out what is causing the crash.
    
    
        ago 22 18:58:52 servidor1 dovecot[904]: auth-worker(12400): Error: sql([email protected],152.172.252.195): Password query failed: Not connected to database
        ago 22 18:58:52 servidor1 dovecot[904]: auth: Error: auth worker: Aborted PASSV request for [email protected]: Lookup timed out
        ago 22 18:58:52 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:58:53 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:58:54 servidor1 dovecot[904]: imap-login: Aborted login (auth failed, 1 attempts in 62 secs): user=<[email protected]>, method=PLAIN, rip=152.172.252.19
        ago 22 18:58:58 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:01 servidor1 CRON[12687]: pam_unix(cron:session): session opened for user root by (uid=0)
        ago 22 18:59:01 servidor1 CRON[12686]: pam_unix(cron:session): session opened for user root by (uid=0)
        ago 22 18:59:01 servidor1 CRON[12688]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log
        ago 22 18:59:01 servidor1 CRON[12689]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.l
        ago 22 18:59:01 servidor1 CRON[12686]: pam_unix(cron:session): session closed for user root
        ago 22 18:59:23 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:25 servidor1 su[12004]: pam_unix(su:session): session closed for user root
        ago 22 18:59:25 servidor1 sudo[12003]: pam_unix(sudo:session): session closed for user root
        ago 22 18:59:27 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:28 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:33 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug
        ago 22 18:59:34 servidor1 sudo[12790]: surempresa : TTY=pts/20 ; PWD=/home/surempresa ; USER=root ; COMMAND=/bin/su
        ago 22 18:59:34 servidor1 sudo[12790]: pam_unix(sudo:session): session opened for user root by (uid=0)
        ago 22 18:59:34 servidor1 su[12809]: Successful su for root by root
        ago 22 18:59:34 servidor1 su[12809]: + /dev/pts/20 root:root
        ago 22 18:59:34 servidor1 su[12809]: pam_unix(su:session): session opened for user root by (uid=0)
        ago 22 18:59:34 servidor1 su[12809]: pam_systemd(su:session): Cannot create session: Already running in a session
    However, I managed to restart the service after I tried the solution from this post:

    https://askubuntu.com/questions/1008298/cant-start-mysql-server-database-corruption

    So I set:

    Code:
        [mysqld]
        innodb_force_recovery = 9
    But, it doesn't work well since some database tables are now read only.

    What should I do instead? Thank you.
     
    Last edited: Aug 26, 2019
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Did you do the power off with the power button? Do not touch the power button on a running server. Log in and do shutdown from root terminal session.
    If you now check those systemctl and journalctl commands, do they still show errors?
     
  3. deividmen

    deividmen New Member

    Yes I did the power off with the power button. I guess that was the problem.

    This is what I got from those commands, with innodb_force_recovery = 9:

    systemctl status mysql.service:

    Code:
    mysql.service - LSB: Start and stop the mysql database server daemon
       Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
       Active: active (running) since jue 2019-08-22 20:41:44 -04; 15h ago
         Docs: man:systemd-sysv-generator(8)
      Process: 1098 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)
       CGroup: /system.slice/mysql.service
               ├─1212 /bin/bash /usr/bin/mysqld_safe
               ├─1366 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/my
               └─1367 logger -t mysqld -p daemon error
    
    ago 23 12:07:04 servidor1 mysqld[1367]: 190823 12:07:04 [ERROR] InnoDB: Failed to find tablespace for table '"phpmyadmin"."pma__export_templates"' in the cache. Attempti
    Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

    journalctl -xe:

    Code:
    lines 991-1013/1013 (END)
    ago 23 12:31:14 servidor1 postfix/smtpd[24059]: disconnect from mail-eopbgr680124.outbound.protection.outlook.com[40.107.68.124] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 q
    ago 23 12:31:20 servidor1 postfix/smtpd[28032]: connect from localhost[127.0.0.1]
    ago 23 12:31:20 servidor1 postfix/smtpd[28032]: A4AAFA6DCE8: client=localhost[127.0.0.1]
    ago 23 12:31:20 servidor1 postfix/cleanup[28002]: A4AAFA6DCE8: message-id=<[email protected]od.outlook.com>
    ago 23 12:31:20 servidor1 postfix/qmgr[1773]: A4AAFA6DCE8: from=<[email protected]>, size=28575, nrcpt=1 (queue active)
    ago 23 12:31:20 servidor1 postfix/smtpd[28032]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
    ago 23 12:31:20 servidor1 amavis[22471]: (22471-06) Passed CLEAN {RelayedOpenRelay}, [40.107.68.124]:38726 [40.107.68.124] <[email protected]> -> <[email protected]
    ago 23 12:31:20 servidor1 postfix/smtp[28015]: 655B1A6DCE7: to=<[email protected]aqdieselchile.cl>, relay=127.0.0.1[127.0.0.1]:10024, delay=8.3, delays=1.6/0.35/0.45/5.8, ds
    ago 23 12:31:20 servidor1 postfix/qmgr[1773]: 655B1A6DCE7: removed
    ago 23 12:31:21 servidor1 postfix/smtpd[26126]: warning: unknown[80.82.77.18]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    ago 23 12:31:21 servidor1 dovecot[28034]: lda([email protected]): sieve: msgid=? <[email protected]od.outlook.
    ago 23 12:31:21 servidor1 postfix/pipe[28033]: A4AAFA6DCE8: to=<[email protected]>, relay=dovecot, delay=0.59, delays=0.1/0.05/0/0.44, dsn=2.0.0, status=sent
    ago 23 12:31:21 servidor1 postfix/qmgr[1773]: A4AAFA6DCE8: removed
    ago 23 12:31:22 servidor1 postfix/smtpd[26126]: disconnect from unknown[80.82.77.18] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
    ago 23 12:31:26 servidor1 sudo[28035]: surempresa : TTY=pts/5 ; PWD=/home/surempresa ; USER=root ; COMMAND=/bin/su
    ago 23 12:31:26 servidor1 sudo[28035]: pam_unix(sudo:session): session opened for user root by (uid=0)
    ago 23 12:31:26 servidor1 su[28037]: Successful su for root by root
    ago 23 12:31:26 servidor1 su[28037]: + /dev/pts/5 root:root
    ago 23 12:31:26 servidor1 su[28037]: pam_unix(su:session): session opened for user root by (uid=0)
    ago 23 12:31:26 servidor1 su[28037]: pam_systemd(su:session): Cannot create session: Already running in a session
    ago 23 12:31:27 servidor1 dovecot[894]: pop3-login: Login: user=<[email protected]>, method=PLAIN, rip=209.85.167.4, lip=192.168.0.11, mpid=28049, session=<kQV7V
    ago 23 12:31:28 servidor1 dovecot[894]: pop3([email protected]): Disconnected: Logged out top=0/0, retr=0/0, del=0/8, size=267944
    ago 23 12:31:51 servidor1 postfix/smtpd[24059]: connect from unknown[80.82.77.18]
     
    Last edited: Aug 26, 2019
  4. concept21

    concept21 Member

    Are you a serious system administrator?!? :D

    Your data is severely corrupted! Restore it to an earlier healthy copy then! It is the only way!
    The best way is to restore it to the latest system image! :p
     

Share This Page