MySQL down

Discussion in 'Server Operation' started by Hans, Jun 7, 2008.

  1. Hans

    Hans Moderator ISPConfig Developer

    Yesterday (it was the first time ever) that the MySQL on one of my ispconfig servers went down.
    Although, monit has been installed. Monit did not start the MySQL again.
    Within the log files i could not find something related to the problem.

    That this could happen was very unplesant.
    I started mysql again with /etc/init.d/mysql start, but i would like to know how it could happen and why monit did not start mysql (as i expect it to).

    I have these lines within the file /etc/monit/monitrc for mysql:

    check process mysql with pidfile /var/run/mysqld/mysqld.pid
    group database
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"
    if failed host 127.0.0.1 port 3306 then restart
    if 5 restarts within 5 cycles then timeout

    Can anyone advise me how to prevent this problem and tell me what the reason might be.
     
  2. falko

    falko Super Moderator ISPConfig Developer

    Did you find any MySQL errors in your logs?

    You can add this to the MySQL stanza in monit to watch the MySQL socket:

    if failed unixsocket /var/run/mysqld/mysqld.sock protocol mysql then restart
     
  3. Hans

    Hans Moderator ISPConfig Developer

    Thanks Falko,
    I added that line as well and i hope it will restart mysql if it ever happens again. I try to find the cause within the log files one more time.
     
  4. Hans

    Hans Moderator ISPConfig Developer

    MySQL-server went down again

    This afternoon it happened again that my MySQL-server went down and I don’t know why.

    This is what I found in the /var/log/syslog:

    Aug 11 14:32:01 server1 mysqld[30789]: 080811 14:32:01 [Note] /usr/sbin/mysqld: Normal shutdown

    Aug 11 14:32:01 server1 mysqld[30789]:

    Aug 11 14:32:03 server1 mysqld[30789]: 080811 14:32:03 InnoDB: Starting shutdown...

    Aug 11 14:32:06 server1 mysqld[30789]: 080811 14:32:06 InnoDB: Shutdown completed; log sequence number 0 1984718085

    Aug 11 14:32:06 server1 mysqld[30789]: 080811 14:32:06 [Note] /usr/sbin/mysqld: Shutdown complete

    Aug 11 14:32:06 server1 mysqld[30789]:

    Aug 11 14:32:06 server1 mysqld_safe[25747]: ended

    Aug 11 14:32:06 server1 mysqld_safe[25818]: started

    Aug 11 14:32:07 server1 mysqld[25821]: 080811 14:32:07 InnoDB: Started; log sequence number 0 1984718085

    Aug 11 14:32:07 server1 mysqld[25821]: 080811 14:32:07 [Note] /usr/sbin/mysqld: ready for connections.

    Aug 11 14:32:07 server1 mysqld[25821]: Version: '5.0.32-Debian_7etch6-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Debian etch distribution

    Aug 11 14:32:07 server1 /etc/mysql/debian-start[25859]: Upgrading MySQL tables if necessary.

    Aug 11 14:32:08 server1 /etc/mysql/debian-start[25871]: Checking for crashed MySQL tables.

    Aug 11 14:32:53 server1 monit[3598]: 'mysql' process PID changed to 25820

    Aug 11 14:32:53 server1 monit[3598]: 'mysql' process PPID changed to 25783


    I was able to restart it with /etc/init.d/mysql restart.

    Monit did not restart MySQL, when MySQL-server went down.
    How is that possible?
     
    Last edited: Aug 11, 2008
  5. Hans

    Hans Moderator ISPConfig Developer

    Problem seems to be solved.

    It seemed that the MySQL threads go up to the maximum value within /etc/my.cnf and then drop. It seems also the reason that Monit could not restart MySQL-server again.

    By increasing the max_connections setting for MySQL to a higher value, this can be avoided.
    I hope that this problem is over now.

    For more monitoring for MySQL, I also changed the MySQL section within /etc/monit/monitrc into:

    #MySQL
    check process mysql with pidfile /var/run/mysqld/mysqld.pid
    group database
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"
    if failed host 127.0.0.1 port 3306 then restart
    if failed unix "/var/lib/mysql/mysql.sock" then restart
    if 5 restarts within 5 cycles then timeout
    depends mysqld_init
    depends mysqld_bin
    depends mysqldsafe_bin
    check file mysqld_init with path /etc/init.d/mysql
    group database
    check file mysqld_bin with path /usr/sbin/mysqld
    group database
    check file mysqldsafe_bin with path /usr/bin/mysqld_safe
    group database

    I want to thank Till & Falko for their help.
     

Share This Page