MySQL down

[Hans]

Yesterday (it was the first time ever) that the MySQL on one of my ispconfig servers went down.
Although, monit has been installed. Monit did not start the MySQL again.
Within the log files i could not find something related to the problem.

That this could happen was very unplesant.
I started mysql again with /etc/init.d/mysql start, but i would like to know how it could happen and why monit did not start mysql (as i expect it to).

I have these lines within the file /etc/monit/monitrc for mysql:

check process mysql with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysql start"
stop program = "/etc/init.d/mysql stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout

Can anyone advise me how to prevent this problem and tell me what the reason might be.

[falko]

Did you find any MySQL errors in your logs?

You can add this to the MySQL stanza in monit to watch the MySQL socket:

if failed unixsocket /var/run/mysqld/mysqld.sock protocol mysql then restart

[Hans]

Thanks Falko,
I added that line as well and i hope it will restart mysql if it ever happens again. I try to find the cause within the log files one more time.

[Hans]

This afternoon it happened again that my MySQL-server went down and I don’t know why.

This is what I found in the /var/log/syslog:

Aug 11 14:32:01 server1 mysqld[30789]: 080811 14:32:01 [Note] /usr/sbin/mysqld: Normal shutdown

Aug 11 14:32:01 server1 mysqld[30789]:

Aug 11 14:32:03 server1 mysqld[30789]: 080811 14:32:03 InnoDB: Starting shutdown...

Aug 11 14:32:06 server1 mysqld[30789]: 080811 14:32:06 InnoDB: Shutdown completed; log sequence number 0 1984718085

Aug 11 14:32:06 server1 mysqld[30789]: 080811 14:32:06 [Note] /usr/sbin/mysqld: Shutdown complete

Aug 11 14:32:06 server1 mysqld[30789]:

Aug 11 14:32:06 server1 mysqld_safe[25747]: ended

Aug 11 14:32:06 server1 mysqld_safe[25818]: started

Aug 11 14:32:07 server1 mysqld[25821]: 080811 14:32:07 InnoDB: Started; log sequence number 0 1984718085

Aug 11 14:32:07 server1 mysqld[25821]: 080811 14:32:07 [Note] /usr/sbin/mysqld: ready for connections.

Aug 11 14:32:07 server1 mysqld[25821]: Version: '5.0.32-Debian_7etch6-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Debian etch distribution

Aug 11 14:32:07 server1 /etc/mysql/debian-start[25859]: Upgrading MySQL tables if necessary.

Aug 11 14:32:08 server1 /etc/mysql/debian-start[25871]: Checking for crashed MySQL tables.

Aug 11 14:32:53 server1 monit[3598]: 'mysql' process PID changed to 25820

Aug 11 14:32:53 server1 monit[3598]: 'mysql' process PPID changed to 25783


I was able to restart it with /etc/init.d/mysql restart.

Monit did not restart MySQL, when MySQL-server went down.
How is that possible?

[Hans]

It seemed that the MySQL threads go up to the maximum value within /etc/my.cnf and then drop. It seems also the reason that Monit could not restart MySQL-server again.

By increasing the max_connections setting for MySQL to a higher value, this can be avoided.
I hope that this problem is over now.

For more monitoring for MySQL, I also changed the MySQL section within /etc/monit/monitrc into:

#MySQL
check process mysql with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysql start"
stop program = "/etc/init.d/mysql stop"
if failed host 127.0.0.1 port 3306 then restart
if failed unix "/var/lib/mysql/mysql.sock" then restart
if 5 restarts within 5 cycles then timeout
depends mysqld_init
depends mysqld_bin
depends mysqldsafe_bin
check file mysqld_init with path /etc/init.d/mysql
group database
check file mysqld_bin with path /usr/sbin/mysqld
group database
check file mysqldsafe_bin with path /usr/bin/mysqld_safe
group database

I want to thank Till & Falko for their help.