Whole server went down

[wxman]

I only have a minute because I'm trying to get my server back up!

It went down about an hour ago and the ony clue I see is in the apache log:

DBI connect('database=dbispconfig;host=localhost:3306','ispconfig',...) failed: Too many connections at /usr/local/ispconfig/server/scripts/vlogger line 255
DBI Error:  at /usr/local/ispconfig/server/scripts/vlogger line 255.
piped log program ' /usr/local/ispconfig/server/scripts/vlogger -s access.log -t "%Y%m%d-access.log" -d "/etc/vlogger-dbi.conf" /var/log/ispconfig/httpd' failed unexpectedly

This shows up over and over.

[till]

Increase the max_connections and max_user_connections setting to e.g. 500 in your mysql my.cnf and restart mysql.

[wxman]

I gave it a try, but I can't tell if it helped. I'm really stumped all day today. The server keeps locking up after being up for an hour or so. I won't be able to log in even to get to the logs, then all I can do is reboot. Stopping apache and mysql does nothing. When it's like that I was able to use top at a command line, and it showed the load averages all over 100. They usually hang around 1.
I get the feeling it has something to do with either apache or mysql, but I can't find anything clear in the logs.

[till]

Which processes cause the load?

[wxman]

Originally Posted by till

Which processes cause the load?

That's half my problem. I'm obviously not as good at running a server as I hoped I was. I don't know now where to find that out.

I've got top open right now, and the load is back to around 1. I do know when the last time it was it went crazy.

[wxman]

It just went down again. While I couldn't get to anything top looked like this:

top - 18:49:56 up  3:15,  2 users,  load average: 69.95, 71.26, 48.81
Tasks: 260 total,   1 running, 259 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  1.7%sy,  0.0%ni, 38.0%id, 59.1%wa,  0.0%hi,  0.0%si,  0.3%st
Mem:   1575132k total,  1562748k used,    12384k free,     8300k buffers
Swap:  7815580k total,  4844568k used,  2971012k free,    57876k cached
PID to renice:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20918 www-data  20   0  270m  43m 3456 S    0  2.9   0:00.66 apache2
20824 www-data  20   0  270m  32m 3444 S    0  2.1   0:00.60 apache2
 6392 mysql     20   0  446m  30m 2924 S    0  2.0   1:17.98 mysqld
20801 www-data  20   0  270m  30m 3424 S    0  2.0   0:00.52 apache2
20749 www-data  20   0  270m  28m 3424 D    0  1.9   0:00.74 apache2
20730 www-data  20   0  270m  26m 3444 S    0  1.7   0:00.76 apache2
20675 www-data  20   0  269m  25m 3416 S    0  1.7   0:00.92 apache2
20710 www-data  20   0  270m  25m 3536 D    0  1.7   0:00.80 apache2
20707 www-data  20   0  270m  25m 3476 S    0  1.6   0:00.56 apache2
20690 www-data  20   0  270m  24m 3428 S    0  1.6   0:00.84 apache2
20684 www-data  20   0  270m  22m 3480 D    0  1.4   0:00.68 apache2
20641 www-data  20   0  270m  20m 3476 S    0  1.4   0:00.82 apache2
20653 www-data  20   0  270m  20m 3480 D    0  1.3   0:00.92 apache2
20539 www-data  20   0  270m  19m 4084 D    0  1.3   0:01.30 apache2
20622 www-data  20   0  269m  18m 3460 S    0  1.2   0:01.06 apache2
20624 www-data  20   0  269m  18m 3464 S    0  1.2   0:00.98 apache2
20553 www-data  20   0  279m  18m 3544 D    0  1.2   0:01.16 apache2
20623 www-data  20   0  270m  16m 3424 S    0  1.1   0:00.94 apache2
20676 www-data  20   0  269m  16m 3416 S    0  1.1   0:00.98 apache2
20302 www-data  20   0  270m  16m 3512 D    0  1.1   0:01.30 apache2
20556 www-data  20   0  269m  15m 3812 S    0  1.0   0:00.94 apache2
19881 www-data  20   0  270m  15m 4292 D    0  1.0   0:03.30 apache2
21225 www-data  20   0  237m  15m 3360 S    0  1.0   0:00.16 apache2
20682 www-data  20   0  270m  14m 3428 S    0  1.0   0:00.92 apache2
20651 www-data  20   0  270m  14m 3428 S    0  1.0   0:00.98 apache2
20225 www-data  20   0  268m  14m 3536 D    1  0.9   0:01.52 apache2
20552 www-data  20   0  274m  14m 3524 S    0  0.9   0:00.92 apache2
20373 www-data  20   0  279m  14m 3500 S    0  0.9   0:01.32 apache2
19903 www-data  20   0  272m  14m 3920 D    0  0.9   0:03.46 apache2

It took less than 5 minutes to go from running normally to overload. The traffic in and out stayed the same until it overloaded. After it did, the traffic died completely. There were a lot more apache COMMAND running but I didn't include it here.