I know. I am just at the point of pointing out any and everything.
The server hit a load average > 50 while I was gone. This is really bs and it seems to be fastcgi or apache related. I watched the Munin graphs for SQL and nothing out of the ordinary is happening with queues, threads, etc. They stay within a range pretty consistently.
My script sent a lot of information and shows TONS of php-cgi processes, a number defunct.
Code:
mysql 16479 5.9 3.8 556428 156500 pts/4 Sl 14:31 10:19 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
root 16480 0.0 0.0 5040 664 pts/4 S 14:31 0:00 \_ logger -p daemon.err -t mysqld_safe -i -t mysqld
root 19829 0.0 0.4 296020 17148 ? Ss 14:53 0:00 /usr/sbin/apache2 -k start
root 19830 0.0 0.1 39184 6928 ? S 14:53 0:01 \_ vlogger (access log)
www-data 19831 0.0 0.1 170056 5388 ? S 14:53 0:00 \_ /usr/sbin/apache2 -k start
web1 19854 0.2 2.1 245876 85716 ? S 14:53 0:24 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web1/web:/var/www/clients/client1/web1/tmp:/var/www/masonicwiki.info/web:/srv/www/masonicwiki.info/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web1/tmp -d session.save_path=/var/www/clients/client1/web1/tmp
web7 20305 0.0 1.3 229008 56012 ? S 14:55 0:04 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web7/web:/var/www/clients/client1/web7/tmp:/var/www/sebastianlodge.com/web:/srv/www/sebastianlodge.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web7/tmp -d session.save_path=/var/www/clients/client1/web7/tmp
web11 20307 0.0 1.3 229256 55156 ? S 14:55 0:03 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web11/web:/var/www/clients/client1/web11/tmp:/var/www/derekgordon.com/web:/srv/www/derekgordon.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web11/tmp -d session.save_path=/var/www/clients/client1/web11/tmp
web11 21013 0.0 1.3 229256 55048 ? S 15:02 0:04 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web11/web:/var/www/clients/client1/web11/tmp:/var/www/derekgordon.com/web:/srv/www/derekgordon.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web11/tmp -d session.save_path=/var/www/clients/client1/web11/tmp
web29 23092 0.0 1.3 228244 56324 ? S 15:18 0:01 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web29/web:/var/www/clients/client1/web29/tmp:/var/www/derekandgaby.com/web:/srv/www/derekandgaby.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web29/tmp -d session.save_path=/var/www/clients/client1/web29/tmp
web5 30283 2.8 1.8 236496 74672 ? S 15:56 2:30 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web5/web:/var/www/clients/client1/web5/tmp:/var/www/areyouliberal.com/web:/srv/www/areyouliberal.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web5/tmp -d session.save_path=/var/www/clients/client1/web5/tmp
web5 30286 1.6 1.7 233160 71184 ? R 15:56 1:24 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web5/web:/var/www/clients/client1/web5/tmp:/var/www/areyouliberal.com/web:/srv/www/areyouliberal.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web5/tmp -d session.save_path=/var/www/clients/client1/web5/tmp
web5 30590 2.0 1.8 236196 74056 ? S 15:58 1:44 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web5/web:/var/www/clients/client1/web5/tmp:/var/www/areyouliberal.com/web:/srv/www/areyouliberal.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web5/tmp -d session.save_path=/var/www/clients/client1/web5/tmp
web8 4242 17.4 1.9 239972 77788 ? S 16:31 9:10 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web9 6159 0.0 1.1 225100 47552 ? S 16:41 0:00 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web9/web:/var/www/clients/client1/web9/tmp:/var/www/Peruvianprincess.org/web:/srv/www/Peruvianprincess.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web9/tmp -d session.save_path=/var/www/clients/client1/web9/tmp
web9 6166 0.0 1.1 225080 47648 ? S 16:41 0:00 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web9/web:/var/www/clients/client1/web9/tmp:/var/www/Peruvianprincess.org/web:/srv/www/Peruvianprincess.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web9/tmp -d session.save_path=/var/www/clients/client1/web9/tmp
web8 8072 15.5 1.5 223504 61448 ? S 16:51 5:00 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 11044 14.7 1.9 240212 78028 ? R 17:06 2:32 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 11073 19.5 1.5 225284 62540 ? R 17:06 3:20 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 11162 15.0 1.9 240240 78384 ? R 17:07 2:27 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13664 5.3 1.4 222704 60336 ? S 17:20 0:12 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13791 14.5 1.9 242224 79976 ? R 17:21 0:23 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13794 14.9 1.5 225024 62280 ? R 17:21 0:24 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13811 13.2 1.5 224772 62168 ? R 17:21 0:20 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13816 13.5 1.9 242220 79972 ? R 17:21 0:20 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13825 2.7 1.4 223228 60804 ? S 17:21 0:03 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13830 14.0 1.9 242240 79984 ? R 17:21 0:20 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13831 4.0 1.4 222708 60332 ? S 17:21 0:05 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13834 8.0 1.9 242216 79824 ? S 17:21 0:11 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13835 4.7 1.9 242200 79804 ? S 17:21 0:06 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13843 6.9 1.5 224496 63416 ? R 17:21 0:09 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13844 6.6 1.5 224220 63468 ? S 17:21 0:09 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13845 11.1 1.5 224240 63736 ? R 17:21 0:15 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13917 6.9 1.5 224216 63500 ? S 17:22 0:08 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13935 8.5 1.5 222956 62052 ? R 17:22 0:09 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13936 6.2 1.5 224476 63760 ? R 17:22 0:06 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web1 13954 0.3 0.7 205732 28848 ? S 17:22 0:00 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web1/web:/var/www/clients/client1/web1/tmp:/var/www/masonicwiki.info/web:/srv/www/masonicwiki.info/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web1/tmp -d session.save_path=/var/www/clients/client1/web1/tmp
web8 13970 15.0 1.9 240140 79592 ? R 17:22 0:13 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 13973 11.3 1.5 224240 63708 ? R 17:22 0:10 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
web8 14142 10.4 2.0 245520 82676 ? R 17:23 0:00 | \_ /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin -d upload_tmp_dir=/var/www/clients/client1/web8/tmp -d session.save_path=/var/www/clients/client1/web8/tmp
....................
server has high load right now.
Time: Wed Oct 27 17:24:02 EDT 2010
This is post 1 of 4 for the timeline of this crazy LOAD.
Nothing in the error logs showing anything of use that I've found. FastCGI is spawning way too many instances and according to the ISPC3 panel (and Till's instructions last week) I set the maximum CHILDREN to 1. This is WAAAAY more than one.