High Disk Usage and Busy Servers from Apache

[lewis_saint]

It seems that about once a week now, my load shoots through the roof and doesn't settle down until I restart apache. At the same time, I can see that the number of 'busy servers' on apache rises dramatically, and the disk usage is very high (3,000kb/s) even though network traffic is practically non-existent. What can I do to troubleshoot this further? Is it an attack, or just perhaps a faulty script?

[lewis_saint]

I just remembered, high swap usage would show higher than normal disk IO. Now I have to understand what is causing so many busy apache2 processes. Any ideas?

[falko]

Do you have munin installed? That might help to find out what is causing the problem. In the worst case it could be a problem with your hard drive.

If this is caused by DOS attacks, you can try this: http://www.faqforge.com/linux/preven...h-mod_evasive/

[lewis_saint]

Thank you for getting back to me. I do have munin installed, and the only spikes I see are in: load, apache 'busy servers' and disk throughput/latency/IO.

The current number of 'free servers' reported by the apache_processes munin plugin is 250, and I can't work out how to lessen this. My 'MaxClients' value is 100, and I am using prefork (on Debian Squeeze). Any ideas?

Also, I have just familiarized myself with the 'apachectl fullstatus' command, but that is only useful when I am experiencing the problem, not in advance or historically.

Thanks for the mod_evasive link. Any disadvantage to it?

[Turbanator]

I had this type of problem, even to the point of shutting down external access into the network. Turned out I had an old install of OsCommerce on a domain and that install has MANY security leaks...so the website got hacked and bad scripts installed.

[lewis_saint]

Originally Posted by Turbanator

I had this type of problem, even to the point of shutting down external access into the network. Turned out I had an old install of OsCommerce on a domain and that install has MANY security leaks...so the website got hacked and bad scripts installed.

Thanks. It just happened again now, and I was able to run 'apachectl fullstatus' while it happened.

Falko - what information from there would help narrow down the issue?

I did see:

.WWWWWWWW_WWWWWWWWWRWWWWWWWWW_WWWWW___WWWWWWWWWW_W WWWWWWWWW.WWWW
_WW_WWWWWWW.WWWWWWWWWWWWWWWWWWWWWWWW.............. ..............
.................................................. ..............
.................................................. ..............

But all of the requests looked genuine. (Usually there wouldn't be that many, but if each request took longer to process, it is understandable that the number of genuine outstanding requests would add up.) Thoughts?

[lewis_saint]

By the way, I do have this line, but only one:

19-0 2034 0/2/150 R 0.00 4 148091 0.0 0.00 1.69 ? ? ..reading..

[lewis_saint]

Perhaps I have this problem: http://www.howtoforge.com/forums/sho...t=37850&page=2
I just counted 37 php-cgi instances...

EDIT: I have now moved all of my sites from FastCGI to mod_php. Given the ridiculous amounts of memory that FastCGI uses, I really have to wonder why there aren't clearer warnings to stay away from it. Unless you're using ISPConfig with only a few sites, I can't see any benefit...

[till]

Fastcgi works fine, I use it for all my sites and its resource efficient on my servers.

Using mod_php is a invitation for hackers as all sites run under the same user. So using mod_php is only a option for inhouse servers or servers that dont host any client websites.

Which ISPConfig version do you use? There was a wrong setting in the vhost and fastcgi master template in a older version which has been fixed with 3.0.3 if I remember correctly. If you use a older version or you created the sites in a older version without having changed or updated them, then it might be that too many fastcgi processes get started by default.

[lewis_saint]

Originally Posted by till

Fastcgi works fine, I use it for all my sites and its resource efficient on my servers.

Using mod_php is a invitation for hackers as all sites run under the same user. So using mod_php is only a option for inhouse servers or servers that dont host any client websites.

Which ISPConfig version do you use? There was a wrong setting in the vhost and fastcgi master template in a older version which has been fixed with 3.0.3 if I remember correctly. If you use a older version or you created the sites in a older version without having changed or updated them, then it might be that too many fastcgi processes get started by default.

Thank you for getting back to me. I am currently using 3.0.3 (it was a fresh install, so all sites were created with this version), with about 40 sites. Do you know exactly which settings change, so I can verify?