Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > ISPConfig 3 > General

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 20th March 2011, 15:18
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
Default High Disk Usage and Busy Servers from Apache

It seems that about once a week now, my load shoots through the roof and doesn't settle down until I restart apache. At the same time, I can see that the number of 'busy servers' on apache rises dramatically, and the disk usage is very high (3,000kb/s) even though network traffic is practically non-existent. What can I do to troubleshoot this further? Is it an attack, or just perhaps a faulty script?
Reply With Quote
Sponsored Links
  #2  
Old 21st March 2011, 01:05
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
Default

I just remembered, high swap usage would show higher than normal disk IO. Now I have to understand what is causing so many busy apache2 processes. Any ideas?
Reply With Quote
  #3  
Old 21st March 2011, 15:24
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,741 Times in 2,575 Posts
Default

Do you have munin installed? That might help to find out what is causing the problem. In the worst case it could be a problem with your hard drive.

If this is caused by DOS attacks, you can try this: http://www.faqforge.com/linux/preven...h-mod_evasive/
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #4  
Old 21st March 2011, 15:33
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
Default

Thank you for getting back to me. I do have munin installed, and the only spikes I see are in: load, apache 'busy servers' and disk throughput/latency/IO.

The current number of 'free servers' reported by the apache_processes munin plugin is 250, and I can't work out how to lessen this. My 'MaxClients' value is 100, and I am using prefork (on Debian Squeeze). Any ideas?

Also, I have just familiarized myself with the 'apachectl fullstatus' command, but that is only useful when I am experiencing the problem, not in advance or historically.

Thanks for the mod_evasive link. Any disadvantage to it?
Reply With Quote
  #5  
Old 21st March 2011, 16:31
Turbanator Turbanator is online now
Senior Member
 
Join Date: Jun 2008
Posts: 218
Thanks: 22
Thanked 16 Times in 16 Posts
Default

I had this type of problem, even to the point of shutting down external access into the network. Turned out I had an old install of OsCommerce on a domain and that install has MANY security leaks...so the website got hacked and bad scripts installed.
Reply With Quote
  #6  
Old 21st March 2011, 16:37
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
Default

Quote:
Originally Posted by Turbanator View Post
I had this type of problem, even to the point of shutting down external access into the network. Turned out I had an old install of OsCommerce on a domain and that install has MANY security leaks...so the website got hacked and bad scripts installed.
Thanks. It just happened again now, and I was able to run 'apachectl fullstatus' while it happened.

Falko - what information from there would help narrow down the issue?

I did see:

.WWWWWWWW_WWWWWWWWWRWWWWWWWWW_WWWWW___WWWWWWWWWW_W WWWWWWWWW.WWWW
_WW_WWWWWWW.WWWWWWWWWWWWWWWWWWWWWWWW.............. ..............
.................................................. ..............
.................................................. ..............

But all of the requests looked genuine. (Usually there wouldn't be that many, but if each request took longer to process, it is understandable that the number of genuine outstanding requests would add up.) Thoughts?
Reply With Quote
  #7  
Old 21st March 2011, 16:44
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
Default

By the way, I do have this line, but only one:

19-0 2034 0/2/150 R 0.00 4 148091 0.0 0.00 1.69 ? ? ..reading..
Reply With Quote
  #8  
Old 21st March 2011, 23:32
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
Default

Perhaps I have this problem: http://www.howtoforge.com/forums/sho...t=37850&page=2
I just counted 37 php-cgi instances...

EDIT: I have now moved all of my sites from FastCGI to mod_php. Given the ridiculous amounts of memory that FastCGI uses, I really have to wonder why there aren't clearer warnings to stay away from it. Unless you're using ISPConfig with only a few sites, I can't see any benefit...

Last edited by lewis_saint; 22nd March 2011 at 00:30. Reason: FastCGI..
Reply With Quote
  #9  
Old 22nd March 2011, 08:20
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 36,194
Thanks: 829
Thanked 5,419 Times in 4,261 Posts
Default

Fastcgi works fine, I use it for all my sites and its resource efficient on my servers.

Using mod_php is a invitation for hackers as all sites run under the same user. So using mod_php is only a option for inhouse servers or servers that dont host any client websites.

Which ISPConfig version do you use? There was a wrong setting in the vhost and fastcgi master template in a older version which has been fixed with 3.0.3 if I remember correctly. If you use a older version or you created the sites in a older version without having changed or updated them, then it might be that too many fastcgi processes get started by default.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #10  
Old 22nd March 2011, 12:29
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
 
Default

Quote:
Originally Posted by till View Post
Fastcgi works fine, I use it for all my sites and its resource efficient on my servers.

Using mod_php is a invitation for hackers as all sites run under the same user. So using mod_php is only a option for inhouse servers or servers that dont host any client websites.

Which ISPConfig version do you use? There was a wrong setting in the vhost and fastcgi master template in a older version which has been fixed with 3.0.3 if I remember correctly. If you use a older version or you created the sites in a older version without having changed or updated them, then it might be that too many fastcgi processes get started by default.
Thank you for getting back to me. I am currently using 3.0.3 (it was a fresh install, so all sites were created with this version), with about 40 sites. Do you know exactly which settings change, so I can verify?
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 18:28.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.