Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Server Operation

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 20th May 2010, 18:14
salami salami is offline
Junior Member
 
Join Date: Sep 2008
Posts: 6
Thanks: 2
Thanked 0 Times in 0 Posts
Default Diagnose sporadic high load (ubuntu 9.10, ISPConfig 3 setup)

Hi everyone

I'm running a server with ubuntu 9.10 set up according to the perfect server guide for ISPConfig 3. There's very little running on the server besides a couple of smaller websites. The box is fairly overpowered and is running just fine usually...

But every few days or so something happens that causes the system load to rise indefinitely, up to the point where the system gets completely unresponsive (I've seen it as high as 60.0).

A look at 'top' doesn't reveal any suspicious process, nothing is using a lot of CPU or memory. The amount of processes in the "waiting" state rises significantly though. My first guess was disk IO, but "iotop" doesn't reveal anything special either.

I have also installed munin, but so far I couldn't make any use of it's output. The spikes are clearly visible in the graphs, but they aren't higher than at other points of the day when the hardware has zero problems handling it.

When I'm quick enough a simple restart of mysqld solves the problem temporarily, the load goes back to normal instantly, but since so many services depend on mysql that's not necessarily the source of the problem.

Some help in hunting down this issue would be very welcome!

Thanks for any tips
Reply With Quote
Sponsored Links
  #2  
Old 21st May 2010, 16:22
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,745 Times in 2,578 Posts
Default

Does this happpen at the same time each day? Maybe a cron job is causing this?
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #3  
Old 21st May 2010, 16:54
Mark_NL Mark_NL is offline
Senior Member
 
Join Date: Sep 2008
Location: The Netherlands
Posts: 912
Thanks: 12
Thanked 100 Times in 96 Posts
Default

Are you pulling data from an NFS mount or anything external called/used?

the high wait and low cpu usage is mostlikely something that's waiting for some external process.
Reply With Quote
  #4  
Old 23rd May 2010, 10:55
salami salami is offline
Junior Member
 
Join Date: Sep 2008
Posts: 6
Thanks: 2
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by falko View Post
Does this happpen at the same time each day? Maybe a cron job is causing this?
Not that I can tell so far. It seems to happen very randomly and "only" once every 2-5 days. Still enough to drive you mad though
Reply With Quote
  #5  
Old 23rd May 2010, 11:00
salami salami is offline
Junior Member
 
Join Date: Sep 2008
Posts: 6
Thanks: 2
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by Mark_NL View Post
Are you pulling data from an NFS mount or anything external called/used?

the high wait and low cpu usage is mostlikely something that's waiting for some external process.
Nope, this is mostly a webserver using locally attached disks. I have now disabled two additional software components, maybe this will help... If not I'll probably just move the websites to a different server and scratch that box. But thanks anyway for your help!
Reply With Quote
  #6  
Old 3rd June 2010, 22:23
BorderAmigos BorderAmigos is offline
Senior Member
 
Join Date: Apr 2008
Location: San Diego & Tijuana
Posts: 302
Thanks: 26
Thanked 33 Times in 31 Posts
Send a message via MSN to BorderAmigos Send a message via Yahoo to BorderAmigos
Default

Probably not a help but I had a similar thing happening on a newly built server. I tracked it to the software RAID 1 disks re-synchronizing.
__________________
System6Hosting.com, ISPConfig 3, Debian.
Reply With Quote
  #7  
Old 11th June 2010, 14:24
zetnsh zetnsh is offline
Senior Member
 
Join Date: Aug 2007
Posts: 111
Thanks: 8
Thanked 5 Times in 5 Posts
 
Default

I have seen this situation before where a server can have a high load for seamingly no reason, and it has in my case been related to disk io. One thing that helped in my case and might help you too given that mysql is somehow implicated, is to make sure your MySQL query cache is well configured, and also the various key caches. Generally sticking a load of RAM in will massively help here, because mysql can put the results of commonly used queries into ram, which dramatically reduces disk I/O.

Not sure this will help, but you never know
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Upgrade Ubuntu 9.04 ISPConfig 3 to 9.10 sheshes Installation/Configuration 0 8th March 2010 15:21
Chrooted environments with Ubuntu and IspConfig hrvbid Installation/Configuration 5 23rd December 2009 19:44
Loads of mysql connections to dbispconfig StrikerNL General 2 5th March 2009 15:31
FC4 Setup DNS and ISPCONFIG issues The General Installation/Configuration 7 15th May 2006 10:45
ISPConfig + Perfect Setup Ubuntu 5.10 + Smtp SPAM problems lyndros Installation/Configuration 11 15th April 2006 18:07


All times are GMT +2. The time now is 11:55.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.