Centos 4.4 32bit Hangs, High Server load
 

Hi I just recently rebuilt my centos server using an IBM Netvista with a 1GB of ram, P4 2.8 GHZ, 250 GB HD Raid (Mirror) Thru a 3ware card. This was running centos 4.3 before nicely for a few months, until I installed some apache/php/mysql things(recently) and the system started to become inoperable with the system time falling behind the hardware time, so I made an attempt to rebuild the server.

I installed centos 4.4, I am running php4.4.x, apache 2.0.52, postfix, etc. I have installed ISPconfig, nothing really out of the ordinary.

Since the latest install the system runs great after a reboot for an hour to several hours before the load appears to rise, and when I check the time the system time starts to fall way behind the hardware time.

Also when I look at the processes in webmin, when the load appears to rise, I don't see it listed in the list of processes, it says 0.0% for all of them down the list.

Anyone please help, and when I choose to reboot, I usually have to Power it off hard, because the reboot takes maybe somewhere between 1-10 minutes to stop each process. So it could/would take somewhere fro 10minutes to hours to reboot this machine if it ever followed the reboot command.

This is urgent as this is for a production server, please help. It is a brand new install, so I don't think it has been compromised.

Do you see errors or warnings in your log files?
Can you post the output of when the losd is high?

Ok sorry for the wait, it took me some time to get this as the server needs some time before it hangs itself. here is an output of "top":
BTW, I checked the logs, and everything seemed to be in order for the most part. I will see how I can post a list of everything thats in top, I am not sure how yet, I am using putty.

I copied this from webmin:

BTW, what does lsod mean?

Heres yet another shot:

here is another shot today.

top - 11:30:58 up 4:31, 1 user, load average: 1.82, 1.51, 1.92
Tasks: 332 total, 2 running, 330 sleeping, 0 stopped, 0 zombie
Cpu(s): 50.0% us, 0.0% sy, 0.0% ni, 50.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1027048k total, 612512k used, 414536k free, 42896k buffers
Swap: 2031608k total, 0k used, 2031608k free, 219708k cached

from webmin:

continued webmin output:

You have lots of Postfix processes running. Are you sure your system isn't abused by spammers? Is there anything strange or inappropriate in your mail log?
Do you have lots of mails in your mailqueue? You can check with Also make sure your system didn't get hacked: http://www.howtoforge.com/faq/1_38_en.html

My system hasn't been hijacked, I used chkrootkit, and rkhunter, both checked out fine.(to my knowledge)

just wanted to post this again:
top - 19:15:16 up 2:18, 1 user, load average: 0.16, 0.07, 0.01
Tasks: 83 total, 1 running, 82 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 98.7% id, 1.3% wa, 0.0% hi, 0.0% si
Mem: 1027048k total, 847448k used, 179600k free, 216580k buffers
Swap: 2031608k total, 0k used, 2031608k free, 154840k cached

I think this may be my problem, i checked my mail logs and I noticed this message.

delivery temporarily suspended: Cannot start TLS: handshake failure

This doesn't mean your system is clean. If the hijackers know what they're doing, they can hide from chkrootkit and rkhunter. The only way to be sure is to run a scan from a Live CD like Knoppix or from a rescue system.