Centos 4.4 32bit Hangs, High Server load

[3cwired_com]

Hi I just recently rebuilt my centos server using an IBM Netvista with a 1GB of ram, P4 2.8 GHZ, 250 GB HD Raid (Mirror) Thru a 3ware card. This was running centos 4.3 before nicely for a few months, until I installed some apache/php/mysql things(recently) and the system started to become inoperable with the system time falling behind the hardware time, so I made an attempt to rebuild the server.

I installed centos 4.4, I am running php4.4.x, apache 2.0.52, postfix, etc. I have installed ISPconfig, nothing really out of the ordinary.

Since the latest install the system runs great after a reboot for an hour to several hours before the load appears to rise, and when I check the time the system time starts to fall way behind the hardware time.

Also when I look at the processes in webmin, when the load appears to rise, I don't see it listed in the list of processes, it says 0.0% for all of them down the list.

Anyone please help, and when I choose to reboot, I usually have to Power it off hard, because the reboot takes maybe somewhere between 1-10 minutes to stop each process. So it could/would take somewhere fro 10minutes to hours to reboot this machine if it ever followed the reboot command.

This is urgent as this is for a production server, please help. It is a brand new install, so I don't think it has been compromised.

[falko]

Do you see errors or warnings in your log files?
Can you post the output of top when the losd is high?

[3cwired_com]

Ok sorry for the wait, it took me some time to get this as the server needs some time before it hangs itself. here is an output of "top":

top - 14:11:17 up 3:19, 1 user, load average: 3.52, 3.85, 2.26
Tasks: 197 total, 2 running, 195 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1027048k total, 422368k used, 604680k free, 37976k buffers
Swap: 2031608k total, 0k used, 2031608k free, 129360k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5799 root 16 0 2888 1024 760 R 0.3 0.1 0:08.85 top
1 root 16 0 1692 548 472 S 0.0 0.1 0:00.52 init


BTW, I checked the logs, and everything seemed to be in order for the most part. I will see how I can post a list of everything thats in top, I am not sure how yet, I am using putty.

[3cwired_com]

I copied this from webmin:


CPU load averages: 3.11 (1 mins) , 3.70 (5 mins) , 2.31 (15 mins)
CPU type: Intel(R) Pentium(R) 4 CPU 2.80GHz (2791 MHz)

Process ID Owner CPU Command
7120 apache 0.1 % /usr/sbin/httpd
1 root 0.0 % init [3]
2 root 0.0 % [ksoftirqd/0]
3 root 0.0 % [events/0]
4 root 0.0 % [khelper]
5 root 0.0 % [kacpid]
20 root 0.0 % [kblockd/0]
38 root 0.0 % [pdflush]
39 root 0.0 % [pdflush]
41 root 0.0 % [aio/0]
21 root 0.0 % [khubd]
40 root 0.0 % [kswapd0]
187 root 0.0 % [kseriod]
296 root 0.0 % [scsi_eh_0]
309 root 0.0 % [kmirrord]
317 root 0.0 % [kjournald]
1443 root 0.0 % udevd
1817 root 0.0 % [kauditd]
1865 root 0.0 % [kjournald]
2670 root 0.0 % syslogd -m 0
2674 root 0.0 % klogd -x
2750 rpc 0.0 % portmap
2769 root 0.0 % rpc.statd
2796 root 0.0 % rpc.idmapd
2865 root 0.0 % /usr/sbin/acpid
2874 root 0.0 % /usr/sbin/sshd
2887 root 0.0 % xinetd -stayalive -pidfile /var/run/xinetd.pid
2899 ntp 0.0 % ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
2960 root 0.0 % /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mys ...
2993 mysql 0.0 % /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/li ...
3021 root 0.0 % /usr/sbin/dovecot
3030 root 0.0 % dovecot-auth
3034 dovecot 0.0 % pop3-login
3036 dovecot 0.0 % pop3-login
3105 root 0.0 % gpm -m /dev/input/mice -t imps2
3137 root 0.0 % crond
3154 root 0.0 % /usr/sbin/atd
3163 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3166 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3167 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3169 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3170 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3176 dbus 0.0 % dbus-daemon-1 --system
3185 root 0.0 % hald
3361 root 0.0 % /root/ispconfig/httpd/bin/ispconfig_httpd -DSSL
3362 root 0.0 % /bin/bash /root/ispconfig/sv/ispconfig_wconf
3364 admispconfig 0.0 % /root/ispconfig/httpd/bin/ispconfig_httpd -DSSL
3461 root 0.0 % /usr/libexec/postfix/master
3468 postfix 0.0 % qmgr -l -t fifo -u
3487 named 0.0 % /usr/sbin/named -u named -t /var/named/chroot
3505 nobody 0.0 % proftpd: (accepting connections)
3512 admispconfig 0.0 % /home/admispconfig/ispconfig/tools/clamav/bin/freshclam -d -c 10 --datadir=/home ...
3521 root 0.0 % /usr/bin/perl /usr/local/webmin/miniserv.pl /etc/webmin/miniserv.conf
3525 root 0.0 % /sbin/mingetty tty1
3526 root 0.0 % /sbin/mingetty tty2
3527 root 0.0 % /sbin/mingetty tty3
3528 root 0.0 % /sbin/mingetty tty4
3529 root 0.0 % /sbin/mingetty tty5
3530 root 0.0 % /sbin/mingetty tty6
4102 postfix 0.0 % tlsmgr -l -t unix -u
4520 admispconfig 0.0 % /root/ispconfig/httpd/bin/ispconfig_httpd -DSSL
4575 dovecot 0.0 % pop3-login
5748 root 0.0 % sshd: root@pts/0
5750 root 0.0 % -bash
6223 dovecot 0.0 % imap-login
6254 dovecot 0.0 % imap-login
6264 dovecot 0.0 % imap-login
7114 root 0.0 % /usr/sbin/httpd
7115 root 0.0 % /root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/ ...
7118 apache 0.0 % /usr/sbin/httpd
7121 apache 0.0 % /usr/sbin/httpd
7122 apache 0.0 % /usr/sbin/httpd
7123 apache 0.0 % /usr/sbin/httpd
7124 apache 0.0 % /usr/sbin/httpd
7125 apache 0.0 % /usr/sbin/httpd
7126 apache 0.0 % /usr/sbin/httpd
7994 apache 0.0 % /usr/sbin/httpd
8043 apache 0.0 % /usr/sbin/httpd
8796 postfix 0.0 % anvil -l -t unix -u
8853 apache 0.0 % /usr/sbin/httpd
8884 postfix 0.0 % smtp -t unix -u
8908 postfix 0.0 % scache -l -t unix -u
8966 apache 0.0 % /usr/sbin/httpd
9038 apache 0.0 % /usr/sbin/httpd
9060 postfix 0.0 % smtpd -n smtp -t inet -u
9062 postfix 0.0 % proxymap -t unix -u
9163 apache 0.0 % /usr/sbin/httpd
9244 apache 0.0 % /usr/sbin/httpd
9577 postfix 0.0 % smtpd -n smtp -t inet -u
9578 postfix 0.0 % smtpd -n smtp -t inet -u
9605 apache 0.0 % /usr/sbin/httpd
9635 postfix 0.0 % smtpd -n smtp -t inet -u
9669 apache 0.0 % /usr/sbin/httpd
9677 postfix 0.0 % smtpd -n smtp -t inet -u
9680 postfix 0.0 % smtpd -n smtp -t inet -u
9681 postfix 0.0 % smtpd -n smtp -t inet -u
9682 postfix 0.0 % smtpd -n smtp -t inet -u
9706 apache 0.0 % /usr/sbin/httpd
9760 postfix 0.0 % smtpd -n smtp -t inet -u
9817 postfix 0.0 % trivial-rewrite -n rewrite -t unix -u
9839 apache 0.0 % sendmail: ./kA7Htffs00
9852 postfix 0.0 % cleanup -z -t unix -u
9897 apache 0.0 % sendmail: ./kA7I4Eq100
9906 apache 0.0 % /usr/sbin/httpd
9987 postfix 0.0 % smtpd -n smtp -t inet -u
10093 postfix 0.0 % smtpd -n smtp -t inet -u
10108 postfix 0.0 % smtpd -n smtp -t inet -u
10117 apache 0.0 % /usr/sbin/sendmail -t -i
10141 postfix 0.0 % smtpd -n smtp -t inet -u
10145 postfix 0.0 % smtpd -n smtp -t inet -u
10151 root 0.0 % crond
10154 root 0.0 % /bin/sh -c /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services. ...
10158 apache 0.0 % sendmail: ./kA7IU3wF01
10160 postfix 0.0 % smtpd -n smtp -t inet -u
10163 postfix 0.0 % smtpd -n smtp -t inet -u
10167 postfix 0.0 % smtpd -n smtp -t inet -u
10170 postfix 0.0 % smtpd -n smtp -t inet -u
10177 postfix 0.0 % smtpd -n smtp -t inet -u
10179 postfix 0.0 % smtpd -n smtp -t inet -u
10194 postfix 0.0 % smtpd -n smtp -t inet -u
10196 postfix 0.0 % smtpd -n smtp -t inet -u
10200 postfix 0.0 % cleanup -z -t unix -u
10220 postfix 0.0 % smtpd -n smtp -t inet -u
10238 postfix 0.0 % smtpd -n smtp -t inet -u
10239 apache 0.0 % /usr/sbin/httpd
10246 postfix 0.0 % smtpd -n smtp -t inet -u
10247 postfix 0.0 % smtpd -n smtp -t inet -u
10249 postfix 0.0 % smtpd -n smtp -t inet -u
10269 postfix 0.0 % smtpd -n smtp -t inet -u
10274 postfix 0.0 % smtpd -n smtp -t inet -u
10294 postfix 0.0 % smtpd -n smtp -t inet -u
10332 postfix 0.0 % smtpd -n smtp -t inet -u
10334 root 0.0 % /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services.php
10343 smmsp 0.0 % /usr/sbin/sendmail -t -i
10346 postfix 0.0 % smtpd -n smtp -t inet -u
10350 postfix 0.0 % smtpd -n smtp -t inet -u
10360 postfix 0.0 % smtpd -n smtp -t inet -u
10372 root 0.0 % crond
10373 root 0.0 % crond
10380 postfix 0.0 % smtpd -n smtp -t inet -u
10385 postfix 0.0 % smtpd -n smtp -t inet -u
10389 postfix 0.0 % smtpd -n smtp -t inet -u
10392 root 0.0 % crond
10393 root 0.0 % crond
10395 postfix 0.0 % smtpd -n smtp -t inet -u
10396 postfix 0.0 % smtpd -n smtp -t inet -u
10398 postfix 0.0 % smtpd -n smtp -t inet -u
10399 postfix 0.0 % smtpd -n smtp -t inet -u
10402 postfix 0.0 % smtpd -n smtp -t inet -u
10405 postfix 0.0 % smtpd -n smtp -t inet -u
10442 postfix 0.0 % smtpd -n smtp -t inet -u
10447 postfix 0.0 % smtpd -n smtp -t inet -u
10451 root 0.0 % crond
10452 root 0.0 % crond
10494 root 0.0 % crond
10495 root 0.0 % crond
10520 apache 0.0 % /usr/sbin/sendmail -t -i
10529 root 0.0 % crond
10530 root 0.0 % crond
10531 root 0.0 % crond
10532 root 0.0 % crond
10538 root 0.0 % crond
10539 root 0.0 % crond
10549 postfix 0.0 % smtpd -n smtp -t inet -u
10551 root 0.0 % crond
10552 root 0.0 % crond
10574 postfix 0.0 % smtpd -n smtp -t inet -u
10576 root 0.0 % crond
10577 root 0.0 % crond
10581 postfix 0.0 % smtpd -n smtp -t inet -u
10594 postfix 0.0 % smtpd -n smtp -t inet -u
10602 apache 0.0 % /usr/sbin/sendmail -t -i
10619 root 0.0 % top
10624 root 0.0 % crond
10625 root 0.0 % crond
10631 postfix 0.0 % pickup -l -t fifo -u
10643 root 0.0 % crond
10644 root 0.0 % crond
10657 root 0.0 % crond
10658 root 0.0 % crond
10668 postfix 0.0 % bounce -z -n defer -t unix -u
10677 root 0.0 % crond
10678 root 0.0 % crond
10679 root 0.0 % crond
10680 root 0.0 % crond
10685 apache 0.0 % /usr/sbin/sendmail -t -i
10692 postfix 0.0 % smtpd -n smtp -t inet -u
10694 apache 0.0 % /usr/sbin/sendmail -t -i
10699 postfix 0.0 % smtpd -n smtp -t inet -u
10714 root 0.0 % sleep 10
10718 root 0.0 % crond
10719 root 0.0 % crond
10722 apache 0.0 % /usr/sbin/sendmail -t -i
10724 root 0.0 % crond
10725 root 0.0 % crond
10726 root 0.0 % /usr/local/webmin/proc/index_cpu.cgi


BTW, what does lsod mean?

[3cwired_com]

Heres yet another shot:

top - 14:45:27 up 3:21, 1 user, load average: 2.75, 3.56, 2.31
Tasks: 196 total, 4 running, 192 sleeping, 0 stopped, 0 zombie
Cpu(s): 100.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1027048k total, 433768k used, 593280k free, 38052k buffers
Swap: 2031608k total, 0k used, 2031608k free, 130412k cached
Unknown command - try 'h' for help
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 16 0 1692 548 472 S -0.0 0.1 0:00.52 init
2 root 34 19 0 0 0 S -0.0 0.0 0:00.00 ksoftirqd/0
3 root 5 -10 0 0 0 S -0.0 0.0 0:00.00 events/0
4 root 9 -10 0 0 0 S -0.0 0.0 0:00.00 khelper
5 root 15 -10 0 0 0 S -0.0 0.0 0:00.00 kacpid
20 root 5 -10 0 0 0 S -0.0 0.0 0:00.00 kblockd/0
38 root 20 0 0 0 0 S -0.0 0.0 0:00.00 pdflush
39 root 15 0 0 0 0 S -0.0 0.0 0:00.01 pdflush
41 root 11 -10 0 0 0 S -0.0 0.0 0:00.00 aio/0
21 root 15 0 0 0 0 S -0.0 0.0 0:00.00 khubd
40 root 25 0 0 0 0 S -0.0 0.0 0:00.00 kswapd0
187 root 25 0 0 0 0 S -0.0 0.0 0:00.00 kseriod
296 root 25 0 0 0 0 S -0.0 0.0 0:00.00 scsi_eh_0
309 root 6 -10 0 0 0 S -0.0 0.0 0:00.00 kmirrord
317 root 15 0 0 0 0 S -0.0 0.0 0:00.28 kjournald
1443 root 6 -10 3400 468 380 S -0.0 0.0 0:00.01 udevd
1817 root 6 -10 0 0 0 S -0.0 0.0 0:00.00 kauditd
1865 root 15 0 0 0 0 S -0.0 0.0 0:00.00 kjournald
2670 root 15 0 2460 548 456 D -0.0 0.1 0:00.20 syslogd
2674 root 16 0 2408 380 316 S -0.0 0.0 0:00.00 klogd
2750 rpc 15 0 3492 544 452 S -0.0 0.1 0:00.00 portmap
2769 root 17 0 3544 728 620 S -0.0 0.1 0:00.00 rpc.statd
2796 root 16 0 5364 340 172 S -0.0 0.0 0:00.00 rpc.idmapd
2865 root 20 0 1936 440 368 S -0.0 0.0 0:00.00 acpid
2874 root 17 0 5112 1136 832 S -0.0 0.1 0:00.08 sshd
2887 root 15 0 2672 756 624 S -0.0 0.1 0:00.00 xinetd
2899 ntp 16 0 6056 6056 3436 S -0.0 0.6 0:00.01 ntpd
2960 root 22 0 6204 1248 1088 S -0.0 0.1 0:00.00 mysqld_safe
2993 mysql 16 0 128m 22m 3932 S -0.0 2.2 0:07.31 mysqld
3021 root 16 0 4668 508 364 S -0.0 0.0 0:01.02 dovecot
3030 root 16 0 6716 1484 1108 S -0.0 0.1 0:00.01 dovecot-auth
3034 dovecot 16 0 4016 1452 1240 S -0.0 0.1 0:00.10 pop3-login
3036 dovecot 16 0 5272 1452 1240 S -0.0 0.1 0:00.10 pop3-login
3105 root 16 0 2948 352 280 S -0.0 0.0 0:00.00 gpm
3137 root 16 0 4612 932 536 S -0.0 0.1 0:00.00 crond
3154 root 16 0 3656 424 308 S -0.0 0.0 0:00.00 atd
3163 root 18 0 4960 432 300 S -0.0 0.0 0:00.00 saslauthd
3166 root 18 0 4960 216 84 S -0.0 0.0 0:00.00 saslauthd

[3cwired_com]

here is another shot today.

top - 11:30:58 up 4:31, 1 user, load average: 1.82, 1.51, 1.92
Tasks: 332 total, 2 running, 330 sleeping, 0 stopped, 0 zombie
Cpu(s): 50.0% us, 0.0% sy, 0.0% ni, 50.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1027048k total, 612512k used, 414536k free, 42896k buffers
Swap: 2031608k total, 0k used, 2031608k free, 219708k cached

from webmin:

1 root 0.0 % init [3]
2 root 0.0 % [ksoftirqd/0]
3 root 0.0 % [events/0]
4 root 0.0 % [khelper]
5 root 0.0 % [kacpid]
20 root 0.0 % [kblockd/0]
38 root 0.0 % [pdflush]
39 root 0.0 % [pdflush]
41 root 0.0 % [aio/0]
21 root 0.0 % [khubd]
40 root 0.0 % [kswapd0]
187 root 0.0 % [kseriod]
296 root 0.0 % [scsi_eh_0]
309 root 0.0 % [kmirrord]
317 root 0.0 % [kjournald]
1443 root 0.0 % udevd
1817 root 0.0 % [kauditd]
1865 root 0.0 % [kjournald]
2670 root 0.0 % syslogd -m 0
2674 root 0.0 % klogd -x
2750 rpc 0.0 % portmap
2769 root 0.0 % rpc.statd
2796 root 0.0 % rpc.idmapd
2865 root 0.0 % /usr/sbin/acpid
2874 root 0.0 % /usr/sbin/sshd
2887 root 0.0 % xinetd -stayalive -pidfile /var/run/xinetd.pid
2899 ntp 0.0 % ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
2960 root 0.0 % /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mys ...
2993 mysql 0.0 % /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/li ...
3021 root 0.0 % /usr/sbin/dovecot
3030 root 0.0 % dovecot-auth
3034 dovecot 0.0 % pop3-login
3036 dovecot 0.0 % pop3-login
3105 root 0.0 % gpm -m /dev/input/mice -t imps2
3137 root 0.0 % crond
3154 root 0.0 % /usr/sbin/atd
3163 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3166 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3167 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3169 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3170 root 0.0 % /usr/sbin/saslauthd -m /var/run/saslauthd -a shadow
3176 dbus 0.0 % dbus-daemon-1 --system
3185 root 0.0 % hald
3361 root 0.0 % /root/ispconfig/httpd/bin/ispconfig_httpd -DSSL
3362 root 0.0 % /bin/bash /root/ispconfig/sv/ispconfig_wconf
3364 admispconfig 0.0 % /root/ispconfig/httpd/bin/ispconfig_httpd -DSSL
3487 named 0.0 % /usr/sbin/named -u named -t /var/named/chroot
3505 nobody 0.0 % proftpd: (accepting connections)
3512 admispconfig 0.0 % /home/admispconfig/ispconfig/tools/clamav/bin/freshclam -d -c 10 --datadir=/home ...
3521 root 0.0 % /usr/bin/perl /usr/local/webmin/miniserv.pl /etc/webmin/miniserv.conf
3525 root 0.0 % /sbin/mingetty tty1
3526 root 0.0 % /sbin/mingetty tty2
3527 root 0.0 % /sbin/mingetty tty3
3528 root 0.0 % /sbin/mingetty tty4
3529 root 0.0 % /sbin/mingetty tty5
3530 root 0.0 % /sbin/mingetty tty6
4520 admispconfig 0.0 % /root/ispconfig/httpd/bin/ispconfig_httpd -DSSL
4575 dovecot 0.0 % pop3-login
5748 root 0.0 % sshd: root@pts/0
5750 root 0.0 % -bash
6223 dovecot 0.0 % imap-login
6254 dovecot 0.0 % imap-login
7114 root 0.0 % /usr/sbin/httpd
7115 root 0.0 % /root/ispconfig/cronolog --symlink=/var/log/httpd/ispconfig_access_log /var/log/ ...
7120 apache 0.0 % /usr/sbin/httpd
7121 apache 0.0 % /usr/sbin/httpd
7122 apache 0.0 % /usr/sbin/httpd
7123 apache 0.0 % /usr/sbin/httpd
7124 apache 0.0 % /usr/sbin/httpd
7994 apache 0.0 % /usr/sbin/httpd
8043 apache 0.0 % /usr/sbin/httpd
8853 apache 0.0 % /usr/sbin/httpd
8966 apache 0.0 % /usr/sbin/httpd
9038 apache 0.0 % /usr/sbin/httpd
9163 apache 0.0 % /usr/sbin/httpd
9244 apache 0.0 % /usr/sbin/httpd
9669 apache 0.0 % /usr/sbin/httpd
9706 apache 0.0 % /usr/sbin/httpd
9906 apache 0.0 % /usr/sbin/httpd
10782 apache 0.0 % /usr/sbin/httpd
10890 apache 0.0 % /usr/sbin/httpd
13535 apache 0.0 % /usr/sbin/httpd
13595 apache 0.0 % /usr/sbin/httpd
13608 apache 0.0 % /usr/sbin/httpd
15513 root 0.0 % /usr/libexec/postfix/master
15515 postfix 0.0 % qmgr -l -t fifo -u
15516 postfix 0.0 % trivial-rewrite -n rewrite -t unix -u
15523 postfix 0.0 % tlsmgr -l -t unix -u
15531 postfix 0.0 % local -t unix
15623 postfix 0.0 % proxymap -t unix -u
15629 postfix 0.0 % anvil -l -t unix -u
15751 postfix 0.0 % smtp -t unix -u
15752 postfix 0.0 % local -t unix
15755 postfix 0.0 % scache -l -t unix -u
15857 postfix 0.0 % local -t unix
15880 postfix 0.0 % local -t unix
17242 postfix 0.0 % smtpd -n smtp -t inet -u
17243 postfix 0.0 % smtpd -n smtp -t inet -u
18802 postfix 0.0 % smtpd -n smtp -t inet -u
19032 postfix 0.0 % smtpd -n smtp -t inet -u
19047 root 0.0 % crond
19048 root 0.0 % /bin/sh -c /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services. ...
19055 postfix 0.0 % smtpd -n smtp -t inet -u
19078 apache 0.0 % sendmail: [127.0.0.1]:
19096 apache 0.0 % sendmail: [127.0.0.1]:
19139 root 0.0 % /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services.php
19142 apache 0.0 % sendmail: ./kA8AseMs01
19147 smmsp 0.0 % sendmail: ./kA8AseHc019147 [127.0.0.1]: client DATA status
19167 apache 0.0 % sendmail: ./kA8B07Bt01
19220 postfix 0.0 % smtpd -n smtp -t inet -u
19222 postfix 0.0 % smtpd -n smtp -t inet -u
19349 postfix 0.0 % smtpd -n smtp -t inet -u
19370 dovecot 0.0 % imap-login
19389 postfix 0.0 % smtpd -n smtp -t inet -u
19564 postfix 0.0 % smtpd -n smtp -t inet -u
19636 postfix 0.0 % smtpd -n smtp -t inet -u
19639 postfix 0.0 % smtpd -n smtp -t inet -u
19641 postfix 0.0 % smtpd -n smtp -t inet -u
19655 postfix 0.0 % smtpd -n smtp -t inet -u
19659 postfix 0.0 % smtpd -n smtp -t inet -u
19677 postfix 0.0 % smtpd -n smtp -t inet -u
19733 postfix 0.0 % smtpd -n smtp -t inet -u
19745 postfix 0.0 % smtpd -n smtp -t inet -u
19868 postfix 0.0 % smtpd -n smtp -t inet -u
19869 postfix 0.0 % smtpd -n smtp -t inet -u
19966 postfix 0.0 % smtpd -n smtp -t inet -u
20047 postfix 0.0 % smtpd -n smtp -t inet -u
20079 postfix 0.0 % smtpd -n smtp -t inet -u
20099 postfix 0.0 % smtpd -n smtp -t inet -u
20114 postfix 0.0 % smtpd -n smtp -t inet -u
20305 postfix 0.0 % smtpd -n smtp -t inet -u
20308 postfix 0.0 % smtpd -n smtp -t inet -u
20320 postfix 0.0 % smtpd -n smtp -t inet -u
20329 postfix 0.0 % smtpd -n smtp -t inet -u
20333 postfix 0.0 % smtpd -n smtp -t inet -u
20385 postfix 0.0 % cleanup -z -t unix -u
20405 postfix 0.0 % smtpd -n smtp -t inet -u
20483 postfix 0.0 % smtpd -n smtp -t inet -u
20516 postfix 0.0 % cleanup -z -t unix -u
20566 postfix 0.0 % smtpd -n smtp -t inet -u
20584 postfix 0.0 % smtpd -n smtp -t inet -u
20610 postfix 0.0 % smtpd -n smtp -t inet -u
20612 postfix 0.0 % smtpd -n smtp -t inet -u
20613 postfix 0.0 % smtpd -n smtp -t inet -u
20631 postfix 0.0 % smtpd -n smtp -t inet -u
20636 postfix 0.0 % smtpd -n smtp -t inet -u
20691 postfix 0.0 % smtpd -n smtp -t inet -u
20710 postfix 0.0 % smtpd -n smtp -t inet -u
20724 postfix 0.0 % smtpd -n smtp -t inet -u
20753 postfix 0.0 % smtpd -n smtp -t inet -u
20759 postfix 0.0 % smtpd -n smtp -t inet -u
20765 postfix 0.0 % smtpd -n smtp -t inet -u
20767 postfix 0.0 % smtpd -n smtp -t inet -u
20775 postfix 0.0 % smtpd -n smtp -t inet -u
20779 postfix 0.0 % smtpd -n smtp -t inet -u
20785 postfix 0.0 % smtpd -n smtp -t inet -u
20795 postfix 0.0 % smtpd -n smtp -t inet -u
20797 postfix 0.0 % smtpd -n smtp -t inet -u
20800 postfix 0.0 % smtpd -n smtp -t inet -u
20803 root 0.0 % crond
20804 root 0.0 % crond
20805 root 0.0 % crond
20806 root 0.0 % crond
20807 root 0.0 % crond
20808 root 0.0 % crond
20811 postfix 0.0 % smtpd -n smtp -t inet -u
20816 postfix 0.0 % smtpd -n smtp -t inet -u
20827 postfix 0.0 % smtpd -n smtp -t inet -u
20829 root 0.0 % crond
20830 root 0.0 % crond
20867 postfix 0.0 % smtpd -n smtp -t inet -u
20881 postfix 0.0 % smtpd -n smtp -t inet -u
20892 postfix 0.0 % smtpd -n smtp -t inet -u
20897 root 0.0 % crond
20898 root 0.0 % crond
20908 postfix 0.0 % smtpd -n smtp -t inet -u
20916 root 0.0 % crond
20917 root 0.0 % crond
20918 root 0.0 % crond
20919 root 0.0 % crond
20924 postfix 0.0 % smtpd -n smtp -t inet -u
20929 root 0.0 % crond
20930 root 0.0 % crond
20932 postfix 0.0 % smtpd -n smtp -t inet -u
20938 postfix 0.0 % smtpd -n smtp -t inet -u
20943 root 0.0 % crond
20944 root 0.0 % crond
20945 postfix 0.0 % smtpd -n smtp -t inet -u
20946 postfix 0.0 % smtpd -n smtp -t inet -u
20952 postfix 0.0 % smtpd -n smtp -t inet -u
20954 postfix 0.0 % smtpd -n smtp -t inet -u
20956 root 0.0 % crond
20957 root 0.0 % crond
20965 postfix 0.0 % smtpd -n smtp -t inet -u
20996 root 0.0 % crond
20997 root 0.0 % crond
21019 root 0.0 % crond
21020 root 0.0 % crond
21030 root 0.0 % crond
21031 root 0.0 % crond
21035 postfix 0.0 % smtpd -n smtp -t inet -u
21043 root 0.0 % crond
21044 root 0.0 % crond

[3cwired_com]

continued webmin output:


21045 root 0.0 % crond
21046 root 0.0 % crond
21047 postfix 0.0 % smtpd -n smtp -t inet -u
21059 root 0.0 % crond
21060 root 0.0 % crond
21061 root 0.0 % crond
21062 root 0.0 % crond
21079 root 0.0 % crond
21081 root 0.0 % crond
21082 root 0.0 % crond
21083 root 0.0 % crond
21090 root 0.0 % crond
21091 root 0.0 % crond
21094 postfix 0.0 % smtpd -n smtp -t inet -u
21099 postfix 0.0 % smtpd -n smtp -t inet -u
21103 root 0.0 % crond
21104 root 0.0 % crond
21115 postfix 0.0 % smtpd -n smtp -t inet -u
21117 postfix 0.0 % smtpd -n smtp -t inet -u
21122 root 0.0 % crond
21123 root 0.0 % crond
21124 root 0.0 % crond
21125 root 0.0 % crond
21133 postfix 0.0 % smtpd -n smtp -t inet -u
21136 root 0.0 % crond
21137 root 0.0 % crond
21141 postfix 0.0 % smtpd -n smtp -t inet -u
21157 root 0.0 % crond
21158 root 0.0 % crond
21168 root 0.0 % crond
21170 root 0.0 % crond
21173 root 0.0 % crond
21174 root 0.0 % crond
21175 root 0.0 % crond
21176 root 0.0 % crond
21185 root 0.0 % crond
21186 root 0.0 % crond
21188 postfix 0.0 % smtpd -n smtp -t inet -u
21205 root 0.0 % crond
21206 root 0.0 % crond
21220 postfix 0.0 % smtpd -n smtp -t inet -u
21224 root 0.0 % crond
21225 root 0.0 % crond
21226 root 0.0 % crond
21227 root 0.0 % crond
21232 postfix 0.0 % smtpd -n smtp -t inet -u
21246 postfix 0.0 % smtpd -n smtp -t inet -u
21249 root 0.0 % crond
21250 root 0.0 % crond
21251 root 0.0 % crond
21252 root 0.0 % crond
21262 postfix 0.0 % smtpd -n smtp -t inet -u
21281 root 0.0 % crond
21282 root 0.0 % crond
21308 postfix 0.0 % smtpd -n smtp -t inet -u
21323 root 0.0 % crond
21324 root 0.0 % crond
21329 postfix 0.0 % smtpd -n smtp -t inet -u
21331 postfix 0.0 % smtpd -n smtp -t inet -u
21338 nobody 0.0 % proftpd: connected: 127.0.0.1 (127.0.0.1:33418)
21340 apache 0.0 % /usr/sbin/sendmail -t -i
21345 root 0.0 % crond
21346 root 0.0 % crond
21347 root 0.0 % crond
21348 root 0.0 % crond
21349 root 0.0 % crond
21350 root 0.0 % crond
21351 root 0.0 % crond
21352 root 0.0 % crond
21385 postfix 0.0 % smtpd -n smtp -t inet -u
21389 root 0.0 % crond
21391 root 0.0 % crond
21392 root 0.0 % crond
21394 root 0.0 % crond
21400 root 0.0 % crond
21401 root 0.0 % crond
21402 root 0.0 % crond
21403 root 0.0 % crond
21452 root 0.0 % crond
21453 root 0.0 % crond
21494 postfix 0.0 % smtpd -n smtp -t inet -u
21545 root 0.0 % crond
21546 root 0.0 % crond
21643 postfix 0.0 % smtpd -n smtp -t inet -u
21653 root 0.0 % crond
21655 root 0.0 % crond
21671 root 0.0 % crond
21672 root 0.0 % crond
21673 root 0.0 % crond
21674 root 0.0 % crond
21711 postfix 0.0 % smtpd -n smtp -t inet -u
21720 apache 0.0 % /usr/sbin/sendmail -t -i
21746 root 0.0 % crond
21747 root 0.0 % crond
21754 postfix 0.0 % smtpd -n smtp -t inet -u
21803 root 0.0 % crond
21805 root 0.0 % crond
21824 postfix 0.0 % smtpd -n smtp -t inet -u
21828 postfix 0.0 % smtpd -n smtp -t inet -u
21834 root 0.0 % crond
21835 root 0.0 % crond
21836 root 0.0 % crond
21837 root 0.0 % crond
21844 root 0.0 % crond
21845 root 0.0 % crond
21846 postfix 0.0 % smtpd -n smtp -t inet -u
21849 apache 0.0 % /usr/sbin/sendmail -t -i
21856 postfix 0.0 % smtpd -n smtp -t inet -u
21858 postfix 0.0 % smtpd -n smtp -t inet -u
21865 root 0.0 % crond
21870 root 0.0 % crond
21875 postfix 0.0 % smtpd -n smtp -t inet -u
21876 postfix 0.0 % smtpd -n smtp -t inet -u
21881 postfix 0.0 % smtpd -n smtp -t inet -u
21888 postfix 0.0 % smtpd -n smtp -t inet -u
21890 postfix 0.0 % smtpd -n smtp -t inet -u
21897 postfix 0.0 % smtpd -n smtp -t inet -u
21908 root 0.0 % crond
21909 root 0.0 % crond
21910 root 0.0 % crond
21911 root 0.0 % crond
21926 postfix 0.0 % smtpd -n smtp -t inet -u
21927 postfix 0.0 % pickup -l -t fifo -u
21950 root 0.0 % sleep 10
21962 root 0.0 % crond
21963 root 0.0 % crond
21966 root 0.0 % top
21967 root 0.0 % /usr/local/webmin/proc/index_cpu.cgi

[falko]

You have lots of Postfix processes running. Are you sure your system isn't abused by spammers? Is there anything strange or inappropriate in your mail log?
Do you have lots of mails in your mailqueue? You can check with postqueue -p
Also make sure your system didn't get hacked: http://www.howtoforge.com/faq/1_38_en.html

[3cwired_com]

My system hasn't been hijacked, I used chkrootkit, and rkhunter, both checked out fine.(to my knowledge)

just wanted to post this again:
top - 19:15:16 up 2:18, 1 user, load average: 0.16, 0.07, 0.01
Tasks: 83 total, 1 running, 82 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 98.7% id, 1.3% wa, 0.0% hi, 0.0% si
Mem: 1027048k total, 847448k used, 179600k free, 216580k buffers
Swap: 2031608k total, 0k used, 2031608k free, 154840k cached

I think this may be my problem, i checked my mail logs and I noticed this message.

delivery temporarily suspended: Cannot start TLS: handshake failure

[falko]

My system hasn't been hijacked, I used chkrootkit, and rkhunter, both checked out fine.(to my knowledge)

This doesn't mean your system is clean. If the hijackers know what they're doing, they can hide from chkrootkit and rkhunter. The only way to be sure is to run a scan from a Live CD like Knoppix or from a rescue system.

[3cwired_com]

Ok so far what I have done is I basically just took my setup(hard drives and raid card) and moved it to a different motherboard, and I switched the ram. I think one of the sticks I had might have been causing the server to behave all funky. because right now, my server is working fine, I did however reinstall postfix/dovecot for the xxth time, and now everything is working fine again. This might have been an issue with a memory stick failing, because I just switched up the modules, and then pulled out one of the modules, and the system has basically been fine ever since, so that is something people should look into in the future.

[falko]

You can test your RAM with memtest86: http://www.memtest86.com/
http://www.memtest.org/