High Load (>4.00), Clamscan / SpamAssassin

[DOCK42]

Hi,

I got a message from my load monitoring script... one of our ispconfig server has a unusual high load since round about 30 minutes…

29296 username  25   0 63924  35m  29m R 22.3  1.7   0:00.67 spamassassin                                                                                   
29320 username  25   0 20216  13m 7040 R 17.6  0.7   0:00.53 clamscan

It's a Athlon X2 4200 CPU with 2GB RAM… so I think it's not „a matter of size“…

top - 16:30:22 up 19 days, 57 min,  1 user,  load average: 3.81, 3.72, 3.32
Tasks: 156 total,   5 running, 150 sleeping,   0 stopped,   1 zombie
 Cpu0 :  1.1% us,  0.1% sy,  0.0% ni, 98.6% id,  0.1% wa,  0.0% hi,  0.0% si
 Cpu1 :  3.3% us,  0.1% sy,  0.0% ni, 96.6% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   2056208k total,  1996320k used,    59888k free,   108964k buffers
Swap:  1004020k total,        4k used,  1004016k free,  1028064k cached

Looks like a process is stuck… (<defunct>)

# ps waux | grep username
username  28233  0.9  0.0 13068 1752 ?        S    16:26   0:03 /usr/bin/imapd Maildir
username  28857  0.0  0.0 12844 1540 ?        S    16:26   0:00 /usr/bin/imapd Maildir
username  28868  0.0  0.0 12860 1584 ?        S    16:26   0:00 /usr/bin/imapd Maildir
username   4931  0.0  0.0  8608  936 ?        Ss   16:32   0:00 /usr/bin/procmail -f-
username   4962  0.0  0.0  8480  800 ?        Ss   16:32   0:00 /usr/bin/procmail -f-
username   4963  0.0  0.0     0    0 ?        Z    16:32   0:00 [sh] <defunct>
username   4967  0.0  0.0  8480  800 ?        S    16:32   0:00 /usr/bin/procmail -f-
username   4968  0.0  0.0  5628 1208 ?        S    16:32   0:00 /bin/sh /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin
username   4977  0.0  0.9 25208 18880 ?       R    16:32   0:00 /home/admispconfig/ispconfig/tools/clamav/bin/clamscan --no-summary --stdout --mbox -
username   4983  0.0  0.0  8608  936 ?        S    16:32   0:00 /usr/bin/procmail -f-
username   4984  0.0  1.7 63392 35340 ?       R    16:32   0:00 /usr/bin/perl -T -w /home/admispconfig/ispconfig/tools/spamassassin/usr/bin/spamassassin --prefs-file=/home/www/web4/user/usernamels/.user_prefs

A few minutes later it's working fine…

# ps waux | grep username
username  28233  0.9  0.0 13068 1752 ?        S    16:26   0:03 /usr/bin/imapd Maildir
username  28857  0.0  0.0 12844 1540 ?        S    16:26   0:00 /usr/bin/imapd Maildir
username  28868  0.0  0.0 12860 1584 ?        S    16:26   0:00 /usr/bin/imapd Maildir
username   6901  0.0  0.0  8608  936 ?        Ss   16:33   0:00 /usr/bin/procmail -f-
username   6932  0.0  0.0  8608  936 ?        S    16:33   0:00 /usr/bin/procmail -f-
username   6933  0.0  1.3 56152 27740 ?       R    16:33   0:00 /usr/bin/perl -T -w /home/admispconfig/ispconfig/tools/spamassassin/usr/bin/spamassassin --prefs-file=/home/www/web4/user/usernamels/.user_prefs
username   6934  0.0  0.0  8480  800 ?        Ss   16:33   0:00 /usr/bin/procmail -f-
username   6935  0.0  0.0  5600 1140 ?        S    16:33   0:00 /bin/sh -c  wc -c | formail -A"X-Loop: ${LOGNAME}@localhost" \?                    -I"Subject: Mailsize: ${LOGNAME}" | \?  $SENDMAIL -oi admispconfig@localhost
username   6938  0.0  0.0 15388 1224 ?        S    16:33   0:00 /usr/sbin/sendmail -oi admispconfig@localhost
username   6939  0.0  0.0  8480  800 ?        S    16:33   0:00 /usr/bin/procmail -f-
username   6940  0.0  0.0  5628 1208 ?        S    16:33   0:00 /bin/sh /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin
username   6941  0.0  0.0 15380 1212 ?        D    16:33   0:00 /usr/sbin/postdrop -r
username   6949  0.0  0.1  8948 2576 ?        R    16:33   0:00 /home/admispconfig/ispconfig/tools/clamav/bin/clamscan --no-summary --stdout --mbox -
root      6951  0.0  0.0  2616  504 pts/0    R+   16:33   0:00 grep username

So… any ideas what's going on there?


Regards,
Yves

[DOCK42]

Without any interaction of mine now the system load is in a "normal" range.

top - 17:25:12 up 19 days,  1:52,  1 user,  load average: 0.00, 0.71, 1.93
Tasks: 130 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
 Cpu0 :  0.3% us,  0.3% sy,  0.0% ni, 99.3% id,  0.0% wa,  0.0% hi,  0.0% si
 Cpu1 :  1.7% us,  0.3% sy,  0.0% ni, 98.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   2056208k total,  1869628k used,   186580k free,   100452k buffers
Swap:  1004020k total,        4k used,  1004016k free,   858756k cached

But eventhough I'm interested in the reasons of such a high load (which isn't just a peak because of the fact that it lasts round about an hour…).
The server has ca. 40 mailboxes yet and I'm looking forward to ten times more mailboxes than now…

Best Regards,
Yves

[till]

I guess you got hit ba a massive spam attack which resulted in the high load.

To reduce the load of spamassassin, you might want to add some global filter rules and dns balcklists to your postfix configuration.

[DOCK42]

There're already DCC, Razor, Pyzor and some RHBL active… and mailgraph doesn't show any massive spam attacks.
I updated to the latest release of ispconfig last night - so I just wait and look for any further load peaks.

[fobicodam]

I had the same issue, even numbers around 60 / 70.. but after a week or so it become to 0.4, 0.3... maybe some underground linux tasks... who knows...

[cjc81]

Could you put in "qhape active" to see the amount of active mails when u experience the high load from spamassassin / clamscan?

I always get high loads when i have large amounts of active mails...

I believe spamassassin / clamscan is taking up a lot of CPU time scanning through the mails.

[minskog]

Here the same, in five servers (2 x opteron, 2x p4 2.8 and 1 x amd64 sempron). All with debian sarge. All spamassasin childs get 100% cpu ... even when there are only one or two mails at queue ...

[till]

If you have also many clamd processes, you should consider to switch to clamd as described here:

http://www.howtoforge.com/forums/showthread.php?t=12864

[minskog]

But, i dont have a lot of processes, the problem is those processes (one, two, ... ten) all take 100% CPU. A single process can increase the load to 4.0, there is no need to be more than one. (sorry for my bad english)