Slow response

[Morons]

HI,
I am without resolve on this one, out of the blue, one of my servers thats been running for weeks now, suddenly become almost impossible to log into. the login process from the consile / ssh and even all ports 110 ans 25 seem tot timeout to a couple minutes!!
My initial thoughts was that his server run out of some resource or another. however I can not detect the resource nor the process in error.

By using top i can indicate an running system - althow slow it seem to function

top - 14:25:40 up 26 min, 2 users, load average: 115.32, 103.99, 69.81
Tasks: 307 total, 2 running, 289 sleeping, 0 stopped, 16 zombie
Cpu(s): 20.9%us, 1.3%sy, 0.0%ni, 0.0%id, 77.2%wa, 0.3%hi, 0.3%si, 0.0%st
Mem: 515896k total, 510960k used, 4936k free, 1416k buffers
Swap: 1510068k total, 1478364k used, 31704k free, 13020k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5264 nyl-cass 17 0 27508 2084 584 D 8.6 0.4 0:13.59 clamscan
6712 top-henn 17 0 10192 8512 1616 D 7.3 1.6 0:00.44 spamassassin
4638 kliek-au 17 0 27244 1792 572 R 2.0 0.3 0:08.93 clamscan
6484 lim-lind 16 0 14908 9084 1680 D 1.7 1.8 0:00.69 spamassassin
5583 root 16 0 2580 1232 788 R 0.7 0.2 0:06.39 top
6414 nyl-flip 16 0 17628 12m 1728 D 0.7 2.4 0:00.70 spamassassin
6514 am-ambro 16 0 18284 10m 1752 D 0.7 2.0 0:00.76 spamassassin
6759 nyl-bouw 16 0 10208 8516 1616 D 0.7 1.7 0:00.41 spamassassin
6767 djd-joha 16 0 10204 8512 1616 D 0.3 1.6 0:00.40 spamassassin
If i reboot the server with

shutdown -r now

It take 30-40 minutes ands some services like MYSQL fail to stop, eventually it seem to reboot.
If i stop postfix, no change.
What can I test for, how can I seek the problem.

I had this issue with an smaller server (mandriva+mailbox), replaced it eventually with (ubuntu 7.04+maildir), but I cant keep on re-building and replacing servers, instead of repairing some missbehaving part.

[till]

1) Do not write in big red letters. This might lead the one or other user to not answer to your post. We are all able to read the normal small black letters

2) The problem is that your server receives to much mail for the currently installed spam scanning and antivirus solution with clamscan and spamassassin. Check your mail log if you get higher email traffic, e.g. a spam.

[Morons]

I have done an

shutdown -r now

and let it do its thing - analysing the top on an terminal and looking at the logfiles to see what processes are closing off, obvious that the once the process causing the issue shuts down the re-boot will speed-up.
All the services stopped, took loooong time from 15:13 to last entry in syslog at 16:01, almost an hour (48 minutes!) the last standing processes was called from ISPConfig clamscan and authdaemond All the time the HDD seem to run non-stop!

[Morons]

Originally Posted by till

1) Do not write in big red letters. This might lead the one or other user to not answer to your post. We are all able to read the normal small black letters

2) The problem is that your server receives to much mail for the currently installed spam scanning and antivirus solution with clamscan and spamassassin. Check your mail log if you get higher email traffic, e.g. a spam.

• Oops big letters came as I did not preview, ancase fixed.
• I have received the same mail 3 x during the day, normally indicating pop3 disconnectin and huge mail on bad lines - however this case i have <1ms latency to server on same LAN segment. Thus I was thinking an re-lapse in the bug of clam were it re-serviced already scanned mails over and over again!
• This last scenario seem more likely as It takes AGES to work through the mail Q,
• Yes it could be the case - what is the solution? this has not happened before on this site as I can indicate with Stats this is an Small site! Normal last week this time's figures.

Grand Totals
------------
messages

2619 received
2787 delivered
67 forwarded
4 deferred (29 deferrals)
2 bounced
2268 rejected (44%)
0 reject warnings
0 held
0 discarded (0%)

237532k bytes received
259623k bytes delivered
838 senders
449 sending hosts/domains
210 recipients
36 recipient hosts/domains

Today's figures (early edition 2/3 day)
Grand Totals
------------
messages

1443 received
1501 delivered
26 forwarded
7 deferred (7 deferrals)
39 bounced
704 rejected (31%)
0 reject warnings
0 held
0 discarded (0%)

146215k bytes received
133408k bytes delivered
508 senders
273 sending hosts/domains
191 recipients
42 recipient hosts/domains

I am convinced that Sumething "Broke" either an update came in or some other misshap. as the traffic to the server seem to be normal etc.

[Morons]

I left the server to work off the load, as another server of mine seem to have had the same issue was re-booted and that sorted it. This is for sure and spam attack of some sort! it kills clamscan and spamassassin, eventually mysql process gets killed as the server run out of memory! and all get grinded to an halt!
I still have no idea how to get rid of this, were to find the "queue to scan as i stop both postfix and ispconfig_server, clamscan keep on spawning scans. Were is that backlog, how do i remove the heaped-up mail to be scaned! It also seem to scan the same mail over and over, as i got some mail 4-5x already. DARn!

[till]

First, clamscan will stop spawning when you stop postfix. But it may take some time, mostly about 5 - 10 minutes.

The emails to be scanned are most likely in the mail queue. You can manage the mailqueue with the postqueue and postsuper commands.

You should try this to get down the load from clamav:

http://www.howtoforge.com/forums/showthread.php?t=12860

clamd is much faster then the current clamscan solution. Sinvce the clamav developers introduced clamav 0.90.x, the antivirus scanning is producing a much higher load. the problem seems to be known according to the clamav mailing lists and the clamav developers recommend to use clamd. There is also some patch which I hope will be integrated in some of the next clamav releases.

[Morons]

My Set-Up consist of: ubuntu 7.04 and ISPConfig Version: 2.2.13

apt-get install clamav-daemon

inside /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin
change the CLAMSCAN line to

CLAMSCAN=/usr/bin/clamdscan

inside /etc/clamav/clamd.conf check the line ScanMail

ScanMail true

inside /etc/clamav/freshclam.conf AND also in /home/admispconfig/ispconfig/tools/clamav/etc/freshclam.conf check NotifyClamd

NotifyClamd /etc/clamav/clamd.conf

Short and sweet!

[Morons]

It seems that clamscan could never eat the chunks of mail sent to it, the server become 100% unresponsive and login took hours. clientsatisfaction became Zero, while you see the faxes arriving from customers cancelling the service and shouting "nOOb" all over.
I Backed up the old server after I managed to quickly stop services postfix and ispconfig_server as well as killed all processes to do with clamscan (It dint go away after even 30 minutes as predicted by Till)
It took me exactly 3H15 minutes from the moment I rebooted on the CD to re-install the server till up and running with "Perfect ubuntu 7.04", ISPConfig and Backup restored / user passwd/shadow/group/gshadow files fixed.
However only to be confronted with Exactly the same problem! At that stage Till's feedback arrived and I was confused, he told me to edit the scritpt /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin and fix it to work with clamd/clamdscan. Well it took me to again untar the ISPConfig installation files and search through install_ispconfig directory and found the README for clamassassin, in there was an nice indication of what to do.
The lessons learned here is that it does not always help to use the "Microsoft Fix" i.e. - Reload, but rather seek the real issue, then start digging/googling.

[cjc81]

Hi,

Thanks for the great post. I followed the instructions, however, i can't seem to get clamdscan to work on Centos 4.4.

Apparently, after yum install, /usr/bin/clamdscan does not exist. I found that executable file in ISPCONFIG's directory, but when i use that path, clamdscan doesn't seem to get invoked.

Could you point me in the right direction?

[cjc81]

hi guys, here's what i found.

In Centos 4.x, there's no need to do an yum install of clamav-daemon.

clamd executable is available in
/home/admispconfig/ispconfig/tools/clamav/sbin/clamd

Therefore, following Morons's instructions above, we just need to do the follows:

inside /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin
change the CLAMSCAN line to

CLAMSCAN=/home/admispconfig/ispconfig/tools/clamav/bin/clamdscan

inside /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf
check the line ScanMail

ScanMail true

inside /home/admispconfig/ispconfig/tools/clamav/etc/freshclam.conf
check the line NotifyClamd

NotifyClamd /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf

Thanks again Moron =)