Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > ISPConfig 3 > Installation/Configuration

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 13th October 2012, 00:33
Quasdunk Quasdunk is offline
Junior Member
 
Join Date: Sep 2012
Posts: 14
Thanks: 1
Thanked 0 Times in 0 Posts
Default ClamAV: Eating memory and stopping mail traffic

My server (ISPConfig 3, Ubuntu Server 12.04 LTS) has been running for nearly a month now, an I've encountered some issues:
From time to time the server would stop delivering emails. At first I did not suspect anything special, but it happened three times now, so I had a closer look at my logs. Here's what /var/log/mail.err says:

Quote:
Oct 12 10:24:53 j31329 postfix/smtpd[27680]: fatal: no SASL authentication mechanisms
Oct 12 10:25:21 j31329 dovecot: master: Error: service(pop3-login): Initial status notification not received in 30 seconds, killing the process
Oct 12 10:25:22 j31329 dovecot: master: Error: service(config): Process 27543 is ignoring idle SIGINT
Oct 12 10:25:22 j31329 dovecot: master: Error: service(pop3-login): child 27701 killed with signal 9
Oct 12 10:25:22 j31329 dovecot: master: Error: service(pop3-login): command startup failed, throttling
Oct 12 10:28:41 j31329 dovecot: master: Error: service(pop3-login): Initial status notification not received in 30 seconds, killing the process
Oct 12 10:28:47 j31329 dovecot: pop3-login: Fatal: Error reading configuration: Timeout reading config from /var/run/dovecot/config
Oct 12 10:28:48 j31329 dovecot: master: Error: service(pop3-login): command startup failed, throttling
Oct 12 10:29:58 j31329 dovecot: auth-worker: Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) - waiting for 1 seconds before retry
Oct 12 10:29:58 j31329 dovecot: auth-worker: Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) - waiting for 1 seconds before retry
Oct 12 10:29:59 j31329 dovecot: auth-worker: Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) - waiting for 5 seconds before retry
Oct 12 10:29:59 j31329 dovecot: auth-worker: Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) - waiting for 5 seconds before retry
So I got a little worried - killing processes and not reaching the DB is not what you want to see in a log. So I had a look at the /var/log/syslog and noticed, that it had something to do with the memory:
Quote:
...
Oct 12 10:24:51 j31329 kernel: 42101 total pagecache pages
Oct 12 10:24:51 j31329 kernel: 5807 pages in swap cache
Oct 12 10:24:51 j31329 kernel: Swap cache stats: add 270215, delete 264408, find 1647700/1659421
Oct 12 10:24:51 j31329 kernel: Free swap = 0kB
Oct 12 10:24:51 j31329 kernel: Total swap = 524284kB
Oct 12 10:24:51 j31329 kernel: 264176 pages RAM
....
Oct 12 10:24:57 j31329 kernel: Out of memory: Kill process 3439 (clamd) score 92 or sacrifice child
Oct 12 10:24:57 j31329 kernel: Killed process 3439 (clamd) total-vm:420464kB, anon-rss:20824kB, file-rss:0kB
Oct 12 10:25:44 j31329 kernel: sh invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
...
Oct 12 10:25:55 j31329 kernel: Out of memory: Kill process 25233 (amavisd (ch1-av) score 61 or sacrifice child
Oct 12 10:25:55 j31329 kernel: Killed process 25233 (amavisd (ch1-av) total-vm:295908kB, anon-rss:1060kB, file-rss:12kB
Oct 12 10:26:15 j31329 kernel: php-cgi invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
...
Oct 12 10:26:18 j31329 kernel: Out of memory: Kill process 2265 (amavisd (master) score 59 or sacrifice child
Oct 12 10:26:18 j31329 kernel: Killed process 27359 (amavisd (virgin) total-vm:216340kB, anon-rss:1084kB, file-rss:48kB
....
Oct 12 10:26:42 j31329 kernel: Out of memory: Kill process 2265 (amavisd (master) score 59 or sacrifice child
Oct 12 10:26:42 j31329 kernel: Killed process 2265 (amavisd (master) total-vm:215052kB, anon-rss:764kB, file-rss:0kB
...
Oct 12 10:27:19 j31329 kernel: Out of memory: Kill process 2138 (mysqld) score 53 or sacrifice child
Oct 12 10:27:19 j31329 kernel: Killed process 2138 (mysqld) total-vm:1359928kB, anon-rss:6228kB, file-rss:0kB
Oct 12 10:27:37 j31329 kernel: init: mysql main process (2138) killed by KILL signal
Oct 12 10:27:37 j31329 kernel: init: mysql main process ended, respawning
...
Oct 12 10:28:47 j31329 kernel: Out of memory: Kill process 27561 (php-cgi) score 42 or sacrifice child
Oct 12 10:28:47 j31329 kernel: Killed process 27561 (php-cgi) total-vm:340900kB, anon-rss:29380kB, file-rss:10188kB
...
So, as far as I understand, the ClamAV processes require too much memory. The server begins to swap the leaking memory and as soon as it runs out of space it starts killing the most "greedy" processes. Among ClamAV that also includes mysql - so my mailserver (dovecot) stops working and queues everything until I restart the server. But even after a restart it doesn't take a minute to fill 850-900MB of memory - what the...?
I have 1GB + 512MB swap, which actually should be quite decent for a web-application with ~700-1000 visitors and 70-100 emails per day - shouldn't it? The CPU rarely touches the 10%-mark.

Running ps aux revealed the following (noteworthy) results:

Quote:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 2130 0.6 3.7 1359812 38324 ? Ssl Oct12 2:20 /usr/sbin/mysqld
amavis 2261 0.0 2.9 215028 30228 ? Ss Oct12 0:01 amavisd (master)
amavis 2830 0.0 6.4 295956 65260 ? S Oct12 0:04 amavisd (ch7-avail)
amavis 2842 0.0 6.5 296640 66612 ? S Oct12 0:05 amavisd (ch7-avail)
clamav 3420 0.0 14.0 412236 143408 ? Ssl Oct12 0:09 /usr/sbin/clamd
clamav 3527 0.0 0.1 44012 1652 ? Ss Oct12 0:12 /usr/bin/freshclam -d --quiet
root 11024 0.0 1.8 386024 18688 ? Ss Oct12 0:02 /usr/sbin/apache2 -k start
www-data 15969 0.0 1.4 387616 14680 ? S 00:01 0:00 /usr/sbin/apache2 -k start
www-data 15991 0.0 1.4 387576 14704 ? S 00:02 0:00 /usr/sbin/apache2 -k start
www-data 16119 0.0 1.3 387468 14076 ? S 00:06 0:00 /usr/sbin/apache2 -k start
web5 16148 0.4 2.8 302072 29380 ? S 00:07 0:02 /usr/bin/php-cgi -d open_basedir=...
web5 16149 0.3 3.1 304152 32220 ? S 00:07 0:01 /usr/bin/php-cgi -d open_basedir=...
www-data 16328 0.0 1.3 387080 13236 ? S 00:13 0:00 /usr/sbin/apache2 -k start
www-data 16329 0.0 1.3 387596 13956 ? S 00:13 0:00 /usr/sbin/apache2 -k start
www-data 16330 0.0 1.3 387488 13748 ? S 00:13 0:00 /usr/sbin/apache2 -k start
www-data 16369 0.0 1.3 387632 13984 ? S 00:14 0:00 /usr/sbin/apache2 -k start
www-data 16370 0.0 1.3 387588 13724 ? S 00:14 0:00 /usr/sbin/apache2 -k start
So the amavis-processes are consuming abot 30% of my memory, which seem way too much.

Is that normal?
Is it required for ISPConfig to work (I couldn't find an option to switch it off...)
If it is, would it be a bad idea to disable/uninstall ClamAV? In fact, I don't quite see the point of an anti-virus-application on a web-server...
Or is the only solution to increase the RAM?
I did some research and it seems I'm not the only one with these observations/complaints - is there an alternative to ClamAV?

Thanks in advance!

Last edited by Quasdunk; 13th October 2012 at 00:50.
Reply With Quote
Sponsored Links
  #2  
Old 13th October 2012, 20:04
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 36,405
Thanks: 834
Thanked 5,496 Times in 4,326 Posts
Default

Your memory is low for a full hosting system, you should eiher increase the ram or extend the swap to 2gb.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #3  
Old 15th October 2012, 04:36
concept21 concept21 is offline
Senior Member
 
Join Date: Dec 2011
Posts: 155
Thanks: 28
Thanked 22 Times in 16 Posts
Talking

Yes, it is.

ClamAV uses 150MB in order to load itself.
Reply With Quote
  #4  
Old 15th October 2012, 04:44
concept21 concept21 is offline
Senior Member
 
Join Date: Dec 2011
Posts: 155
Thanks: 28
Thanked 22 Times in 16 Posts
Default

You should also fine tune apache2 and mysql memory usage.

Search the web info.


My ispconfig 3 can run at full load at about 500MB ram out of 1GB ram, and 1GB swap, without clamav loaded.
Reply With Quote
  #5  
Old 13th December 2012, 16:39
glenstewart glenstewart is offline
Junior Member
 
Join Date: Dec 2012
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
 
Default The fun realities of ClamAV

This chart should bring some realities home.



This particular server has 4GB RAM, and is losing almost 10% of that to ClamAV. ClamAV is a great tool - perhaps still the best on Linux, but this is simply the price you have to pay to run it.

4GB RAM would be the very least I'd put up with to run an internet-facing email server these days.

Remember, Just because you can doesn't mean you should.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Virtual Users And Domains With Postfix, Courier, MySQL And SquirrelMail -Ubuntu 8.04 c4rdinal HOWTO-Related Questions 112 23rd August 2011 10:49
ClamAV and out of memory jaripetteri Installation/Configuration 4 7th December 2010 19:28


All times are GMT +2. The time now is 06:47.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.