Massive memory usage

[voltron81]

Hi to everybody.
I'm running 2 mailservers with ISPConfig 3.0.1.6, master-master replicated (master-master replication of mysql and /var/vmail with glusterfs).
Yesterday one of the server crashed, without giving me even the possibility to enter in ssh. Luckly my server farm gave me the opportunity to enter with serial, and I saw some segmentation fault for clamd, but I was able to see just that error, because the server was not responding to commands.
I'm checking the memory to see if it was an hardware fault.
The other server is up and running. It is less powerfull that the other one (just 1 gb of ram). I can see that I've a massive usage of memory... and I'm just scare that probably the other server crashed for the same reason...
From the server that is now running, I've this usage of memory:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
2578 clamav    20   0  295m 168m 5596 S    0 17.3   0:16.85 clamd
31677 amavis    20   0  202m  85m 4244 S    0  8.8   0:04.68 amavisd-new
31743 amavis    20   0  200m  83m 4296 S    0  8.6   0:04.34 amavisd-new
 2144 amavis    20   0  196m  79m 2736 S    0  8.1   0:01.64 amavisd-new
 2204 mysql     20   0  230m  45m 6104 S    0  4.7   1:12.65 mysqld
 2272 root      20   0  103m  42m 2664 S    0  4.4   0:06.72 spamd
 2576 root      20   0  103m  40m  588 S    0  4.2   0:00.02 spamd
 2577 root      20   0  103m  40m  588 S    0  4.2   0:00.04 spamd
 9667 www-data  20   0  239m  20m 4800 S    0  2.2   0:00.48 apache2
26938 www-data  20   0  239m  20m 4788 S    0  2.1   0:00.30 apache2
27851 www-data  20   0  238m  20m 4816 S    0  2.1   0:01.06 apache2
 9665 www-data  20   0  236m  20m 4316 S    0  2.1   0:00.46 apache2
 9664 www-data  20   0  236m  19m 4272 S    0  2.0   0:00.26 apache2
27855 www-data  20   0  236m  19m 4252 S    0  2.0   0:00.26 apache2
27852 www-data  20   0  234m  17m 4072 S    0  1.8   0:00.24 apache2
 4339 root      20   0  227m  15m 7696 S    0  1.6   0:01.12 apache2
27854 www-data  20   0  228m  12m 3640 S    0  1.3   0:00.14 apache2
 9663 www-data  20   0  227m  10m 2500 S    0  1.1   0:00.06 apache2
31093 www-data  20   0  227m 9476 1444 S    0  1.0   0:00.00 apache2
 1825 root      20   0  179m 9236 1232 S    0  0.9  16:11.01 glusterfs
 9662 www-data  20   0  155m 7452  456 S    0  0.7   0:00.08 apache2
 9661 root      20   0 39092 6892 2740 S    0  0.7   0:00.20 vlogger
 3007 root      20   0 73356 6224 1984 S    0  0.6   0:10.83 fail2ban-server
31613 postfix   20   0  103m 6024 4468 S    0  0.6   0:00.16 smtpd
31598 postfix   20   0  103m 5816 4292 S    0  0.6   0:00.12 smtpd
31629 postfix   20   0  103m 5812 4296 S    0  0.6   0:00.12 smtpd
 2749 root      20   0  184m 4584 1084 S    0  0.5  24:50.60 glusterfsd
 2684 clamav    20   0 42776 3884 2644 S    0  0.4   0:00.74 freshclam
31910 postfix   20   0 56504 3456 2672 S    0  0.3   0:00.02 cleanup
31904 postfix   20   0 52268 3224 2432 S    0  0.3   0:00.06 proxymap
31903 postfix   20   0 52152 3204 2460 S    0  0.3   0:00.02 trivial-rewrite
 2969 postfix   20   0 41404 3180 2248 S    0  0.3   0:00.12 tlsmgr
 3183 root      20   0 66072 3144 2484 S    0  0.3   0:01.94 sshd
31253 root      20   0 66068 3108 2468 S    0  0.3   0:00.32 sshd
31915 postfix   20   0 56408 3044 2332 S    0  0.3   0:00.02 smtp
 2871 postfix   20   0 52056 2948 2272 S    0  0.3   0:00.92 qmgr
31751 root      20   0 10828 2416 1052 T    0  0.2   0:00.00 nano
31767 root      20   0 10828 2396 1036 T    0  0.2   0:00.00 nano
 2844 root      20   0 36840 2304 1804 S    1  0.2   0:06.36 master
 2970 postfix   20   0 38900 2288 1772 S    0  0.2   0:01.00 anvil
31679 postfix   20   0 38900 2272 1788 S    0  0.2   0:00.00 pickup
31999 postfix   20   0 38900 2264 1772 S    0  0.2   0:00.00 showq
 3112 vmail     20   0 22920 2060 1316 S    0  0.2   0:18.38 imapd

As you can see, clamd is taking 17% of memory, even without usage of CPU.
also Amavis is using quite a lot of memory. Is it normal that amavis is using 3 daemons?
I saw before that the free memory is jumping from 10MB to 300MB...

Is it normal?

Michele

[damir]

It's a little higher than mines. Do you have any heavy rules or lists that you have added lately?

[voltron81]

Originally Posted by damir

It's a little higher than mines. Do you have any heavy rules or lists that you have added lately?

No, nothing

[damir]

Maybe your traffic has increased, that can affect load. Have you seen any spikes in your graphs? Anything strange in log files?

[voltron81]

Well, the traffic is increased a lot because before all the traffic was shared between the 2 servers... yesterday one of the servers went down and now all the emails are managed from just one server.
Anyway still sound strange to me that clamd is reserving all this memory...

thanks
Michele

[till]

In general, 1 GB of ram is very low for todays hardware. Especiall when you take in account that a GB of ram costs just a few euros. You should increase ram to 2 or 4 GB on that server.

You should stop spamd daemon. Spamd is not used by ispconfig as the mail scanning is done with amavisd.

Is it normal that amavis is using 3 daemons?

Yes. You can set the number of daemons in the amavisd configuration file.

[damir]

From my own experiences adding more memory will help the increased traffic. Monitor your traffic and try to fine tune your filter and rules.

[voltron81]

Originally Posted by till

In general, 1 GB of ram is very low for todays hardware. Especiall when you take in account that a GB of ram costs just a few euros. You should increase ram to 2 or 4 GB on that server.

It was a kind of backup server... the other one, that now is crashed and it's under investigation, has got 2 gb.

You should stop spamd daemon. Spamd is not used by ispconfig as the mail scanning is done with amavisd.

That is strange, because I was following the perfect server debian how-to... so I don't know then why spamd is up and running.


Thanks
Michele

[damir]

Originally Posted by voltron81

That is strange, because I was following the perfect server debian how-to... so I don't know then why spamd is up and running.

In tutorial there is line with following packages

apt-get install amavisd-new spamassassin clamav clamav-daemon zoo unzip bzip2 arj nomarch lzop cabextract apt-listchanges libnet-ldap-perl libauthen-sasl-perl clamav-docs daemon libio-string-perl libio-socket-ssl-perl libnet-ident-perl zip libnet-dns-perl

As you see spamassassin is installed and run script starts spamd daemon.

[voltron81]

So why it's in the how-to if at the end ispconfig is not using it?
Can I even remove it?

Anyway the other server is up and running again... and also him have a very strange usage of memory...
have a look here:

top - 13:20:17 up 32 min, 2 users, load average: 1.38, 1.32, 0.97
Tasks: 143 total, 2 running, 141 sleeping, 0 stopped, 0 zombie
Cpu(s): 28.5%us, 4.0%sy, 0.0%ni, 45.4%id, 19.7%wa, 0.7%hi, 1.7%si, 0.0%st
Mem: 4042296k total, 4002716k used, 39580k free, 840k buffers
Swap: 1959920k total, 608k used, 1959312k free, 3362320k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2548 clamav 20 0 231m 168m 5408 S 0 4.3 0:03.81 clamd
2222 amavis 20 0 205m 88m 4388 S 0 2.2 0:10.54 amavisd-new
2223 amavis 20 0 201m 84m 4332 S 0 2.1 0:08.30 amavisd-new
2106 amavis 20 0 196m 79m 2736 S 0 2.0 0:00.82 amavisd-new
2233 root 20 0 103m 42m 2664 S 0 1.1 0:01.18 spamd
2524 root 20 0 103m 40m 588 S 0 1.0 0:00.00 spamd
2527 root 20 0 103m 40m 588 S 0 1.0 0:00.00 spamd
2166 mysql 20 0 226m 29m 5732 S 0 0.8 0:03.84 mysqld
1784 root 20 0 182m 18m 1224 S 5 0.5 0:47.50 glusterfs
2962 root 20 0 225m 12m 6460 S 0 0.3 0:00.12 apache2
3745 www-data 20 0 227m 11m 3708 S 0 0.3 0:00.16 apache2
3747 www-data 20 0 227m 11m 3636 S 0 0.3 0:00.08 apache2
3746 www-data 20 0 227m 11m 3624 S 0 0.3 0:00.06 apache2
3748 www-data 20 0 226m 10m 3652 S 0 0.3 0:00.08 apache2
3042 www-data 20 0 226m 10m 3640 S 0 0.3 0:00.08 apache2
3041 www-data 20 0 226m 10m 3640 S 0 0.3 0:00.08 apache2
3040 www-data 20 0 226m 10m 2860 S 0 0.3 0:00.04 apache2
3043 www-data 20 0 226m 8992 2392 S 0 0.2 0:00.00 apache2
3319 www-data 20 0 225m 7584 1332 S 0 0.2 0:00.00 apache2
3811 www-data 20 0 225m 6876 684 S 0 0.2 0:00.00 apache2
2963 root 20 0 39092 6836 2736 S 0 0.2 0:00.08 vlogger
2975 postfix 20 0 104m 6088 4472 S 0 0.2 0:00.28 smtpd
3857 postfix 20 0 104m 5984 4472 S 0 0.1 0:00.08 smtpd
3051 postfix 20 0 103m 5848 4292 S 0 0.1 0:00.24 smtpd
3370 postfix 20 0 103m 5844 4292 S 0 0.1 0:00.20 smtpd
2981 root 20 0 68832 5604 1792 S 0 0.1 0:01.24 fail2ban-server
2720 root 20 0 184m 4900 1080 S 6 0.1 1:35.43 glusterfsd
3962 postfix 20 0 101m 4704 3636 S 0 0.1 0:00.02 smtpd
2977 www-data 20 0 151m 4032 468 S 0 0.1 0:00.00 apache2
2646 clamav 20 0 42776 3880 2644 S 0 0.1 0:00.32 freshclam
3958 postfix 20 0 56504 3444 2672 S 0 0.1 0:00.00 cleanup
3788 postfix 20 0 52252 3228 2432 S 0 0.1 0:00.14 proxymap
3942 postfix 20 0 52152 3200 2460 S 0 0.1 0:00.02 trivial-rewrite
2976 postfix 20 0 41404 3156 2232 S 0 0.1 0:00.00 tlsmgr
3945 root 20 0 66068 3104 2468 S 0 0.1 0:00.02 sshd
3071 root 20 0 66068 3100 2468 S 0 0.1 0:00.14 sshd
3963 postfix 20 0 56280 3100 2408 S 0 0.1 0:00.00 smtp
3959 postfix 20 0 56408 3048 2332 S 0 0.1 0:00.00 smtp
2845 postfix 20 0 52056 2932 2268 S 0 0.1 0:00.04 qmgr
2825 root 20 0 36840 2300 1804 S 0 0.1 0:00.26 master
3044 postfix 20 0 38900 2280 1772 S 0 0.1 0:00.02 anvil
3956 postfix 20 0 38900 2268 1772 S 0 0.1 0:00.00 showq
2844 postfix 20 0 38900 2200 1736 S 0 0.1 0:00.00 pickup
3075 root 20 0 18844 1968 1408 S 0 0.0 0:00.06 bash
3949 root 20 0 18836 1900 1352 S 0 0.0 0:00.00 bash
2758 nobody 20 0 26356 1768 776 S 0 0.0 0:00.02 mydns
3894 vmail 20 0 22668 1768 1316 S 0 0.0 0:00.04 imapd
3912 vmail 20 0 22672 1756 1312 S 0 0.0 0:00.04 imapd

Than few minutes later, the memory free was jumping to 1626024k free...

Thanks
Michele