CentOS 7.5 clamd consuming too much memory

Discussion in 'Installation/Configuration' started by kyferez, Aug 12, 2018.

  1. kyferez

    kyferez Member

    Using CentOS 7.4 perfect server guide. This issue with clamd was occurring before upgrade from 7.4 to 7.5. Have found that clamd is consuming memory under the amavis user.

    Currently memory consumption of clamd is 30% and has been bouncing up and down, but stays high rather than low. Server uptime is 4 days. I cannot run the server for more than 7 or 8 days without having out of memory issues in the logs, and the only item that seems to consistently consume excessive amounts of memory is clamd. If it starts having out of memory issues in the logs it will eventually go till it brings most of the server down.

    This system has 2GB ram, 2 cores, and SSD based storage - it's a VM on a 3rd party provider. CPU generally is not high utilization, though there are some spikes.

    I have found that if I kill the clamd process, then I free up enormous amounts of memory and can extend the server uptime. I would like to keep running clamd for mail scanning, but need to keep the server and email running stable. I have already modifying settings per this with little success in resolving this: https://www.centos.org/forums/viewtopic.php?t=45552

    Here's a little from my logs where I log top output with it organized per Mem:
    Code:
    Sun Aug 12 14:39:02 EDT 2018 top - 14:39:02 up 4 days,  4:32,  1 user,  load average: 0.76, 0.58, 0.48
    Sun Aug 12 14:39:02 EDT 2018 Tasks: 177 total,   5 running, 172 sleeping,   0 stopped,   0 zombie
    Sun Aug 12 14:39:02 EDT 2018 %Cpu(s): 23.1 us, 30.8 sy,  0.0 ni,  3.8 id, 19.2 wa,  0.0 hi,  0.0 si, 23.1 st
    Sun Aug 12 14:39:02 EDT 2018 KiB Mem :  1015296 total,   199156 free,   546440 used,   269700 buff/cache
    Sun Aug 12 14:39:02 EDT 2018 KiB Swap:  1048572 total,   267592 free,   780980 used.   260472 avail Mem
    Sun Aug 12 14:39:02 EDT 2018
    Sun Aug 12 14:39:02 EDT 2018 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    Sun Aug 12 14:39:02 EDT 2018 8031 amavis    20   0  742752 351664      0 S   0.0 34.6   0:00.00 clamd
    Sun Aug 12 14:39:02 EDT 2018 8041 web1      20   0  514352  67788  50052 S   0.0  6.7   0:01.20 php-cgi
    Sun Aug 12 14:39:02 EDT 2018 3138 web1      20   0  333096  20552   1648 S   0.0  2.0   0:01.70 otrs.Daemo+
    Sun Aug 12 14:39:02 EDT 2018 8108 root      20   0  490860  17120  10820 R  31.2  1.7   0:00.05 php
    Sun Aug 12 14:39:02 EDT 2018 8094 root      20   0  490860  17116  10820 R  37.5  1.7   0:00.06 php
    Sun Aug 12 14:39:02 EDT 2018 1351 mysql     20   0 1653960  15412   1808 S   0.0  1.5  19:58.02 mysqld
    Sun Aug 12 14:39:02 EDT 2018 4542 web1      20   0  324320   8788   1232 S   0.0  0.9   0:01.50 otrs.Daemo+
    Sun Aug 12 14:39:02 EDT 2018 4284 root      20   0  154680   6692   6556 S  12.5  0.7  19:45.12 systemd-jo+
    Sun Aug 12 14:39:02 EDT 2018 6165 web1      20   0  322032   6284   1004 S   0.0  0.6   0:01.42 otrs.Daemo+
    Sun Aug 12 14:39:02 EDT 2018 4550 web1      20   0  321904   5220    840 S   0.0  0.5   0:01.41 otrs.Daemo+
    
    Sun Aug 12 14:40:01 EDT 2018 top - 14:40:01 up 4 days,  4:33,  1 user,  load average: 0.56, 0.58, 0.49
    Sun Aug 12 14:40:01 EDT 2018 Tasks: 186 total,  10 running, 176 sleeping,   0 stopped,   0 zombie
    Sun Aug 12 14:40:01 EDT 2018 %Cpu(s): 33.3 us, 55.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 11.1 st
    Sun Aug 12 14:40:01 EDT 2018 KiB Mem :  1015296 total,    74120 free,   662512 used,   278664 buff/cache
    Sun Aug 12 14:40:01 EDT 2018 KiB Swap:  1048572 total,   273736 free,   774836 used.   141572 avail Mem
    Sun Aug 12 14:40:01 EDT 2018
    Sun Aug 12 14:40:01 EDT 2018 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    Sun Aug 12 14:40:01 EDT 2018 8031 amavis    20   0  742752 351664      0 S   0.0 34.6   0:00.00 clamd
    Sun Aug 12 14:40:01 EDT 2018 8041 web1      20   0  514664  68176  50056 S   0.0  6.7   0:01.42 php-cgi
    Sun Aug 12 14:40:01 EDT 2018 8227 web3      20   0  452416  45060  14696 S   0.0  4.4   0:00.53 php-cgi
    Sun Aug 12 14:40:01 EDT 2018 8222 web3      20   0  450520  41740  14772 S   0.0  4.1   0:01.05 php-cgi
    Sun Aug 12 14:40:01 EDT 2018 3138 web1      20   0  333100  20632   1652 S   0.0  2.0   0:01.77 otrs.Daemo+
    Sun Aug 12 14:40:01 EDT 2018 1351 mysql     20   0 1653960  18272   2480 S   0.0  1.8  19:58.25 mysqld
    Sun Aug 12 14:40:01 EDT 2018 4284 root      20   0  154680  11560  11420 R  35.3  1.1  19:45.42 systemd-jo+
    Sun Aug 12 14:40:01 EDT 2018 8308 root      20   0  340056  10180   6896 D  17.6  1.0   0:00.03 php
    Sun Aug 12 14:40:01 EDT 2018 4542 web1      20   0  324320   8788   1232 S   0.0  0.9   0:01.58 otrs.Daemo+
    Sun Aug 12 14:40:01 EDT 2018 6165 web1      20   0  322032   6696   1304 S   0.0  0.7   0:01.55 otrs.Daemo+
    
    Sun Aug 12 14:41:01 EDT 2018 top - 14:41:01 up 4 days,  4:34,  1 user,  load average: 0.50, 0.60, 0.50
    Sun Aug 12 14:41:01 EDT 2018 Tasks: 183 total,   4 running, 179 sleeping,   0 stopped,   0 zombie
    Sun Aug 12 14:41:01 EDT 2018 %Cpu(s): 24.1 us, 37.9 sy,  0.0 ni, 13.8 id, 13.8 wa,  0.0 hi,  0.0 si, 10.3 st
    Sun Aug 12 14:41:01 EDT 2018 KiB Mem :  1015296 total,    66032 free,   742304 used,   206960 buff/cache
    Sun Aug 12 14:41:01 EDT 2018 KiB Swap:  1048572 total,   249672 free,   798900 used.    64232 avail Mem
    Sun Aug 12 14:41:01 EDT 2018
    Sun Aug 12 14:41:01 EDT 2018 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    Sun Aug 12 14:41:01 EDT 2018 8031 amavis    20   0  742752 310084      0 S   0.0 30.5   0:00.00 clamd
    Sun Aug 12 14:41:01 EDT 2018 8227 web3      20   0  482044  65000   6856 S   0.0  6.4   0:01.83 php-cgi
    Sun Aug 12 14:41:01 EDT 2018 8222 web3      20   0  480472  63276   6848 S   0.0  6.2   0:02.44 php-cgi
    Sun Aug 12 14:41:01 EDT 2018 8041 web1      20   0  515040  62200  43812 S   0.0  6.1   0:01.73 php-cgi
    Sun Aug 12 14:41:01 EDT 2018 8569 web3      20   0  440084  32236   5124 S   0.0  3.2   0:00.61 php-cgi
    Sun Aug 12 14:41:01 EDT 2018 1351 mysql     20   0 1653960  31956   2388 S   0.0  3.1  19:58.76 mysqld
    Sun Aug 12 14:41:01 EDT 2018 3138 web1      20   0  333080  20608   1648 S   0.0  2.0   0:01.83 otrs.Daemo+
    Sun Aug 12 14:41:01 EDT 2018 8649 root      20   0  396084  11896   7948 R  25.0  1.2   0:00.04 php
    Sun Aug 12 14:41:01 EDT 2018 8639 root      20   0  391568  11716   7828 R  25.0  1.2   0:00.04 php
    Sun Aug 12 14:41:01 EDT 2018 4542 web1      20   0  324320   8768   1212 S   0.0  0.9   0:01.68 otrs.Daemo+
    
    I was thinking a script to run daily to restart amavsid. Looking for some expert guidance. What do you think? Recommendations?
     
    Last edited: Aug 12, 2018
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    You wrote
    But top output included shows the host has 1 GB.
    Have you followed swap usage? If also swap is full host really is out of memory. You can make larger swap, that should get rid of the out of memory condition.
    If 100% uptime is not required, create cron job to reboot nightly.
     
    ahrasis likes this.
  3. kyferez

    kyferez Member

    Taleman,

    I was mistaken, it indeed has 1GB. Nightly reboots are not an option due to the fact there is some sort of issue that causes the Kernel to crash at reboot about once out of 7 or 8 reboots; I have a thread about it here: https://www.centos.org/forums/viewtopic.php?f=47&t=67928

    I will look into swap, thanks.
     
  4. kyferez

    kyferez Member

    Well, looks like I'm using quite a bit of swap and that I'm hitting swap pretty regular. So def a mem issue. The concern though is if I add more memory, is that going to fix this or is clamd just going to consume more and more memory? It seems it's consuming more and more memory due to the fact that it has no issue for 5 days. I killed it yesterday and it has remained low so far. I'll wait a few more days and see if it starts creeping up again.

    How much memory does clamd consume on your system after running for 10+ days?

    Code:
    # free
                  total        used        free      shared  buff/cache   available
    Mem:        1015296      599536      124544       84568      291216      158764
    Swap:       1048572      987340       61232
    
    # vmstat 3 100
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  0 1011976 124084      0 364300   35   20   441   100   27   23  2  1 97  0  1
     7  3 1011976 111888      0 368712   21    0  1577     1  415  373  2  2 96  0  1
     0  0 1013056  75880      0 430732  128  359 173613  5453 1474 1296  5 11 75  6  4
     0  0 1013056 114588      0 391140  304    0   476   592  490  515  4  2 93  1  1
     0  0 1013056 116172      0 389556    0    0     0     0  192  167  0  0 100  0  0
     0  0 1013056 116172      0 389560    0    0     0     1  184  166  0  0 99  0  1
     0  0 1013056 116172      0 389560    0    0     0     0  198  177  0  0 100  0  0
     0  0 1013056 116048      0 389848    0    0    96     0  226  206  0  0 99  0  1
     0  0 1013056 115924      0 389848    0    0    37    70  222  205  0  0 100  0  0
     0  0 1013056 113692      0 391560    0    0   571     0  301  256  1  0 98  0  1
     0  0 1013056 113612      0 391560    0    0     5     0  260  234  1  0 99  0  0
     0  0 1013056 113596      0 391560    0    0     0     0  220  198  0  0 100  0  0
     0  0 1013056 113612      0 391508   21    0    21     5  234  205  0  0 100  0  0
     0  0 1013056 113484      0 391684    0    0     0   600  260  334  0  0 99  0  0
     0  0 1013056 113508      0 391684    0    0     0     0  237  213  0  0 99  0  0
     0  0 1013056 116452      0 391860    0    0    77     1  232  208  0  0 100  0  0
     0  0 1013056 116220      0 391908    0    0    52     0  345  346  2  0 98  0  0
     0  0 1013056 116220      0 391908    0    0     0     0  230  228  0  0 99  0  0
     0  0 1013056 116220      0 392060    0    0    56    17  234  212  0  0 100  0  0
     0  0 1013056 116220      0 392060    0    0     0     0  227  215  0  0 100  0  0
     0  0 1013056 114112      0 394232   11    0   735     4  429  433  3  1 96  0  0
     4  0 1013056  91112      0 406096    0    0  9997     1  962 1110  7  5 87  0  1
     0  0 1013056  98852      0 405964    0    0    24     1  371  452  1  1 98  0  0
     0  0 1013056  99056      0 405936    0    0     1   505  452  440  5  1 93  0  0
     0  0 1013056  89008      0 411888    0    0  1833     1  620  524  2  1 96  0  1
     0  0 1013056  81788      0 413192    0    0     0  1024  249  322  0  0 99  0  0
     0  0 1013056  85972      0 413248   27    0   197    82  332  287  1  1 98  0  0
     0  0 1012800  84868      0 413292   53    0   112    39  332  296  1  0 99  0  0
     0  0 1012800  87228      0 413072    0    0    75   189  263  243  0  0 99  0  0
     0  0 1012800  93912      0 412608    0    0     0     0  309  240  2  0 98  0  0
     0  0 1012800  93928      0 412604    0    0     0     0  280  271  1  0 99  0  0
     3  0 1012800  93804      0 412600   43    0    65     0  260  229  0  0 99  0  0
     0  0 1012800  89440      0 413024    0    0   116    33  368  300  0  0 99  0  1
     0  0 1012800  89068      0 413024   11    0    11     0  268  252  0  0 99  0  0
     0  0 1011264  94784      0 412588  137    0   137     0  271  230  1  1 98  0  0
     0  0 1011264  94548      0 412856    0    0    89     1  222  200  0  0 100  0  0
     0  0 1011264  94548      0 412856    0    0     0     0  252  231  1  0 99  0  0
     0  0 1011264  69516      0 424852   81    0  2195   287  638  647  3  1 94  1  1
     0  0 1011264  78312      0 380632   43    0  1643   387  582  498  4  2 93  0  1
     0  0 1011264  79256      0 379688    0    0     0     0  237  220  0  0 100  0  0
     0  0 1010496  81636      0 379532   52    0    56     0  467  445  3  1 95  0  0
     0  0 1010496  81292      0 379532    0    0     0     1  295  266  1  0 99  0  1
     0  0 1010496  94696      0 365820    0    0     5    65  751  895  7  5 87  0  1
    
     
  5. ahrasis

    ahrasis Well-Known Member

    I think it mainly depends on how high is your mail trafiic. If it is not, you can try reading and implement this idea.

    Also, I think for heavy mail user, having a mail service together with web and other services sometimes may not be a very good idea. You can try separating your mail server from the rest and set it up to be on its own. ISPConfig 3.1 is ever ready for multi server setup.

    Or you can use third party mail server too.
     
  6. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Memory used on my system.
    Testing on Debian GNU/Linux 9.5, ISPConfig 3.1.11. Host is virtual machine with 6 CPU and 12 G memory.
    Code:
    [email protected]:/var/www/clients# uptime
     12:49:05 up 7 days,  2:20,  1 user,  load average: 0,83, 1,11, 1,06
    [email protected]:/var/www/clients# free
                  total        used        free      shared  buff/cache   available
    Mem:       12307736     3643032     1856032      480232     6808672     7862356
    Swap:       2096124           0     2096124
    [email protected]:/var/www/clients#
    
    Host has somewhat slow disk system, I compensate by adding ample memory so
    most reads happen from cache.

    Code:
    [email protected]:/var/www/clients# ps -ef | egrep "(clam|STIME)"
    UID        PID  PPID  C STIME TTY          TIME CMD
    clamav    1022     1  0 elo08 ?        00:01:32 /usr/bin/freshclam -d --foreground=true
    clamav    1045     1  0 elo08 ?        00:06:57 /usr/sbin/clamd --foreground=true
    root     27318  1824  0 12:52
    
    [email protected]:/var/www/clients# top -p 1045 -p 1022  -n 2
    
    top - 13:10:06 up 7 days,  2:41,  1 user,  load average: 1,28, 0,97, 1,01
    Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
    %Cpu(s): 15,4 us,  2,7 sy,  0,0 ni, 75,8 id,  6,0 wa,  0,0 hi,  0,2 si,  0,0 st
    KiB Mem : 12307736 total,  1879116 free,  3638196 used,  6790424 buff/cache
    KiB Swap:  2096124 total,  2096124 free,        0 used.  7897552 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                              
     1045 clamav    20   0  980676 587524  32044 S   0,0  4,8   6:57.93 clamd                                                                
     1022 clamav    20   0  152876  26516  21672 S   0,0  0,2   1:32.20 freshclam  
    
     
    ahrasis likes this.
  7. kyferez

    kyferez Member

    It's not mail heavy, but there is semi-regular mail flowing, so I have setup Cron jobs to restart several services to see if that helps. If it does not, I will try the guide you provided ahrasis, thank you.

    That said, I still may have to bite the bullet and increase mem; I'm hesitant to because the cost doubles for the server. I'm curious on adding another server, however I'm not clear on best method or how it would avoid memory issues? Would I follow the perfect server guide, install all same services, and then what? Won't I see the same issue? I assume I can't disable httpd/php (the other most memory consuming processes) due to ISPConfig...? So where would I have any memory usage reduction?
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    You can disable httpd, but you will not be able to host websites on that server and not be able to access ISPConfig interface. E.g. when you have a multiserver setup, then the email node(s) normally don't run apache as the ISPConfig UI is on a separate server or on the web server.
     
  9. kyferez

    kyferez Member

    Thanks again Till.

    So I just noticed that clam started pegging my CPUs and the PID kept changing every few seconds, and my IO skyrocketed. This started about the same time as these logs, and I found past logs with the same CPU and IO spike and same logs, which was after restarting amavisd and httpd so I'll be increasing the swap and watching the logs for a few days again:
    Code:
    Aug 15 12:07:19 s02 clamd[6290]: Received 0 file descriptor(s) from systemd.
    Aug 15 12:07:19 s02 clamd[6290]: clamd daemon 0.100.1 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
    Aug 15 12:07:19 s02 clamd[6290]: Running as user amavis (UID 994, GID 991)
    Aug 15 12:07:19 s02 clamd[6290]: Log file size limited to 1048576 bytes.
    Aug 15 12:07:19 s02 clamd[6290]: Reading databases from /var/lib/clamav
    Aug 15 12:07:19 s02 clamd[6290]: Not loading PUA signatures.
    Aug 15 12:07:19 s02 clamd[6290]: Bytecode: Security mode set to "TrustSigned".
    Aug 15 12:07:32 s02 clamd[6290]: Loaded 6608882 signatures.
    Aug 15 12:07:33 s02 clamd[6290]: LOCAL: Unix socket file /var/run/clamd.amavisd/clamd.sock
    Aug 15 12:07:33 s02 clamd[6290]: LOCAL: Setting connection queue length to 200
    Aug 15 12:07:33 s02 clamd[6290]: daemonize() failed: Cannot allocate memory
    Aug 15 12:07:33 s02 clamd[6290]: Socket file removed.
    
     
  10. kyferez

    kyferez Member

    Looks like increasing the swap did the trick. It's running great with 9 days of uptime now. Thanks all!
     
    ahrasis likes this.

Share This Page