clamav-daemon keeps crashing

Discussion in 'Server Operation' started by anark10n, Aug 5, 2019.

Tags:
  1. anark10n

    anark10n Member

    Hey there
    So as the title says, clamav-daemon keeps crashing on my server. and when it crashes, ***UNCHECKED*** gets appended to the subject line in outgoing email messages. The logs don't report anything, the just have database checks up until the point of crashing.
    This is the most recent status on the daemon:
    Any ideas as to why this keeps happening, or what else to check.
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Try starting clamav-daemon to get error messages why it fails to start.
    Code:
    systemctl start clamav-daemon.service
     
  3. anark10n

    anark10n Member

    Hi, apologies for waiting so long, the issue isn't that it doesn't start, it's that it starts, and in roughly a month's time, it stops, and I'm unable to trace why. The daemon starts without an error code.; although, while running, there is this line in the status check:
    Code:
    Process: 22027 ExecStartPre=/bin/mkdir /run/clamav (code=exited, status=1/FAILURE)
    The part in brackets appears in red. Is this a problem?
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    If you do not care about this occasional crashing, restart clamav every week.
    If you want to find out why crashing happens, increase log level and wait for the crash to happen. That hopefully gives more info on what happens.
     
  5. anark10n

    anark10n Member

    Alright, I will keep monitoring and report back.
     
  6. anark10n

    anark10n Member

    Hello again, so I'm not sure i increased the appropriate log level or whether i'm looking at the appropriate logs, but there are still no errors in the logs for clamav-daemon. Are these the appropriate levels to catch any errors:
    Code:
    7   4   1   7
    
    I am seeing memory allocation errors when i checked the status after a recent crash.
    clamav-daemon status:
    Code:
    ● clamav-daemon.service - Clam AntiVirus userspace daemon
       Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/clamav-daemon.service.d
               └─extend.conf
       Active: failed (Result: signal) since Tue 2019-09-24 11:36:08 UTC; 1h 20min ago
         Docs: man:clamd(8)
               man:clamd.conf(5)
               https://www.clamav.net/documents/
      Process: 5016 ExecStart=/usr/sbin/clamd --foreground=true (code=killed, signal=KILL)
      Process: 5011 ExecStartPre=/bin/chown clamav /run/clamav (code=exited, status=0/SUCCESS)
      Process: 5008 ExecStartPre=/bin/mkdir /run/clamav (code=exited, status=1/FAILURE)
     Main PID: 5016 (code=killed, signal=KILL)
    
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Error: CRITICAL: fmap() failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: Tue Sep 24 11:12:04 2019 -> ~/var/lib/amavis/tmp/amavis-20190923T160521-02854-_8g63HPN/parts/p001: Can't allocate memory ERROR
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Error: CRITICAL: fmap() failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: Tue Sep 24 11:12:04 2019 -> ~/var/lib/amavis/tmp/amavis-20190923T181201-06827-x0LHl9d9/parts/p001: Can't allocate memory ERROR
    Sep 24 11:36:03 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed
    Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Main process exited, code=killed, status=9/KILL
    Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Unit entered failed state.
    Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Failed with result 'signal'.
    
     
  7. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    increase the memory?
     
  8. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Last edited: Sep 24, 2019
  9. anark10n

    anark10n Member

    free -h reports:
    Code:
                  total        used        free      shared  buff/cache   available
    Mem:           2.0G        1.5G         85M        164M        399M        182M
    Swap:            0B          0B          0B
    
    Definitely going to look into more RAM, but i'm unable to right now. I will look into adding a swap partition for now and see if that solves the issue
     
  10. brainsys

    brainsys New Member

    I have the same issue except its daily when Freshclam gets a new update. I believe it to be a lack of RAM but paradoxically increasing RAM is not the solution.

    I have five ISPConfig servers installed using the same Perfect Server tutorial so should be identical. The issue only occurs on two - a lightly loaded 2Gb system and a medium 4Gb. I shouldn't need 4Gb as the other servers are happy with 2Gb.

    What is characteristic of the two servers is that when running normally the free RAM is always significantly smaller than the other systems so, I guess, when update time occurs the clamav-daemon just runs out of RAM and is killed.

    I have to assume it was my user error when installing ISPConfig which makes these two servers misbehave. But I can't find what I might have done wrong.

    Current workaround is crontab restarting clamav-daemon daily at 13:10. (Freshclam appears to randomly update just after 12:00 or 13:00).

    Servers are Debian 10/Apache.
     
  11. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    What about swap? Is there swap and how much is it used?
    If you have 2 GB ram and 4 GB swap, it is 6 GB virtual memory space for the system.
     
  12. brainsys

    brainsys New Member

    Seemingly identical systems work flawlessly on 2GB, no swap. If I need more than 4GB something is very wrong but I can't put my finger on it.

    I have VMSTed TOPed etc but I can't find where the memory is leaking.
     
  13. Steini86

    Steini86 Active Member

    What are the logs telling you why clamd exits? It could be that freshclam needs some additional ram when doing the update. Since you don't have swap, the system kills the process it thinks it should to free memory.
     
  14. brainsys

    brainsys New Member

    code=killed, status=9/KILL

    But the point I'm making is that something is soaking up any memory available on two servers out of five.
    A 2GB & 4GB run out of memory for 2 x 2GB and a 1 x 4GB do not. So increasing RAM/swap isn't the solution. Finding and stopping the memory leak is. The Perfect Server setup for all five servers should be identical. The probable solution is I did something wrong on these two - but there is no obvious app swallowing more on the two servers - but the amount of Free memory under identical conditions is significantly less than the good servers so its disappering somewhere.

    ClamAV & Freshclam are playing the role of the canary in the coal mine methinks. Note that restarting clamav-daemon on fail works instantly.
     
    Last edited: Jan 30, 2020
  15. tfboy

    tfboy Member

    I've started having this problem too.
    Code:
    $ free -h
                  total        used        free      shared  buff/cache   available
    Mem:           3.8G        1.7G        347M        702M        1.8G        1.2G
    Swap:          511M        511M          0B
    Happens around once a month. Will dig a little more.
    Ubuntu 16.04.6 LTS
     
  16. nhybgtvfr

    nhybgtvfr Active Member

    this is probably more useful, if late, for brainsys. but still applicable to anyone else.
    the perfect sever install may well be identical. but the users files on each server aren't.
    if you have a webserver with postfix to send out website sourced mail, and no mailboxes, and a mailserver with no websites but lots of mailboxes, they're going to have very different load patterns and memory usage.
    and the amount of memory that clamav will need for itself when doing a full scan is going to be completely different on each server.
    add a mailserver experiencing a slightly higher load than normal due to user activity, a very recently updated virus library, and a full clamav scan all coinciding at the same time and say goodbye to all your free ram....

    also it's debatable if a swap partition is helpful, at least on a vps. you're trying to write memory contents to a disk partition that is itself a file on a disk partition. that could just be too slow, more processes get backed up waiting... more ram gets needed, more swap gets used, more slowness..... could be a vicious circle that just takes out your vps.


    @tfboy,
    when did you run free -h? if it's after the processes have already been killed it's too late, if it's while your system is working ok, it's too early.
    to properly identify any culprits, you'll need something that will regularly monitor your system and alert you before the critical point.
    then you may be able to see some useful information.
    cosider looking at munin, monit, Nagios, icinga and many others like those.
     
  17. Steini86

    Steini86 Active Member

    Well, your swap was full. So the system killed some process to free up ram (otherwise the system would have crashed). Swapping in general is bad for performance. Reduce your memory footprint (find out which program uses all your memory) or upgrade your server with more memory.
    If you don't care about performance, just increase your swap.
     
  18. tfboy

    tfboy Member

    I'm not too familiar with the swap management but as I had 1.2G of RAM free, should this be an issue? I can force a purge of the swap by turning off and back on again, but should I have to do that?
     
  19. Steini86

    Steini86 Active Member

    At the time you made the "free" command, you had 1.2G RAM free. However, swap is only used when necessary (when RAM is full). When your swap is full too, your system kills applications to free some RAM. Usually, swap is not moved back to RAM if not needed. So your swap being full suggests, that in the past all your RAM+SWAP was completely used and your system hat to kill applications to keep the system running. This freed up some ram, but killed your clamav.
    So the question is how to prevent this in the future:
    1) Install more ram
    2) Decrease ram needed by applications (you need to investigate which applications used up your ram and why. Maybe you can decrease the needed memory or have a memory leak..)
    3) Increase swap. If swap does not get full, no applications will be killed. However, your system performance will degrade
     
  20. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    If the problem is running out of memory. Purging swap does not help.
    You wrote
    , but that was when you ran free -h. Previously the host may have had memory completely used and so swap was used to help that.
     

Share This Page