clamav-daemon keeps crashing

Discussion in 'Server Operation' started by anark10n, Aug 5, 2019.

Tags:
  1. anark10n

    anark10n Member

    Hey there
    So as the title says, clamav-daemon keeps crashing on my server. and when it crashes, ***UNCHECKED*** gets appended to the subject line in outgoing email messages. The logs don't report anything, the just have database checks up until the point of crashing.
    This is the most recent status on the daemon:
    Any ideas as to why this keeps happening, or what else to check.
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Try starting clamav-daemon to get error messages why it fails to start.
    Code:
    systemctl start clamav-daemon.service
     
  3. anark10n

    anark10n Member

    Hi, apologies for waiting so long, the issue isn't that it doesn't start, it's that it starts, and in roughly a month's time, it stops, and I'm unable to trace why. The daemon starts without an error code.; although, while running, there is this line in the status check:
    Code:
    Process: 22027 ExecStartPre=/bin/mkdir /run/clamav (code=exited, status=1/FAILURE)
    The part in brackets appears in red. Is this a problem?
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    If you do not care about this occasional crashing, restart clamav every week.
    If you want to find out why crashing happens, increase log level and wait for the crash to happen. That hopefully gives more info on what happens.
     
  5. anark10n

    anark10n Member

    Alright, I will keep monitoring and report back.
     
  6. anark10n

    anark10n Member

    Hello again, so I'm not sure i increased the appropriate log level or whether i'm looking at the appropriate logs, but there are still no errors in the logs for clamav-daemon. Are these the appropriate levels to catch any errors:
    Code:
    7   4   1   7
    
    I am seeing memory allocation errors when i checked the status after a recent crash.
    clamav-daemon status:
    Code:
    ● clamav-daemon.service - Clam AntiVirus userspace daemon
       Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/clamav-daemon.service.d
               └─extend.conf
       Active: failed (Result: signal) since Tue 2019-09-24 11:36:08 UTC; 1h 20min ago
         Docs: man:clamd(8)
               man:clamd.conf(5)
               https://www.clamav.net/documents/
      Process: 5016 ExecStart=/usr/sbin/clamd --foreground=true (code=killed, signal=KILL)
      Process: 5011 ExecStartPre=/bin/chown clamav /run/clamav (code=exited, status=0/SUCCESS)
      Process: 5008 ExecStartPre=/bin/mkdir /run/clamav (code=exited, status=1/FAILURE)
     Main PID: 5016 (code=killed, signal=KILL)
    
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Error: CRITICAL: fmap() failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: Tue Sep 24 11:12:04 2019 -> ~/var/lib/amavis/tmp/amavis-20190923T160521-02854-_8g63HPN/parts/p001: Can't allocate memory ERROR
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Error: CRITICAL: fmap() failed
    Sep 24 11:12:04 opensrvr01 clamd[5016]: Tue Sep 24 11:12:04 2019 -> ~/var/lib/amavis/tmp/amavis-20190923T181201-06827-x0LHl9d9/parts/p001: Can't allocate memory ERROR
    Sep 24 11:36:03 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed
    Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Main process exited, code=killed, status=9/KILL
    Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Unit entered failed state.
    Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Failed with result 'signal'.
    
     
  7. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    increase the memory?
     
  8. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Last edited: Sep 24, 2019
  9. anark10n

    anark10n Member

    free -h reports:
    Code:
                  total        used        free      shared  buff/cache   available
    Mem:           2.0G        1.5G         85M        164M        399M        182M
    Swap:            0B          0B          0B
    
    Definitely going to look into more RAM, but i'm unable to right now. I will look into adding a swap partition for now and see if that solves the issue
     
  10. brainsys

    brainsys New Member

    I have the same issue except its daily when Freshclam gets a new update. I believe it to be a lack of RAM but paradoxically increasing RAM is not the solution.

    I have five ISPConfig servers installed using the same Perfect Server tutorial so should be identical. The issue only occurs on two - a lightly loaded 2Gb system and a medium 4Gb. I shouldn't need 4Gb as the other servers are happy with 2Gb.

    What is characteristic of the two servers is that when running normally the free RAM is always significantly smaller than the other systems so, I guess, when update time occurs the clamav-daemon just runs out of RAM and is killed.

    I have to assume it was my user error when installing ISPConfig which makes these two servers misbehave. But I can't find what I might have done wrong.

    Current workaround is crontab restarting clamav-daemon daily at 13:10. (Freshclam appears to randomly update just after 12:00 or 13:00).

    Servers are Debian 10/Apache.
     
  11. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    What about swap? Is there swap and how much is it used?
    If you have 2 GB ram and 4 GB swap, it is 6 GB virtual memory space for the system.
     
  12. brainsys

    brainsys New Member

    Seemingly identical systems work flawlessly on 2GB, no swap. If I need more than 4GB something is very wrong but I can't put my finger on it.

    I have VMSTed TOPed etc but I can't find where the memory is leaking.
     
  13. Steini86

    Steini86 Active Member

    What are the logs telling you why clamd exits? It could be that freshclam needs some additional ram when doing the update. Since you don't have swap, the system kills the process it thinks it should to free memory.
     
  14. brainsys

    brainsys New Member

    code=killed, status=9/KILL

    But the point I'm making is that something is soaking up any memory available on two servers out of five.
    A 2GB & 4GB run out of memory for 2 x 2GB and a 1 x 4GB do not. So increasing RAM/swap isn't the solution. Finding and stopping the memory leak is. The Perfect Server setup for all five servers should be identical. The probable solution is I did something wrong on these two - but there is no obvious app swallowing more on the two servers - but the amount of Free memory under identical conditions is significantly less than the good servers so its disappering somewhere.

    ClamAV & Freshclam are playing the role of the canary in the coal mine methinks. Note that restarting clamav-daemon on fail works instantly.
     
    Last edited: Jan 30, 2020

Share This Page