Slow response

Discussion in 'Installation/Configuration' started by Morons, May 22, 2007.

  1. Morons

    Morons New Member

    HI,
    I am without resolve on this one, out of the blue, one of my servers thats been running for weeks now, suddenly become almost impossible to log into. the login process from the consile / ssh and even all ports 110 ans 25 seem tot timeout to a couple minutes!!
    My initial thoughts was that his server run out of some resource or another. however I can not detect the resource nor the process in error.

    By using top i can indicate an running system - althow slow it seem to function

    top - 14:25:40 up 26 min, 2 users, load average: 115.32, 103.99, 69.81
    Tasks: 307 total, 2 running, 289 sleeping, 0 stopped, 16 zombie
    Cpu(s): 20.9%us, 1.3%sy, 0.0%ni, 0.0%id, 77.2%wa, 0.3%hi, 0.3%si, 0.0%st
    Mem: 515896k total, 510960k used, 4936k free, 1416k buffers
    Swap: 1510068k total, 1478364k used, 31704k free, 13020k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    5264 nyl-cass 17 0 27508 2084 584 D 8.6 0.4 0:13.59 clamscan
    6712 top-henn 17 0 10192 8512 1616 D 7.3 1.6 0:00.44 spamassassin
    4638 kliek-au 17 0 27244 1792 572 R 2.0 0.3 0:08.93 clamscan
    6484 lim-lind 16 0 14908 9084 1680 D 1.7 1.8 0:00.69 spamassassin
    5583 root 16 0 2580 1232 788 R 0.7 0.2 0:06.39 top
    6414 nyl-flip 16 0 17628 12m 1728 D 0.7 2.4 0:00.70 spamassassin
    6514 am-ambro 16 0 18284 10m 1752 D 0.7 2.0 0:00.76 spamassassin
    6759 nyl-bouw 16 0 10208 8516 1616 D 0.7 1.7 0:00.41 spamassassin
    6767 djd-joha 16 0 10204 8512 1616 D 0.3 1.6 0:00.40 spamassassin

    If i reboot the server with
    Code:
    shutdown -r now
    It take 30-40 minutes ands some services like MYSQL fail to stop, eventually it seem to reboot.
    If i stop postfix, no change.
    What can I test for, how can I seek the problem.

    I had this issue with an smaller server (mandriva+mailbox), replaced it eventually with (ubuntu 7.04+maildir), but I cant keep on re-building and replacing servers, instead of repairing some missbehaving part.:cool:
     
    Last edited: May 22, 2007
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    1) Do not write in big red letters. This might lead the one or other user to not answer to your post. We are all able to read the normal small black letters ;)

    2) The problem is that your server receives to much mail for the currently installed spam scanning and antivirus solution with clamscan and spamassassin. Check your mail log if you get higher email traffic, e.g. a spam.
     
  3. Morons

    Morons New Member

    I have done an
    Code:
    shutdown -r now
    and let it do its thing - analysing the top on an terminal and looking at the logfiles to see what processes are closing off, obvious that the once the process causing the issue shuts down the re-boot will speed-up.
    All the services stopped, took loooong time from 15:13 to last entry in syslog at 16:01, almost an hour (48 minutes!) the last standing processes was called from ISPConfig clamscan and authdaemond All the time the HDD seem to run non-stop!
     
  4. Morons

    Morons New Member

    Spam

    • Oops big letters came as I did not preview, ancase fixed.
    • I have received the same mail 3 x during the day, normally indicating pop3 disconnectin and huge mail on bad lines - however this case i have <1ms latency to server on same LAN segment. Thus I was thinking an re-lapse in the bug of clam were it re-serviced already scanned mails over and over again!
    • This last scenario seem more likely as It takes AGES to work through the mail Q,
    • Yes it could be the case - what is the solution? this has not happened before on this site as I can indicate with Stats this is an Small site! Normal last week this time's figures.
    Grand Totals
    ------------
    messages

    2619 received
    2787 delivered
    67 forwarded
    4 deferred (29 deferrals)
    2 bounced
    2268 rejected (44%)
    0 reject warnings
    0 held
    0 discarded (0%)

    237532k bytes received
    259623k bytes delivered
    838 senders
    449 sending hosts/domains
    210 recipients
    36 recipient hosts/domains

    Today's figures (early edition 2/3 day)
    Grand Totals
    ------------
    messages

    1443 received
    1501 delivered
    26 forwarded
    7 deferred (7 deferrals)
    39 bounced
    704 rejected (31%)
    0 reject warnings
    0 held
    0 discarded (0%)

    146215k bytes received
    133408k bytes delivered
    508 senders
    273 sending hosts/domains
    191 recipients
    42 recipient hosts/domains


    I am convinced that Sumething "Broke" either an update came in or some other misshap. as the traffic to the server seem to be normal etc.
     
    Last edited: May 22, 2007
  5. Morons

    Morons New Member

    I left the server to work off the load, as another server of mine seem to have had the same issue was re-booted and that sorted it. This is for sure and spam attack of some sort! it kills clamscan and spamassassin, eventually mysql process gets killed as the server run out of memory! and all get grinded to an halt!
    I still have no idea how to get rid of this, were to find the "queue to scan as i stop both postfix and ispconfig_server, clamscan keep on spawning scans. Were is that backlog, how do i remove the heaped-up mail to be scaned! It also seem to scan the same mail over and over, as i got some mail 4-5x already. DARn!
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    First, clamscan will stop spawning when you stop postfix. But it may take some time, mostly about 5 - 10 minutes.

    The emails to be scanned are most likely in the mail queue. You can manage the mailqueue with the postqueue and postsuper commands.

    You should try this to get down the load from clamav:

    http://www.howtoforge.com/forums/showthread.php?t=12860

    clamd is much faster then the current clamscan solution. Sinvce the clamav developers introduced clamav 0.90.x, the antivirus scanning is producing a much higher load. the problem seems to be known according to the clamav mailing lists and the clamav developers recommend to use clamd. There is also some patch which I hope will be integrated in some of the next clamav releases.
     
  7. Morons

    Morons New Member

    Short HOWTO switching to clamd/clamdscan in ISPConfig

    My Set-Up consist of: ubuntu 7.04 and ISPConfig Version: 2.2.13
    Code:
    apt-get install clamav-daemon
    inside /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin
    change the CLAMSCAN line to
    Code:
    CLAMSCAN=/usr/bin/clamdscan
    inside /etc/clamav/clamd.conf check the line ScanMail
    Code:
    ScanMail true
    inside /etc/clamav/freshclam.conf AND also in /home/admispconfig/ispconfig/tools/clamav/etc/freshclam.conf check NotifyClamd
    Code:
    NotifyClamd /etc/clamav/clamd.conf
    Short and sweet! :cool:
     
  8. Morons

    Morons New Member

    The Fix/solution

    It seems that clamscan could never eat the chunks of mail sent to it, the server become 100% unresponsive and login took hours. clientsatisfaction became Zero, while you see the faxes arriving from customers cancelling the service and shouting "nOOb" all over.
    I Backed up the old server after I managed to quickly stop services postfix and ispconfig_server as well as killed all processes to do with clamscan (It dint go away after even 30 minutes as predicted by Till)
    It took me exactly 3H15 minutes from the moment I rebooted on the CD to re-install the server till up and running with "Perfect ubuntu 7.04", ISPConfig and Backup restored / user passwd/shadow/group/gshadow files fixed.
    However only to be confronted with Exactly the same problem! At that stage Till's feedback arrived and I was confused, he told me to edit the scritpt /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin and fix it to work with clamd/clamdscan. Well it took me to again untar the ISPConfig installation files and search through install_ispconfig directory and found the README for clamassassin, in there was an nice indication of what to do.
    The lessons learned here is that it does not always help to use the "Microsoft Fix" i.e. - Reload, but rather seek the real issue, then start digging/googling.
     
  9. cjc81

    cjc81 New Member

    Hi,

    Thanks for the great post. I followed the instructions, however, i can't seem to get clamdscan to work on Centos 4.4.

    Apparently, after yum install, /usr/bin/clamdscan does not exist. I found that executable file in ISPCONFIG's directory, but when i use that path, clamdscan doesn't seem to get invoked.

    Could you point me in the right direction?
     
  10. cjc81

    cjc81 New Member

    Switching to CLAMD on Centos 4.x with Ispconfig 2.2.13

    hi guys, here's what i found.

    In Centos 4.x, there's no need to do an yum install of clamav-daemon.

    clamd executable is available in
    /home/admispconfig/ispconfig/tools/clamav/sbin/clamd

    Therefore, following Morons's instructions above, we just need to do the follows:

    inside /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin
    change the CLAMSCAN line to
    Code:
    CLAMSCAN=/home/admispconfig/ispconfig/tools/clamav/bin/clamdscan
    inside /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf
    check the line ScanMail
    Code:
    ScanMail true
    inside /home/admispconfig/ispconfig/tools/clamav/etc/freshclam.conf
    check the line NotifyClamd
    Code:
    NotifyClamd /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf
    Thanks again Moron =)
     
    Last edited: May 31, 2007
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Have you tested your solution with the eicar virus test pattern and does it really filter viruses? You need a running clamd daemon for this and I do not see where you configuare and start clamd automatically as ISPConfig does not ahs a clamd daemon.
     
  12. Morons

    Morons New Member

    Carefull

    cjc81, Althow there is an clamd script installed with the ispconfig (comes with clamassasin) it require and clamd daemon to be running.
    Code:
    updatedb
    locate clamd
    ps ax |grep clamd
    check whether the daemon is running, as till sais use eicar test!
    If not running switch on the yum installed clamd daemon and change the config to use it according to the locate you did above.
     
  13. Morons

    Morons New Member

    I would like to ask ISPConfig to perhaps consider using clamd instead of clamscan by default. I only relised now that I had to upgrade hardware on 2 other installs before this that would have been unnessesary if we switched!
    Maybe add an option to switch the two methods.
     
  14. cjc81

    cjc81 New Member

    Yes sir, i tried the test pattern and the mail got filtered out. (ie. its working properly)

    Also, clamd daemon is running. Do i need to add that to chkconfig to make sure that clamd daemon is running when system starts?

    I am also monitoring my running processes via "top". I only see clamd coming active there are mail activities.
     
  15. Morons

    Morons New Member

    Yes this is were the issue comes in. Currently you have 2 installations of clamd, one installev via ISPC / clamassasin part and one using yum, Also 2 x freshclam. each updating its database residing in diferent places.
    Now, that being the case the versions might be diffrent! also make sure the you inspect the headers of an mail comming through to see the scan successfull! I have this in my header. The No mean there is no virus found!
    Code:
    X-Virus-Status: No
    X-Virus-Checker-Version: clamassassin 1.2.4 with clamdscan / ClamAV 0.90.2/3334/Thu May 31 10:54:25 2007
    Not all distro's have the chkconfig command, it seems to be maintained by the "redhat" derivatives and yes to make sure the yum installed version is running use
    Code:
    chkconfig clamd on
    The ISPC version will not start automatically, if my analysis is correct. Till can perhaps comment on this.
     
  16. cjc81

    cjc81 New Member

    I was wondering if i could copy /home/admispconfig/ispconfig/tools/clamav/sbin/clamd
    to
    /etc/init.d
    then add it to chkconfig to get it to run when system start?
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    This will not work because in /etc/init.d are the start scripts for the daemins and not the daemin binaries itself. If you want to run the clamd daemon /home/admispconfig/ispconfig/tools/clamav/sbin/clamd, you will have to craete a start script for it in /etc/init.d.
     
  18. plucchetti

    plucchetti New Member

    clamd.amavisd is the same as clamd???
     
    Last edited: Jun 1, 2007
  19. cjc81

    cjc81 New Member

    Ah i c.

    Thanks. I second Morons's suggestion of havig CLAMD being used as default in ISPCONFIG.
     
  20. XPinG

    XPinG New Member

    Hi all, I post my little contribution on how I did on my FC5 server

    first I followed the steps posted by cjc81 :) (thanx man) then :

    I enabled logging in /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf

    Code:
    LogFile /var/log/clamd.log
    I created and changed the owner of the log file to admispconfig

    Code:
    touch /var/log/clamd.log
    chmod admispconfig.admispconfig /var/log/clamd.log
    I've put a starting script in /etc/init.d/clamd that I hacked to run the clamd from ispconfig

    Code:
    #!/bin/sh
    #
    # clamd         Script to start/stop clamd.
    #
    # chkconfig:    - 61 39
    # description:  clamd is an antivirus daemon.
    #
    # processname: clamd
    # config: /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf
    # pidfile: /var/run/clamd.pid
    #
    # Source function library
    . /etc/rc.d/init.d/functions
    # Get network config
    . /etc/sysconfig/network
    #
    test -f /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf || exit 0
    #
    RETVAL=0
    #
    start() {
           echo -n $"Starting Clam AV daemon: "
           daemon /home/admispconfig/ispconfig/tools/clamav/sbin/clamd
           RETVAL=$?
           echo
           [ $RETVAL -eq 0 ] && touch /var/lock/subsys/clamd
           return $RETVAL
    }
    #
    stop() {
           echo -n $"Stopping Clam AV daemon: "
           killproc clamd
           RETVAL=$?
           echo
           [ $RETVAL -eq 0 ] && rm -f /var/run/clamd.pid /var/lock/subsys/clamd
           return $RETVAL
    }
    #
    restart() {
           stop
           start
    }
    
    reload() {
           echo -n $"Reloading DB: "
           killproc clamd -USR2
           RETVAL=$?
           echo
           return $RETVAL
    }
    #
    case "$1" in
     start)
           start
           ;;
     stop)
           stop
           ;;
     status)
           status clamd
           ;;
     restart)
           restart
           ;;
     condrestart)
           [ -f /var/lock/subsys/clamd ] && restart || :
           ;;
     reload)
           reload
           ;;
     *)
           echo $"Usage: $0 {start|stop|status|restart|condrestart|reload}"
           exit 1
    esac
    #
    exit $?
    and execute

    Code:
    chmod 700 /etc/init.d/clamd
    chkconfig --add clamd
    chkconfig --level 2345 clamd on
    service clamd start
     

Share This Page