ISPConfig 3, Debian 7 - high load

Discussion in 'ISPConfig 3 Priority Support' started by inside83, May 14, 2015.

  1. inside83

    inside83 Member HowtoForge Supporter

    Hello,
    One of my production servers has recently began having high load.
    When I run 'uptime' command it shows a value of 30 or even 40.
    That particular server has two websites (WordPress and a custom app) and owncloud.
    It serves 90 mailboxes.
    Hardware specification is: CPU Q6600, 8 GB RAM, 250 GB HDD (160 GB used).

    When users send mail (if it goes through), they have 3 or 4 copies of the sent mail in sent folder.
    Database sync takes forever.
    The server is generally unresponsive.

    What should I do?
    Please help.
     
  2. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    You should first check the output of top, iotop and ps aux to check which processes are using that much power.
    Often this caused by hacked webs or mail accounts spamming or attacking other servers. Or it might be a hardware problem (e.g. hdd failure), you could check with smartctl if you have high CPU wait time.
     
  3. inside83

    inside83 Member HowtoForge Supporter

    Thank you very much for a fast response.
    I just ran 'smartctl --all /dev/sda' and this is what it returned: http://paste.ofcode.org/TfeL7T6AigEJGe8ZUg2ytS
    Could you please help me out?
    Does this mean that I have to replace HDD?
    If so, how to migrate data and should I use 250 GB HDD or I could go with 1 TB?
     
  4. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    No, this output doesn't mean your HDD is defective. Have you checked the other commands?
     
  5. inside83

    inside83 Member HowtoForge Supporter

    iostat shows:
    Code:
    [email protected]:~# iostat
    Linux 3.2.0-4-686-pae (vps3.goinfobl.com)       05/15/2015      _i686_  (4 CPU)
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.83    0.53    0.28   23.47    0.00   74.89
    
    Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
    sda              15.69       158.30        98.29    1452663     901992
    
    [email protected]:~#
     
  6. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    What about running processes with high cpu wait time and run time?
    Check for processes running as web user or non-root in general.
     
  7. inside83

    inside83 Member HowtoForge Supporter

    top shows:
     

    Attached Files:

  8. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Okay, for mee that seems to be quite too much load and cpu wait for the listed processes. Could you please install "iotop" and watch it's output for processes that have high i/o?
    Additionally check syslog for error messages.
     
  9. inside83

    inside83 Member HowtoForge Supporter

    This is 'iotop' next to 'uptime'
     

    Attached Files:

  10. till

    till Super Moderator Staff Member ISPConfig Developer

    did you check the server for rootkits with rkhunter and chkrootkit programs?
     
    Last edited: May 15, 2015
  11. inside83

    inside83 Member HowtoForge Supporter

    Till,
    How to do that?
     
  12. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Install chkrootkit and rkhunter programs, than run them.
     
  13. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    You can also install htop and sort the procs by cpu and state
     
  14. inside83

    inside83 Member HowtoForge Supporter

    chkrootkit: http://paste.ofcode.org/bQTSZd5dxeSHFPD5EibDvr
    rkhunter: http://paste.ofcode.org/GLEQtd5qkHuQiXvgNCQbW3
    rkhunter summary:
    Code:
    System checks summary
    =====================
    
    File properties checks...
        Required commands check failed
        Files checked: 137
        Suspect files: 13
    
    Rootkit checks...
        Rootkits checked : 307
        Possible rootkits: 0
    
    Applications checks...
        All checks skipped
    
    The system checks took: 26 minutes and 54 seconds
    
    All results have been written to the log file: /var/log/rkhunter.log
    
    One or more warnings have been found while checking the system.
    Please check the log file (/var/log/rkhunter.log)
    
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    Thats both ok, as far as I can see. The BINDSHELL ifection is a known false positive from chkrootkit, so nothing to worry about.

    This is a physical server or is it virtualized?
     
  16. inside83

    inside83 Member HowtoForge Supporter

    Yeah, I checked that too and I see it's a false positive.
    It's physical. At my client's server room.
     
  17. inside83

    inside83 Member HowtoForge Supporter

    Just a note: I've put owncloud in inactive state before writing the first post in this thread.
    So all the measurements shown are with owncloud disabled.
     
  18. till

    till Super Moderator Staff Member ISPConfig Developer

    Try to find out which service is causing the lod. Gets the load back to normal when you stop apache, when you stop postfix or when you stop dovecot or dos none of these services has a real impact?
     
  19. inside83

    inside83 Member HowtoForge Supporter

    Till,
    I didn't have the chance to find that out. This morning I couldn't log on to server ant went to my client's server room.
    The message on the display was 'Oh no! Something has gone wrong' :(
    So I got a new HDD and started cloning with CloneZilla, since I'm pretty sure that it's due to a faulty HDD.
    All of this started to happen when ownCloud caused HDD to be 92% full.
    Than I freed out some space (to 72%), put ownCloud in maintenance mode and started this thread.
    I just hope the cloning goes well. I'll update when it's done.
     
  20. inside83

    inside83 Member HowtoForge Supporter

    Update:
    CloneZilla finished (with -rescue option) and everything works fine. :)
    Now I have 1TB HDD with these partitions:
    Code:
    NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    sda      8:0    0 931.5G  0 disk
    ├─sda1   8:1    0 223.4G  0 part /
    ├─sda2   8:2    0     1K  0 part
    └─sda5   8:5    0   9.5G  0 part [SWAP]
    [email protected]:~#
    
    Should I enlarge sda1 or sda1 AND sda5?
     

Share This Page