Random very high server loads

Discussion in 'ISPConfig 3 Priority Support' started by tomek, Aug 28, 2013.

  1. tomek

    tomek New Member

    Hi, I hope somebody can help me to solve a problem I've been struggling with for a while now. The problem I am having is random very high server load. This seems to usually happen during the day several times per hour, and tends to get worse as the day goes by. All my websites start to get noticeably slower and slower, in ISP config server load it jumps from 10, to 20, 30 and sometimes reaches over 90, at this point it is at a standstill. The server seems to crush and the only solution is to stop apache, wait at least 10 minutes and restart it, than things get back to normal for a while and than it starts again, and than suddenly just stops spiking up and is normal for a long period.

    I did some research to check MySQL long query logs, etc but that is not the case, there is no long queries running. Looking at server logs I noticed an error mod_fcgid: can't apply process slot for.

    Based on research I did, the solution is to adjust mod_fcgid settings. Based o the info available I should do that under ISPCONFIG under Sites >> Mysite >> Options >> Apache Directives. So in there I put this in there, There wasn't enought time for me to see if this makes a differance, because as of now the server is running fine. But i know this issue will come back:

    FcgidMaxRequestsPerProcess 500
    FcgidMaxProcessCount 3
    FcgidMaxRequestLen 33554432
    FcgidIPCCommTimeout 60
    FcgidIdleTimeout 120
    FcgidIdleScanInterval 30
    FcgidBusyTimeout 120
    FcgidBusyScanInterval 90
    FcgidErrorScanInterval 3
    FcgidZombieScanInterval 3
    FcgidProcessLifeTime 3600

    This is some info about what I'm running:

    Hardware: Intel Core i3 2GZ, 8GB Ram
    OS: CentOS 6.4 64bit
    ISPCONFIG: 3.0.5.3
    Apache
    Domains: Currently hosting 6 websites
    Traffic: about 3000/day nothing very significant

    Can somebody point me in the right directions so I can finally solve this problem?

    Thanks
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    With your traffic, the default settings should work. On my servers I did not had to change them yet, but it might be that you have to adjust them on your server for some sites depending on their needs and resource usage.

    What you might try is to switch the sites from php-fcgi to php-fpm if you have that installed. php-fpm is a newer form of the fcgi interface and is a bit more efficient then fcgi as far as I know.
     
  3. tomek

    tomek New Member

    I am still having this problem despite trying to adjust the FCGI settings. It just doesn't work. This server overload happens at least once or twice per day, the server just gets overloaded jumping to over 120. Weird thing is that my website is not getting that many visits. It is around 3000 per day which is not that much, I don't see a reason why there would be such a problem for it to handle, all I'm running is a wordpress website. In my error logs when this happens I see errors: mod_fcgid: can't apply process slot for .... with reference to multiple files on my wordpress site. In ISPCONFIG i put settings to adjust FCGID, this is what I have in Apache Directives in the settings of my website:

    FcgidMaxRequestLen 33554432
    FcgidIdleTimeout 3600
    FcgidProcessLifeTime 7200
    FcgidMaxProcessCount 1000
    FcgidDefaultMinClassProcessCount 3
    FcgidDefaultMaxClassProcessCount 1000
    FcgidIPCConnectTimeout 8
    FcgidIPCCommTimeout 600
    FcgidBusyTimeout 300

    I keep adjusting these settings but this overload is still happening. I am pulling my hair out trying to figure out the problem I don't know what else to do. Can somebody please help me.
     
  4. Turbanator

    Turbanator Member HowtoForge Supporter

    Is this a CentOS VM on a windows box or direct CentOS on hardware?
     
  5. tomek

    tomek New Member

    This is a natively running CentOS 6.4 on Intel core i3 8GB ram no VMs involved
     
  6. Turbanator

    Turbanator Member HowtoForge Supporter

    Next time is loads up, can you get a snapshot of 'top' with all cpu's showing and an 'iotop' snapshot as well?

    I'm not an expert, but for your own reference check ps -aux when it happen to make sure you don't have a rogue script/bot sitting in /tmp or something that launches when it wants.

    (just tips that I checked when had a similar problem)...but mine was a VM and windows was the issue.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    If you use suexec in all sites, you can see with "top" the users that run the fcgi processes. When you know the username, you can find the corresponding website and there you shuld lookinto the access and error.log of the site. Such temporary high loads are often caused by bots that spider your website too fast or by script kids that try a dos. From the log, you can see the IP addresses and block access to your site.

    Another helpful tool to automate this is the apache module mod_evasive. It can temporarily block requests if someone tries to load / reload pages from your sites too fast.
     
  8. tomek

    tomek New Member

    I only have 3 websites on this server and I know exactly which one is causing the problem as it has the most visits and in the error logs. The error: mod_fcgid: can't apply process slot for points to this site when viewing it's error logs, and this error is generated at roughly the same time the overload happens. This problem accoured today as well and I am attaching a screen shot of the top command during when this is happening.

    As far as mod_evasive, can I just install it using yum? Is there anything else I need to do it to make it work with ISPCONFIG?
     

    Attached Files:

    • top.jpg
      top.jpg
      File size:
      93.4 KB
      Views:
      6
  9. Turbanator

    Turbanator Member HowtoForge Supporter

    What are your settings in my.cnf?

    Also what type of site/rfamework is that website with the errors?
     
  10. tomek

    tomek New Member

    Here is my my.cnf

    [[email protected] ~]# vi /etc/my.cnf
    [mysqld]
    datadir=/var/lib/mysql
    socket=/var/lib/mysql/mysql.sock
    user=mysql
    # Disabling symbolic-links is recommended to prevent assorted security risks
    symbolic-links=0

    [mysqld_safe]
    log-error=/var/log/mysqld.log
    pid-file=/var/run/mysqld/mysqld.pid


    The site is a Wordpress site
     
  11. Turbanator

    Turbanator Member HowtoForge Supporter

    sounds like a resource issue to me. You probably need a lot more tuning in your my.cnf.
    I'm sure others have posted some examples here or google wordpress my.cnf .
     

Share This Page