Apache2 FCGID locking up the box?

Discussion in 'General' started by crypted, Apr 11, 2011.

  1. crypted

    crypted New Member

    My box continually locks up with loads of over 190.

    In the Apache error log, I see this for several hours prior to the lockup:

    Code:
    [Mon Apr 11 13:43:28 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
    .....
    [Mon Apr 11 13:53:51 2011] [warn] mod_fcgid: process 13068 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:53:51 2011] [warn] mod_fcgid: process 13066 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:54:44 2011] [warn] mod_fcgid: process 13081 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:55:06 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL    
    [Mon Apr 11 13:55:20 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL   
    [Mon Apr 11 13:55:26 2011] [warn] mod_fcgid: process 13112 graceful kill fail, sending SIGKILL
    [Mon Apr 11 13:55:26 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL
    
    In the end, I have to have the ISP hard-reboot the server so that it will be back online and accessible.

    Any ideas? This has been destroying my productivity since switching to ISPC3 last year...
     
  2. till

    till Super Moderator

    Therae are no known problems with ispconfig and fcgid in the current versions. Which ISPConfig version do you use, what are the hardware specifications of the server etc. And please post the fcgi starter file of a affected website.
     
  3. crypted

    crypted New Member

    I'm running Debian Squeeze x64 (happened on Lenny x64 also) Linux 2.6.32-5-amd64 on x86_64.

    It's a Intel(R) Pentium(R) D CPU 3.00GHz 2 cores, 4 GB RAM, and 2x 300GB SATA2 hdds.

    I've got ISPC 3.0.3.2 installed.

    Here are several FCGI starter scripts from almost all of the sites that end up with the crazy spawning of processes and lag crashing.
    Code:
    cat /var/www/php-fcgi-scripts/web11/.php-fcgi-starter 
    #!/bin/sh
    PHPRC="/var/www/conf/web11"
    export PHPRC
    PHP_DOCUMENT_ROOT="/var/www/clients/client1/web11"
    export PHP_DOCUMENT_ROOT
    # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache 
    # mod_fcgi will control the number of childs themself and never use the additional processes.
    # PHP_FCGI_CHILDREN=1
    # export PHP_FCGI_CHILDREN
    PHP_FCGI_MAX_REQUESTS=5000
    export PHP_FCGI_MAX_REQUESTS
    exec /usr/bin/php-cgi \
     -d open_basedir="/var/www/clients/client1/web11/web:/var/www/clients/client1/web11/tmp:/var/www/derekgordon.com/web:/srv/www/derekgordon.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
    -d upload_tmp_dir=/var/www/clients/client1/web11/tmp \
    -d session.save_path=/var/www/clients/client1/web11/tmp \
     $1
    
    Code:
    my:/home/dingle# cat /var/www/php-fcgi-scripts/web5/.php-fcgi-starter 
    #!/bin/sh
    PHPRC="/var/www/conf/web5"
    export PHPRC
    PHP_DOCUMENT_ROOT="/var/www/clients/client1/web5"
    export PHP_DOCUMENT_ROOT
    # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache 
    # mod_fcgi will control the number of childs themself and never use the additional processes.
    # PHP_FCGI_CHILDREN=1
    # export PHP_FCGI_CHILDREN
    PHP_FCGI_MAX_REQUESTS=5000
    export PHP_FCGI_MAX_REQUESTS
    exec /usr/bin/php-cgi \
     -d open_basedir="/var/www/clients/client1/web5/web:/var/www/clients/client1/web5/tmp:/var/www/areyouliberal.com/web:/srv/www/areyouliberal.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
    -d upload_tmp_dir=/var/www/clients/client1/web5/tmp \
    -d session.save_path=/var/www/clients/client1/web5/tmp \
    export LD_PRELOAD = amon.so
     $1
    Code:
    my:/home/dingle# cat /var/www/php-fcgi-scripts/web8/.php-fcgi-starter 
    #!/bin/sh
    PHPRC="/var/www/conf/web8"
    export PHPRC
    PHP_DOCUMENT_ROOT="/var/www/clients/client1/web8"
    export PHP_DOCUMENT_ROOT
    # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache 
    # mod_fcgi will control the number of childs themself and never use the additional processes.
    # PHP_FCGI_CHILDREN=1
    # export PHP_FCGI_CHILDREN
    PHP_FCGI_MAX_REQUESTS=5000
    export PHP_FCGI_MAX_REQUESTS
    exec /usr/bin/php-cgi \
     -d open_basedir="/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
    -d upload_tmp_dir=/var/www/clients/client1/web8/tmp \
    -d session.save_path=/var/www/clients/client1/web8/tmp \
    export LD_PRELOAD = amon.so
     $1
    Code:
    my:/home/dingle# cat /var/www/php-fcgi-scripts/web35/.php-fcgi-starter 
    #!/bin/sh
    PHPRC="/var/www/conf/web35"
    export PHPRC
    PHP_DOCUMENT_ROOT="/var/www/clients/client1/web35"
    export PHP_DOCUMENT_ROOT
    # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache 
    # mod_fcgi will control the number of childs themself and never use the additional processes.
    # PHP_FCGI_CHILDREN=1
    # export PHP_FCGI_CHILDREN
    PHP_FCGI_MAX_REQUESTS=5000
    export PHP_FCGI_MAX_REQUESTS
    exec /usr/bin/php-cgi \
     -d open_basedir="/var/www/clients/client1/web35/web:/var/www/clients/client1/web35/tmp:/var/www/masonicleaks.com/web:/srv/www/masonicleaks.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
    -d upload_tmp_dir=/var/www/clients/client1/web35/tmp \
    -d session.save_path=/var/www/clients/client1/web35/tmp \
     $1my:/home/dingle# 
     
  4. till

    till Super Moderator

    Does these lines are really like this:

    -d session.save_path=/var/www/clients/client1/web5/tmp \
    export LD_PRELOAD = amon.so
    $1

    it has to be:

    -d session.save_path=/var/www/clients/client1/web5/tmp \
    $1

    The other parts of the file look fine. You should check your apache configuration file and limit the number of apache instances that get spawned.
     
  5. crypted

    crypted New Member

    I was playing around with amon earlier. So, those entries weren't always there.

    When you say the max instances, are you talking about the FCGI part in the vhosts files?

    If so, what would you recommend the settings to be?
     
  6. till

    till Super Moderator

    I'am talking about the mpm settings in the apache2.conf file.

    But you can take a look in the vhost file as well, be default the values in the vhost file should be:

    # IdleTimeout n (3600 seconds)
    # An idle fastcgi application will be terminated after IdleTimeout seconds.
    IdleTimeout 3600

    # ProcessLifeTime n (7200 seconds)
    # A fastcgi application will be terminated if lifetime expired, even no error is detected.
    ProcessLifeTime 7200

    # MaxProcessCount n (1000)
    # The max count of total fastcgi process count.
    # MaxProcessCount 1000

    # DefaultMinClassProcessCount n (3)
    # The minimum number of fastcgi application instances for any one fastcgi application.
    # Idle fastcgi will not be killed if their count is less than n
    # Set this to 0, and tweak IdleTimeout
    DefaultMinClassProcessCount 0

    # DefaultMaxClassProcessCount n (100)
    # The maximum number of fastcgi application instances allowed to run for
    # particular one fastcgi application.
    DefaultMaxClassProcessCount 100

    # IPCConnectTimeout n (3 seconds)
    # The connect timeout to a fastcgi application.
    IPCConnectTimeout 8

    # IPCCommTimeout n (20 seconds)
    # The communication timeout to a fastcgi application. Please increase this
    # value if your CGI have a slow initialization or slow respond.
    IPCCommTimeout 360

    # BusyTimeout n (300 seconds)
    # A fastcgi application will be terminated if handing a single request
    # longer than busy timeout.
    BusyTimeout 300


    you can e.g. tune the MaxProcessCount and DefaultMaxClassProcessCount parameters.
     
  7. crypted

    crypted New Member

    Right, and in apache2 I see the following with mpm involved:

    Code:
    IfModule mpm_prefork_module>
        StartServers          5
        MinSpareServers       5
        MaxSpareServers      10           
        MaxClients          150
        MaxRequestsPerChild   0
    </IfModule>
    Code:
    <IfModule mpm_worker_module>
        StartServers          2
        MaxClients          150
        MinSpareThreads      25
        MaxSpareThreads      75
        ThreadsPerChild      25
        MaxRequestsPerChild   0
    </IfModule>
    what would you recommend for changing? Would max clients of 50 be too low? With a MaxSpareThreads of 45?
     

Share This Page