HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   General (http://www.howtoforge.com/forums/forumdisplay.php?f=25)
-   -   Apache2 FCGID locking up the box? (http://www.howtoforge.com/forums/showthread.php?t=52238)

crypted 11th April 2011 20:18

Apache2 FCGID locking up the box?
 
My box continually locks up with loads of over 190.

In the Apache error log, I see this for several hours prior to the lockup:

Code:

[Mon Apr 11 13:43:28 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL
.....
[Mon Apr 11 13:53:51 2011] [warn] mod_fcgid: process 13068 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:53:51 2011] [warn] mod_fcgid: process 13066 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:54:44 2011] [warn] mod_fcgid: process 13081 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:55:06 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL   
[Mon Apr 11 13:55:20 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL 
[Mon Apr 11 13:55:26 2011] [warn] mod_fcgid: process 13112 graceful kill fail, sending SIGKILL
[Mon Apr 11 13:55:26 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL

In the end, I have to have the ISP hard-reboot the server so that it will be back online and accessible.

Any ideas? This has been destroying my productivity since switching to ISPC3 last year...

till 12th April 2011 10:24

Therae are no known problems with ispconfig and fcgid in the current versions. Which ISPConfig version do you use, what are the hardware specifications of the server etc. And please post the fcgi starter file of a affected website.

crypted 12th April 2011 14:23

I'm running Debian Squeeze x64 (happened on Lenny x64 also) Linux 2.6.32-5-amd64 on x86_64.

It's a Intel(R) Pentium(R) D CPU 3.00GHz 2 cores, 4 GB RAM, and 2x 300GB SATA2 hdds.

I've got ISPC 3.0.3.2 installed.

Here are several FCGI starter scripts from almost all of the sites that end up with the crazy spawning of processes and lag crashing.
Code:

cat /var/www/php-fcgi-scripts/web11/.php-fcgi-starter
#!/bin/sh
PHPRC="/var/www/conf/web11"
export PHPRC
PHP_DOCUMENT_ROOT="/var/www/clients/client1/web11"
export PHP_DOCUMENT_ROOT
# The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache
# mod_fcgi will control the number of childs themself and never use the additional processes.
# PHP_FCGI_CHILDREN=1
# export PHP_FCGI_CHILDREN
PHP_FCGI_MAX_REQUESTS=5000
export PHP_FCGI_MAX_REQUESTS
exec /usr/bin/php-cgi \
 -d open_basedir="/var/www/clients/client1/web11/web:/var/www/clients/client1/web11/tmp:/var/www/derekgordon.com/web:/srv/www/derekgordon.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
-d upload_tmp_dir=/var/www/clients/client1/web11/tmp \
-d session.save_path=/var/www/clients/client1/web11/tmp \
 $1

Code:

my:/home/dingle# cat /var/www/php-fcgi-scripts/web5/.php-fcgi-starter
#!/bin/sh
PHPRC="/var/www/conf/web5"
export PHPRC
PHP_DOCUMENT_ROOT="/var/www/clients/client1/web5"
export PHP_DOCUMENT_ROOT
# The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache
# mod_fcgi will control the number of childs themself and never use the additional processes.
# PHP_FCGI_CHILDREN=1
# export PHP_FCGI_CHILDREN
PHP_FCGI_MAX_REQUESTS=5000
export PHP_FCGI_MAX_REQUESTS
exec /usr/bin/php-cgi \
 -d open_basedir="/var/www/clients/client1/web5/web:/var/www/clients/client1/web5/tmp:/var/www/areyouliberal.com/web:/srv/www/areyouliberal.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
-d upload_tmp_dir=/var/www/clients/client1/web5/tmp \
-d session.save_path=/var/www/clients/client1/web5/tmp \
export LD_PRELOAD = amon.so
 $1

Code:

my:/home/dingle# cat /var/www/php-fcgi-scripts/web8/.php-fcgi-starter
#!/bin/sh
PHPRC="/var/www/conf/web8"
export PHPRC
PHP_DOCUMENT_ROOT="/var/www/clients/client1/web8"
export PHP_DOCUMENT_ROOT
# The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache
# mod_fcgi will control the number of childs themself and never use the additional processes.
# PHP_FCGI_CHILDREN=1
# export PHP_FCGI_CHILDREN
PHP_FCGI_MAX_REQUESTS=5000
export PHP_FCGI_MAX_REQUESTS
exec /usr/bin/php-cgi \
 -d open_basedir="/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
-d upload_tmp_dir=/var/www/clients/client1/web8/tmp \
-d session.save_path=/var/www/clients/client1/web8/tmp \
export LD_PRELOAD = amon.so
 $1

Code:

my:/home/dingle# cat /var/www/php-fcgi-scripts/web35/.php-fcgi-starter
#!/bin/sh
PHPRC="/var/www/conf/web35"
export PHPRC
PHP_DOCUMENT_ROOT="/var/www/clients/client1/web35"
export PHP_DOCUMENT_ROOT
# The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache
# mod_fcgi will control the number of childs themself and never use the additional processes.
# PHP_FCGI_CHILDREN=1
# export PHP_FCGI_CHILDREN
PHP_FCGI_MAX_REQUESTS=5000
export PHP_FCGI_MAX_REQUESTS
exec /usr/bin/php-cgi \
 -d open_basedir="/var/www/clients/client1/web35/web:/var/www/clients/client1/web35/tmp:/var/www/masonicleaks.com/web:/srv/www/masonicleaks.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \
-d upload_tmp_dir=/var/www/clients/client1/web35/tmp \
-d session.save_path=/var/www/clients/client1/web35/tmp \
 $1my:/home/dingle#


till 12th April 2011 14:56

Does these lines are really like this:

-d session.save_path=/var/www/clients/client1/web5/tmp \
export LD_PRELOAD = amon.so
$1

it has to be:

-d session.save_path=/var/www/clients/client1/web5/tmp \
$1

The other parts of the file look fine. You should check your apache configuration file and limit the number of apache instances that get spawned.

crypted 12th April 2011 15:05

I was playing around with amon earlier. So, those entries weren't always there.

When you say the max instances, are you talking about the FCGI part in the vhosts files?

If so, what would you recommend the settings to be?

till 12th April 2011 15:24

Quote:

When you say the max instances, are you talking about the FCGI part in the vhosts files?
I'am talking about the mpm settings in the apache2.conf file.

But you can take a look in the vhost file as well, be default the values in the vhost file should be:

# IdleTimeout n (3600 seconds)
# An idle fastcgi application will be terminated after IdleTimeout seconds.
IdleTimeout 3600

# ProcessLifeTime n (7200 seconds)
# A fastcgi application will be terminated if lifetime expired, even no error is detected.
ProcessLifeTime 7200

# MaxProcessCount n (1000)
# The max count of total fastcgi process count.
# MaxProcessCount 1000

# DefaultMinClassProcessCount n (3)
# The minimum number of fastcgi application instances for any one fastcgi application.
# Idle fastcgi will not be killed if their count is less than n
# Set this to 0, and tweak IdleTimeout
DefaultMinClassProcessCount 0

# DefaultMaxClassProcessCount n (100)
# The maximum number of fastcgi application instances allowed to run for
# particular one fastcgi application.
DefaultMaxClassProcessCount 100

# IPCConnectTimeout n (3 seconds)
# The connect timeout to a fastcgi application.
IPCConnectTimeout 8

# IPCCommTimeout n (20 seconds)
# The communication timeout to a fastcgi application. Please increase this
# value if your CGI have a slow initialization or slow respond.
IPCCommTimeout 360

# BusyTimeout n (300 seconds)
# A fastcgi application will be terminated if handing a single request
# longer than busy timeout.
BusyTimeout 300


you can e.g. tune the MaxProcessCount and DefaultMaxClassProcessCount parameters.

crypted 12th April 2011 16:16

Right, and in apache2 I see the following with mpm involved:

Code:

IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers      5
    MaxSpareServers      10         
    MaxClients          150
    MaxRequestsPerChild  0
</IfModule>

Code:

<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadsPerChild      25
    MaxRequestsPerChild  0
</IfModule>

what would you recommend for changing? Would max clients of 50 be too low? With a MaxSpareThreads of 45?


All times are GMT +2. The time now is 07:58.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.