Problem with a site that kills my server

Discussion in 'General' started by Skyflash, Jan 31, 2012.

  1. Skyflash

    Skyflash New Member

    (sorry for my bad english)

    I have a ISPConfig 3 installation on a VPS i configured my own
    I've configured 6-7 sites, but i'm really using only one, my personal blog (http://www.skyflash.it). I'm having a strange trouble on it :(

    When i publish a "normal" post that contains no external links, everything goes well. When i publish a post that contains external links, trackbacks or internal links with the full address (ie: http://www.skyflash.it/linked-older-post), my server hangs :(

    "Top" tells me that the server consumes ALL the 2GB ram, and the CPU goes to 99% of the wait process

    [​IMG]

    So i have to restart apache2 service (which requires maaaaaany time, because the server in on heavy load). After that, the server returns in good state. Also, i have to manually restart clamav-daemon and freshclam processes that die unexpectedly

    I'm using Fast-CGI mode and i really have no idea where the problem is :(
     
  2. till

    till Super Moderator

    1) Which CMS system do you use in that website?
    2) Do you get any errors in the error.log of that site?
    3) Which Linux distribution is used on the server?
    4) Are the nameservers that are listed in /etc/resolv.conf corect an reachable from the server?
     
  3. Skyflash

    Skyflash New Member

    Wordpress 3.3.1

    Code:
    root@server01:/var/www/skyflash.it/log# tail -f error.log
    [Tue Jan 31 10:05:21 2012] [warn] [client 176.31.136.82] mod_fcgid: read data timeout in 360 seconds
    [Tue Jan 31 10:05:21 2012] [error] [client 176.31.136.82] Premature end of script headers: wp-cron.php
    [Tue Jan 31 10:05:21 2012] [warn] [client 176.31.136.82] mod_fcgid: read data timeout in 360 seconds
    [Tue Jan 31 10:05:21 2012] [error] [client 176.31.136.82] Premature end of script headers: wp-cron.php
    [Tue Jan 31 10:05:21 2012] [warn] [client 176.31.136.82] mod_fcgid: read data timeout in 360 seconds
    [Tue Jan 31 10:05:22 2012] [error] [client 176.31.136.82] Premature end of script headers: wp-cron.php
    [Tue Jan 31 10:05:22 2012] [warn] [client 176.31.136.82] mod_fcgid: read data timeout in 360 seconds
    [Tue Jan 31 10:05:22 2012] [error] [client 176.31.136.82] Premature end of script headers: wp-cron.php
    [Tue Jan 31 10:05:24 2012] [warn] [client 176.31.136.82] mod_fcgid: read data timeout in 360 seconds
    [Tue Jan 31 10:05:24 2012] [error] [client 176.31.136.82] Premature end of script headers: wp-cron.php
    
    Ubuntu 11.10 x64, on a VmWare ESX node

    Code:
    root@server01:/etc# more /etc/resolv.conf
    nameserver 127.0.0.1
    nameserver 213.186.33.99
    nameserver 208.67.222.222
    nameserver 8.8.8.8
    
     
  4. Skyflash

    Skyflash New Member

    I'm trying the same thing on a test site i developed for test purposes (http://www.skyflashtest.biz) and everything seems ok

    The test site is brand new, a standard installation with about 10 posts. My blog is a site with 1000 post and 3000 comments, and a db size about of 3MB (tgz)

    Do you think may be related to a max memory reserved for MySQL istance?
     
  5. Skyflash

    Skyflash New Member

    I have another log. Now the server is on heavy load (no post published), and this is the error.log
    Code:
    [Tue Jan 31 10:49:04 2012] [warn] [client 85.17.73.40] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server, referer: http://www.skyflash.it/af/2005/03/colori-ds
    [Tue Jan 31 10:49:20 2012] [warn] [client 85.17.73.40] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function, referer: http://www.skyflash.it/af/2005/03/colori-ds
    [Tue Jan 31 10:49:29 2012] [warn] [client 85.17.73.40] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server, referer: http://www.skyflash.it/af/2006/09/sony-ps3-senza-hdmi
    [Tue Jan 31 10:49:30 2012] [warn] [client 85.17.73.40] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function, referer: http://www.skyflash.it/af/2006/09/sony-ps3-senza-hdmi
    [Tue Jan 31 10:49:35 2012] [warn] [client 85.17.73.40] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server, referer: http://www.skyflash.it/af/2006/12/lancio-del-wii
    [Tue Jan 31 10:49:36 2012] [warn] [client 85.17.73.40] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function, referer: http://www.skyflash.it/af/2006/12/lancio-del-wii
    
    I remember some months ago i installaed php-apc on the server, because it was requested by the translation plugin i installed on the blog. Maybe this pear extension the responsible?
     
  6. till

    till Super Moderator

    How much memory do you allow for php processes in php.ini? Try to increae it to 128 or even 256MB and restart apache if it is set to a lower value at the moment.

    Regarding the php binary cache, I use php5-xcache (available as DEbian and Ubuntu package from the debian repositories) on my wordpress servers. I would try to uninstall apc and try xcache instead.
     
  7. Skyflash

    Skyflash New Member

    root@server01:/etc/php5/cgi# vi php.ini
    Code:
    ;;;;;;;;;;;;;;;;;
    ; Miscellaneous ;
    ;;;;;;;;;;;;;;;;;
    
    ; Decides whether PHP may expose the fact that it is installed on the server
    ; (e.g. by adding its signature to the Web server header).  It is no security
    ; threat in any way, but it makes it possible to determine whether you use PHP
    ; on your server or not.
    ; http://php.net/expose-php
    expose_php = On
    
    ;;;;;;;;;;;;;;;;;;;
    ; Resource Limits ;
    ;;;;;;;;;;;;;;;;;;;
    
    ; Maximum execution time of each script, in seconds
    ; http://php.net/max-execution-time
    ; Note: This directive is hardcoded to 0 for the CLI SAPI
    max_execution_time = 300
    
    ; Maximum amount of time each script may spend parsing request data. It's a good
    ; idea to limit this time on productions servers in order to eliminate unexpectedly
    ; long running scripts.
    ; Note: This directive is hardcoded to -1 for the CLI SAPI
    ; Default Value: -1 (Unlimited)
    ; Development Value: 60 (60 seconds)
    ; Production Value: 60 (60 seconds)
    ; http://php.net/max-input-time
    max_input_time = 60
    
    ; Maximum input variable nesting level
    ; http://php.net/max-input-nesting-level
    ;max_input_nesting_level = 64
    
    ; Maximum amount of memory a script may consume (128MB)
    ; http://php.net/memory-limit
    memory_limit = 256M
    It was to 256MB

    Ok, thank you. Now i've uninstalled apc and installed xcache
     
  8. Skyflash

    Skyflash New Member

    The cache plugin has not resolved anything :(

    In this moment the server is on heavy load. No post was published. I woke up this morning ad i've found the server locked.

    Top returns me this:

    [​IMG]

    22k zombie processes!!!
    The CPU wait % is ok, but i've no free ram.

    Can you help me, please?
     
  9. till

    till Super Moderator

    Please post the vhost file of the affected website and the fcgi starter script of that site.
     
  10. Skyflash

    Skyflash New Member

    Code:
    root@server01:/var/www/skyflash.it/web# vi /etc/apache2/sites-enabled/100-skyflash.it.vhost 
    <Directory /var/www/skyflash.it>
        AllowOverride None
        Order Deny,Allow
        Deny from all
    </Directory>
    
    <VirtualHost *:80>
          DocumentRoot /var/www/skyflash.it/web
      
        ServerName skyflash.it
        ServerAlias www.skyflash.it
        ServerAdmin webmaster@skyflash.it
    
        ErrorLog /var/log/ispconfig/httpd/skyflash.it/error.log
    
    
        ErrorDocument 400 /error/400.html
        ErrorDocument 401 /error/401.html
        ErrorDocument 403 /error/403.html
        ErrorDocument 404 /error/404.html
        ErrorDocument 405 /error/405.html
        ErrorDocument 500 /error/500.html
        ErrorDocument 502 /error/502.html
        ErrorDocument 503 /error/503.html
    
        <IfModule mod_ssl.c>
        </IfModule>
        <Directory /var/www/skyflash.it/web>
            Options FollowSymLinks
            AllowOverride All
            Order allow,deny
            Allow from all
            
            # ssi enabled
            AddType text/html .shtml
            AddOutputFilter INCLUDES .shtml
            Options +Includes
        </Directory>
        <Directory /var/www/clients/client1/web2/web>
            Options FollowSymLinks
            AllowOverride All
            Order allow,deny
            Allow from all
            
            # ssi enabled
            AddType text/html .shtml
            AddOutputFilter INCLUDES .shtml
            Options +Includes
        </Directory>
    
        <IfModule mod_ruby.c>
          <Directory /var/www/skyflash.it/web>
            Options +ExecCGI
          </Directory>
          RubyRequire apache/ruby-run
          #RubySafeLevel 0
          <Files *.rb>
            SetHandler ruby-object
            RubyHandler Apache::RubyRun.instance
          </Files>
          <Files *.rbx>
            SetHandler ruby-object
            RubyHandler Apache::RubyRun.instance
          </Files>
        </IfModule>
    
    
        # cgi enabled
    	<Directory /var/www/clients/client1/web2/cgi-bin>
          Order allow,deny
          Allow from all
        </Directory>
        ScriptAlias  /cgi-bin/ /var/www/clients/client1/web2/cgi-bin/
        AddHandler cgi-script .cgi
        AddHandler cgi-script .pl
        # suexec enabled
        SuexecUserGroup web2 client1
        # Clear PHP settings of this website
        <FilesMatch "\.ph(p3?|tml)$">
            SetHandler None
        </FilesMatch>
        # php as fast-cgi enabled
    	# For config options see: http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html
        <IfModule mod_fcgid.c>
            FcgidIdleTimeout 300
            FcgidProcessLifeTime 3600
            # FcgidMaxProcesses 1000
            FcgidMinProcessesPerClass 0
            FcgidMaxProcessesPerClass 100
            FcgidConnectTimeout 3
            FcgidIOTimeout 360
            FcgidBusyTimeout 300
        </IfModule>
        <Directory /var/www/skyflash.it/web>
            AddHandler fcgid-script .php .php3 .php4 .php5
            FCGIWrapper /var/www/php-fcgi-scripts/web2/.php-fcgi-starter .php
            Options +ExecCGI
            AllowOverride All
            Order allow,deny
            Allow from all
        </Directory>
        <Directory /var/www/clients/client1/web2/web>
            AddHandler fcgid-script .php .php3 .php4 .php5
            FCGIWrapper /var/www/php-fcgi-scripts/web2/.php-fcgi-starter .php
            Options +ExecCGI
            AllowOverride All
            Order allow,deny
            Allow from all
        </Directory>
    
    
        # add support for apache mpm_itk
        <IfModule mpm_itk_module>
          AssignUserId web2 client1
        </IfModule>
    
        <IfModule mod_dav_fs.c>
    	  # Do not execute PHP files in webdav directory
          <Directory /var/www/clients/client1/web2/webdav>
    	    <FilesMatch "\.ph(p3?|tml)$">
              SetHandler None
            </FilesMatch>
          </Directory>
          # DO NOT REMOVE THE COMMENTS!
          # IF YOU REMOVE THEM, WEBDAV WILL NOT WORK ANYMORE!
          # WEBDAV BEGIN
          # WEBDAV END
        </IfModule>
    
    
    </VirtualHost>
    <VirtualHost *:443>
          DocumentRoot /var/www/skyflash.it/web
      
        ServerName skyflash.it
        ServerAlias www.skyflash.it
        ServerAdmin webmaster@skyflash.it
    
        ErrorLog /var/log/ispconfig/httpd/skyflash.it/error.log
    
    
        ErrorDocument 400 /error/400.html
        ErrorDocument 401 /error/401.html
        ErrorDocument 403 /error/403.html
        ErrorDocument 404 /error/404.html
        ErrorDocument 405 /error/405.html
        ErrorDocument 500 /error/500.html
        ErrorDocument 502 /error/502.html
        ErrorDocument 503 /error/503.html
    
        <IfModule mod_ssl.c>
    	SSLEngine on
        SSLCertificateFile /var/www/clients/client1/web2/ssl/skyflash.it.crt
        SSLCertificateKeyFile /var/www/clients/client1/web2/ssl/skyflash.it.key
        </IfModule>
        <Directory /var/www/skyflash.it/web>
            Options FollowSymLinks
            AllowOverride All
            Order allow,deny
            Allow from all
            
            # ssi enabled
            AddType text/html .shtml
            AddOutputFilter INCLUDES .shtml
            Options +Includes
        </Directory>
        <Directory /var/www/clients/client1/web2/web>
            Options FollowSymLinks
            AllowOverride All
            Order allow,deny
            Allow from all
            
            # ssi enabled
            AddType text/html .shtml
            AddOutputFilter INCLUDES .shtml
            Options +Includes
        </Directory>
    
        <IfModule mod_ruby.c>
          <Directory /var/www/skyflash.it/web>
            Options +ExecCGI
          </Directory>
          RubyRequire apache/ruby-run
          #RubySafeLevel 0
          <Files *.rb>
            SetHandler ruby-object
            RubyHandler Apache::RubyRun.instance
          </Files>
          <Files *.rbx>
            SetHandler ruby-object
            RubyHandler Apache::RubyRun.instance
          </Files>
        </IfModule>
    
    
        # cgi enabled
    	<Directory /var/www/clients/client1/web2/cgi-bin>
          Order allow,deny
          Allow from all
        </Directory>
        ScriptAlias  /cgi-bin/ /var/www/clients/client1/web2/cgi-bin/
        AddHandler cgi-script .cgi
        AddHandler cgi-script .pl
        # suexec enabled
        SuexecUserGroup web2 client1
        # Clear PHP settings of this website
        <FilesMatch "\.ph(p3?|tml)$">
            SetHandler None
        </FilesMatch>
        # php as fast-cgi enabled
    	# For config options see: http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html
        <IfModule mod_fcgid.c>
            FcgidIdleTimeout 300
            FcgidProcessLifeTime 3600
            # FcgidMaxProcesses 1000
            FcgidMinProcessesPerClass 0
            FcgidMaxProcessesPerClass 100
            FcgidConnectTimeout 3
            FcgidIOTimeout 360
            FcgidBusyTimeout 300
        </IfModule>
        <Directory /var/www/skyflash.it/web>
            AddHandler fcgid-script .php .php3 .php4 .php5
            FCGIWrapper /var/www/php-fcgi-scripts/web2/.php-fcgi-starter .php
            Options +ExecCGI
            AllowOverride All
            Order allow,deny
            Allow from all
        </Directory>
        <Directory /var/www/clients/client1/web2/web>
            AddHandler fcgid-script .php .php3 .php4 .php5
            FCGIWrapper /var/www/php-fcgi-scripts/web2/.php-fcgi-starter .php
            Options +ExecCGI
            AllowOverride All
            Order allow,deny
            Allow from all
        </Directory>
    
    
        # add support for apache mpm_itk
        <IfModule mpm_itk_module>
          AssignUserId web2 client1
        </IfModule>
    
        <IfModule mod_dav_fs.c>
    	  # Do not execute PHP files in webdav directory
          <Directory /var/www/clients/client1/web2/webdav>
    	    <FilesMatch "\.ph(p3?|tml)$">
              SetHandler None
            </FilesMatch>
          </Directory>
          # DO NOT REMOVE THE COMMENTS!
          # IF YOU REMOVE THEM, WEBDAV WILL NOT WORK ANYMORE!
          # WEBDAV BEGIN
          # WEBDAV END
        </IfModule>
    
    
    </VirtualHost>
    
    Where i can find the fcgi starter script of the affected site?
     
  11. Skyflash

    Skyflash New Member

    I think i've found it:

    Code:
    root@server01:/var/www/skyflash.it/web# vi /var/www/php-fcgi-scripts/web2/.php-fcgi-starter 
    
    #!/bin/sh
    PHPRC="/etc/php5/cgi/"
    export PHPRC
    PHP_DOCUMENT_ROOT="/var/www/clients/client1/web2"
    export PHP_DOCUMENT_ROOT
    # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache 
    # mod_fcgi will control the number of childs themself and never use the additional processes.
    # PHP_FCGI_CHILDREN=1
    # export PHP_FCGI_CHILDREN
    PHP_FCGI_MAX_REQUESTS=1000
    export PHP_FCGI_MAX_REQUESTS
    exec /usr/bin/php-cgi \
     $1
    
     
  12. pititis

    pititis Member

    You don't use open_basedir restrictions??
     
  13. Skyflash

    Skyflash New Member

    In the site's configuration document there are:
    Code:
    /var/www/clients/client1/web2/web:/var/www/clients/client1/web2/tmp:/var/www/skyflash.it/web:/srv/www/skyflash.it/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin
    [​IMG]
     
  14. pititis

    pititis Member

    Yeah sure but see your /var/www/php-fcgi-scripts/web2/.php-fcgi-starter

    ..nothing.


    I don't know if its ispcofig related issue but try to apply or change the website options tab.

    ...e poi ci racconti
     
  15. Skyflash

    Skyflash New Member

    Re-applied and nothing has been changed :(
    Otherwise, other websites has the correct openbasedir instruction

    ... e non cambia, il fetecchione :)
     
  16. Skyflash

    Skyflash New Member

    I tried deleting the .php-fcgi-starter, then modify and re-apply the configuration in ISPConfig

    ISPConfigr recreated the file, but its the same, without ANY openbasedir instruction :(

    Heeeeelp! :D
     
  17. till

    till Super Moderator

    1) Which ISPConfig version do you use?
    2) Post the outout of:

    ls /usr/local/ispconfig/server/conf-custom/
     
  18. Skyflash

    Skyflash New Member

    Latest, 3.0.4.2

    Code:
    root@server01:/var/www/php-fcgi-scripts/web2# ls /usr/local/ispconfig/server/conf-custom/
    error  index  mail
     
  19. Skyflash

    Skyflash New Member

    I think i've a trouble in the ISPConfig configuration.

    I've just created a new site. Created the client, the site, db and ftp access.
    This is the configuration document:
    [​IMG]

    Now i'm checking site's .php-fgci-starter file, and it has no openbasedir directive:

    Code:
    #!/bin/sh
    PHPRC="/etc/php5/cgi/"
    export PHPRC
    PHP_DOCUMENT_ROOT="/var/www/clients/client2/web14"
    export PHP_DOCUMENT_ROOT
    # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache 
    # mod_fcgi will control the number of childs themself and never use the additional processes.
    # PHP_FCGI_CHILDREN=1
    # export PHP_FCGI_CHILDREN
    PHP_FCGI_MAX_REQUESTS=1000
    export PHP_FCGI_MAX_REQUESTS
    exec /usr/bin/php-cgi \
     $1
    
    Maybe something gone wrong during the latest ISPConfig upgrade?

    Please, help me :(
     
  20. Guavaman

    Guavaman New Member

    I know this is an old thread, but this is exactly the same problem I'm seeing with all Wordpress sites hosted on GoDaddy.com. I haven't found a single other mention of this problem on the web. Did you ever find a resolution?

    URLs in posts will eventually cause Apache or FastCGI to crash when saving the post. The more URL's in the post, the most likely a crash, but it appears to only happen with URL's of certain formats. For example, I've verified URL's with port numbers like :80 will always make it crash after only a few saves. Also, very long URL's over about 200 characters will also crash it quickly. This even happens with a fresh, clean Wordpress installation. Posts with no http:// in them never crash it.

    GoDaddy is also using FastCGI and there's no way to disable it. The behavior of the crash makes me think something is going wrong on POST and that the URLs are being parsed or sanitized or something that's causing the apache processes to choke. I've done these tests on two other servers not on GoDaddy and they never crash.

    I get these error very frequently in the logs:

    Code:
    [Thu Aug 29 16:07:04 2013] [4675188] [fcgid:warn] (9)Bad file descriptor: [client 98.210.131.207:3694] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function, referer http://site.com/test/wp-admin/post.php?post=22&action=edit&message=10
    
    [Thu Aug 29 16:12:12 2013] [4675188] [fcgid:warn] (104)Connection reset by peer: [client 98.210.131.207:3721] mod_fcgid: error reading data from FastCGI server, referer http://site.com/test/wp-admin/post.php?post=22&action=edit&message=10
    I've tried increasing memory, and setting fcgi variables in php5.ini have no effect.
     

Share This Page