Problem with mod_fcgid

Discussion in 'ISPConfig 3 Priority Support' started by ITFixt, Jan 6, 2016.

  1. ITFixt

    ITFixt New Member HowtoForge Supporter

    I had a perfectly working server. A couple of sites were using a self-signed SSL certificate which worked as expected - the browser baulked at it but accepting the certificate meant I could test everything under SSL.

    I bought a multi-domain Comodo SSL certificate and installed it on the two relevant sites, At that point I started getting a raft of errors:

    Code:
    [Wed Jan 06 07:39:30 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
    [Wed Jan 06 07:39:30 2016] [warn] RSA server certificate CommonName (CN) `Webmaster' does NOT match server name!?
    [Wed Jan 06 07:39:30 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
    [Wed Jan 06 07:39:31 2016] [notice] Digest: generating secret for digest authentication ...
    [Wed Jan 06 07:39:31 2016] [notice] Digest: done
    [Wed Jan 06 07:39:31 2016] [notice] FastCGI: process manager initialized (pid 22447)
    [Wed Jan 06 07:39:31 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
    [Wed Jan 06 07:39:31 2016] [warn] RSA server certificate CommonName (CN) `Webmaster' does NOT match server name!?
    [Wed Jan 06 07:39:32 2016] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
    [Wed Jan 06 07:39:32 2016] [notice] Apache/2.2.22 (Ubuntu) DAV/2 mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_fcgid/2.3.6 PHP/5.3.10-1ubuntu3.21 with Suhosin-Patch mod_ssl/2.2.22 OpenSSL/1.0.1 mod_qos/9.76 mod_ruby/1.2.6 Ruby/1.8.7(2011-06-30) mod_wsgi/3.3 Python/2.7.3 configured -- resuming normal operations
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Wed Jan 06 07:40:02 2016] [alert] Child 22452 returned a Fatal error... Apache is exiting!
    [Wed Jan 06 07:54:13 2016] [emerg] (22)Invalid argument: couldn't release the accept mutex
    Having done some searches for what the cause might be I made several changes (yes, I wish I'd kept a note of all of them :/ ).
    Two major changes which allowed the server to work (sort of) were:
    (1) Load the right key file for the two SSL sites, one which doesn't have the passphrase in it!
    (2) Add "AcceptMutex fcntl" to Apache's config file.
    Now what happens is that Apache runs but once/minute this happens:
    Code:
    [emerg] (22)Invalid argument: mod_fcgid: can't lock process table in PM, pid 1680
    [error] mod_fcgid: fcgid process manager died, restarting the server
    [notice] SIGHUP received.  Attempting to restart
    and another process like this appears:
    Code:
      3378     ispconfig     18:06     /usr/bin/php-cgi -d magic_quotes_gpc=off -d session.save_path=/usr/local/ispconf ...
          3380     ispconfig     18:06     /usr/bin/php-cgi -d magic_quotes_gpc=off -d session.save_path=/usr/local/ispconf ...
    ...and Apache restarts. Using web sites being served, you don't see this happening and as the server is, currently, low traffic, it's not a big deal... apart from, I have to kill a load of processes every now & again.

    I have searched for an answer to this problem and done some investigating, so far without success. I'm guessing that, unless you've seen this before, you'd need more information on the environment. This is the basic stuff - let me know what else you might need.
    Operating system Ubuntu Linux 12.04.5
    Kernel and CPU Linux 3.14.32-xxxx-grs-ipv6-64 on x86_64
    Processor Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz, 8 cores
    RAM 32 GB
    Disc 2 TB, mirrored
    Apache version 2.2.22
    MySQL version 5.5.46
    php5 5.3.10-1ubuntu3.21
    libapache2-mod-fcgid 1:2.3.6-1.1ubuntu0.1

    ISPConfig Apache VHost:
    Code:
      ServerAdmin [email protected]
     
      <FilesMatch "\.ph(p3?|tml)$">
        SetHandler None
      </FilesMatch>
     
      <IfModule mod_fcgid.c>
    DocumentRoot /var/www/ispconfig/
        SuexecUserGroup ispconfig ispconfig
        <Directory /var/www/ispconfig/>
          Options -Indexes +FollowSymLinks +MultiViews +ExecCGI
          AllowOverride AuthConfig Indexes Limit Options FileInfo
          <FilesMatch "\.php$">
              SetHandler fcgid-script
          </FilesMatch>
          FCGIWrapper /var/www/php-fcgi-scripts/ispconfig/.php-fcgi-starter .php
                Order allow,deny
          Allow from all
              </Directory>
        IPCCommTimeout  7200
        MaxRequestLen 15728640
      </IfModule>
     
      <IfModule mpm_itk_module>
        DocumentRoot /usr/local/ispconfig/interface/web/
        AssignUserId ispconfig ispconfig
        AddType application/x-httpd-php .php
        <Directory /usr/local/ispconfig/interface/web>
          # php_admin_value open_basedir "/usr/local/ispconfig/interface:/usr/share:/tmp"
          Options +FollowSymLinks
          AllowOverride None
                Order allow,deny
          Allow from all
                php_value magic_quotes_gpc        0
        </Directory>
      </IfModule>
     
      # ErrorLog /var/log/apache2/error.log
      # CustomLog /var/log/apache2/access.log combined
      ServerSignature Off
     
      <IfModule mod_security2.c>
        SecRuleEngine Off
      </IfModule>
    
      # SSL Configuration
      SSLEngine On
      SSLProtocol All -SSLv2 -SSLv3
      SSLCertificateFile /usr/local/ispconfig/interface/ssl/ispserver.crt
      SSLCertificateKeyFile /usr/local/ispconfig/interface/ssl/ispserver.key
      #SSLCACertificateFile /usr/local/ispconfig/interface/ssl/ispserver.bundle
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    You have to load always a key without password like ispconfig is doing it. Otherwise apache cant start as it will wait that you enter a password.

    Never had to do that on any server yet.

    Regarding the fcgid problem, try to switch to php-fpm and set the mode to ondemand on the options tab of the website.

    Beside that, you should consider to update to a recent Ubuntu version, 12.04 is really old. Update e.g. to 14.04 LTS.
     
  3. ITFixt

    ITFixt New Member HowtoForge Supporter

    I know - I just made a mistake at first & used the wrong file contents. In fact, it did start but no SSL sites were
    accessible as Mod_SSL simply refused to work. Other sites were OK. I was surprised at that - I thought it wouldn't have started :/
    Later: I did an Apache Reload - I now know that that's what happens if you only reload. If you stop/start Apache, it won't.
    Neither have I! But, according to the posts I found elsewhere it's not that uncommon. I find it very odd that the thing that "broke" Apache was changing from a self-signed CRT to a real one :?
    As far as I can see, the website that is causing the problem is the ISPConfig UI - as time goes on, the number of processes like this:
    Code:
    /usr/bin/php-cgi -d magic_quotes_gpc=off -d session.save_path=/usr/local/ispconfig/interface/temp
    increases. What I haven't though to check, up until now, is if a new one occurs *only* if I am using the UI whilst Apache crashes. If I'm wrong, I can't see how to find out which site is causing the problem - the log entries don't identify it.
    It is - but the whole point of an LTS release is that all important updates are backported so you don't have to change release every two years. A case of not fixing it if it isn't broken (yes, I know, mine now appears to be :} ) I have to admit, if I could be totally sure that an upgrade wouldn't break my server,I would seriously consider it... but I can't, so the only way I would do it would be to rent a new server, clean install the latest LTS version and migrate domains one at time. However, that said, I'd value your opinion on the subject.
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    As far as I have seen in the posts on that error, there was a bug in apache so that mod_fcgi processes pile up. Therefore my recommendation to try to switch to php-fpm and / r update Ubuntu.

    What you can try is that you take the php fcgi parameters from one of the other websites and add these into the ispconfig vhost. ispconfig is just a normal website, so there is nothing special in regards of php handling.
     
  5. ITFixt

    ITFixt New Member HowtoForge Supporter

    Yes and yes. I only had 3 or 4 sites using FCGID and I'd changed them to SuPHP. I have some sites on PHP-FPM but TBH I can't make up my mind which is best for each site. Given their low traffic rate ATM, it probably makes no difference. Anyway... as the FCGID errors persist, it has to be the ISPConfig UI causing the crash - I don't mean it's ISPConfig's fault, I think it more likely to be an FCGID problem (or a configuration error on my part :/).
    Damn - as I now don't have any FCGI sites running, I've nothing t copy from! Is there any reason why the ISPConfig UI won't run under SuPHP, as long as I get the directory, Group & User names etc. right?
    I've put together an SuPHP vhost config file for ISPCOnfig but it needs the location of a php.ini file. Any ideas, Till?
    Code:
    <VirtualHost XX.XX.XX.XX:8080>
    
    DocumentRoot /var/www/ispconfig/
    ServerAdmin [email protected]
    
    ErrorLog /var/log/ispconfig/httpd/a1shopdirect.com/error.log
    Alias /error/ "/var/www/a1shopdirect.com/web/error/"
    
      <FilesMatch "\.ph(p3?|tml)$">
        SetHandler None
      </FilesMatch>
    
        <Directory /var/www/ispconfig/>
          Options -Indexes +FollowSymLinks +MultiViews +ExecCGI
          AllowOverride AuthConfig Indexes Limit Options FileInfo
            Order allow,deny
          Allow from all
        </Directory>
    
    # suphp enabled
    <Directory /var/www/ispconfig/>
      <IfModule mod_suphp.c>
        suPHP_Engine on
        suPHP_UserGroup ispconfig ispconfig
        suPHP_ConfigPath ?????????????????????????
        <FilesMatch "\.php[345]?$">
          SetHandler x-httpd-suphp
        </FilesMatch>
        suPHP_AddHandler x-httpd-suphp
      </IfModule>
    </Directory>
    
    # add support for apache mpm_itk
    <IfModule mpm_itk_module>
      DocumentRoot /usr/local/ispconfig/interface/web/
      AssignUserId ispconfig ispconfig
      AddType application/x-httpd-php .php
      <Directory /usr/local/ispconfig/interface/web>
        Options +FollowSymLinks
        AllowOverride None
        Order allow,deny
        Allow from all
        php_value magic_quotes_gpc 0
      </Directory>
    </IfModule>
    
    <IfModule mod_security2.c>
        SecRuleEngine Off
      </IfModule>
    
      ServerSignature Off
    
      # SSL Configuration
      SSLEngine On
      SSLProtocol All -SSLv2 -SSLv3
      SSLCertificateFile /usr/local/ispconfig/interface/ssl/ispserver.crt
      SSLCertificateKeyFile /usr/local/ispconfig/interface/ssl/ispserver.key
      #SSLCACertificateFile /usr/local/ispconfig/interface/ssl/ispserver.bundle
    </VirtualHost>
    In parallel, I am going to remove the "AcceptMutex fcntl" and get a capture of the original error messages, then try again to see if there is a solution for that. I will also look into updating FCGID as the Ubuntu version is way behind, as is so often the case.

    Thanks again for your time & patience :)
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Suphp is outdated and dead slow, it should not be used anymore these days. When your server supports php-fpm then php-fpm is the best option as it is fast, efficient, secure and works with all sites, don't use suphp at all, this option exist for legacy reasons as suphp was a good option 10 years ago.

    Theoretically ISPConfig should work on suphp, it will be slow and has a high resource usage but if you really want to use it, then reconfigure the vhost. There are more than a million servers that use ISPConfig and none of them has such a problem as far as I know, so your issue is definitely something in your config as all ispconfig servers use the exact same vhost file without an issue.

    You can try these fchgi options:

    FcgidIdleTimeout 300
    FcgidProcessLifeTime 3600
    # FcgidMaxProcesses 1000
    FcgidMaxRequestsPerProcess 250
    FcgidMinProcessesPerClass 0
    FcgidMaxProcessesPerClass 10
    FcgidConnectTimeout 3
    FcgidIOTimeout 600
    FcgidBusyTimeout 3600
    FcgidMaxRequestLen 1073741824

    and if they have no effect try these ones (which are for a older fcgi version):

    IdleTimeout 300
    ProcessLifeTime 3600
    # MaxProcessCount 1000
    DefaultMinClassProcessCount 0
    DefaultMaxClassProcessCount 100
    IPCConnectTimeout 3
    IPCCommTimeout 600
    BusyTimeout 3600
     
  7. ITFixt

    ITFixt New Member HowtoForge Supporter

    Thanks Till. Yes, I finally figured out that this is almost certainly not an ISPC problem but something odd about the configuration of my server.

    I have updated mod_fcgi to the latest version from the Apache source and that made no difference. I tried your FCGID param suggestions, both types, both in the VHost definition and in the mod_fcgid.conf file but none of those changes made any difference.

    I then had to reboot the machine anyway for an OpenSSL update. On looking at the Apache error log after that, I noticed that the FCGID error messages were missing but the SIGTERM to Apache was still happening once/minute. As soon as I used the ISPC UI, the FCGI message started appearing again, too.

    So: I realised that the FCGID errors were not the cause of the problem but a *Symptom*. Too cut a long story short: if I disable the one/minute ISPC CRON job, Apache does not get restarted every minute. I have had a quick glance at the four files server.php, check.php, app.inc.php and config.inc.php and I can't see anything untoward there. I am still getting these messages in the ISPC CRON log, at one minute intervals:

    Code:
    Fri Jan 8 16:45:02 CET 2016 sh: -c: line 0: unexpected EOF while looking for matching `&quot;'
    Fri Jan 8 16:45:02 CET 2016 sh: -c: line 1: syntax error: unexpected end of file
    Fri Jan 8 16:50:02 CET 2016 sh: -c: line 0: unexpected EOF while looking for matching `&quot;'
    Fri Jan 8 16:50:02 CET 2016 sh: -c: line 1: syntax error: unexpected end of file
    
    and a new one:
    Code:
    Fri Jan 8 16:41:01 CET 2016 usage: ipcrm [ [-q msqid] [-m shmid] [-s semid]
    Fri Jan 8 16:41:01 CET 2016 [-Q msgkey] [-M shmkey] [-S semkey] ... ]
    which happens irregularly - I suspect when I have manually stopped Apache to fix something.

    I did discover that there is a setting for FCGID syntax 2.0/2.2 which was set to 2.0. As I'm running 2.2, I changed that and, lo, the FCGID errors have stopped! So, a partial result :)

    And now I'm stuck again :/
     
  8. ITFixt

    ITFixt New Member HowtoForge Supporter

    I finally remembered Debug mode. If I set it and run the 1/minute CRON job manually (through Webmin), this is what I get in the ISPC log:

    Code:
    08.01.2016-22:11 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    08.01.2016-22:11 - DEBUG - Found 2 changes, starting update process.
    08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
    08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
    08.01.2016-22:11 - DEBUG - Network configuration disabled in server settings.
    08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
    08.01.2016-22:11 - DEBUG - Processed datalog_id 2294
    08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
    08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
    08.01.2016-22:11 - DEBUG - Network configuration disabled in server settings.
    08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
    08.01.2016-22:11 - DEBUG - Processed datalog_id 2295
    08.01.2016-22:11 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    08.01.2016-22:11 - DEBUG - Restarting httpd: service apache2 restart
    08.01.2016-22:11 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    There are 6 web_domain updates in the jobqueue but they are for another server and they are not being processed because the other server is not currently available.

    The reason Network Settings is disabled is because I administer that in Webmin (been doing that longer than I've been using ISPC & just never changed).
     
  9. ITFixt

    ITFixt New Member HowtoForge Supporter

    Sorry to add to this... I've just discovered that most web processes are running under www-data and not their "own" group/userid. Apart from the security aspect, this has broken some websites. This happens if suexec is set or not in ISPC for the site. I *think* this happened since I updated mod_fcgid but it might be since I changed the syntax setting in ISPC from 2.0 to 2.2. Looking at a VHost file, there's some extra lines that weren't there last time I looked... and some I would not expect to see if the site were running under PHP-FPM. I did wonder if it was changing the sites to On Demand for PHP-FPM, so I tried changing one back to Dynamic but that had no effect.

    I won't be around most of tomorrow to take any action wit the web server, so if you can spare some time to think about this lot, no rush :}

    I'm beginning to think there's something seriously broken on this server :(
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Please dont mix up apache and php processes. apache processes run alsways as www-data as they dont execute scripts, they have only read access to the website. Only the website php-fpm or fcgi processes run as web user.
     
  11. ITFixt

    ITFixt New Member HowtoForge Supporter

    Sorry Till, I know that but wasn't being very clear. I shouldn't have tried fixing that at 1 in the morning :/
    Sites which *should* be running under php-fpm and should therefore be running under clientxx/webyy (which is now almost all of them) are not - they're running under apache (I presume) and www-data.

    I cannot find anything in the logs which would indicate why this is happening. I have restarted php-fpm then restarted Apache after doing a web site resync in ISPC.

    Is there any information I can post that might make it easier to see what is going on?

    I have been using a similar setup for several years now and hadn't really gone into things very deeply - they just worked. I think it's now about time I learnt a lot more :/
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    I guess you mix up the apache and php processes here, apache processes have to run as www-data. and not the web user, only PHP processes shall run as webX user. If you followed the perfect server guide to install your server and created a website by just clicking on new website and enetring the domain name, then all processes will run under their correct users (apache as www-data and php under the web user), there is no a need to reconfigure anything as the defaults are the right configuration in ISPConfig.

    Create a phpinfo() file in the site and check the output, you can see the php handler used by that site there.

    And you can test easiyl if a site runs under the right user. Add a file test.php in the web folder of s site with this content:

    <?php
    touch('test.txt');
    ?>

    open the test.php file in a browser, then check the owner of the newly created file test.txt (not test.php!) in the web folder.
     
  13. ITFixt

    ITFixt New Member HowtoForge Supporter

    Hi Till - I wasn't expecting a reply on a Sunday :)
    .
    I understand that perfectly - that was my attempt to explain what was happening. I'll try again....

    The sites were created normally in ISPConfig. Originally, some were running under php-fpm, some under fcgi and some under suphp. There were historical reasons for that, as well as me not doing enough research into the best way to run them and keeping up with developments.

    When you said about php-fpm being the best way to run sites generally (at least, that's what I understood), I changed virtually all of them to php-fpm. Once all the updates had been applied, I checked and virtually all of them were running under Apache as www. The only one running under php-fpm was one which was not enabled in ISPC!
    That's what used to happen but it isn't now. If I create a new site and set it to run under php-fpm, it runs under Apache instead. Just in case, I tried it both with suexec on and off. Just FYI, the server setup isn't totally as per the Perfect setup - I don't use either of the two firewalls recommended, nor fail2ban: I use CSF.

    I have overcome the problem temporarily by reverting to fcgid and suexec. Performance is not an issue ATM, so I can now resolve the real problem without rushing and I have an operational server. BTW, I'm pretty sure this problem originated with one of two events:
    Either
    1) Changing the fcgi syntax in ISPC settings from 2.0 to 2.2
    or
    2) updating mod-fcgid manually (downloading the source/config/make/install)
    I did a web site resync in ISPC but that hasn't helped. Oh, and the ISPC job queue is now empty.

    One other thing 9and this may be a separate problem but it's what started this off): the 1/minute ISPC Cron job is still restarting Apache every time it runs, regardless of whether there is anything in the job queue. For the moment, I have that CRON job de-activated and I run it manually if I make any change in ISPC.

    I did write a bit of php myself to check what was going on with a couple of the sites and it confirmed that they were running under www-data despite being configured to run under php-fpm. I can actually see this easily anyway by using the process display in Webmin but I wanted to be sure.

    I have a feeling that I've done something that has broken the server in a way I probably can't fix, so I will probably have to rebuild it in the near future, although if you have any suggestions that might help me fix the problems, I'd be very grateful.

    Thanks again for all your help & patience.
     

Share This Page