HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   Installation/Configuration (http://www.howtoforge.com/forums/forumdisplay.php?f=16)
-   -   Apache2 server stopped working and fails restart without error (http://www.howtoforge.com/forums/showthread.php?t=19942)

chillifire 30th January 2008 09:43

Apache2 server stopped working and fails restart without error
 
Hi,

I am running ISPConfig 2.1.18 on Ubuntu 7.10.

After months of troublefree operations Apache stopped working today. It fails on restart - with no error messages, which does not help. I made a number of changes today through ISPConfig and I am uncertain what may have triggered the issue.

I added a new IP address, I added a new SSL certificate for a subdomain (which has the IP address assigned). I created a few more DNS records. That's about it. The new IP address and the new SSL certificate seemed to be working with no problems. the new DNS entries were mainly CNAME entries and an update of the SPF entry for the new IP address. Again, nothing earthshattering one would have thought.

Yes, I also tried to insatll ImageMagick from source, which failed, uninstalled it, and reinstalled it with apt-get install. I cannot think that that might have had any relevant though, as apache was working for several hours after that.

The apache errorlog shows nothing too exciting. Whether the issues are related to the server going down is not clear to me. Attempts to restart the server just fail without any entry in the log or message in ssh mode.

netstat -tap output is
Code:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address          Foreign Address        State      PID/Program name
tcp        0      0 *:mysql                *:*                    LISTEN    1184/mysqld
tcp        0      0 *:81                    *:*                    LISTEN    1610/ispconfig_http
tcp        0      0 210-48-62-43.sit:domain *:*                    LISTEN    1783/named
tcp        0      0 blackbird.chilli:domain *:*                    LISTEN    1783/named
tcp        0      0 localhost.locala:domain *:*                    LISTEN    1783/named
tcp        0      0 localhost.localadmi:953 *:*                    LISTEN    1783/named
tcp        0      0 *:smtp                  *:*                    LISTEN    1735/master
tcp        0      0 blackbird.chillifi:4095 210-48-62-11.site:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillif:mysql blackbird.chillifi:1892 ESTABLISHED1184/mysqld
tcp        0      0 blackbird.chillif:mysql blackbird.chillifi:1893 ESTABLISHED1184/mysqld
tcp        0      0 blackbird.chillif:mysql blackbird.chillifi:1894 ESTABLISHED1184/mysqld
tcp        0      0 blackbird.chillifi:1891 blackbird.chillif:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillifi:1890 blackbird.chillif:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillifi:1893 blackbird.chillif:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillifi:1892 blackbird.chillif:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillifi:1894 blackbird.chillif:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillif:mysql blackbird.chillifi:1890 ESTABLISHED1184/mysqld
tcp        0      0 blackbird.chillif:mysql blackbird.chillifi:1891 ESTABLISHED1184/mysqld
tcp        0      0 blackbird.chillifi:4099 210-48-62-11.site:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillifi:4098 210-48-62-11.site:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillifi:4097 210-48-62-11.site:mysql ESTABLISHED1494/freeradius
tcp        0      0 blackbird.chillifi:4096 210-48-62-11.site:mysql ESTABLISHED1494/freeradius
tcp6      0      0 *:imaps                *:*                    LISTEN    1299/couriertcpd
tcp6      0      0 *:pop3s                *:*                    LISTEN    1343/couriertcpd
tcp6      0      0 *:pop3                  *:*                    LISTEN    1319/couriertcpd
tcp6      0      0 *:imap2                *:*                    LISTEN    1275/couriertcpd
tcp6      0      0 *:ftp                  *:*                    LISTEN    1816/proftpd: (acce
tcp6      0      0 *:ssh                  *:*                    LISTEN    1088/sshd
tcp6      0      0 ip6-localhost:953      *:*                    LISTEN    1783/named
tcp6      0      0 *:smtp                  *:*                    LISTEN    1735/master
tcp6      0  3700 blackbird.chillifir:ssh 60-234-129-51.bit:53274 ESTABLISHED1884/0

httpd -t returns:
Code:

Syntax OK
df -h returns
Code:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1            5.0G  1.6G  3.1G  35% /
varrun                65M  88K  64M  1% /var/run
varlock                65M    0  65M  0% /var/lock
udev                  65M  16K  65M  1% /dev
devshm                65M    0  65M  0% /dev/shm

Last half-hour of the /var/log/apache2/error.log:
Code:

[Wed Jan 30 00:56:21 2008] [notice] Graceful restart requested, doing restart
[Wed Jan 30 00:56:21 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 00:56:21 2008] [notice] Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6.2 mod_ssl/2.2.4 OpenSSL/0.9.8e configured -- resuming normal operations
[Wed Jan 30 00:56:21 2008] [warn] long lost child came home! (pid 8808)
[Wed Jan 30 00:57:03 2008] [notice] Graceful restart requested, doing restart
[Wed Jan 30 00:57:03 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 00:57:03 2008] [notice] Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6.2 mod_ssl/2.2.4 OpenSSL/0.9.8e configured -- resuming normal operations
[Wed Jan 30 00:57:03 2008] [warn] long lost child came home! (pid 9539)
[Wed Jan 30 00:58:45 2008] [notice] caught SIGWINCH, shutting down gracefully
[Wed Jan 30 00:58:45 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 00:58:45 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 00:58:45 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 00:58:45 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 00:58:45 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 00:58:56 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Wed Jan 30 00:58:56 2008] [notice] Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6.2 mod_ssl/2.2.4 OpenSSL/0.9.8e configured -- resuming normal operations
[Wed Jan 30 01:15:58 2008] [notice] caught SIGWINCH, shutting down gracefully
[Wed Jan 30 01:15:58 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:15:58 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:15:58 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:15:58 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:15:58 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:16:08 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Wed Jan 30 01:16:08 2008] [notice] Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6.2 mod_ssl/2.2.4 OpenSSL/0.9.8e configured -- resuming normal operations
[Wed Jan 30 01:16:19 2008] [notice] caught SIGWINCH, shutting down gracefully
[Wed Jan 30 01:16:19 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:16:19 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:16:19 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:16:19 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:16:19 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:16:29 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Wed Jan 30 01:16:30 2008] [notice] Apache/2.2.4 (Ubuntu) PHP/5.2.3-1ubuntu6.2 mod_ssl/2.2.4 OpenSSL/0.9.8e configured -- resuming normal operations
[Wed Jan 30 01:23:53 2008] [error] an unknown filter was not added: PHP
[Wed Jan 30 01:23:53 2008] [error] an unknown filter was not added: PHP
[Wed Jan 30 01:24:11 2008] [error] [client 60.234.129.51] File does not exist: /var/www/sharedip/hotspotlogin.php
[Wed Jan 30 01:25:00 2008] [error] an unknown filter was not added: PHP
[Wed Jan 30 01:25:00 2008] [error] an unknown filter was not added: PHP
[Wed Jan 30 01:25:01 2008] [error] an unknown filter was not added: PHP
[Wed Jan 30 01:25:01 2008] [error] an unknown filter was not added: PHP
[Wed Jan 30 01:28:43 2008] [notice] caught SIGWINCH, shutting down gracefully
[Wed Jan 30 01:28:43 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:43 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:43 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:43 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:43 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:44 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:44 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:44 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:44 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Wed Jan 30 01:28:44 2008] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)


Needless to say I am in a medium size panic:eek: . Any input is welcome.

chillifire

chillifire 30th January 2008 22:31

Resolved
 
Hi,
the mistery is resolved - but remains a mistery :)

I switched off apache modules one by one to see what caused the issue.

PHP was fine it seems, so in my view the failed Imagick installation from source did not impact the server.

SSL finally prooved to be the problem. Apache started fine without that module. Out of curiosity I restarted SMTP and POP3 - they also failed to restart - I believe they also use SSL libraries. Interesting. This would explain (although not very satisfactorily) why Apache did not report a problem - the problem was within SSL itself.

So I figured it was the new SSL key or the new IP address. I reissued an new key for the certificate, enteredit into ISPconfig and lo - it works again.

I find this really bemusing. A signed SSL certificate (maybe faulty? maybe not?, who knows, but certainly from a reputable vendor) brings down apache so it fails on restart without error. I would not have htought this was possible. One would have thought exceptions like thesee are cought, rather than bring down all services using SSL.

Anyway - we live and learn.

Thanks for everyone who read the post - although I had no response. I hope this is going to help someone else in the future.

bswinnerton 13th March 2008 16:21

I'm having this same problem right now. It could be the way that I put in the certificate i suppose, so how did you get this working?

bswinnerton 13th March 2008 16:26

neeeever mind i got it =)


All times are GMT +2. The time now is 10:36.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.