HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   General (http://www.howtoforge.com/forums/forumdisplay.php?f=25)
-   -   Production Server "httpd is down! Rescue will not help!" (http://www.howtoforge.com/forums/showthread.php?t=59938)

lusso64 12th December 2012 18:10
Production Server "httpd is down! Rescue will not help!"
 
Hi all,

I have a production server that until this morning was stable and trouble free. Ubuntu 10.10, ISPConfig 3.0.4.6

I can login to the control panel and view the "httpd is down! Rescue will not help!" error in the ISPConfig Log.

websites are down
email is down
FTP is up

Status of Services tab shows everything working.

I am a little lost on where to start with this problem. I have tried to restart apache and get the following:

Code:
root@websrv:/home/administrator# apache2 -k restart
apache2: bad user name ${APACHE_RUN_USER}
I have checked /etc/apache2/apache2.conf and it refers me to /etc/apache2/envvars which is as follows:

Code:
# envvars - default environment variables for apache2ctl

# this won't be correct after changing uid
unset HOME

# for supporting multiple apache2 instances
if [ "${APACHE_CONFDIR##/etc/apache2-}" != "${APACHE_CONFDIR}" ] ; then
        SUFFIX="-${APACHE_CONFDIR##/etc/apache2-}"
else
        SUFFIX=
fi

# Since there is no sane way to get the parsed apache2 config in scripts, some
# settings are defined via environment variables and then used in apache2ctl,
# /etc/init.d/apache2, /etc/logrotate.d/apache2, etc.
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_PID_FILE=/var/run/apache2$SUFFIX.pid
export APACHE_RUN_DIR=/var/run/apache2$SUFFIX
export APACHE_LOCK_DIR=/var/lock/apache2$SUFFIX
# Only /var/log/apache2 is handled by /etc/logrotate.d/apache2.
export APACHE_LOG_DIR=/var/log/apache2$SUFFIX

## The locale used by some modules like mod_dav
export LANG=C
## Uncomment the following line to use the system default locale instead:
#. /etc/default/locale

export LANG

## The command to get the status for 'apache2ctl status'.
## Some packages providing 'www-browser' need '--dump' instead of '-dump'.
#export APACHE_LYNX='www-browser -dump'
I'm kinda lost so any help would be gratefully appreciated!

Thanks, Dave

till 12th December 2012 19:47
To restart apache, please run:

/etc/init.d/apache2 restart

If it does not restart, then post the errors that you get on the screen and the ones that you get in the apache error.log

Regarding the mail system: Please post the errors from your mail.log and restart postfix, amavis and clamav. Use the init scripts to restart the services like I explained above for apache.

lusso64 12th December 2012 20:34
Apache:
Code:
root@websrv:/var/log/ispconfig# /etc/init.d/apache2 restart
 * Restarting web server apache2                                                                                                                            [Wed Dec 12 11:30:21 2012] [warn] NameVirtualHost 192.168.2.25:80 has no VirtualHosts
[Wed Dec 12 11:30:21 2012] [warn] NameVirtualHost 192.168.2.25:443 has no VirtualHosts
 ... waiting [Wed Dec 12 11:30:22 2012] [warn] NameVirtualHost 192.168.2.25:80 has no VirtualHosts
[Wed Dec 12 11:30:22 2012] [warn] NameVirtualHost 192.168.2.25:443 has no VirtualHosts
                                                                                                                                                      [ OK ]
Apache error.log:

Code:
[Wed Dec 12 11:30:21 2012] [notice] caught SIGTERM, shutting down
[Wed Dec 12 11:30:22 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Wed Dec 12 11:30:22 2012] [notice] Digest: generating secret for digest authentication ...
[Wed Dec 12 11:30:22 2012] [notice] Digest: done
[Wed Dec 12 11:30:22 2012] [notice] Apache/2.2.16 (Ubuntu) DAV/2 mod_fcgid/2.3.5 PHP/5.3.3-1ubuntu9.5 with Suhosin-Patch mod_ruby/1.2.6 Ruby/1.8.7(2010-06-23) configured -- resuming normal operations

lusso64 12th December 2012 20:44
Email log BEFORE restart:

Code:
Dec 12 11:33:01 websrv pop3d: Disconnected, ip=[::1]
Dec 12 11:33:01 websrv imapd: Connection, ip=[::1]
Dec 12 11:33:01 websrv imapd: Disconnected, ip=[::1], time=0
Dec 12 11:34:01 websrv postfix/smtpd[4960]: connect from localhost[127.0.0.1]
Dec 12 11:34:01 websrv postfix/smtpd[4960]: lost connection after CONNECT from localhost[127.0.0.1]
Dec 12 11:34:01 websrv postfix/smtpd[4960]: disconnect from localhost[127.0.0.1]
Dec 12 11:34:01 websrv pop3d: Connection, ip=[::1]
Dec 12 11:34:01 websrv pop3d: Disconnected, ip=[::1]
Dec 12 11:34:01 websrv imapd: Connection, ip=[::1]
Dec 12 11:34:01 websrv imapd: Disconnected, ip=[::1], time=0
Dec 12 11:35:01 websrv postfix/smtpd[4960]: connect from localhost[127.0.0.1]
Dec 12 11:35:01 websrv postfix/smtpd[4960]: lost connection after CONNECT from localhost[127.0.0.1]
Dec 12 11:35:01 websrv postfix/smtpd[4960]: disconnect from localhost[127.0.0.1]
Dec 12 11:35:01 websrv pop3d: Connection, ip=[::1]
Dec 12 11:35:01 websrv pop3d: Disconnected, ip=[::1]
Dec 12 11:35:01 websrv imapd: Connection, ip=[::1]
Dec 12 11:35:01 websrv imapd: Disconnected, ip=[::1], time=0
Dec 12 11:35:01 websrv postfix/smtpd[4960]: connect from localhost[127.0.0.1]
Dec 12 11:35:01 websrv postfix/smtpd[4960]: lost connection after CONNECT from localhost[127.0.0.1]
Dec 12 11:35:01 websrv postfix/smtpd[4960]: disconnect from localhost[127.0.0.1]
Dec 12 11:35:01 websrv pop3d: Connection, ip=[::1]
Dec 12 11:35:01 websrv pop3d: Disconnected, ip=[::1]
Dec 12 11:35:01 websrv imapd: Connection, ip=[::1]
Dec 12 11:35:01 websrv imapd: Disconnected, ip=[::1], time=0
Dec 12 11:36:01 websrv postfix/smtpd[4960]: connect from localhost[127.0.0.1]
Dec 12 11:36:01 websrv postfix/smtpd[4960]: lost connection after CONNECT from localhost[127.0.0.1]
Dec 12 11:36:01 websrv postfix/smtpd[4960]: disconnect from localhost[127.0.0.1]
Dec 12 11:36:01 websrv pop3d: Connection, ip=[::1]
Dec 12 11:36:01 websrv pop3d: Disconnected, ip=[::1]
Dec 12 11:36:01 websrv imapd: Connection, ip=[::1]
Dec 12 11:36:01 websrv imapd: Disconnected, ip=[::1], time=0

Restarting:

Code:
root@websrv:/etc/init.d# /etc/init.d/clamav-daemon restart
 * Stopping ClamAV daemon clamd                                                                                                                      [ OK ]
 * Starting ClamAV daemon clamd                                                                                                                              LibClamAV Warning: ***********************************************************
LibClamAV Warning: ***  This version of the ClamAV engine is outdated.    ***
LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***
LibClamAV Warning: ***********************************************************
LibClamAV Warning: ***********************************************************
LibClamAV Warning: ***  This version of the ClamAV engine is outdated.    ***
LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***
LibClamAV Warning: ***********************************************************
LibClamAV Warning: ***********************************************************
LibClamAV Warning: ***  This version of the ClamAV engine is outdated.    ***
LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***
LibClamAV Warning: ***********************************************************
                                                                                                                                                      [ OK ]
root@websrv:/etc/init.d# /etc/init.d/clamav-freshclam restart
 * Stopping ClamAV virus database updater freshclam                                                                                                  [ OK ]
 * Starting ClamAV virus database updater freshclam                                                                                                  [ OK ]
root@websrv:/etc/init.d# /etc/init.d/amavis restart
Stopping amavisd: amavisd-new.
Starting amavisd: amavisd-new.
root@websrv:/etc/init.d# /etc/init.d/postfix restart
 * Stopping Postfix Mail Transport Agent postfix                                                                                                      [ OK ]
 * Starting Postfix Mail Transport Agent postfix                                                                                                      [ OK ]
root@websrv:/etc/init.d#

Mail.log of restarts:
Code:
Dec 12 11:40:39 websrv amavis[7210]: starting.  /usr/sbin/amavisd-new at websrv.tynerroberts.com amavisd-new-2.6.4 (20090625), Unicode aware, LANG="en_US.UTF-8"
Dec 12 11:40:39 websrv amavis[7210]: Perl version              5.010001
Dec 12 11:40:40 websrv amavis[7215]: Net::Server: Group Not Defined.  Defaulting to EGID '116 116'
Dec 12 11:40:40 websrv amavis[7215]: Net::Server: User Not Defined.  Defaulting to EUID '107'
Dec 12 11:40:40 websrv amavis[7215]: Module Amavis::Conf        2.207
Dec 12 11:40:40 websrv amavis[7215]: Module Archive::Zip        1.30
Dec 12 11:40:40 websrv amavis[7215]: Module BerkeleyDB          0.42
Dec 12 11:40:40 websrv amavis[7215]: Module Compress::Zlib      2.02
Dec 12 11:40:40 websrv amavis[7215]: Module Convert::TNEF      0.17
Dec 12 11:40:40 websrv amavis[7215]: Module Convert::UUlib      1.12
Dec 12 11:40:40 websrv amavis[7215]: Module Crypt::OpenSSL::RSA 0.25
Dec 12 11:40:40 websrv amavis[7215]: Module DBD::mysql          4.016
Dec 12 11:40:40 websrv amavis[7215]: Module DBI                1.611
Dec 12 11:40:40 websrv amavis[7215]: Module DB_File            1.82
Dec 12 11:40:40 websrv amavis[7215]: Module Digest::MD5        2.39
Dec 12 11:40:40 websrv amavis[7215]: Module Digest::SHA        5.47
Dec 12 11:40:40 websrv amavis[7215]: Module IO::Socket::INET6  2.54
Dec 12 11:40:40 websrv amavis[7215]: Module MIME::Entity        5.427
Dec 12 11:40:40 websrv amavis[7215]: Module MIME::Parser        5.427
Dec 12 11:40:40 websrv amavis[7215]: Module MIME::Tools        5.427
Dec 12 11:40:40 websrv amavis[7215]: Module Mail::DKIM::Signer  0.38
Dec 12 11:40:40 websrv amavis[7215]: Module Mail::DKIM::Verifier 0.38
Dec 12 11:40:40 websrv amavis[7215]: Module Mail::Header        2.06
Dec 12 11:40:40 websrv amavis[7215]: Module Mail::Internet      2.06
Dec 12 11:40:40 websrv amavis[7215]: Module Mail::SPF          v2.007
Dec 12 11:40:40 websrv amavis[7215]: Module Mail::SpamAssassin  3.003001
Dec 12 11:40:40 websrv amavis[7215]: Module Net::DNS            0.66
Dec 12 11:40:40 websrv amavis[7215]: Module Net::Server        0.97
Dec 12 11:40:40 websrv amavis[7215]: Module NetAddr::IP        4.028
Dec 12 11:40:40 websrv amavis[7215]: Module Socket6            0.23
Dec 12 11:40:40 websrv amavis[7215]: Module Time::HiRes        1.9719
Dec 12 11:40:40 websrv amavis[7215]: Module URI                1.54
Dec 12 11:40:40 websrv amavis[7215]: Module Unix::Syslog        1.1
Dec 12 11:40:40 websrv amavis[7215]: Amavis::DB code      loaded
Dec 12 11:40:40 websrv amavis[7215]: Amavis::Cache code  loaded
Dec 12 11:40:40 websrv amavis[7215]: SQL base code        loaded
Dec 12 11:40:40 websrv amavis[7215]: SQL::Log code        NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: SQL::Quarantine      NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: Lookup::SQL code    loaded
Dec 12 11:40:40 websrv amavis[7215]: Lookup::LDAP code    NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: AM.PDP-in proto code loaded
Dec 12 11:40:40 websrv amavis[7215]: SMTP-in proto code  loaded
Dec 12 11:40:40 websrv amavis[7215]: Courier proto code  NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: SMTP-out proto code  loaded
Dec 12 11:40:40 websrv amavis[7215]: Pipe-out proto code  NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: BSMTP-out proto code NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: Local-out proto code loaded
Dec 12 11:40:40 websrv amavis[7215]: OS_Fingerprint code  NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: ANTI-VIRUS code      loaded
Dec 12 11:40:40 websrv amavis[7215]: ANTI-SPAM code      loaded
Dec 12 11:40:40 websrv amavis[7215]: ANTI-SPAM-EXT code  NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: ANTI-SPAM-C code    NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: ANTI-SPAM-SA code    loaded
Dec 12 11:40:40 websrv amavis[7215]: Unpackers code      loaded
Dec 12 11:40:40 websrv amavis[7215]: DKIM code            loaded
Dec 12 11:40:40 websrv amavis[7215]: Tools code          NOT loaded
Dec 12 11:40:40 websrv amavis[7215]: Found $file            at /usr/bin/file
Dec 12 11:40:40 websrv amavis[7215]: No $altermime,        not using it
Dec 12 11:40:40 websrv amavis[7215]: Internal decoder for .mail
Dec 12 11:40:40 websrv amavis[7215]: No decoder for      .F
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .Z    at /bin/uncompress
Dec 12 11:40:40 websrv amavis[7215]: Internal decoder for .gz
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .bz2  at /bin/bzip2 -d
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .lzo  at /usr/bin/lzop -d
Dec 12 11:40:40 websrv amavis[7215]: No decoder for      .rpm  tried: rpm2cpio.pl, rpm2cpio
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .cpio at /usr/bin/pax
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .tar  at /usr/bin/pax
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .deb  at /usr/bin/ar
Dec 12 11:40:40 websrv amavis[7215]: Internal decoder for .zip
Dec 12 11:40:40 websrv amavis[7215]: No decoder for      .7z  tried: 7zr, 7za, 7z
Dec 12 11:40:40 websrv amavis[7215]: No decoder for      .rar  tried: unrar-free
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .arj  at /usr/bin/arj
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .arc  at /usr/bin/nomarch
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .zoo  at /usr/bin/zoo
Dec 12 11:40:40 websrv amavis[7215]: No decoder for      .lha
Dec 12 11:40:40 websrv amavis[7215]: No decoder for      .doc  tried: ripole
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .cab  at /usr/bin/cabextract
Dec 12 11:40:40 websrv amavis[7215]: No decoder for      .tnef
Dec 12 11:40:40 websrv amavis[7215]: Internal decoder for .tnef
Dec 12 11:40:40 websrv amavis[7215]: Found decoder for    .exe  at /usr/bin/arj
Dec 12 11:40:40 websrv amavis[7215]: Using primary internal av scanner code for ClamAV-clamd
Dec 12 11:40:40 websrv amavis[7215]: Found secondary av scanner ClamAV-clamscan at /usr/bin/clamscan
Dec 12 11:40:40 websrv amavis[7215]: Creating db in /var/lib/amavis/db/; BerkeleyDB 0.42, libdb 4.8
Dec 12 11:40:47 websrv postfix/master[6040]: terminating on signal 15
Dec 12 11:40:48 websrv postfix/master[7326]: daemon started -- version 2.7.1, configuration /etc/postfix
Dec 12 11:41:01 websrv pop3d: Connection, ip=[::1]
Dec 12 11:41:01 websrv pop3d: Disconnected, ip=[::1]
Dec 12 11:41:01 websrv imapd: Connection, ip=[::1]
Dec 12 11:41:01 websrv imapd: Disconnected, ip=[::1], time=0
Dec 12 11:41:01 websrv postfix/smtpd[7337]: connect from localhost[127.0.0.1]
Dec 12 11:41:01 websrv postfix/smtpd[7337]: lost connection after CONNECT from localhost[127.0.0.1]
Dec 12 11:41:01 websrv postfix/smtpd[7337]: disconnect from localhost[127.0.0.1]

lusso64 12th December 2012 22:36
This may be more useful...

This error is from the cron log after disabling and re-enabling the web service through the control panel.

Code:
ERROR: /var/www/clients is writable by group or others!mkdir: cannot create directory `/var/www/clients/client1/web26/var': File exists
mkdir: cannot create directory `/var/www/clients/client1/web26/var/run': File exists
mkdir: cannot create directory `/var/www/clients/client1/web26/var/run/mysqld': File exists
ln: creating hard link `/var/www/clients/client1/web26/var/run/mysqld/mysqld.sock' => `/var/run/mysqld/mysqld.sock': Invalid cross-device link
ERROR: /var/www/clients is writable by group or others!ERROR: /var/www/clients is writable by group or others!invalid shell, /var/www/clients/client1/web26/bin/bash does not exist
usermod: no changes
PHP Warning: mkdir(): File exists in /usr/local/ispconfig/server/plugins-available/cron_jailkit_plugin.inc.php on line 293

till 13th December 2012 08:37
Please run:

chmod 755 /var/www/clients

to fix the permission issue of the clients directory.

The mail.log looks fine, the services are running without errors. To check the mail system, login to webmail with an account that is hosted on the server and send a email to the same account, wait a minute and check if you received the email in webmail and then post the lines that have been added in the mail.log

lusso64 13th December 2012 08:50
Okay - did that, restarted apache and still nothing. I can't get to webmail - no websites other than the ISPconfig control panel itself. No webmail interface, no phpMyAdmin.
This is bizarre..... It has been rock solid up until now.

WAIT!

I just had a flash of inspiration and decided to add an entry into my hosts file to direct one of the domains to the internal IP address, and it is working! Now.... where to start trouble shooting this new problem.

At least ISPConfig retains its rock solid reputation with me :) Thanks Till for your help with this.


All times are GMT +2. The time now is 05:52.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2013, vBulletin Solutions, Inc.