HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   Server Operation (http://www.howtoforge.com/forums/forumdisplay.php?f=5)
-   -   CentOS 5 Server - Keeps going offline (http://www.howtoforge.com/forums/showthread.php?t=41849)

Matty B 15th December 2009 17:57

CentOS 5 Server - Keeps going offline
 
Hello,

I have been running my website on a dedicated server for around 3 years now with Fasthosts.

Everything had been running fine until recently and I believe that the server is now being attacked.

I have checked my Apache error_log, it has a huge list of errors which are mostly repeated, but when searching them in Google to find out what they mean and how to fix them all I have found is people saying they are minor issues which makes me believe they are not the reason for the server going down.

I do know that it seems to go offline mainly between 3 - 12PM GMT meaning if I restart the server on a morning when I wake up, it can be online for several hours before going offline again, but once it does go offline I can restart it several times and it will just keep going back offline each time.

So could someone please help me find out what is going wrong and how to rectify it?

I would also like to be able to get it so that if the server itself or any of the services on it could start themselves back up if they where ever to go offline in the future aswell if possible. I have tried installing something called SIM but that doesn't seem to be working.

Thanks.

topdog 16th December 2009 07:45

check the logs to see what actually happens look at /var/log/messages as well as other logs not just the apache log

Matty B 16th December 2009 16:31

Hi thanks for your reply, I have checked through /var/log/messages and quite far down the log it has the list of errors below which caught my attention, I didn't really understand any of the messages which I seen within the log but these seem as though something is going wrong:

Code:

Dec 13 21:14:59 localhost avahi-daemon[2623]: Network interface enumeration completed.
Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-2>
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-3>
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.

When checking the contents of /var/log/mysqld.log I seem to get these messages repeating over and over again:

Code:

091215 02:23:02  mysqld started
091215  2:23:03 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
091215  2:23:03  InnoDB: Started; log sequence number 0 377946
091215  2:23:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.86'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
091215 16:31:35  mysqld started
091215 16:31:36 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
091215 16:31:36  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
091215 16:31:37  InnoDB: Started; log sequence number 0 380788
091215 16:31:37 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.86'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
091215 16:39:02  mysqld started
091215 16:39:02 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
091215 16:39:02  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.

Obviously something is happening with that as it states that the database was not shut down properly and that is had crashed, is that enough to take the entire server offline or just a minor issue?

And here at the main error messages from /var/log/httpd/error_log:

This one seems to repeat in big blocks, not sure what exactly it means but my public files are stored within /user/htdocs not /var/www/html/ do I need to change something to remove that error?
Code:

[Sun Dec 13 21:04:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
I also seem to get this error repeated quite a lot aswell:
Code:

[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted

This one appears once that I have noticed:
Code:

[Tue Dec 15 02:23:10 2009] [notice] Graceful restart requested, doing restart
Then this seems to be a typical block of code which gets repeated over and over hundreds of times per day:
Code:

[Tue Dec 15 16:31:51 2009] [notice] mod_python: using mutex_directory /tmp
[Tue Dec 15 16:31:52 2009] [notice] Apache/2.2.3 (FH) configured -- resuming normal operations
[Tue Dec 15 16:32:43 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:32:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:32:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:39:10 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Dec 15 16:39:11 2009] [notice] Digest: generating secret for digest authentication ...
[Tue Dec 15 16:39:11 2009] [notice] Digest: done
[Tue Dec 15 16:39:12 2009] [notice] mod_python: Creating 4 session mutexes based on 3000 max processes and 0 max threads.

Are those the right logs to check and have I supplied enough useful information? Not sure what other logs their are to check.

Thanks for your time and help :)

topdog 16th December 2009 16:36

turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.

Matty B 16th December 2009 16:48

Quote:

Originally Posted by topdog (Post 213701)
turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.

Thanks again for the reply, I'm a complete newbie when it comes to servers so could you please tell me what I would need to do to configure my network interfaces? My installation at the moment is basically an "out of the box" package from Fasthosts.co.uk and their system automatically installed and setup my CentOS.

Thanks

topdog 16th December 2009 16:55

Unfortunately that is well beyond the scope of what a forum post can provide, i would advise that you read up on the documentation.

http://www.centos.org/docs/5/html/5....ed-config.html

topdog 16th December 2009 17:01

Sorry i did not actually see this

Code:

[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted

Something is crushing your php/apache stack u need to investigate what it is.

Matty B 16th December 2009 17:03

Quote:

Originally Posted by topdog (Post 213711)
Unfortunately that is well beyond the scope of what a forum post can provide, i would advise that you read up on the documentation.

http://www.centos.org/docs/5/html/5....ed-config.html

Thanks for all your help, after quickly checking my hosts file and a few other files mentioned on the first page of that configuration documentation I went ahead and disabled avahi-daemon anyway and then also stopped the service aswell and everything still seems to be working fine.

Hopefully that will also stop the server from going offline, but if not I will post an update within this topic.

Thanks again :D

Matty B 16th December 2009 17:05

Quote:

Originally Posted by topdog (Post 213712)
Sorry i did not actually see this

Code:

[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted

Something is crushing your php/apache stack u need to investigate what it is.

Woops, did not see this post.

I have just quickly done a search for that error and came across this, would you recommend trying their idea as a solution? http://ubuntuforums.org/archive/index.php/t-18490.html

topdog 16th December 2009 17:06

are u by any chance using the apc php module ? as there seems to be a bug similer to what you are experiencing.

http://pecl.php.net/bugs/bug.php?id=13511


All times are GMT +2. The time now is 11:43.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.