Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Server Operation

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 15th December 2009, 17:57
Matty B Matty B is offline
Junior Member
 
Join Date: Dec 2009
Posts: 18
Thanks: 2
Thanked 0 Times in 0 Posts
Default CentOS 5 Server - Keeps going offline

Hello,

I have been running my website on a dedicated server for around 3 years now with Fasthosts.

Everything had been running fine until recently and I believe that the server is now being attacked.

I have checked my Apache error_log, it has a huge list of errors which are mostly repeated, but when searching them in Google to find out what they mean and how to fix them all I have found is people saying they are minor issues which makes me believe they are not the reason for the server going down.

I do know that it seems to go offline mainly between 3 - 12PM GMT meaning if I restart the server on a morning when I wake up, it can be online for several hours before going offline again, but once it does go offline I can restart it several times and it will just keep going back offline each time.

So could someone please help me find out what is going wrong and how to rectify it?

I would also like to be able to get it so that if the server itself or any of the services on it could start themselves back up if they where ever to go offline in the future aswell if possible. I have tried installing something called SIM but that doesn't seem to be working.

Thanks.
Reply With Quote
Sponsored Links
  #2  
Old 16th December 2009, 07:45
topdog topdog is offline
Senior Member
 
Join Date: Jan 2008
Location: South Africa
Posts: 1,352
Thanks: 0
Thanked 153 Times in 150 Posts
Default

check the logs to see what actually happens look at /var/log/messages as well as other logs not just the apache log
__________________
----
http://www.topdog.za.net - Got Linux problems ? - I can help.
http://www.baruwa.org - Try it.
Reply With Quote
The Following User Says Thank You to topdog For This Useful Post:
Matty B (16th December 2009)
  #3  
Old 16th December 2009, 16:31
Matty B Matty B is offline
Junior Member
 
Join Date: Dec 2009
Posts: 18
Thanks: 2
Thanked 0 Times in 0 Posts
Default

Hi thanks for your reply, I have checked through /var/log/messages and quite far down the log it has the list of errors below which caught my attention, I didn't really understand any of the messages which I seen within the log but these seem as though something is going wrong:

Code:
Dec 13 21:14:59 localhost avahi-daemon[2623]: Network interface enumeration completed.
Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-2>
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-3>
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
When checking the contents of /var/log/mysqld.log I seem to get these messages repeating over and over again:

Code:
091215 02:23:02  mysqld started
091215  2:23:03 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
091215  2:23:03  InnoDB: Started; log sequence number 0 377946
091215  2:23:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.86'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
091215 16:31:35  mysqld started
091215 16:31:36 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
091215 16:31:36  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
091215 16:31:37  InnoDB: Started; log sequence number 0 380788
091215 16:31:37 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.86'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
091215 16:39:02  mysqld started
091215 16:39:02 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
091215 16:39:02  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
Obviously something is happening with that as it states that the database was not shut down properly and that is had crashed, is that enough to take the entire server offline or just a minor issue?

And here at the main error messages from /var/log/httpd/error_log:

This one seems to repeat in big blocks, not sure what exactly it means but my public files are stored within /user/htdocs not /var/www/html/ do I need to change something to remove that error?
Code:
[Sun Dec 13 21:04:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
I also seem to get this error repeated quite a lot aswell:
Code:
[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted
This one appears once that I have noticed:
Code:
[Tue Dec 15 02:23:10 2009] [notice] Graceful restart requested, doing restart
Then this seems to be a typical block of code which gets repeated over and over hundreds of times per day:
Code:
[Tue Dec 15 16:31:51 2009] [notice] mod_python: using mutex_directory /tmp
[Tue Dec 15 16:31:52 2009] [notice] Apache/2.2.3 (FH) configured -- resuming normal operations
[Tue Dec 15 16:32:43 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:32:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:32:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:39:10 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Dec 15 16:39:11 2009] [notice] Digest: generating secret for digest authentication ...
[Tue Dec 15 16:39:11 2009] [notice] Digest: done
[Tue Dec 15 16:39:12 2009] [notice] mod_python: Creating 4 session mutexes based on 3000 max processes and 0 max threads.
Are those the right logs to check and have I supplied enough useful information? Not sure what other logs their are to check.

Thanks for your time and help
Reply With Quote
  #4  
Old 16th December 2009, 16:36
topdog topdog is offline
Senior Member
 
Join Date: Jan 2008
Location: South Africa
Posts: 1,352
Thanks: 0
Thanked 153 Times in 150 Posts
Default

turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.
__________________
----
http://www.topdog.za.net - Got Linux problems ? - I can help.
http://www.baruwa.org - Try it.
Reply With Quote
The Following User Says Thank You to topdog For This Useful Post:
Matty B (16th December 2009)
  #5  
Old 16th December 2009, 16:48
Matty B Matty B is offline
Junior Member
 
Join Date: Dec 2009
Posts: 18
Thanks: 2
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by topdog View Post
turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.
Thanks again for the reply, I'm a complete newbie when it comes to servers so could you please tell me what I would need to do to configure my network interfaces? My installation at the moment is basically an "out of the box" package from Fasthosts.co.uk and their system automatically installed and setup my CentOS.

Thanks
Reply With Quote
  #6  
Old 16th December 2009, 16:55
topdog topdog is offline
Senior Member
 
Join Date: Jan 2008
Location: South Africa
Posts: 1,352
Thanks: 0
Thanked 153 Times in 150 Posts
Default

Unfortunately that is well beyond the scope of what a forum post can provide, i would advise that you read up on the documentation.

http://www.centos.org/docs/5/html/5....ed-config.html
__________________
----
http://www.topdog.za.net - Got Linux problems ? - I can help.
http://www.baruwa.org - Try it.
Reply With Quote
  #7  
Old 16th December 2009, 17:01
topdog topdog is offline
Senior Member
 
Join Date: Jan 2008
Location: South Africa
Posts: 1,352
Thanks: 0
Thanked 153 Times in 150 Posts
Default

Sorry i did not actually see this

Code:
[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted
Something is crushing your php/apache stack u need to investigate what it is.
__________________
----
http://www.topdog.za.net - Got Linux problems ? - I can help.
http://www.baruwa.org - Try it.
Reply With Quote
  #8  
Old 16th December 2009, 17:03
Matty B Matty B is offline
Junior Member
 
Join Date: Dec 2009
Posts: 18
Thanks: 2
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by topdog View Post
Unfortunately that is well beyond the scope of what a forum post can provide, i would advise that you read up on the documentation.

http://www.centos.org/docs/5/html/5....ed-config.html
Thanks for all your help, after quickly checking my hosts file and a few other files mentioned on the first page of that configuration documentation I went ahead and disabled avahi-daemon anyway and then also stopped the service aswell and everything still seems to be working fine.

Hopefully that will also stop the server from going offline, but if not I will post an update within this topic.

Thanks again
Reply With Quote
  #9  
Old 16th December 2009, 17:05
Matty B Matty B is offline
Junior Member
 
Join Date: Dec 2009
Posts: 18
Thanks: 2
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by topdog View Post
Sorry i did not actually see this

Code:
[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted
Something is crushing your php/apache stack u need to investigate what it is.
Woops, did not see this post.

I have just quickly done a search for that error and came across this, would you recommend trying their idea as a solution? http://ubuntuforums.org/archive/index.php/t-18490.html
Reply With Quote
  #10  
Old 16th December 2009, 17:06
topdog topdog is offline
Senior Member
 
Join Date: Jan 2008
Location: South Africa
Posts: 1,352
Thanks: 0
Thanked 153 Times in 150 Posts
 
Default

are u by any chance using the apc php module ? as there seems to be a bug similer to what you are experiencing.

http://pecl.php.net/bugs/bug.php?id=13511
__________________
----
http://www.topdog.za.net - Got Linux problems ? - I can help.
http://www.baruwa.org - Try it.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
User unknown in relay recipient table Taxick Installation/Configuration 12 9th April 2013 12:31
Virtual Users And Domains With Postfix, Courier, MySQL And SquirrelMail -Ubuntu 8.04 c4rdinal HOWTO-Related Questions 112 23rd August 2011 10:49
Connection dropped by IMAP server gublym Server Operation 5 23rd January 2009 09:47
Howtoforge the perfect Server(links) rini90 Installation/Configuration 0 14th December 2008 10:58
cacti problem - graphs have huge gaps Chip Installation/Configuration 7 7th February 2008 23:24


All times are GMT +2. The time now is 20:30.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.