CentOS 5 Server - Keeps going offline

[Matty B]

Hello,

I have been running my website on a dedicated server for around 3 years now with Fasthosts.

Everything had been running fine until recently and I believe that the server is now being attacked.

I have checked my Apache error_log, it has a huge list of errors which are mostly repeated, but when searching them in Google to find out what they mean and how to fix them all I have found is people saying they are minor issues which makes me believe they are not the reason for the server going down.

I do know that it seems to go offline mainly between 3 - 12PM GMT meaning if I restart the server on a morning when I wake up, it can be online for several hours before going offline again, but once it does go offline I can restart it several times and it will just keep going back offline each time.

So could someone please help me find out what is going wrong and how to rectify it?

I would also like to be able to get it so that if the server itself or any of the services on it could start themselves back up if they where ever to go offline in the future aswell if possible. I have tried installing something called SIM but that doesn't seem to be working.

Thanks.

[topdog]

check the logs to see what actually happens look at /var/log/messages as well as other logs not just the apache log

[Matty B]

Hi thanks for your reply, I have checked through /var/log/messages and quite far down the log it has the list of errors below which caught my attention, I didn't really understand any of the messages which I seen within the log but these seem as though something is going wrong:

Dec 13 21:14:59 localhost avahi-daemon[2623]: Network interface enumeration completed.
Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-2>
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-3>
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.

When checking the contents of /var/log/mysqld.log I seem to get these messages repeating over and over again:

091215 02:23:02 mysqld started
091215 2:23:03 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
091215 2:23:03 InnoDB: Started; log sequence number 0 377946
091215 2:23:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.86' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
091215 16:31:35 mysqld started
091215 16:31:36 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
091215 16:31:36 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
091215 16:31:37 InnoDB: Started; log sequence number 0 380788
091215 16:31:37 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.86' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
091215 16:39:02 mysqld started
091215 16:39:02 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
091215 16:39:02 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.


Obviously something is happening with that as it states that the database was not shut down properly and that is had crashed, is that enough to take the entire server offline or just a minor issue?

And here at the main error messages from /var/log/httpd/error_log:

This one seems to repeat in big blocks, not sure what exactly it means but my public files are stored within /user/htdocs not /var/www/html/ do I need to change something to remove that error?
[Sun Dec 13 21:04:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/

I also seem to get this error repeated quite a lot aswell:
[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted


This one appears once that I have noticed:
[Tue Dec 15 02:23:10 2009] [notice] Graceful restart requested, doing restart


Then this seems to be a typical block of code which gets repeated over and over hundreds of times per day:
[Tue Dec 15 16:31:51 2009] [notice] mod_python: using mutex_directory /tmp
[Tue Dec 15 16:31:52 2009] [notice] Apache/2.2.3 (FH) configured -- resuming normal operations
[Tue Dec 15 16:32:43 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:32:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:32:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Tue Dec 15 16:39:10 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Dec 15 16:39:11 2009] [notice] Digest: generating secret for digest authentication ...
[Tue Dec 15 16:39:11 2009] [notice] Digest: done
[Tue Dec 15 16:39:12 2009] [notice] mod_python: Creating 4 session mutexes based on 3000 max processes and 0 max threads.


Are those the right logs to check and have I supplied enough useful information? Not sure what other logs their are to check.

Thanks for your time and help :)

[topdog]

turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.

[Matty B]

turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.

Thanks again for the reply, I'm a complete newbie when it comes to servers so could you please tell me what I would need to do to configure my network interfaces? My installation at the moment is basically an "out of the box" package from Fasthosts.co.uk and their system automatically installed and setup my CentOS.

Thanks

[topdog]

Unfortunately that is well beyond the scope of what a forum post can provide, i would advise that you read up on the documentation.

http://www.centos.org/docs/5/html/5.2/Deployment_Guide/pt-network-related-config.html

[topdog]

Sorry i did not actually see this

[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted

Something is crushing your php/apache stack u need to investigate what it is.

[Matty B]

Unfortunately that is well beyond the scope of what a forum post can provide, i would advise that you read up on the documentation.

http://www.centos.org/docs/5/html/5.2/Deployment_Guide/pt-network-related-config.html

Thanks for all your help, after quickly checking my hosts file and a few other files mentioned on the first page of that configuration documentation I went ahead and disabled avahi-daemon anyway and then also stopped the service aswell and everything still seems to be working fine.

Hopefully that will also stop the server from going offline, but if not I will post an update within this topic.

Thanks again :D

[Matty B]

Sorry i did not actually see this

[Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
zend_mm_heap corrupted

Something is crushing your php/apache stack u need to investigate what it is.

Woops, did not see this post.

I have just quickly done a search for that error and came across this, would you recommend trying their idea as a solution? http://ubuntuforums.org/archive/index.php/t-18490.html

[topdog]

are u by any chance using the apc php module ? as there seems to be a bug similer to what you are experiencing.

http://pecl.php.net/bugs/bug.php?id=13511

[topdog]

Try that and see if you actually have the python module installed.
yum remove mod_python
service httpd restart

[Matty B]

Try that and see if you actually have the python module installed.
yum remove mod_python
service httpd restart

Yes it is installed, after typing yum remove mod_python it displays some data with the name, status and size.

Should I go ahead with the uninstall?

are u by any chance using the apc php module ? as there seems to be a bug similer to what you are experiencing.

http://pecl.php.net/bugs/bug.php?id=13511

I'm not sure, how can I check?

[topdog]

On second thoughts i do not really think you problem is the same though because your crush is happening inside the php module zend_mm_heap corrupted if you are not running any python code in apache then uninstall it.

You can check the installed php modules using

php -m

[Matty B]

On second thoughts i do not really think you problem is the same though because your crush is happening inside the php module zend_mm_heap corrupted if you are not running any python code in apache then uninstall it.

You can check the installed php modules using

php -m

I have uninstalled python as I was not using it.

I don't seem to have the APC module installed, but just as a bit more information here is a list of the modules which where returned:

bz2
calendar
ctype
curl
date
dbase
exif
filter
ftp
gd
gettext
gmp
hash
iconv
ionCube Loader
json
ldap
libxml
mysql
mysqli
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
posix
readline
Reflection
session
shmop
SimpleXML
sockets
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
zip
zlib

[topdog]

The problem could be the ionCube Loader as it hooks into the zend memory manager i think.

[Matty B]

The problem could be the ionCube Loader as it hooks into the zend memory manager i think.

I'll try removing that aswell then, and just reinstall it if something I need stops working. I think I installed it so that Cast Control would run on my website which I no longer use.

[Matty B]

Hmm I can't seem to find anything online explaining how to uninstall ionCube Loader, just found a few people mentioning that having both ionCube & Zend installed takes their servers offline.

[Matty B]

I have managed to stop the service from starting by commenting out the start up lines for it in php.ini.

The server has just died on me again there.

[topdog]

what info can u get from the logs ?

[Matty B]

what info can u get from the logs ?

Just restarting it now so that I can access it again.

[topdog]

are u sure it is not hardware related ? Have you got your hosting company to check the hardware ?

[Matty B]

Their support is ridiculous, when I contact them asking about it they reply with:
We can have one of our technicians look into the situation for £60 + VAT per half hour, if the cause if hardware related we will refund you the costs.

I don't really trust them for wanting me to pay them first, incase it is a hardware problem and they don't own up to it so they can keep the money.

I'm with www.fasthosts.co.uk and only just recently I decided to look at a few reviews about them and a lot of people have done nothing but complain about their service.

I will try contacting them again though.

[Matty B]

When I am clicking to restart the server from my control panel its not even starting up, it's going straight back down again, over and over. But if I leave it until the morning it will start to work again straight away, it's been doing the same thing for the past week or two.

[topdog]

Then it is surely hardware related, could be overheating of components.

[Matty B]

I have emailed my host asking them to take a look at the server.

Will more than likely be sometime tomorrow when I get a reply now though since its after 5 PM here, I'll post an update once I receive a reply from them.

Thanks for all your support up to now, I really appreciate it, thanks :)