CentOS 5 Server - Keeps going offline

Discussion in 'Server Operation' started by Matty B, Dec 15, 2009.

  1. Matty B

    Matty B New Member

    Hello,

    I have been running my website on a dedicated server for around 3 years now with Fasthosts.

    Everything had been running fine until recently and I believe that the server is now being attacked.

    I have checked my Apache error_log, it has a huge list of errors which are mostly repeated, but when searching them in Google to find out what they mean and how to fix them all I have found is people saying they are minor issues which makes me believe they are not the reason for the server going down.

    I do know that it seems to go offline mainly between 3 - 12PM GMT meaning if I restart the server on a morning when I wake up, it can be online for several hours before going offline again, but once it does go offline I can restart it several times and it will just keep going back offline each time.

    So could someone please help me find out what is going wrong and how to rectify it?

    I would also like to be able to get it so that if the server itself or any of the services on it could start themselves back up if they where ever to go offline in the future aswell if possible. I have tried installing something called SIM but that doesn't seem to be working.

    Thanks.
     
  2. topdog

    topdog HowtoForge Supporter

    check the logs to see what actually happens look at /var/log/messages as well as other logs not just the apache log
     
  3. Matty B

    Matty B New Member

    Hi thanks for your reply, I have checked through /var/log/messages and quite far down the log it has the list of errors below which caught my attention, I didn't really understand any of the messages which I seen within the log but these seem as though something is going wrong:

    Code:
    Dec 13 21:14:59 localhost avahi-daemon[2623]: Network interface enumeration completed.
    Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
    Dec 13 21:14:59 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-2>
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Registering HINFO record with values 'I686'/'LINUX'.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.130 on eth0.
    Dec 13 21:15:00 localhost avahi-daemon[2623]: Withdrawing address record for 88.208.230.131 on eth0.
    Dec 13 21:15:01 localhost avahi-daemon[2623]: Host name conflict, retrying with <localhost-3>
    Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for fe80::230:5ff:fee5:2a90 on eth0.
    Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.130 on eth0.
    Dec 13 21:15:01 localhost avahi-daemon[2623]: Registering new address record for 88.208.230.131 on eth0.
    When checking the contents of /var/log/mysqld.log I seem to get these messages repeating over and over again:

    Code:
    091215 02:23:02  mysqld started
    091215  2:23:03 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
    091215  2:23:03  InnoDB: Started; log sequence number 0 377946
    091215  2:23:03 [Note] /usr/libexec/mysqld: ready for connections.
    Version: '5.0.86'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
    091215 16:31:35  mysqld started
    091215 16:31:36 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    091215 16:31:36  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    091215 16:31:37  InnoDB: Started; log sequence number 0 380788
    091215 16:31:37 [Note] /usr/libexec/mysqld: ready for connections.
    Version: '5.0.86'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
    091215 16:39:02  mysqld started
    091215 16:39:02 [Warning] option 'max_connections': unsigned value 20000 adjusted to 16384
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    091215 16:39:02  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    
    Obviously something is happening with that as it states that the database was not shut down properly and that is had crashed, is that enough to take the entire server offline or just a minor issue?

    And here at the main error messages from /var/log/httpd/error_log:

    This one seems to repeat in big blocks, not sure what exactly it means but my public files are stored within /user/htdocs not /var/www/html/ do I need to change something to remove that error?
    Code:
    [Sun Dec 13 21:04:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
    I also seem to get this error repeated quite a lot aswell:
    Code:
    [Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
    zend_mm_heap corrupted
    
    This one appears once that I have noticed:
    Code:
    [Tue Dec 15 02:23:10 2009] [notice] Graceful restart requested, doing restart
    
    Then this seems to be a typical block of code which gets repeated over and over hundreds of times per day:
    Code:
    [Tue Dec 15 16:31:51 2009] [notice] mod_python: using mutex_directory /tmp
    [Tue Dec 15 16:31:52 2009] [notice] Apache/2.2.3 (FH) configured -- resuming normal operations
    [Tue Dec 15 16:32:43 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
    [Tue Dec 15 16:32:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
    [Tue Dec 15 16:32:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
    [Tue Dec 15 16:39:10 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
    [Tue Dec 15 16:39:11 2009] [notice] Digest: generating secret for digest authentication ...
    [Tue Dec 15 16:39:11 2009] [notice] Digest: done
    [Tue Dec 15 16:39:12 2009] [notice] mod_python: Creating 4 session mutexes based on 3000 max processes and 0 max threads.
    
    Are those the right logs to check and have I supplied enough useful information? Not sure what other logs their are to check.

    Thanks for your time and help :)
     
  4. topdog

    topdog HowtoForge Supporter

    turn of the avahi-daemon you should not be running that on a server, use static configuration for your network interfaces.
     
  5. Matty B

    Matty B New Member

    Thanks again for the reply, I'm a complete newbie when it comes to servers so could you please tell me what I would need to do to configure my network interfaces? My installation at the moment is basically an "out of the box" package from Fasthosts.co.uk and their system automatically installed and setup my CentOS.

    Thanks
     
  6. topdog

    topdog HowtoForge Supporter

  7. topdog

    topdog HowtoForge Supporter

    Sorry i did not actually see this

    Code:
    [Tue Dec 15 00:57:25 2009] [notice] child pid 3488 exit signal Segmentation fault (11)
    zend_mm_heap corrupted
    Something is crushing your php/apache stack u need to investigate what it is.
     
  8. Matty B

    Matty B New Member

    Thanks for all your help, after quickly checking my hosts file and a few other files mentioned on the first page of that configuration documentation I went ahead and disabled avahi-daemon anyway and then also stopped the service aswell and everything still seems to be working fine.

    Hopefully that will also stop the server from going offline, but if not I will post an update within this topic.

    Thanks again :D
     
  9. Matty B

    Matty B New Member

    Woops, did not see this post.

    I have just quickly done a search for that error and came across this, would you recommend trying their idea as a solution? http://ubuntuforums.org/archive/index.php/t-18490.html
     
  10. topdog

    topdog HowtoForge Supporter

  11. topdog

    topdog HowtoForge Supporter

    Try that and see if you actually have the python module installed.
    Code:
    yum remove mod_python
    service httpd restart
     
  12. Matty B

    Matty B New Member

    Yes it is installed, after typing yum remove mod_python it displays some data with the name, status and size.

    Should I go ahead with the uninstall?

    I'm not sure, how can I check?
     
  13. topdog

    topdog HowtoForge Supporter

    On second thoughts i do not really think you problem is the same though because your crush is happening inside the php module
    Code:
    zend_mm_heap corrupted
    if you are not running any python code in apache then uninstall it.

    You can check the installed php modules using

    Code:
    php -m
     
  14. Matty B

    Matty B New Member

    I have uninstalled python as I was not using it.

    I don't seem to have the APC module installed, but just as a bit more information here is a list of the modules which where returned:

    bz2
    calendar
    ctype
    curl
    date
    dbase
    exif
    filter
    ftp
    gd
    gettext
    gmp
    hash
    iconv
    ionCube Loader
    json
    ldap
    libxml
    mysql
    mysqli
    openssl
    pcntl
    pcre
    PDO
    pdo_mysql
    pdo_sqlite
    posix
    readline
    Reflection
    session
    shmop
    SimpleXML
    sockets
    SPL
    standard
    sysvmsg
    sysvsem
    sysvshm
    tokenizer
    wddx
    xml
    zip
    zlib
     
  15. topdog

    topdog HowtoForge Supporter

    The problem could be the ionCube Loader as it hooks into the zend memory manager i think.
     
  16. Matty B

    Matty B New Member

    I'll try removing that aswell then, and just reinstall it if something I need stops working. I think I installed it so that Cast Control would run on my website which I no longer use.
     
  17. Matty B

    Matty B New Member

    Hmm I can't seem to find anything online explaining how to uninstall ionCube Loader, just found a few people mentioning that having both ionCube & Zend installed takes their servers offline.
     
  18. Matty B

    Matty B New Member

    I have managed to stop the service from starting by commenting out the start up lines for it in php.ini.

    The server has just died on me again there.
     
  19. topdog

    topdog HowtoForge Supporter

    what info can u get from the logs ?
     
  20. Matty B

    Matty B New Member

    Just restarting it now so that I can access it again.
     

Share This Page