Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Server Operation

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 23rd May 2008, 19:14
v2k v2k is offline
Member
 
Join Date: Sep 2007
Location: Vancouver, Canada
Posts: 93
Thanks: 3
Thanked 1 Time in 1 Post
Default Server brought to a crawl

I'm pretty sure MySQL isthe culprit, but I'm not sure how to find the problem.

I hope that someone can suggest some ideas for how to debug this.

Basically my web server goes down to its knees after a while... or so it seems. When I last ran top when it was crawling, mysqld was taking 8.2% of the mem and there were a lot of httpd (more than usual) and sendmail tasks listed at the top when sotred by memory.

The machine has 2GB of memory, but when I run free -m it reports:
Mem: Total/2013 used/1996 free/17

Is this a red flag right there?

Any suggestions on how to approach the issue?


Thanks.
Reply With Quote
Sponsored Links
  #2  
Old 23rd May 2008, 20:36
Leszek Leszek is offline
Senior Member
 
Join Date: Nov 2006
Location: Poland,Włocławek
Posts: 369
Thanks: 22
Thanked 42 Times in 35 Posts
Send a message via Skype™ to Leszek
Default

Are there any unusual (errors) entries in the logs (Apache,MySQL) ?
Reply With Quote
  #3  
Old 23rd May 2008, 21:32
v2k v2k is offline
Member
 
Join Date: Sep 2007
Location: Vancouver, Canada
Posts: 93
Thanks: 3
Thanked 1 Time in 1 Post
Default

There are bunch of these in the mysql log, but I think they're just after the problem when I was trying to shutdown/restart mysql:
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11

In the apache logs, there's a ton of ModSecutiry spam that I probably need to clean up somehow:
[error] [client X] ModSecurity: Warning. Operator EQ match: 0.
[error] [client X] ModSecurity: Could not set variable "resource.alerted_960903_compression" as the collection does not exist.

Lots of other random msgs..

Warnings:
Module pgsql already loaded
Module gd already loaded


My database is about 2.2GB, 1.4GB in MyIASM and the rest in InnoDB
Reply With Quote
  #4  
Old 24th May 2008, 12:52
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,714 Times in 2,556 Posts
Default

You can try to repair your tables, either in phpMyAdmin,, or on the command line:

http://dev.mysql.com/doc/refman/5.0/...air-table.html
http://dev.mysql.com/doc/en/Repair.html
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #5  
Old 24th May 2008, 20:04
v2k v2k is offline
Member
 
Join Date: Sep 2007
Location: Vancouver, Canada
Posts: 93
Thanks: 3
Thanked 1 Time in 1 Post
Default

I happened again when I woke up; the server was pretty dead.

When I ran top, mysql was at the top with 7.2% of memory, but there were like an infinite number of http processes; I thought that might be the issue.

restarting httpd was very slow:

Code:
Stopping httpd:                                            [  OK  ]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

But shutting down and restarting http, didnt seem to fix it. It was back up, bust still slow.. simple pages with very little mysql worked fine, but anything else was still pretty much dead.

I'll try running the repairs.
Reply With Quote
  #6  
Old 24th May 2008, 20:41
v2k v2k is offline
Member
 
Join Date: Sep 2007
Location: Vancouver, Canada
Posts: 93
Thanks: 3
Thanked 1 Time in 1 Post
Default

I ran check and repair on my largest tables and everything reported "OK".
Reply With Quote
  #7  
Old 25th May 2008, 09:11
v2k v2k is offline
Member
 
Join Date: Sep 2007
Location: Vancouver, Canada
Posts: 93
Thanks: 3
Thanked 1 Time in 1 Post
Default

Is it possible this is a memory issue -- or would I have seen errors in the logs about memory?

I bumped up the memory on the server from 2GB to 8GB, but I still have no idea what is causing this issue.
Reply With Quote
  #8  
Old 25th May 2008, 18:34
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,714 Times in 2,556 Posts
Default

Quote:
Unable to open logs
Do all log files/log directories exist?
What's the output of
Code:
df -h
?
Any errors in Apache's error log?
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #9  
Old 26th May 2008, 03:34
v2k v2k is offline
Member
 
Join Date: Sep 2007
Location: Vancouver, Canada
Posts: 93
Thanks: 3
Thanked 1 Time in 1 Post
Default

20 hr uptime since the memory upgrade.

There are logs, I only got that error when the server was in it's "messed up state".

sudo cat /etc/httpd/logs/error_log | grep Error
and
sudo cat /etc/httpd/logs/error_log | grep ERROR

return nothing. The apache log is just cluttered with mod_security warnings and such. I can't seem to find anything relevant.

Code:
df -h 
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      224G   15G  198G   7% /
/dev/sda1              99M   20M   75M  21% /boot
tmpfs                 4.0G     0  4.0G   0% /dev/shm
Reply With Quote
  #10  
Old 26th May 2008, 15:27
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,714 Times in 2,556 Posts
 
Default

Do the log files that are referenced in your Apache configuration (e.g. the access log) exist?
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Site very slow after server migraion dimitar General 2 22nd February 2008 12:23
Terrible server speeds gAnDo HOWTO-Related Questions 3 14th February 2008 18:31
Problem with keeping Apache alive bobeq Server Operation 3 29th November 2007 16:11
Rejecting outbound mail tristanlee85 General 11 20th May 2007 17:04
Dns question Mahir Installation/Configuration 48 14th November 2006 10:19


All times are GMT +2. The time now is 13:32.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.