Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > ISPConfig 3 > General

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 11th August 2009, 02:40
MrM MrM is offline
Member
 
Join Date: Oct 2008
Location: Ljubljana, Slovenia
Posts: 58
Thanks: 6
Thanked 0 Times in 0 Posts
Exclamation Swap usage issue

Today I started getting loads of Error connecting to MySQL server at localhost: Too many connections error messages on our production server. After connecting via SSH, I noticed that the server was completely swamped (load average >50). I barely managed to reboot it.

After a closer examination, I found out that for the past month or so, swap usage has been persistently growing. Today it shot up to more than 2GB (see attachments) and "commited" to 5G (the server has only 512M of RAM). The strange part is that "apps" usage never exceeded 300M and it was more or less around 100M on average.

The server is running ISPConfig 3.0.1.3 on Ubuntu Jaunty. Does anyone have any idea why this has been happening?

I can post additional information about the server if it will help determining the cause.

Any help will be much appreciated.
Attached Images
  
Reply With Quote
Sponsored Links
  #2  
Old 11th August 2009, 04:42
syadnom syadnom is offline
Junior Member
 
Join Date: Mar 2009
Posts: 21
Thanks: 0
Thanked 0 Times in 0 Posts
Default memory leak. any unstable software?

Are you running anything from jaunty proposed or backports or any thing that is not part of jaunty?

have you looked at the processes and seen what is actually eating up ram and getting pushed to swap?

Do you have something writing to tmpfs?
Reply With Quote
  #3  
Old 11th August 2009, 09:50
till till is online now
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,486
Thanks: 813
Thanked 5,256 Times in 4,121 Posts
Default

Quote:
Today I started getting loads of Error connecting to MySQL server at localhost: Too many connections error messages on our production server.
http://www.howtoforge.com/forums/showthread.php?t=32533
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #4  
Old 11th August 2009, 10:56
MrM MrM is offline
Member
 
Join Date: Oct 2008
Location: Ljubljana, Slovenia
Posts: 58
Thanks: 6
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by syadnom View Post
Are you running anything from jaunty proposed or backports or any thing that is not part of jaunty?
I don't think I'm running any unstable software. It's basically a clean ISPConfig sysyem, with some extras like munin, subversion and some php modules (see attachment for a full list of packages). I have not added any repositories either. This is the full list of repositories (with deb-src counterparts removed):
Code:
deb http://si.archive.ubuntu.com/ubuntu/ jaunty main restricted
deb http://si.archive.ubuntu.com/ubuntu/ jaunty-updates main restricted
deb http://si.archive.ubuntu.com/ubuntu/ jaunty universe
deb http://si.archive.ubuntu.com/ubuntu/ jaunty-updates universe
deb http://si.archive.ubuntu.com/ubuntu/ jaunty multiverse
deb http://si.archive.ubuntu.com/ubuntu/ jaunty-updates multiverse
deb http://security.ubuntu.com/ubuntu jaunty-security main restricted
deb http://security.ubuntu.com/ubuntu jaunty-security universe
deb http://security.ubuntu.com/ubuntu jaunty-security multiverse
Quote:
Originally Posted by syadnom View Post
have you looked at the processes and seen what is actually eating up ram and getting pushed to swap?
As mentioned in my first post, I don't think the processes are eating up ram at all. The 'apps' memory usage is about 100M on average and does not rise over time. The only suspicious process I noticed in htop is /usr/sbin/console-kit-daemon. There are about 65 active instances, each reportedly using 1.2% of ram.

Quote:
Originally Posted by syadnom View Post
Do you have something writing to tmpfs?
How can I check this?

I don't believe the problem is actually in MySQL. I think the too many connections error is a consequence rather than a reason for this problem. I can set the max_connections setting though, if you think it'll help.
Attached Files
File Type: txt packages.txt (21.5 KB, 162 views)
Reply With Quote
  #5  
Old 11th August 2009, 11:13
till till is online now
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,486
Thanks: 813
Thanked 5,256 Times in 4,121 Posts
Default

Please do what I suggested and you wil see that your problem is solved. The reasom for this is simply a lot of spam spam or a similar incident which causes postfix to open up more connections then your mysql settings allow which causes a lot of waiting processes which then fill up your swap.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #6  
Old 11th August 2009, 11:18
MrM MrM is offline
Member
 
Join Date: Oct 2008
Location: Ljubljana, Slovenia
Posts: 58
Thanks: 6
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by till View Post
Please do what I suggested and you wil see that your problem is solved. The reasom for this is simply a lot of spam spam or a similar incident which causes postfix to open up more connections then your mysql settings allow which causes a lot of waiting processes which then fill up your swap.
I've set the following settings in my.cnf:
Code:
max_connections = 500
max_user_connections = 500
Was there anything else you had in mind? What about postfix?
Reply With Quote
  #7  
Old 11th August 2009, 11:21
till till is online now
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,486
Thanks: 813
Thanked 5,256 Times in 4,121 Posts
Default

Just restart mysql. There has nothing to be changed in postfix.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
The Following User Says Thank You to till For This Useful Post:
MrM (11th August 2009)
  #8  
Old 11th August 2009, 11:24
MrM MrM is offline
Member
 
Join Date: Oct 2008
Location: Ljubljana, Slovenia
Posts: 58
Thanks: 6
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by till View Post
Just restart mysql. There has nothing to be changed in postfix.
OK, have already done that, thanks.

Just one observation: Could the max connection limit affect websites? The way I see it is that if postfix still tries to open up a lot of connections, legitimate connections from websites could get blocked when the limit is reached.
Reply With Quote
  #9  
Old 11th August 2009, 12:02
MrM MrM is offline
Member
 
Join Date: Oct 2008
Location: Ljubljana, Slovenia
Posts: 58
Thanks: 6
Thanked 0 Times in 0 Posts
Default

Just to clarify my reasoning as to why I thought MySQL was not the culprit here, but rather a consequence. As you can see in the (first two) attachments to this post, MySQL thread count was more or less stable at about 2 throughout the past month, and never exceeded 10. When the overload happened last night at around midnight it shot up to over a 100.

In the third attachment (memory usage), you can see a strange thing happening. Apparently last night after I rebooted the server, swap usage grew rapidly to about 1G, then dropped at around 5 AM.

P.S.: All times are GMT+2 (Paris time).
Attached Images
   
Reply With Quote
  #10  
Old 23rd August 2009, 18:51
MrM MrM is offline
Member
 
Join Date: Oct 2008
Location: Ljubljana, Slovenia
Posts: 58
Thanks: 6
Thanked 0 Times in 0 Posts
 
Default

Hi, till,

Changing max_connections and max_user_connections did not help. Swap usage is currently at 150M and still gradually rising at the same rate as before.

Any other suggestions? I'm at a loss as to the reason for this behaviour.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
High swap usage. tonysch General 3 28th June 2009 22:28
Apache2 fills my memory and my swap! batman Server Operation 1 10th February 2009 17:29
Squid Proxy Caching on Linux obzerver Installation/Configuration 4 13th August 2008 19:51
Memory usage on debian etch devnull3d Installation/Configuration 4 17th January 2008 16:19
iptables issue with xen perfect setup - debian alexnz HOWTO-Related Questions 3 25th November 2006 13:49


All times are GMT +2. The time now is 21:01.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.