Swap usage issue

[MrM]

Today I started getting loads of Error connecting to MySQL server at localhost: Too many connections error messages on our production server. After connecting via SSH, I noticed that the server was completely swamped (load average >50). I barely managed to reboot it.

After a closer examination, I found out that for the past month or so, swap usage has been persistently growing. Today it shot up to more than 2GB (see attachments) and "commited" to 5G (the server has only 512M of RAM). The strange part is that "apps" usage never exceeded 300M and it was more or less around 100M on average.

The server is running ISPConfig 3.0.1.3 on Ubuntu Jaunty. Does anyone have any idea why this has been happening?

I can post additional information about the server if it will help determining the cause.

Any help will be much appreciated.

[syadnom]

Are you running anything from jaunty proposed or backports or any thing that is not part of jaunty?

have you looked at the processes and seen what is actually eating up ram and getting pushed to swap?

Do you have something writing to tmpfs?

[till]

Today I started getting loads of Error connecting to MySQL server at localhost: Too many connections error messages on our production server.

http://www.howtoforge.com/forums/showthread.php?t=32533

[MrM]

Originally Posted by syadnom

Are you running anything from jaunty proposed or backports or any thing that is not part of jaunty?

I don't think I'm running any unstable software. It's basically a clean ISPConfig sysyem, with some extras like munin, subversion and some php modules (see attachment for a full list of packages). I have not added any repositories either. This is the full list of repositories (with deb-src counterparts removed):

deb http://si.archive.ubuntu.com/ubuntu/ jaunty main restricted
deb http://si.archive.ubuntu.com/ubuntu/ jaunty-updates main restricted
deb http://si.archive.ubuntu.com/ubuntu/ jaunty universe
deb http://si.archive.ubuntu.com/ubuntu/ jaunty-updates universe
deb http://si.archive.ubuntu.com/ubuntu/ jaunty multiverse
deb http://si.archive.ubuntu.com/ubuntu/ jaunty-updates multiverse
deb http://security.ubuntu.com/ubuntu jaunty-security main restricted
deb http://security.ubuntu.com/ubuntu jaunty-security universe
deb http://security.ubuntu.com/ubuntu jaunty-security multiverse

Originally Posted by syadnom

have you looked at the processes and seen what is actually eating up ram and getting pushed to swap?

As mentioned in my first post, I don't think the processes are eating up ram at all. The 'apps' memory usage is about 100M on average and does not rise over time. The only suspicious process I noticed in htop is /usr/sbin/console-kit-daemon. There are about 65 active instances, each reportedly using 1.2% of ram.

Originally Posted by syadnom

Do you have something writing to tmpfs?

How can I check this?

Originally Posted by till

http://www.howtoforge.com/forums/showthread.php?t=32533

I don't believe the problem is actually in MySQL. I think the too many connections error is a consequence rather than a reason for this problem. I can set the max_connections setting though, if you think it'll help.

[till]

Please do what I suggested and you wil see that your problem is solved. The reasom for this is simply a lot of spam spam or a similar incident which causes postfix to open up more connections then your mysql settings allow which causes a lot of waiting processes which then fill up your swap.

[MrM]

Originally Posted by till

Please do what I suggested and you wil see that your problem is solved. The reasom for this is simply a lot of spam spam or a similar incident which causes postfix to open up more connections then your mysql settings allow which causes a lot of waiting processes which then fill up your swap.

I've set the following settings in my.cnf:

max_connections = 500
max_user_connections = 500

Was there anything else you had in mind? What about postfix?

[till]

Just restart mysql. There has nothing to be changed in postfix.

[MrM]

Originally Posted by till

Just restart mysql. There has nothing to be changed in postfix.

OK, have already done that, thanks.

Just one observation: Could the max connection limit affect websites? The way I see it is that if postfix still tries to open up a lot of connections, legitimate connections from websites could get blocked when the limit is reached.

[MrM]

Just to clarify my reasoning as to why I thought MySQL was not the culprit here, but rather a consequence. As you can see in the (first two) attachments to this post, MySQL thread count was more or less stable at about 2 throughout the past month, and never exceeded 10. When the overload happened last night at around midnight it shot up to over a 100.

In the third attachment (memory usage), you can see a strange thing happening. Apparently last night after I rebooted the server, swap usage grew rapidly to about 1G, then dropped at around 5 AM.

P.S.: All times are GMT+2 (Paris time).

[MrM]

Hi, till,

Changing max_connections and max_user_connections did not help. Swap usage is currently at 150M and still gradually rising at the same rate as before.

Any other suggestions? I'm at a loss as to the reason for this behaviour.