Swap usage issue

[till]

Check with ps aux which processes consume the memory.

[MrM]

Originally Posted by till

Check with ps aux which processes consume the memory.

The thing is, no process uses an excessive amount of memory. In fact, there is always a significant amount of free memory on the server. That's why this is so bewildering to me.

Here is an excerpt from the top command (sorted by memory usage):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10960 mysql     20   0  237m  45m 3744 S  0.0  9.3  76:52.11 mysqld
 2447 root      20   0  244m  38m 1140 S  0.0  7.9  28:14.47 console-kit-dae
 2627 root      20   0  278m  13m 7240 S  0.0  2.7   1:15.99 apache2
17871 www-data  20   0  278m 8420 1496 S  0.0  1.7   0:00.55 apache2
22976 www-data  20   0  278m 8404 1492 S  0.0  1.7   0:00.11 apache2
22977 www-data  20   0  279m 8364 1496 S  0.0  1.7   0:00.11 apache2
22276 www-data  20   0  279m 8356 1456 S  0.0  1.7   0:00.16 apache2
32108 www-data  20   0  279m 8344 1496 S  0.0  1.7   0:00.06 apache2
32107 www-data  20   0  279m 8324 1460 S  0.3  1.7   0:00.06 apache2
 6905 www-data  20   0  279m 8284 1488 S  0.0  1.7   0:00.00 apache2
 7404 www-data  20   0  279m 8284 1444 S  0.0  1.7   0:00.03 apache2
 7279 www-data  20   0  279m 8272 1452 S  0.0  1.7   0:00.01 apache2
 5044 www-data  20   0  278m 8176 1492 S  0.0  1.6   0:00.07 apache2                                                                                          
 6906 www-data  20   0  278m 8128 1452 S  0.0  1.6   0:00.03 apache2
 7407 www-data  20   0  278m 8120 1452 S  0.0  1.6   0:00.01 apache2
 7410 www-data  20   0  278m 8060 1420 S  0.0  1.6   0:00.00 apache2
 7408 www-data  20   0  278m 7780 1304 S  0.0  1.6   0:00.01 apache2
 7409 www-data  20   0  278m 7780 1292 S  0.0  1.6   0:00.00 apache2
 7402 www-data  20   0  278m 7728 1272 S  0.0  1.5   0:00.00 apache2
 7403 www-data  20   0  278m 7592 1160 S  0.0  1.5   0:00.00 apache2
29980 www-data  20   0  162m 5824  464 S  0.0  1.2   0:00.02 apache2
 2905 root      20   0 55100 4132 1080 S  0.0  0.8   0:39.71 fail2ban-server
29979 root      20   0 20304 3804 1892 S  0.0  0.8   0:08.65 vlogger
21009 root      20   0 76688 3452 2696 S  0.0  0.7   0:00.04 sshd
26665 root      20   0 76688 3452 2696 R  0.0  0.7   0:00.07 sshd
 7384 postfix   20   0 56576 3256 2508 S  0.0  0.7   0:00.00 smtp
21399 root      20   0 30692 3108 1964 S  0.0  0.6   0:00.02 mc
 2759 root      20   0 40552 2312  644 S  0.0  0.5   0:15.48 munin-node
26814 root      20   0 20080 2212 1540 S  0.0  0.4   0:00.03 bash
 2158 nobody    20   0 27756 2208  528 S  0.0  0.4   0:17.54 mydns
21017 root      20   0 20080 2208 1540 S  0.0  0.4   0:00.01 bash
 7387 postfix   20   0 39136 2180 1712 S  0.0  0.4   0:00.01 bounce
21401 root      20   0 20064 2176 1528 S  0.0  0.4   0:00.05 bash
21474 postfix   20   0 39104 2136 1680 S  0.0  0.4   0:00.00 pickup
 7294 root      20   0 18984 1308  988 R  0.3  0.3   0:00.18 top
12292 root      20   0 76688 1032  808 S  0.0  0.2   0:05.58 sshd
23820 root      20   0 76688 1032  808 S  0.0  0.2   0:05.58 sshd
 1842 messageb  20   0 22596  868  420 S  0.0  0.2   0:45.53 dbus-daemon

I've also created a temporary login for munin here:
http://munin.protobit.net/protobit.n...tobit.net.html
username: test
password: test

You can check the graphs for yourself, to see if there's anything out of the ordinary.

[till]

Looks quite normal.

[MrM]

Originally Posted by till

Looks quite normal.

It does to me as well. What I can't get my head around is why swap is being used, when there's plenty of RAM left.

Should I try to reduce MaxClients in apache2.conf? I think this shouldn't be a problem, since the server hosts no site with very high traffic. It is currently set to 150 (the default).

Any other ideas?

[bajodel]

Originally Posted by MrM

Any other ideas?

Hi Mrm .. take a look at your actual 'swappiness' kernel parameter ..

# cat /proc/sys/vm/swappiness

I don't know your distro .. (debian default is 60)

.. maybe you can test a lower value (for a day or two) setting to 20 (or 0 ..even better)

# echo "0" > /proc/sys/vm/swappiness

(These settings are applied instantly by the kernel and are not persistent after a reboot)

..you asked for 'another idea' ..here you have :-)

Bye..

bajodel.

[MrM]

Hi, bajodel,

I do appreciate any ideas at this point. First of all, let me say that I was not aware this setting existed. After reading up on it a bit, I'm pretty sure I understand what it does.

So, if I understand it correctly, I'm not sure how it could have an effect in my situation, where the swap size gradually and persistently grows over time:


Again, if I understand it correctly, increasing this value should increase swap usage (with the same memory usage), while decreasing it should decrease swap usage. However, by my understanding, this setting should not affect the growth of swap over time.

But, as I said at the beginning, I do appreciate every idea, so I'm going to set it to 10 and report back in a couple of days, when I see if it will have had any effect.

[bajodel]

Originally Posted by MrM

..i do appreciate every idea, so I'm going to set it to 10 and report back in a couple of days, when I see if it will have had any effect.

The "Swap FAQ" you have read look like a 'simple explanation' ..but ..yes .. that parameter is (in few words) the kernel "trend" (over time) to swap.

Just try it .. but '0' is better in my opinion.

I've re-read the entire post ..your distro is ubuntu (server) and the default is the same ad debian (60) .. which is quite good, but i've also experienced more swap than what i was used to on my (debian) test server with ISPConfig on-board

I've also experienced bad performances related to clamav scanning .. in particular if you have gunzipped attachmentes (cpio is the hungry app).

Try to understand which app is swapped:
# top
then press: SHIFT-O
then press: P <enter>

you can see sorted apps for swap usage .. copy & paste here.

Bye..

bajodel.

[MrM]

Originally Posted by bajodel

Just try it .. but '0' is better in my opinion.

OK, I've changed it to 0.

EDIT: Should the swap usage decrease without rebooting the server? Or will it only stop growing (in case this setting helps)?

Originally Posted by bajodel

I've re-read the entire post ..your distro is ubuntu (server) and the default is the same ad debian (60) .. which is quite good, but i've also experienced more swap than what i was used to on my (debian) test server with ISPConfig on-board

The default was indeed 60.

I think the problem here isn't so much that swap is being used, the more serious problem is that it keeps growing. This is what I really don't understand.

Originally Posted by bajodel

I've also experienced bad performances related to clamav scanning .. in particular if you have gunzipped attachmentes (cpio is the hungry app).

Since this is basically not a mail server (it only has postfix installed for the websites to use it), I have disabled clamav, spamassassin, pop3 and imap. Especially clamav was indeed a huge memory hog, which was the primary reason I disabled it.

Originally Posted by bajodel

Try to understand which app is swapped:
# top
then press: SHIFT-O
then press: P <enter>

you can see sorted apps for swap usage .. copy & paste here.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
 5761 www-data  20   0  278m 8516 1488 S  0.0  1.7   0:00.41 270m apache2
 5652 www-data  20   0  279m 8728 1544 S  0.0  1.7   0:00.39 270m apache2
 5731 www-data  20   0  279m 8716 1536 S  0.0  1.7   0:00.37 270m apache2
 4456 www-data  20   0  279m 8732 1544 S  0.0  1.7   0:00.52 270m apache2
 6223 www-data  20   0  278m 8536 1496 S  0.0  1.7   0:00.33 270m apache2
 6347 www-data  20   0  278m 8576 1488 S  0.0  1.7   0:00.36 270m apache2
 6229 www-data  20   0  278m 8332 1488 S  0.3  1.7   0:00.32 270m apache2
 6225 www-data  20   0  278m 8408 1516 S  0.0  1.7   0:00.32 270m apache2
 5780 www-data  20   0  278m 8520 1532 S  0.0  1.7   0:00.40 270m apache2
 1867 www-data  20   0  278m 8496 1548 S  0.0  1.7   0:00.36 270m apache2
16034 root      20   0  278m  14m 8700 S  0.0  3.1   0:15.43 263m apache2
 2447 root      20   0  259m  37m 1140 S  0.0  7.6  35:49.74 221m console-kit-dae
10960 mysql     20   0  237m  45m 3788 S  0.7  9.4  84:01.18 191m mysqld
25706 www-data  20   0  162m 5816  468 S  0.0  1.2   0:00.01 156m apache2
26718 postfix   20   0  106m 5188 3840 S  0.0  1.0   0:00.01 101m smtpd
12292 root      20   0 76688 1012  788 S  0.0  0.2   0:06.99  73m sshd
23820 root      20   0 76688 1012  788 S  0.0  0.2   0:06.95  73m sshd
25557 root      20   0 76688 3444 2696 R  0.0  0.7   0:00.10  71m sshd
 2294 root      20   0 56428  332  328 S  0.0  0.1   0:00.05  54m saslauthd
 2295 root      20   0 56428  332  328 S  0.0  0.1   0:00.04  54m saslauthd
 2299 root      20   0 56428  332  328 S  0.0  0.1   0:00.02  54m saslauthd
 2301 root      20   0 56428  332  328 S  0.0  0.1   0:00.02  54m saslauthd
 2302 root      20   0 56428  332  328 S  0.0  0.1   0:00.03  54m saslauthd
 2905 root      20   0 55100 3336 1016 S  0.0  0.7   0:44.08  50m fail2ban-server
 8520 postfix   20   0 52220 1668 1020 S  0.0  0.3   0:00.03  49m qmgr
 1861 root      20   0 48940  396  284 S  0.0  0.1   0:00.23  47m sshd
 8894 postfix   20   0 41612 2156 1248 S  0.0  0.4   0:00.04  38m tlsmgr
 2759 root      20   0 40552 2156  644 S  0.0  0.4   0:17.27  37m munin-node
 2819 postfix   20   0 39104 2132 1680 S  0.0  0.4   0:00.00  36m pickup
26734 postfix   20   0 39104 2136 1684 S  0.0  0.4   0:00.01  36m showq
 2230 root      20   0 37048  724  472 S  0.0  0.1   0:11.92  35m master
30926 root      20   0 31796  384  316 S  0.0  0.1   0:01.50  30m pure-ftpd-mysql
 2157 nobody    20   0 26192  308  200 S  0.0  0.1   0:00.36  25m mydns
 2158 nobody    20   0 27756 2256  520 S  0.0  0.5   0:19.08  24m mydns
 1842 messageb  20   0 22596  836  420 S  0.0  0.2   0:49.90  21m dbus-daemon
14166 ntp       20   0 21384 1188  776 S  0.0  0.2   0:00.11  19m ntpd
 2403 root      20   0 19972  536  384 S  0.0  0.1   0:11.72  18m cron
12350 root      20   0 19056  304  300 S  0.0  0.1   0:00.04  18m bash

I don't think I quite understand the swap column here. What does 270m mean? 270MB? Surely not?!

[bajodel]

Originally Posted by MrM

OK, I've changed it to 0.
EDIT: Should the swap usage decrease without rebooting the server? Or will it only stop growing (in case this setting helps)?

(in case) ..i think it should stop growing.. at least.
No reboot is required.

But ..if you want to test the 'trend' from the initial status (low swap) you can:
# echo "0" > /proc/sys/vm/swappiness (mod swappiness behaviour)
# sync (recommended before drop cached memory)
# echo "3" > /proc/sys/vm/drop_caches (drop chached memory)
# swapoff -a (disable swap)
# swapon -a (re-enable swap)

It's heavy to swallow for your server ..but i think it's (quite) equivalent to rebooting. At worst you can cron that in a script :-) if you cannot find a solution

Originally Posted by MrM

.. 270m apache2
.. 270m apache2
.. 270m apache2

I don't think I quite understand the swap column here. What does 270m mean? 270MB? Surely not?!

For 'top' the default view is kb (when not explicit) ..in your case is surely mb .. but consider:


p: SWAP -- Swapped size (kb)
The swapped out portion of a taskâs total virtual memory image.

o: VIRT -- Virtual Image (kb)
The total amount of virtual memory used by the task. It includes
all code, data and shared libraries plus pages that have been
swapped out.

VIRT = SWAP + RES.

q: RES -- Resident size (kb)
The non-swapped physical memory a task has used.

RES = CODE + DATA.


Bye..

bajodel.

[bajodel]

So.. MrM ..how things are going ?
Is swap devouring your expensive server ?

Bye..

bajodel.