yet another apache2 tuning thread

[Ovidiu]

down to a crawl again :-( check it live here: http://zice.ro)

the current relevant apache settings are:



Timeout 120
MaxKeepAliveRequests 100
KeepAliveTimeout 3

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
MaxClients 45
MaxRequestsPerChild 2000
</IfModule>

I am unsure about one thing though: why are these apache settigns relevant at all? After all all my vhosts are using fastCGI + suexec with its own children spawning settings so these shouldn't impact anything!


|-apache2-+-45*[apache2]
| |-apache2---67*[php-cgi]
| `-sh---vlogger


why 45 apache processes?

does this help? http://pastebin.com/WzJsZxxj its the memory dump from yesterday and today... countless apache2 and php-CGI processes...

[erosbk]

Ok, did it eat your memory more quickly than before???? or it looks like always (same speed, same memory consumption, etc). Was there any change after changing the setting? (possitive or negative).

[Ovidiu]

it feels the same. lagging like crazy. although, according to top there was still some ram left swapping still hit more than 2GB :-(

http://zice.ro/munin/serverkompetenz...net/index.html that is where you can verify it.

[erosbk]

Just for testing purposes... decrease MaxClient to 25 (and MaxRequestPerChild to 100). See memory consumtion after, see if something changes.

MaxClient sets how many connections at same time can apache serve. And MaxRequestsperChild defines after how many requests each of those threads will be recycled. If you have apache set to MaxRequestsPerChild 100 and one thread serves 100 requests, then it will be destroyed, and new apache thread will replace it.

Each time apache serves a request, memory consumption grows, and at the same time apache keep parts of the request in memory to get better performance...

You server is working ok, this is a configuration issue at worst (see swap and ram as a total... memory consumption will increase while you have free memory, trying to satisfay the freedom that you gave in settings for each app...).

[Ovidiu]

I know what these values do and how they work. of course I can limit myself to very few apache processes, but do you explain those 45 apache2 and 67 php-cgi processes below?
the issue is that there is hardly any traffic so how does this result?


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11682 web1 20 0 421m 167m 106m S 0 4.2 0:53.67 php-cgi

and
|-apache2-+-45*[apache2]
| |-apache2---67*[php-cgi]

that's 67 processes each with a huge memory consumption, or do I misunderstand above values?
I'm sure something (a wordpress plugin or some other script) is running rogues here - how would I track what web site is spawning most connections or maybe even which script does it?

[erosbk]

If this doesn't change the mem behavior, we could think that apache2 is not the main problem (or not at all). I intended to decrease the apache process loaded in memory (and make them to die and reborn quickly) to see if apache was using cache (and your OS swaping as it increased). This is more a "try and error" than other thing (to isolate the problem).

Next step I think could be to play with fast cgi setting for your sites. I think that in this forum you can find people with huge more knowledge than me to guide you, but if you want, we can try... (always work with backup)

In my opinion, the correct aproach right now would be to identified the site that eats the large memory portion, and work with its conf vhost file (/etc/apache2/sites-available/site.com...)

This are the default options of fast-cgi in ispconfig for each site. So, I think that the best place to work is here... if you can manage to tune up one site, next you can modify the entire fastcgi configuration for all your vsites... (correct me please I am wrong...)

<IfModule mod_fcgid.c>
IdleTimeout 3600
ProcessLifeTime 7200
# MaxProcessCount 1000
DefaultMinClassProcessCount 0
DefaultMaxClassProcessCount 100
IPCConnectTimeout 8
IPCCommTimeout 360
BusyTimeout 300
</IfModule>

I will try this weekend to load my servers with lot of sites trying to re-create your problem and test myself... If I can get to do that, I will try this settings (please, be my guess to try them xD)

<IfModule mod_fcgid.c>
IdleTimeout 600
ProcessLifeTime 7200
# MaxProcessCount 1000
DefaultMinClassProcessCount 0
DefaultMaxClassProcessCount 100
IPCConnectTimeout 8
IPCCommTimeout 360
BusyTimeout 300

IdleScanInterval 60
BusyScanInterval 30
ErrorScanInterval 8
ZombieScanInterval 8
</IfModule>

I think that IdleTimeout 300/600 should help you a lot. In theory, after 5' or 10' an idle fcgi process should be killed... (freeing memory).

Falko, Till, could you explain a little more about the default configuration please? if good to work on them? I think that the problem here is that a lot of fast cgi process are being created, eating memory continuosly without killing them if them are idle (1 hour idle as default).

Edit: maybe could be a good idea to add too: MaxRequestsPerProcess 500
Edit2: read my post on your squirrelmail thread

Edit3: If you have just 1 user accessing a site, it is enought for apache for loading it in memory. You don't need a lot of people accessing a site...

[Ovidiu]

In my opinion, the correct aproach right now would be to identified the site that eats the large memory portion,

how could I determine that? that would already be a huge help for me

[erosbk]

I think that web9 is the chosen one (or pick two or three clients with lots of proccesses and high memory usage)

Modify the setting, reload apache (or restart? help here please? :P) and CONTROL EXCLUSIVELY what happens with this clients (not the general behavior of the server, only the behavior of the php-cgi proccess of the choosen clients).

be carefull with this, you will be playing with your clients! :P

- another way could be to make a copy of the most ram eater clients, and play with the copies in a more safer environment. What you do is your decition.-

[erosbk]

Well, if you decrease the life time of an idle process, it will be destroyed soon... si, the OS will not swap it (because it was not used for a while...).

I was researching about how kernel controls memory... I think that the other part of the solution is to change the swap behavior of the operative system.

http://lwn.net/Articles/83588/ (google for more articles, there are lots of web with info about this)

To change to a "try to not swap until strictly necessary" (wait until distress value get really high)

echo 0 > /proc/sys/vm/swappiness

If you want to make this changes persistent at every reboot, add this:

echo "vm/swappiness=0" >> /etc/sysctl.conf

You can play with value between 0 and 100 with swappines (60 default, you MUST decrease it right now... ). In any way, you are changing HOW and WHEN the kernell will swap, you are NOT disabling swaping. 0-10 is recommended for production servers.

Another thing, is ALWAYS recommenden that swap space be separated of your user file space.

Regards

Edit: just a correction in some words... sorry for my poor english xD

[Ovidiu]

thanks for the input. will read up on this, experiment with the swappiness value and look for a client to test with.

web9 isn't good as the guy is moving away in a few days besides, just wanted to mention, this server is just hosting my and some friend's sites so no real customers on it so I can experiment a bit...

##edit###
isn't this deprecated? MaxRequestsPerProcess I only found FCGI_MaxRequestsPerProcess ....