Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Server Operation

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #31  
Old 12th May 2011, 13:49
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,266
Thanks: 82
Thanked 24 Times in 20 Posts
Default

down to a crawl again :-( check it live here: http://zice.ro)

the current relevant apache settings are:



Timeout 120
MaxKeepAliveRequests 100
KeepAliveTimeout 3

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
MaxClients 45
MaxRequestsPerChild 2000
</IfModule>

I am unsure about one thing though: why are these apache settigns relevant at all? After all all my vhosts are using fastCGI + suexec with its own children spawning settings so these shouldn't impact anything!


|-apache2-+-45*[apache2]
| |-apache2---67*[php-cgi]
| `-sh---vlogger


why 45 apache processes?

does this help? http://pastebin.com/WzJsZxxj its the memory dump from yesterday and today... countless apache2 and php-CGI processes...

Last edited by Ovidiu; 12th May 2011 at 14:07.
Reply With Quote
Sponsored Links
  #32  
Old 12th May 2011, 16:36
erosbk erosbk is offline
Senior Member
 
Join Date: Mar 2011
Posts: 337
Thanks: 49
Thanked 36 Times in 30 Posts
Default

Ok, did it eat your memory more quickly than before???? or it looks like always (same speed, same memory consumption, etc). Was there any change after changing the setting? (possitive or negative).
Reply With Quote
  #33  
Old 12th May 2011, 16:39
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,266
Thanks: 82
Thanked 24 Times in 20 Posts
Default

it feels the same. lagging like crazy. although, according to top there was still some ram left swapping still hit more than 2GB :-(

http://zice.ro/munin/serverkompetenz...net/index.html that is where you can verify it.
Reply With Quote
  #34  
Old 12th May 2011, 17:04
erosbk erosbk is offline
Senior Member
 
Join Date: Mar 2011
Posts: 337
Thanks: 49
Thanked 36 Times in 30 Posts
Default

Just for testing purposes... decrease MaxClient to 25 (and MaxRequestPerChild to 100). See memory consumtion after, see if something changes.

MaxClient sets how many connections at same time can apache serve. And MaxRequestsperChild defines after how many requests each of those threads will be recycled. If you have apache set to MaxRequestsPerChild 100 and one thread serves 100 requests, then it will be destroyed, and new apache thread will replace it.

Each time apache serves a request, memory consumption grows, and at the same time apache keep parts of the request in memory to get better performance...

You server is working ok, this is a configuration issue at worst (see swap and ram as a total... memory consumption will increase while you have free memory, trying to satisfay the freedom that you gave in settings for each app...).
Reply With Quote
  #35  
Old 12th May 2011, 17:09
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,266
Thanks: 82
Thanked 24 Times in 20 Posts
Default

I know what these values do and how they work. of course I can limit myself to very few apache processes, but do you explain those 45 apache2 and 67 php-cgi processes below?
the issue is that there is hardly any traffic so how does this result?


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11682 web1 20 0 421m 167m 106m S 0 4.2 0:53.67 php-cgi

and
|-apache2-+-45*[apache2]
| |-apache2---67*[php-cgi]

that's 67 processes each with a huge memory consumption, or do I misunderstand above values?
I'm sure something (a wordpress plugin or some other script) is running rogues here - how would I track what web site is spawning most connections or maybe even which script does it?
Reply With Quote
  #36  
Old 12th May 2011, 20:04
erosbk erosbk is offline
Senior Member
 
Join Date: Mar 2011
Posts: 337
Thanks: 49
Thanked 36 Times in 30 Posts
Thumbs down

If this doesn't change the mem behavior, we could think that apache2 is not the main problem (or not at all). I intended to decrease the apache process loaded in memory (and make them to die and reborn quickly) to see if apache was using cache (and your OS swaping as it increased). This is more a "try and error" than other thing (to isolate the problem).

Next step I think could be to play with fast cgi setting for your sites. I think that in this forum you can find people with huge more knowledge than me to guide you, but if you want, we can try... (always work with backup)

In my opinion, the correct aproach right now would be to identified the site that eats the large memory portion, and work with its conf vhost file (/etc/apache2/sites-available/site.com...)

This are the default options of fast-cgi in ispconfig for each site. So, I think that the best place to work is here... if you can manage to tune up one site, next you can modify the entire fastcgi configuration for all your vsites... (correct me please I am wrong...)

<IfModule mod_fcgid.c>
IdleTimeout 3600
ProcessLifeTime 7200
# MaxProcessCount 1000
DefaultMinClassProcessCount 0
DefaultMaxClassProcessCount 100
IPCConnectTimeout 8
IPCCommTimeout 360
BusyTimeout 300
</IfModule>

I will try this weekend to load my servers with lot of sites trying to re-create your problem and test myself... If I can get to do that, I will try this settings (please, be my guess to try them xD)

<IfModule mod_fcgid.c>
IdleTimeout 600
ProcessLifeTime 7200
# MaxProcessCount 1000
DefaultMinClassProcessCount 0
DefaultMaxClassProcessCount 100
IPCConnectTimeout 8
IPCCommTimeout 360
BusyTimeout 300

IdleScanInterval 60
BusyScanInterval 30
ErrorScanInterval 8
ZombieScanInterval 8
</IfModule>

I think that IdleTimeout 300/600 should help you a lot. In theory, after 5' or 10' an idle fcgi process should be killed... (freeing memory).

Falko, Till, could you explain a little more about the default configuration please? if good to work on them? I think that the problem here is that a lot of fast cgi process are being created, eating memory continuosly without killing them if them are idle (1 hour idle as default).

Edit: maybe could be a good idea to add too: MaxRequestsPerProcess 500
Edit2: read my post on your squirrelmail thread

Edit3: If you have just 1 user accessing a site, it is enought for apache for loading it in memory. You don't need a lot of people accessing a site...

Last edited by erosbk; 12th May 2011 at 21:48.
Reply With Quote
The Following User Says Thank You to erosbk For This Useful Post:
Ovidiu (19th May 2011)
  #37  
Old 12th May 2011, 22:40
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,266
Thanks: 82
Thanked 24 Times in 20 Posts
Default

Quote:
In my opinion, the correct aproach right now would be to identified the site that eats the large memory portion,
how could I determine that? that would already be a huge help for me
Reply With Quote
  #38  
Old 12th May 2011, 22:54
erosbk erosbk is offline
Senior Member
 
Join Date: Mar 2011
Posts: 337
Thanks: 49
Thanked 36 Times in 30 Posts
Default

I think that web9 is the chosen one (or pick two or three clients with lots of proccesses and high memory usage)

Modify the setting, reload apache (or restart? help here please? :P) and CONTROL EXCLUSIVELY what happens with this clients (not the general behavior of the server, only the behavior of the php-cgi proccess of the choosen clients).

be carefull with this, you will be playing with your clients! :P

- another way could be to make a copy of the most ram eater clients, and play with the copies in a more safer environment. What you do is your decition.-
Reply With Quote
  #39  
Old 13th May 2011, 14:30
erosbk erosbk is offline
Senior Member
 
Join Date: Mar 2011
Posts: 337
Thanks: 49
Thanked 36 Times in 30 Posts
Default

Well, if you decrease the life time of an idle process, it will be destroyed soon... si, the OS will not swap it (because it was not used for a while...).

I was researching about how kernel controls memory... I think that the other part of the solution is to change the swap behavior of the operative system.

http://lwn.net/Articles/83588/ (google for more articles, there are lots of web with info about this)

To change to a "try to not swap until strictly necessary" (wait until distress value get really high)

echo 0 > /proc/sys/vm/swappiness

If you want to make this changes persistent at every reboot, add this:

echo "vm/swappiness=0" >> /etc/sysctl.conf

You can play with value between 0 and 100 with swappines (60 default, you MUST decrease it right now... ). In any way, you are changing HOW and WHEN the kernell will swap, you are NOT disabling swaping. 0-10 is recommended for production servers.

Another thing, is ALWAYS recommenden that swap space be separated of your user file space.

Regards

Edit: just a correction in some words... sorry for my poor english xD

Last edited by erosbk; 13th May 2011 at 16:17.
Reply With Quote
  #40  
Old 13th May 2011, 14:58
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,266
Thanks: 82
Thanked 24 Times in 20 Posts
 
Default

thanks for the input. will read up on this, experiment with the swappiness value and look for a client to test with.

web9 isn't good as the guy is moving away in a few days besides, just wanted to mention, this server is just hosting my and some friend's sites so no real customers on it so I can experiment a bit...

##edit###
isn't this deprecated? MaxRequestsPerProcess I only found FCGI_MaxRequestsPerProcess ....

Last edited by Ovidiu; 13th May 2011 at 15:05.
Reply With Quote
Reply

Bookmarks

Tags
apache2, server, tuning

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
ISPConfig 3.0.1.4 causes Apache to have too many open files gkovacs Installation/Configuration 29 27th February 2013 08:59
Whole server went down wxman General 25 28th September 2012 18:32
about: tuning apache2 Ovidiu HOWTO-Related Questions 26 2nd February 2011 12:28
ISPConfig 3 Security mnzava Installation/Configuration 8 8th March 2010 11:00
Swap usage issue MrM General 23 7th October 2009 10:52


All times are GMT +2. The time now is 18:38.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.