yet another apache2 tuning thread

Discussion in 'Server Operation' started by Ovidiu, Jan 24, 2011.

  1. falko

    falko Super Moderator

    There are spikes in disk latency and disk usage for /dev/sdb. Maybe there's something wrong with the drive?
     
  2. Ovidiu

    Ovidiu Active Member

    thanks, I saw that. but postfix was busy sending out a huge newsletter and the HD always gets that busy during that period without crashing everything.

    since I moved to this new server just a few months ago, I assumed the HDs were ok...

    I'll run the Strato HW test again tonight, makes the server unreachable for two hours but at least I'll know if there is something wrong...

    btw. that server runs on a software raid1 that came pre-installed on the minimal debian OS they offer. anyone has any links for raid optimizing? any parameters to tune? I haven't touched it yet...
     
  3. erosbk

    erosbk New Member

  4. Ovidiu

    Ovidiu Active Member

    thx, there are a few nice tips in there, will see if it helps figure it all out...
     
  5. Ovidiu

    Ovidiu Active Member

    didn't have time to check the HDs yet, but here we go again:


    |-amavisd-new---2*[amavisd-new]
    |-apache2-+-50*[apache2]
    | |-apache2---60*[php-cgi]

    swapping like crazy.

    top with M pressed: http://screencast.com/t/P4m7xKvs8

    everything is soooo slow :-(

    ###edit###
    managed to dump some info here: http://pastebin.com/qErr4PjE - does that help?
     
    Last edited: May 11, 2011
  6. erosbk

    erosbk New Member

    Are you using prefork or worker? Could you post your apache2.conf
     
  7. Ovidiu

    Ovidiu Active Member

  8. erosbk

    erosbk New Member

    Backup your apache2.conf, make this changes:

    MaxKeepAliveRequests 100
    KeepAlivetimeout 15

    <IfModule mpm_prefork_module>
    StartServers 2
    MinSpareServers 2
    MaxSpareServers 3
    MaxClients 45
    MaxRequestsPerChild 500
    </IfModule>

    I am not an expert, but as I can see apache2 is eating all your memory... I don't know your entire configuration and if this could or not work for you. But I think that you must limit the resources destinated to apache, in order to evade swap (and lack of memory... see what happen in four or five days, your memory will not be enough if you let it as it is right now).

    Try to see if it works better or not than before. Take note of memory consumtion and memory consumption incrementation curve (more slow? more quickly?). If you note any change, it is just a setting problem with this directives. If this not work, you always can rollback the conf file.

    Restart apache2 this night, it was recently restarted, so... it must be running ok right now.

    Regards
     
  9. Ovidiu

    Ovidiu Active Member

    not sure if your settings will help.
    I see you want me to start fewer children and limit the maxrequestsperchild but it doesn't look like some script has a memory leak as I don't have a few apache children with high memory consumption, I have heaps of them...

    default KeepAlivetimeout is 5 - why set it higher? that will tie my ressources down even more :-(

    still, giving some of your changes a try.

    thanks
     
  10. erosbk

    erosbk New Member

    What I want is to isolate you problem (not to find the exactly settings). If you can manage to do that, next you know where to work (if you have changes after this, you can see if you have problems only with apache, or with other process, or whatever)

    When I am blind about a problem, always worked for me to discard little pieces one at a time.

    I give special atention to your case, I will use debian with apache2 in production server in a few days, finding a solution for you will be great for me too. I will keep working on your problem.

    Right know, in my test server, with just 2 test sites with almost no traffic, my ram consumption has growth from 340mb to 1.4gb max in 2 days (with no swaping because I have 4gb ram).

    I think that apache2 debian package is not prepared out of the box to host vhosts... we will see for your health and mine :p

    Regards
     
  11. Ovidiu

    Ovidiu Active Member

    down to a crawl again :-( check it live here: http://zice.ro)

    the current relevant apache settings are:



    Timeout 120
    MaxKeepAliveRequests 100
    KeepAliveTimeout 3

    <IfModule mpm_prefork_module>
    StartServers 2
    MinSpareServers 2
    MaxSpareServers 5
    MaxClients 45
    MaxRequestsPerChild 2000
    </IfModule>

    I am unsure about one thing though: why are these apache settigns relevant at all? After all all my vhosts are using fastCGI + suexec with its own children spawning settings so these shouldn't impact anything!


    |-apache2-+-45*[apache2]
    | |-apache2---67*[php-cgi]
    | `-sh---vlogger


    why 45 apache processes?

    does this help? http://pastebin.com/WzJsZxxj its the memory dump from yesterday and today... countless apache2 and php-CGI processes...
     
    Last edited: May 12, 2011
  12. erosbk

    erosbk New Member

    Ok, did it eat your memory more quickly than before???? or it looks like always (same speed, same memory consumption, etc). Was there any change after changing the setting? (possitive or negative).
     
  13. Ovidiu

    Ovidiu Active Member

  14. erosbk

    erosbk New Member

    Just for testing purposes... decrease MaxClient to 25 (and MaxRequestPerChild to 100). See memory consumtion after, see if something changes.

    MaxClient sets how many connections at same time can apache serve. And MaxRequestsperChild defines after how many requests each of those threads will be recycled. If you have apache set to MaxRequestsPerChild 100 and one thread serves 100 requests, then it will be destroyed, and new apache thread will replace it.

    Each time apache serves a request, memory consumption grows, and at the same time apache keep parts of the request in memory to get better performance...

    You server is working ok, this is a configuration issue at worst (see swap and ram as a total... memory consumption will increase while you have free memory, trying to satisfay the freedom that you gave in settings for each app...).
     
  15. Ovidiu

    Ovidiu Active Member

    I know what these values do and how they work. of course I can limit myself to very few apache processes, but do you explain those 45 apache2 and 67 php-cgi processes below?
    the issue is that there is hardly any traffic so how does this result?


    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    11682 web1 20 0 421m 167m 106m S 0 4.2 0:53.67 php-cgi

    and
    |-apache2-+-45*[apache2]
    | |-apache2---67*[php-cgi]

    that's 67 processes each with a huge memory consumption, or do I misunderstand above values?
    I'm sure something (a wordpress plugin or some other script) is running rogues here - how would I track what web site is spawning most connections or maybe even which script does it?
     
  16. erosbk

    erosbk New Member

    If this doesn't change the mem behavior, we could think that apache2 is not the main problem (or not at all). I intended to decrease the apache process loaded in memory (and make them to die and reborn quickly) to see if apache was using cache (and your OS swaping as it increased). This is more a "try and error" than other thing (to isolate the problem).

    Next step I think could be to play with fast cgi setting for your sites. I think that in this forum you can find people with huge more knowledge than me to guide you, but if you want, we can try... (always work with backup)

    In my opinion, the correct aproach right now would be to identified the site that eats the large memory portion, and work with its conf vhost file (/etc/apache2/sites-available/site.com...)

    This are the default options of fast-cgi in ispconfig for each site. So, I think that the best place to work is here... if you can manage to tune up one site, next you can modify the entire fastcgi configuration for all your vsites... (correct me please I am wrong...)

    <IfModule mod_fcgid.c>
    IdleTimeout 3600
    ProcessLifeTime 7200
    # MaxProcessCount 1000
    DefaultMinClassProcessCount 0
    DefaultMaxClassProcessCount 100
    IPCConnectTimeout 8
    IPCCommTimeout 360
    BusyTimeout 300
    </IfModule>

    I will try this weekend to load my servers with lot of sites trying to re-create your problem and test myself... If I can get to do that, I will try this settings (please, be my guess to try them xD)

    <IfModule mod_fcgid.c>
    IdleTimeout 600
    ProcessLifeTime 7200
    # MaxProcessCount 1000
    DefaultMinClassProcessCount 0
    DefaultMaxClassProcessCount 100
    IPCConnectTimeout 8
    IPCCommTimeout 360
    BusyTimeout 300

    IdleScanInterval 60
    BusyScanInterval 30
    ErrorScanInterval 8
    ZombieScanInterval 8
    </IfModule>

    I think that IdleTimeout 300/600 should help you a lot. In theory, after 5' or 10' an idle fcgi process should be killed... (freeing memory).

    Falko, Till, could you explain a little more about the default configuration please? if good to work on them? I think that the problem here is that a lot of fast cgi process are being created, eating memory continuosly without killing them if them are idle (1 hour idle as default).

    Edit: maybe could be a good idea to add too: MaxRequestsPerProcess 500
    Edit2: read my post on your squirrelmail thread

    Edit3: If you have just 1 user accessing a site, it is enought for apache for loading it in memory. You don't need a lot of people accessing a site...
     
    Last edited: May 12, 2011
  17. Ovidiu

    Ovidiu Active Member

    how could I determine that? that would already be a huge help for me
     
  18. erosbk

    erosbk New Member

    I think that web9 is the chosen one (or pick two or three clients with lots of proccesses and high memory usage)

    Modify the setting, reload apache (or restart? help here please? :p) and CONTROL EXCLUSIVELY what happens with this clients (not the general behavior of the server, only the behavior of the php-cgi proccess of the choosen clients).

    be carefull with this, you will be playing with your clients! :p

    - another way could be to make a copy of the most ram eater clients, and play with the copies in a more safer environment. What you do is your decition.-
     
  19. erosbk

    erosbk New Member

    Well, if you decrease the life time of an idle process, it will be destroyed soon... si, the OS will not swap it (because it was not used for a while...).

    I was researching about how kernel controls memory... I think that the other part of the solution is to change the swap behavior of the operative system.

    http://lwn.net/Articles/83588/ (google for more articles, there are lots of web with info about this)

    To change to a "try to not swap until strictly necessary" (wait until distress value get really high)

    echo 0 > /proc/sys/vm/swappiness

    If you want to make this changes persistent at every reboot, add this:

    echo "vm/swappiness=0" >> /etc/sysctl.conf

    You can play with value between 0 and 100 with swappines (60 default, you MUST decrease it right now... ). In any way, you are changing HOW and WHEN the kernell will swap, you are NOT disabling swaping. 0-10 is recommended for production servers.

    Another thing, is ALWAYS recommenden that swap space be separated of your user file space.

    Regards

    Edit: just a correction in some words... sorry for my poor english xD
     
    Last edited: May 13, 2011
  20. Ovidiu

    Ovidiu Active Member

    thanks for the input. will read up on this, experiment with the swappiness value and look for a client to test with.

    web9 isn't good as the guy is moving away in a few days besides, just wanted to mention, this server is just hosting my and some friend's sites so no real customers on it so I can experiment a bit...

    ##edit###
    isn't this deprecated? MaxRequestsPerProcess I only found FCGI_MaxRequestsPerProcess ....
     
    Last edited: May 13, 2011

Share This Page