yet another apache2 tuning thread

[Ovidiu]

Thanks Till.
Its not quite as you say, I do know how linux manages its RAM and was not worried about having no free RAM, more worried about the fact that it started swaping again.
BEfore I changed all vhosts to fastCGI it constantly had 500MB-1GB of swap and now its already at 123MB again. IS that really OK?

Besides if now with a meager 10 sites with really low traffic pstree tells me that I have -apache2---26*[php-cgi] processes running consuming RAM like hell see here for top sorted by memory consumption:

top - 10:33:57 up 2 days, 22:47,  1 user,  load average: 0.24, 0.33, 0.34
Tasks: 153 total,   1 running, 149 sleeping,   3 stopped,   0 zombie
Cpu(s): 17.6%us,  1.5%sy,  0.0%ni, 80.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4047116k total,  3530392k used,   516724k free,    41828k buffers
Swap:  2104496k total,   424036k used,  1680460k free,  1435564k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30543 web18     20   0  423m 171m  64m S    0  4.4   0:28.68 php-cgi
27067 web18     20   0  423m 170m  62m S    0  4.3   0:51.07 php-cgi
 5160 web18     20   0  418m 164m  61m S    0  4.2   0:07.62 php-cgi
28407 web9      20   0  389m 142m  66m S    0  3.6   0:50.21 php-cgi
23905 web9      20   0  380m 138m  71m S    0  3.5   0:55.83 php-cgi
23881 web9      20   0  374m 129m  67m S    0  3.3   0:49.37 php-cgi
28404 web9      20   0  376m 129m  65m S    0  3.3   0:24.16 php-cgi
28408 web9      20   0  375m 129m  65m S    0  3.3   0:21.66 php-cgi
28406 web9      20   0  375m 125m  60m S    0  3.2   0:15.12 php-cgi
24487 web28     20   0  381m 114m  50m S    0  2.9   0:29.18 php-cgi
23878 web6      20   0  377m 111m  46m S    0  2.8   0:54.99 php-cgi
25104 web6      20   0  377m 111m  46m S   31  2.8   0:49.27 php-cgi
28405 web9      20   0  364m 111m  58m S    4  2.8   0:43.81 php-cgi
24488 web28     20   0  379m 110m  49m S    0  2.8   0:41.04 php-cgi
32341 web6      20   0  377m 110m  45m S    0  2.8   0:10.72 php-cgi
32342 web6      20   0  377m 110m  45m S    0  2.8   0:10.40 php-cgi
 3077 web42     20   0  373m 107m  51m S    0  2.7   0:05.56 php-cgi
 3078 web42     20   0  366m 102m  48m S    0  2.6   0:02.82 php-cgi
24552 web2      20   0  364m  97m  44m S    0  2.5   0:18.12 php-cgi
30980 web2      20   0  363m  90m  38m S    0  2.3   0:07.48 php-cgi
25759 web10     20   0  363m  89m  38m S    0  2.3   0:01.96 php-cgi
32395 web10     20   0  360m  88m  37m S    0  2.2   0:01.82 php-cgi
 3086 web7      20   0  356m  80m  35m S    0  2.0   0:04.80 php-cgi
24526 web7      20   0  352m  75m  32m S    0  1.9   0:21.42 php-cgi
19383 mysql     20   0  533m  58m 3080 S    3  1.5  48:35.04 mysqld
32377 amavis    20   0  227m  57m 3728 S    0  1.5   0:02.36 amavisd-new
 3156 amavis    20   0  227m  57m 3712 S    0  1.4   0:00.96 amavisd-new
23862 clamav    20   0  328m  27m  932 S    0  0.7   0:40.84 clamd
29017 web39     20   0  319m  20m  11m S    0  0.5   0:00.46 php-cgi
 1177 web14     20   0  319m  16m 9.9m S    0  0.4   0:00.10 php-cgi
 5793 postfix   20   0  105m 5976 4456 S    0  0.1   0:00.02 smtpd
 4471 www-data  20   0  389m 5316 2292 S    0  0.1   0:00.26 apache2
 5710 www-data  20   0  389m 5092 2124 S    0  0.1   0:00.06 apache2
 5162 www-data  20   0  389m 5068 2272 S    0  0.1   0:00.06 apache2
 5157 www-data  20   0  389m 4916 2128 S    0  0.1   0:00.04 apache2
28580 postgrey  20   0 62200 4392 1640 S    0  0.1   0:02.94 postgrey
 5795 www-data  20   0  389m 4288 2004 S    0  0.1   0:00.00 apache2

this is the latest top, we are already almost at 500MB swapping. But maybe I am totally wrong? feel free to correct me please.

[Ovidiu]

And now it happened: server is down to sluggish crawling, swapping 2GB.

top - 12:26:09 up 3 days, 39 min,  1 user,  load average: 26.52, 10.09, 3.88
Tasks: 256 total,   1 running, 252 sleeping,   3 stopped,   0 zombie
Cpu(s):  6.6%us,  3.3%sy,  0.8%ni,  0.0%id, 89.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4047116k total,  4025332k used,    21784k free,     5576k buffers
Swap:  2104496k total,  2104460k used,       36k free,   917000k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
20259 web7      20   0  637m  53m  23m S    0  1.3   0:10.42 584m php-cgi
19383 mysql     20   0  578m 125m 3056 S    0  3.2  53:02.56 452m mysqld
14236 web18     20   0  428m  10m  10m S    0  0.3   0:21.74 417m php-cgi
 5160 web18     20   0  418m  11m  11m S    0  0.3   2:18.09 407m php-cgi
19805 root      20   0  388m 1156  216 S    0  0.0   0:17.24 387m apache2
20314 www-data  20   0  389m 3096 2092 S    0  0.1   0:00.02 386m apache2
20316 www-data  20   0  389m 3108 2088 S    0  0.1   0:00.02 386m apache2
20319 www-data  20   0  389m 2860 1900 S    0  0.1   0:00.02 386m apache2
19692 www-data  20   0  389m 3396 2284 S    0  0.1   0:00.06 386m apache2
18979 www-data  20   0  389m 3500 2292 S    0  0.1   0:00.14 386m apache2
20315 www-data  20   0  389m 3404 2104 S    0  0.1   0:00.02 386m apache2
20325 www-data  20   0  389m 3052 1952 S    0  0.1   0:00.00 386m apache2
20276 www-data  20   0  389m 3436 2124 S    0  0.1   0:00.04 385m apache2
19687 www-data  20   0  389m 3668 2128 S    0  0.1   0:00.08 385m apache2
20424 www-data  20   0  389m 4140 1904 S    0  0.1   0:00.00 385m apache2
20350 www-data  20   0  389m 4152 1920 S    0  0.1   0:00.00 385m apache2
20334 www-data  20   0  389m 4164 1952 S    0  0.1   0:00.00 385m apache2
20333 www-data  20   0  389m 4180 1952 S    0  0.1   0:00.00 385m apache2
20257 www-data  20   0  389m 4732 2120 S    0  0.1   0:00.10 385m apache2
20352 www-data  20   0  389m 4192 1956 S    0  0.1   0:00.00 385m apache2
20340 www-data  20   0  389m 4196 1952 S    0  0.1   0:00.00 385m apache2
20341 www-data  20   0  389m 4196 1952 S    0  0.1   0:00.00 385m apache2
20344 www-data  20   0  389m 4196 1952 S    0  0.1   0:00.00 385m apache2
20353 www-data  20   0  389m 4196 1960 S    0  0.1   0:00.00 385m apache2
20356 www-data  20   0  389m 4196 1960 S    0  0.1   0:00.00 385m apache2
20343 www-data  20   0  389m 4200 1956 S    0  0.1   0:00.00 385m apache2
20354 www-data  20   0  389m 4200 1956 S    0  0.1   0:00.00 385m apache2
20357 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20358 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20373 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20378 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20379 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20383 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20385 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20390 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20393 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20394 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2

|-apache2-+-45*[apache2]
| |-apache2---68*[php-cgi]
| `-vlogger

[till]

The above top can not be from normal traffic to that sites if they are low traffic (Or do you mean a few hundred thousand pageviews per day and site with low traffic?) or one of the sites has programming issues. Either its a dos or there is a issue that pages can not be delivered e.g. due to problems in scripts or that e.g. the harddisk has problems.

Please post the apache2.conf file of your server.

Also you should install munin and monit to see if this is e.g. caused by a harddisk which has errors.

[Ovidiu]

No, I really mean low-traffic as in a few GB per month. Don't have any figures handy to show what I mean by low-traffic.

Munin is accessible here, it you can make it come up :-( I don't see any anomaly

http://h1550830.stratoserver.net/mun...net/index.html

apache2.conf:

ServerRoot "/etc/apache2"
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
#<IfModule !mpm_winnt.c>
#<IfModule !mpm_netware.c>
LockFile /var/lock/apache2/accept.lock
#</IfModule>
#</IfModule>

# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

# Timeout: The number of seconds before receives and sends time out.
#
Timeout 120

# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 500

# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 3

##
## Server-Pool Size Regulation (MPM specific)
## 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       3
    MaxSpareServers       5
    MaxClients           45
    MaxRequestsPerChild   5000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          2
    MaxClients           75
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#

AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain


#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog /var/log/apache2/error.log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

# Include module configuration:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

# Include all the user configurations:
Include /etc/apache2/httpd.conf

# Include ports listing
Include /etc/apache2/ports.conf

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

#
# Define an access log for VirtualHosts that don't define their own logfile
CustomLog /var/log/apache2/other_vhosts_access.log vhost_combined

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# Putting this all together, we can internationalize error responses.
#
# We use Alias to redirect any /error/HTTP_<error>.html.var response to
# our collection of by-error message multi-language collections.  We use 
# includes to substitute the appropriate text.
#
# You can modify the messages' appearance without changing any of the
# default HTTP_<error>.html.var files by adding the line:
#
#   Alias /error/include/ "/your/include/path/"
#
# which allows you to create your own set of files by starting with the
# /usr/share/apache2/error/include/ files and copying them to /your/include/path/, 
# even on a per-VirtualHost basis.  The default include files will display
# your Apache version number and your ServerAdmin email address regardless
# of the setting of ServerSignature.
#
# The internationalized error documents require mod_alias, mod_include
# and mod_negotiation.  To activate them, uncomment the following 30 lines.

#    Alias /error/ "/usr/share/apache2/error/"
#
#    <Directory "/usr/share/apache2/error">
#        AllowOverride None
#        Options IncludesNoExec
#        AddOutputFilter Includes html
#        AddHandler type-map var
#        Order allow,deny
#        Allow from all
#        LanguagePriority en cs de es fr it nl sv pt-br ro
#        ForceLanguagePriority Prefer Fallback
#    </Directory>
#
#    ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
#    ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
#    ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
#    ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
#    ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
#    ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
#    ErrorDocument 410 /error/HTTP_GONE.html.var
#    ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
#    ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
#    ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
#    ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
#    ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
#    ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
#    ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
#    ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
#    ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
#    ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var



# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
Include /etc/apache2/conf.d/

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/

#<VirtualHost 85.214.146.68:80>
#ServerName webmail
#ServerAlias mail.*
#ErrorLog /var/www/webmail/log/error.log
#ServerAdmin ovidiu@pacura.ru
#DocumentRoot /var/www/webmail
#DirectoryIndex index.html index.htm index.php index.php5 index.php4 index.php3 $
#AddType application/x-httpd-php .php .php3 .php4 .php5
#</VirtualHost>

#Alias /webmail /var/www/webmail/

#<Directory "/var/www/webmail/">
#Options Indexes FollowSymlinks
#AllowOverride None
#Order allow,deny
#Allow from all
#AddDefaultCharset off
#</Directory>

Alias /piwik /var/www/piwik/

<Directory "/var/www/piwik/">
Options Indexes FollowSymlinks
AllowOverride None
Order allow,deny
Allow from all
AddDefaultCharset off
</Directory>

Alias /x-cache/ /usr/share/xcache/admin/

<Directory "/usr/share/xcache/admin/">
Options Indexes FollowSymlinks
AllowOverride None
Order allow,deny
Allow from all
AddDefaultCharset off
</Directory>

had to reboot/reset, the whole server became totally unresponsive.

[Ovidiu]

ran the strato hardware test with the result: you need to change your hardware :-)

hopefully that will take care of a few things... will post back here afterwards.

[lewis_saint]

What's the strato hardware test? I've having the exact same issue here...

[Ovidiu]

you can find the strato hardware test inside your configuration panel at https://config.stratoserver.net/

head to the menu: Serverkonfiguration => hardwaretest

[Ovidiu]

Got problems again:

4GB of RAM used and $GB of SWAP used!?

have a look here http://screencast.com/t/mWLBf0f48Vu and here please: http://screencast.com/t/xTaF30kvvBzk

any suggestions what else to check?

###edit###
=> |-apache2-+-apache2-+-62*[php-cgi]
| | `-10*[suexec]
| |-50*[apache2]

[erosbk]

Please, use TOP and press M (not m). Post next pls.

[Ovidiu]

thanks but I don't think this will help at all. The system became so unresponsive yesterday, swapped al 4GB and I had to force-restart it. couldn't even connect via ssh anymore.

here is the graphical view of what happened yesterday, check the graphs where the gap is. that is where the system got so unresponsive it wasn't even graphing stuff and where I restarted it: http://zice.ro/munin/serverkompetenz...net/index.html

still, here is hte info from now, everything clean and running smooth:


http://screencast.com/t/QqRr4ep7UU