Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > Linux Forums > Server Operation

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #11  
Old 3rd March 2011, 10:35
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

Thanks Till.
Its not quite as you say, I do know how linux manages its RAM and was not worried about having no free RAM, more worried about the fact that it started swaping again.
BEfore I changed all vhosts to fastCGI it constantly had 500MB-1GB of swap and now its already at 123MB again. IS that really OK?

Besides if now with a meager 10 sites with really low traffic pstree tells me that I have -apache2---26*[php-cgi] processes running consuming RAM like hell see here for top sorted by memory consumption:

Code:
top - 10:33:57 up 2 days, 22:47,  1 user,  load average: 0.24, 0.33, 0.34
Tasks: 153 total,   1 running, 149 sleeping,   3 stopped,   0 zombie
Cpu(s): 17.6%us,  1.5%sy,  0.0%ni, 80.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4047116k total,  3530392k used,   516724k free,    41828k buffers
Swap:  2104496k total,   424036k used,  1680460k free,  1435564k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30543 web18     20   0  423m 171m  64m S    0  4.4   0:28.68 php-cgi
27067 web18     20   0  423m 170m  62m S    0  4.3   0:51.07 php-cgi
 5160 web18     20   0  418m 164m  61m S    0  4.2   0:07.62 php-cgi
28407 web9      20   0  389m 142m  66m S    0  3.6   0:50.21 php-cgi
23905 web9      20   0  380m 138m  71m S    0  3.5   0:55.83 php-cgi
23881 web9      20   0  374m 129m  67m S    0  3.3   0:49.37 php-cgi
28404 web9      20   0  376m 129m  65m S    0  3.3   0:24.16 php-cgi
28408 web9      20   0  375m 129m  65m S    0  3.3   0:21.66 php-cgi
28406 web9      20   0  375m 125m  60m S    0  3.2   0:15.12 php-cgi
24487 web28     20   0  381m 114m  50m S    0  2.9   0:29.18 php-cgi
23878 web6      20   0  377m 111m  46m S    0  2.8   0:54.99 php-cgi
25104 web6      20   0  377m 111m  46m S   31  2.8   0:49.27 php-cgi
28405 web9      20   0  364m 111m  58m S    4  2.8   0:43.81 php-cgi
24488 web28     20   0  379m 110m  49m S    0  2.8   0:41.04 php-cgi
32341 web6      20   0  377m 110m  45m S    0  2.8   0:10.72 php-cgi
32342 web6      20   0  377m 110m  45m S    0  2.8   0:10.40 php-cgi
 3077 web42     20   0  373m 107m  51m S    0  2.7   0:05.56 php-cgi
 3078 web42     20   0  366m 102m  48m S    0  2.6   0:02.82 php-cgi
24552 web2      20   0  364m  97m  44m S    0  2.5   0:18.12 php-cgi
30980 web2      20   0  363m  90m  38m S    0  2.3   0:07.48 php-cgi
25759 web10     20   0  363m  89m  38m S    0  2.3   0:01.96 php-cgi
32395 web10     20   0  360m  88m  37m S    0  2.2   0:01.82 php-cgi
 3086 web7      20   0  356m  80m  35m S    0  2.0   0:04.80 php-cgi
24526 web7      20   0  352m  75m  32m S    0  1.9   0:21.42 php-cgi
19383 mysql     20   0  533m  58m 3080 S    3  1.5  48:35.04 mysqld
32377 amavis    20   0  227m  57m 3728 S    0  1.5   0:02.36 amavisd-new
 3156 amavis    20   0  227m  57m 3712 S    0  1.4   0:00.96 amavisd-new
23862 clamav    20   0  328m  27m  932 S    0  0.7   0:40.84 clamd
29017 web39     20   0  319m  20m  11m S    0  0.5   0:00.46 php-cgi
 1177 web14     20   0  319m  16m 9.9m S    0  0.4   0:00.10 php-cgi
 5793 postfix   20   0  105m 5976 4456 S    0  0.1   0:00.02 smtpd
 4471 www-data  20   0  389m 5316 2292 S    0  0.1   0:00.26 apache2
 5710 www-data  20   0  389m 5092 2124 S    0  0.1   0:00.06 apache2
 5162 www-data  20   0  389m 5068 2272 S    0  0.1   0:00.06 apache2
 5157 www-data  20   0  389m 4916 2128 S    0  0.1   0:00.04 apache2
28580 postgrey  20   0 62200 4392 1640 S    0  0.1   0:02.94 postgrey
 5795 www-data  20   0  389m 4288 2004 S    0  0.1   0:00.00 apache2
this is the latest top, we are already almost at 500MB swapping. But maybe I am totally wrong? feel free to correct me please.
Reply With Quote
Sponsored Links
  #12  
Old 3rd March 2011, 12:28
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

And now it happened: server is down to sluggish crawling, swapping 2GB.

Code:
top - 12:26:09 up 3 days, 39 min,  1 user,  load average: 26.52, 10.09, 3.88
Tasks: 256 total,   1 running, 252 sleeping,   3 stopped,   0 zombie
Cpu(s):  6.6%us,  3.3%sy,  0.8%ni,  0.0%id, 89.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4047116k total,  4025332k used,    21784k free,     5576k buffers
Swap:  2104496k total,  2104460k used,       36k free,   917000k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
20259 web7      20   0  637m  53m  23m S    0  1.3   0:10.42 584m php-cgi
19383 mysql     20   0  578m 125m 3056 S    0  3.2  53:02.56 452m mysqld
14236 web18     20   0  428m  10m  10m S    0  0.3   0:21.74 417m php-cgi
 5160 web18     20   0  418m  11m  11m S    0  0.3   2:18.09 407m php-cgi
19805 root      20   0  388m 1156  216 S    0  0.0   0:17.24 387m apache2
20314 www-data  20   0  389m 3096 2092 S    0  0.1   0:00.02 386m apache2
20316 www-data  20   0  389m 3108 2088 S    0  0.1   0:00.02 386m apache2
20319 www-data  20   0  389m 2860 1900 S    0  0.1   0:00.02 386m apache2
19692 www-data  20   0  389m 3396 2284 S    0  0.1   0:00.06 386m apache2
18979 www-data  20   0  389m 3500 2292 S    0  0.1   0:00.14 386m apache2
20315 www-data  20   0  389m 3404 2104 S    0  0.1   0:00.02 386m apache2
20325 www-data  20   0  389m 3052 1952 S    0  0.1   0:00.00 386m apache2
20276 www-data  20   0  389m 3436 2124 S    0  0.1   0:00.04 385m apache2
19687 www-data  20   0  389m 3668 2128 S    0  0.1   0:00.08 385m apache2
20424 www-data  20   0  389m 4140 1904 S    0  0.1   0:00.00 385m apache2
20350 www-data  20   0  389m 4152 1920 S    0  0.1   0:00.00 385m apache2
20334 www-data  20   0  389m 4164 1952 S    0  0.1   0:00.00 385m apache2
20333 www-data  20   0  389m 4180 1952 S    0  0.1   0:00.00 385m apache2
20257 www-data  20   0  389m 4732 2120 S    0  0.1   0:00.10 385m apache2
20352 www-data  20   0  389m 4192 1956 S    0  0.1   0:00.00 385m apache2
20340 www-data  20   0  389m 4196 1952 S    0  0.1   0:00.00 385m apache2
20341 www-data  20   0  389m 4196 1952 S    0  0.1   0:00.00 385m apache2
20344 www-data  20   0  389m 4196 1952 S    0  0.1   0:00.00 385m apache2
20353 www-data  20   0  389m 4196 1960 S    0  0.1   0:00.00 385m apache2
20356 www-data  20   0  389m 4196 1960 S    0  0.1   0:00.00 385m apache2
20343 www-data  20   0  389m 4200 1956 S    0  0.1   0:00.00 385m apache2
20354 www-data  20   0  389m 4200 1956 S    0  0.1   0:00.00 385m apache2
20357 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20358 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20373 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20378 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20379 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20383 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20385 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20390 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20393 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2
20394 www-data  20   0  389m 4204 1960 S    0  0.1   0:00.00 385m apache2

|-apache2-+-45*[apache2]
| |-apache2---68*[php-cgi]
| `-vlogger
Reply With Quote
  #13  
Old 3rd March 2011, 12:40
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 35,446
Thanks: 813
Thanked 5,216 Times in 4,089 Posts
Default

The above top can not be from normal traffic to that sites if they are low traffic (Or do you mean a few hundred thousand pageviews per day and site with low traffic?) or one of the sites has programming issues. Either its a dos or there is a issue that pages can not be delivered e.g. due to problems in scripts or that e.g. the harddisk has problems.

Please post the apache2.conf file of your server.

Also you should install munin and monit to see if this is e.g. caused by a harddisk which has errors.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #14  
Old 3rd March 2011, 13:03
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

No, I really mean low-traffic as in a few GB per month. Don't have any figures handy to show what I mean by low-traffic.

Munin is accessible here, it you can make it come up :-( I don't see any anomaly

http://h1550830.stratoserver.net/mun...net/index.html

apache2.conf:

Code:
ServerRoot "/etc/apache2"
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
#<IfModule !mpm_winnt.c>
#<IfModule !mpm_netware.c>
LockFile /var/lock/apache2/accept.lock
#</IfModule>
#</IfModule>

# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

# Timeout: The number of seconds before receives and sends time out.
#
Timeout 120

# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 500

# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 3

##
## Server-Pool Size Regulation (MPM specific)
## 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       3
    MaxSpareServers       5
    MaxClients           45
    MaxRequestsPerChild   5000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          2
    MaxClients           75
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#

AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain


#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog /var/log/apache2/error.log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

# Include module configuration:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

# Include all the user configurations:
Include /etc/apache2/httpd.conf

# Include ports listing
Include /etc/apache2/ports.conf

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

#
# Define an access log for VirtualHosts that don't define their own logfile
CustomLog /var/log/apache2/other_vhosts_access.log vhost_combined

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# Putting this all together, we can internationalize error responses.
#
# We use Alias to redirect any /error/HTTP_<error>.html.var response to
# our collection of by-error message multi-language collections.  We use 
# includes to substitute the appropriate text.
#
# You can modify the messages' appearance without changing any of the
# default HTTP_<error>.html.var files by adding the line:
#
#   Alias /error/include/ "/your/include/path/"
#
# which allows you to create your own set of files by starting with the
# /usr/share/apache2/error/include/ files and copying them to /your/include/path/, 
# even on a per-VirtualHost basis.  The default include files will display
# your Apache version number and your ServerAdmin email address regardless
# of the setting of ServerSignature.
#
# The internationalized error documents require mod_alias, mod_include
# and mod_negotiation.  To activate them, uncomment the following 30 lines.

#    Alias /error/ "/usr/share/apache2/error/"
#
#    <Directory "/usr/share/apache2/error">
#        AllowOverride None
#        Options IncludesNoExec
#        AddOutputFilter Includes html
#        AddHandler type-map var
#        Order allow,deny
#        Allow from all
#        LanguagePriority en cs de es fr it nl sv pt-br ro
#        ForceLanguagePriority Prefer Fallback
#    </Directory>
#
#    ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
#    ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
#    ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
#    ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
#    ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
#    ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
#    ErrorDocument 410 /error/HTTP_GONE.html.var
#    ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
#    ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
#    ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
#    ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
#    ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
#    ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
#    ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
#    ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
#    ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
#    ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var



# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
Include /etc/apache2/conf.d/

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/

#<VirtualHost 85.214.146.68:80>
#ServerName webmail
#ServerAlias mail.*
#ErrorLog /var/www/webmail/log/error.log
#ServerAdmin ovidiu@pacura.ru
#DocumentRoot /var/www/webmail
#DirectoryIndex index.html index.htm index.php index.php5 index.php4 index.php3 $
#AddType application/x-httpd-php .php .php3 .php4 .php5
#</VirtualHost>

#Alias /webmail /var/www/webmail/

#<Directory "/var/www/webmail/">
#Options Indexes FollowSymlinks
#AllowOverride None
#Order allow,deny
#Allow from all
#AddDefaultCharset off
#</Directory>

Alias /piwik /var/www/piwik/

<Directory "/var/www/piwik/">
Options Indexes FollowSymlinks
AllowOverride None
Order allow,deny
Allow from all
AddDefaultCharset off
</Directory>

Alias /x-cache/ /usr/share/xcache/admin/

<Directory "/usr/share/xcache/admin/">
Options Indexes FollowSymlinks
AllowOverride None
Order allow,deny
Allow from all
AddDefaultCharset off
</Directory>
had to reboot/reset, the whole server became totally unresponsive.
Reply With Quote
  #15  
Old 3rd March 2011, 19:36
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

ran the strato hardware test with the result: you need to change your hardware :-)

hopefully that will take care of a few things... will post back here afterwards.
Reply With Quote
  #16  
Old 31st March 2011, 20:33
lewis_saint lewis_saint is offline
Member
 
Join Date: Mar 2007
Posts: 69
Thanks: 0
Thanked 3 Times in 2 Posts
Default

What's the strato hardware test? I've having the exact same issue here...
Reply With Quote
  #17  
Old 2nd April 2011, 21:45
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

you can find the strato hardware test inside your configuration panel at https://config.stratoserver.net/

head to the menu: Serverkonfiguration => hardwaretest
Reply With Quote
  #18  
Old 5th May 2011, 16:09
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
Default

Got problems again:

4GB of RAM used and $GB of SWAP used!?

have a look here http://screencast.com/t/mWLBf0f48Vu and here please: http://screencast.com/t/xTaF30kvvBzk

any suggestions what else to check?

###edit###
=> |-apache2-+-apache2-+-62*[php-cgi]
| | `-10*[suexec]
| |-50*[apache2]

Last edited by Ovidiu; 5th May 2011 at 17:06.
Reply With Quote
  #19  
Old 6th May 2011, 03:54
erosbk erosbk is offline
Senior Member
 
Join Date: Mar 2011
Posts: 337
Thanks: 49
Thanked 36 Times in 30 Posts
Default

Please, use TOP and press M (not m). Post next pls.
Reply With Quote
  #20  
Old 6th May 2011, 09:07
Ovidiu Ovidiu is offline
Senior Member
 
Join Date: Sep 2005
Posts: 1,257
Thanks: 75
Thanked 22 Times in 18 Posts
 
Default

thanks but I don't think this will help at all. The system became so unresponsive yesterday, swapped al 4GB and I had to force-restart it. couldn't even connect via ssh anymore.

here is the graphical view of what happened yesterday, check the graphs where the gap is. that is where the system got so unresponsive it wasn't even graphing stuff and where I restarted it: http://zice.ro/munin/serverkompetenz...net/index.html

still, here is hte info from now, everything clean and running smooth:


http://screencast.com/t/QqRr4ep7UU
Reply With Quote
Reply

Bookmarks

Tags
apache2, server, tuning

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
ISPConfig 3.0.1.4 causes Apache to have too many open files gkovacs Installation/Configuration 29 27th February 2013 08:59
Whole server went down wxman General 25 28th September 2012 18:32
about: tuning apache2 Ovidiu HOWTO-Related Questions 26 2nd February 2011 12:28
ISPConfig 3 Security mnzava Installation/Configuration 8 8th March 2010 11:00
Swap usage issue MrM General 23 7th October 2009 10:52


All times are GMT +2. The time now is 04:28.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.