yet another apache2 tuning thread

[Ovidiu]

The problem seems to be memory. Running Debian latest with ispcfg3. tried optimizing apache, then placed nginx as a front-end proxy still the same problem. details about the server can be found here, its nice graphs and you will see the dire memory probs I am having: http://h1550830.stratoserver.net/mun...net/index.html

server details:


model name : Dual-Core AMD Opteron(tm) Processor 1214 HE
stepping : 3
cpu MHz : 2200.000
cache size : 1024 KB


cat /proc/meminfo
MemTotal: 4047116 kB
MemFree: 944716 kB
Buffers: 118796 kB
Cached: 1522848 kB
SwapCached: 407656 kB
Active: 2422260 kB
Inactive: 529960 kB
SwapTotal: 2104496 kB
SwapFree: 1046996 kB

current situation:

free
total used free shared buffers cached
Mem: 4047116 3219260 827856 0 118808 1555688
-/+ buffers/cache: 1544764 2502352
Swap: 2104496 1057392 1047104

running about 7-8 low traffic sites, mostly wordpress based. most of the sites didn't have more than 2GB of traffic the whole last year. only one of them had 30GB traffic over the last year.
One of them sends out a daily newsletter at night to about 8000 recipients.

current status looks like this:

ls /etc/apache2/mods-enabled/
actions.conf alias.load authz_default.load authz_user.load cgi.load dir.conf fcgid.conf headers.load mime.load php5.conf rpaf.conf setenvif.conf ssl.load suexec.load
actions.load auth_basic.load authz_groupfile.load autoindex.conf deflate.conf dir.load fcgid.load include.load negotiation.conf php5.load rpaf.load setenvif.load status.conf suphp.conf
alias.conf authn_file.load authz_host.load autoindex.load deflate.load env.load headers.conf mime.conf negotiation.load rewrite.load ruby.load ssl.conf status.load suphp.load

please also see screenshots.

http://screencast.com/t/vg78MAuXi
http://screencast.com/t/zxKoNAtohOP
http://screencast.com/t/WbLjTqw7Q

Please let me know if you need more info or if you have any suggestions.

[till]

First I would recommend that you remove nginx from that setup. Your current hardware is able to run at least 100 or more wordpress blogs.

After you removed nginx, we can start to debug your setup to find out why it uses so much ram.

- Which php mode have you selected for the websites?
- Is suexec enabled?
- Post the output of:

ps aux

[Ovidiu]

thx. lets get started:

- will get to work and disable nginx proxy tonight, meanwhile ehre are the answers to your other questions:

- the 2 biggest sites are running fastcgi + suexec while the rest are running suPhp

ps aux:


USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 10316 88 ? Ss 2010 3:20 init [2]
root 2 0.0 0.0 0 0 ? S< 2010 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S< 2010 0:20 [migration/0]
root 4 0.0 0.0 0 0 ? S< 2010 0:42 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 2010 0:31 [watchdog/0]
root 6 0.0 0.0 0 0 ? S< 2010 0:15 [migration/1]
root 7 0.0 0.0 0 0 ? S< 2010 0:44 [ksoftirqd/1]
root 8 0.0 0.0 0 0 ? S< 2010 0:02 [watchdog/1]
root 9 0.0 0.0 0 0 ? S< 2010 4:19 [events/0]
root 10 0.0 0.0 0 0 ? S< 2010 5:07 [events/1]
root 11 0.0 0.0 0 0 ? S< 2010 0:00 [khelper]
root 46 0.0 0.0 0 0 ? S< 2010 0:11 [kblockd/0]
root 47 0.0 0.0 0 0 ? S< 2010 1:10 [kblockd/1]
root 49 0.0 0.0 0 0 ? S< 2010 0:00 [kacpid]
root 50 0.0 0.0 0 0 ? S< 2010 0:00 [kacpi_notify]
root 144 0.0 0.0 0 0 ? S< 2010 0:00 [ksuspend_usbd]
root 150 0.0 0.0 0 0 ? S< 2010 0:00 [khubd]
root 153 0.0 0.0 0 0 ? S< 2010 0:00 [kseriod]
root 203 0.0 0.0 0 0 ? S< 2010 24:10 [kswapd0]
root 204 0.0 0.0 0 0 ? S< 2010 0:00 [aio/0]
root 205 0.0 0.0 0 0 ? S< 2010 0:00 [aio/1]
root 398 0.0 0.0 0 0 ? S< 2010 0:00 [ata/0]
root 399 0.0 0.0 0 0 ? S< 2010 0:00 [ata/1]
root 400 0.0 0.0 0 0 ? S< 2010 0:00 [ata_aux]
root 404 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_0]
root 406 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_1]
root 414 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_2]
root 416 0.0 0.0 0 0 ? S< 2010 0:00 [scsi_eh_3]
root 936 0.0 0.0 0 0 ? S< 2010 0:01 [md0_raid1]
root 944 0.1 0.0 0 0 ? S< 2010 157:02 [md1_raid1]
root 979 0.0 0.0 0 0 ? S< 2010 94:15 [kjournald]
ovi 984 0.0 0.0 22588 8 ttyS0 S 2010 0:00 -sh
root 1060 0.0 0.0 16936 4 ? S<s 2010 0:00 udevd --daemon
root 1109 0.0 0.0 34536 8 ttyS0 S 2010 0:00 su
root 1110 0.0 0.0 20904 8 ttyS0 S+ 2010 0:00 bash
www-data 1474 0.0 0.0 274440 1692 ? S 03:37 0:06 /usr/sbin/apache2 -k start
root 1477 0.0 0.0 0 0 ? S< 2010 0:00 [kpsmoused]
root 1940 0.0 0.0 8080 384 ? Ss 2010 3:08 /sbin/dhcpcd-bin -Y -N -H eth0
root 2085 0.0 0.0 12368 240 ? Ss 2010 2:26 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
root 2102 0.0 0.0 188284 1244 ? Sl 2010 33:13 /usr/sbin/rsyslogd -c3
root 2129 0.0 0.0 179164 8 ? Ss 2010 0:34 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 2130 0.0 0.0 95152 2020 ? Ss 2010 27:33 /usr/bin/perl -w /usr/sbin/mailgraph -l /var/log/mail.log -d --daemon_rrd=/var/lib/mailgraph
root 2131 0.0 0.0 172440 8 ? S 2010 0:34 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 2132 0.0 0.0 180964 588 ? S 2010 0:35 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 2133 0.0 0.0 177604 8 ? S 2010 0:41 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 2134 0.0 0.0 177964 660 ? S 2010 0:34 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 2142 0.0 0.0 3800 4 ? Ss 2010 0:00 /usr/sbin/acpid
amavis 2157 0.0 0.0 216400 924 ? Ss 2010 4:37 amavisd (master)
web39 2220 0.0 0.1 200432 5504 ? S 00:39 0:00 /usr/bin/php-cgi
root 2430 0.0 0.0 6064 4 ? S 2010 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/authdaemon/pid -start /usr/lib/courier/courier-authlib/authdaemond
root 2431 0.0 0.0 29600 44 ? S 2010 0:45 /usr/lib/courier/courier-authlib/authdaemond
root 2438 0.0 0.0 31696 488 ? S 2010 0:52 /usr/lib/courier/courier-authlib/authdaemond
root 2439 0.0 0.0 31696 488 ? S 2010 0:56 /usr/lib/courier/courier-authlib/authdaemond
root 2440 0.0 0.0 31696 488 ? S 2010 0:51 /usr/lib/courier/courier-authlib/authdaemond
root 2441 0.0 0.0 31696 488 ? S 2010 0:50 /usr/lib/courier/courier-authlib/authdaemond
root 2442 0.0 0.0 31696 488 ? S 2010 0:49 /usr/lib/courier/courier-authlib/authdaemond
root 2446 0.0 0.0 6064 248 ? S 2010 0:08 /usr/sbin/courierlogger -pid=/var/run/courier/imapd.pid -start -name=imapd /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=40 -nodnslo
okup -noidentlookup 143 /usr/lib/courier/courier/imaplogin /usr/bin/imapd Maildir
root 2447 0.0 0.0 9236 128 ? S 2010 0:19 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=40 -nodnslookup -noidentlookup 143 /usr/lib/courier/courier/imaplogin /usr/bin/imapd M
--More--

aildir
root 2458 0.0 0.0 6064 248 ? S 2010 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/imapd-ssl.pid -start -name=imapd-ssl /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=40
-nodnslookup -noidentlookup 993 /usr/bin/couriertls -server -tcpd /usr/lib/courier/courier/imaplogin /usr/bin/imapd Maildir
root 2459 0.0 0.0 9236 128 ? S 2010 0:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=40 -nodnslookup -noidentlookup 993 /usr/bin/couriertls -server -tcpd /usr/lib/courier/
courier/imaplogin /usr/bin/imapd Maildir
root 2464 0.0 0.0 6064 248 ? S 2010 0:08 /usr/sbin/courierlogger -pid=/var/run/courier/pop3d.pid -start -name=pop3d /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noiden
tlookup -address=0 110 /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir
root 2465 0.0 0.0 9236 124 ? S 2010 0:19 /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 /usr/lib/courier/courier/courierpop3login /usr/lib/c
ourier/courier/courierpop3d Maildir
root 2476 0.0 0.0 6064 224 ? S 2010 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/pop3d-ssl.pid -start -name=pop3d-ssl /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -
nodnslookup -noidentlookup 995 /usr/bin/couriertls -server -tcpd /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir
root 2477 0.0 0.0 9236 120 ? S 2010 0:04 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup 995 /usr/bin/couriertls -server -tcpd /usr/lib/courier/c
ourier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir
root 2492 0.0 0.0 21920 248 ? Ss 2010 4:49 /usr/sbin/cron
root 2507 0.0 0.1 155696 4128 ? Sl 2010 49:38 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
nobody 2511 0.0 0.0 31028 2352 ? S 04:04 0:06 /usr/local/sbin/mydns -b
nobody 2539 0.0 0.0 68096 340 ? Sl 2010 2:03 /usr/sbin/imapproxyd -f /etc/imapproxy.conf
ntp 2554 0.0 0.0 22384 592 ? Ss 2010 4:42 /usr/sbin/ntpd -p /var/run/ntpd.pid -u 102:104 -g
postgrey 2563 0.0 0.1 57100 4496 ? Ss 2010 1:14 /usr/sbin/postgrey --pidfile=/var/run/postgrey.pid --daemonize --inet=127.0.0.1:60000
root 2571 0.0 0.0 3792 136 ? SNs 2010 94:55 /usr/sbin/powernowd -q
root 2588 0.0 0.0 11924 232 ? S 2010 0:35 /usr/sbin/smartd --pidfile /var/run/smartd.pid
root 2589 0.0 0.0 137548 1508 ? S 04:08 0:00 spamd child
root 2603 0.0 0.0 48868 344 ? Ss 2010 0:32 /usr/sbin/sshd
web7 2727 0.0 0.2 312628 9712 ? S 04:09 0:03 /usr/bin/php-cgi
root 2757 0.0 0.0 137548 1504 ? S 04:09 0:00 spamd child
root 2920 0.0 0.0 36844 476 ? Ss 2010 20:51 /usr/lib/postfix/master
nobody 2933 0.0 0.4 184816 18488 ? Sl 2010 59:36 /usr/bin/memcached -m 64 -p 11211 -u nobody -l 127.0.0.1
root 3342 0.0 0.0 3800 8 tty1 Ss+ 2010 0:00 /sbin/getty 38400 tty1
root 3343 0.0 0.0 45636 8 ttyS0 Ss 2010 0:00 /bin/login --
postfix 3357 0.0 0.0 41568 864 ? S 2010 1:45 tlsmgr -l -t unix -u -c
root 3986 0.0 0.0 273516 916 ? Ss Jan23 0:16 /usr/sbin/apache2 -k start
root 3987 0.0 0.0 39180 2872 ? S Jan23 0:36 vlogger (access log)
www-data 3988 0.0 0.0 160072 1116 ? S Jan23 0:11 /usr/sbin/apache2 -k start
web10 4170 0.0 0.0 266492 8 ? Ss Jan23 0:00 /usr/bin/php-cgi
web10 4172 0.0 0.7 275800 30512 ? S Jan23 0:16 /usr/bin/php-cgi
web10 4173 0.0 0.7 301336 29980 ? S Jan23 0:18 /usr/bin/php-cgi
web10 4175 0.0 0.6 277516 28052 ? S Jan23 0:17 /usr/bin/php-cgi
web10 4176 0.0 0.6 273860 28224 ? S Jan23 0:16 /usr/bin/php-cgi
web10 4177 0.0 0.7 275700 29892 ? S Jan23 0:17 /usr/bin/php-cgi
web10 4178 0.0 0.6 275724 28092 ? S Jan23 0:17 /usr/bin/php-cgi
web10 4179 0.0 0.6 275596 28104 ? S Jan23 0:17 /usr/bin/php-cgi
web10 4182 0.0 0.7 275724 29728 ? S Jan23 0:17 /usr/bin/php-cgi
root 7874 0.0 0.0 30468 16 ? S Jan19 0:00 /USR/SBIN/CRON
munin 7875 0.0 0.0 0 0 ? Zs Jan19 0:00 [sh] <defunct>
munin 7943 0.0 0.0 117892 8 ? S Jan19 0:06 /usr/share/munin/munin-update [Munin::Master::UpdateWorker<stratoserver.net;h1550 830.stratoserver.net>]
amavis 9343 0.0 0.6 221620 25832 ? S 15:20 0:06 amavisd (ch18-accept)
root 9411 0.0 0.0 214464 212 ? Ss Jan04 0:00 nginx: master process /usr/sbin/nginx
www-data 9412 0.0 0.0 215668 1772 ? S Jan04 10:51 nginx: worker process
www-data 9413 0.0 0.0 215428 1672 ? S Jan04 11:06 nginx: worker process
www-data 9414 0.0 0.0 214620 292 ? S Jan04 1:34 nginx: cache manager process
web10 9962 0.0 0.1 266492 7880 ? Ss 09:18 0:00 /usr/bin/php-cgi
web10 9963 0.0 0.7 300784 30512 ? S 09:18 0:06 /usr/bin/php-cgi
web10 9964 0.0 0.7 273748 29948 ? S 09:18 0:05 /usr/bin/php-cgi
web10 9965 0.0 0.7 274780 29824 ? S 09:18 0:05 /usr/bin/php-cgi
web10 9966 0.0 0.7 277512 28956 ? S 09:18 0:05 /usr/bin/php-cgi
web10 9967 0.0 0.7 272856 29112 ? S 09:18 0:04 /usr/bin/php-cgi
web10 9968 0.0 0.7 273324 29436 ? S 09:18 0:04 /usr/bin/php-cgi
web10 9969 0.0 0.7 275700 30444 ? S 09:18 0:05 /usr/bin/php-cgi
web10 9970 0.0 0.7 273112 28932 ? S 09:18 0:05 /usr/bin/php-cgi
munin 11959 0.0 0.0 36836 8 ? S Jan19 0:00 /usr/sbin/sendmail -i -FCronDaemon -oem root
--More--

munin 11962 0.0 0.0 36824 8 ? S Jan19 0:00 /usr/sbin/postdrop -r
root 12389 0.0 0.0 0 0 ? S 21:36 0:00 [pdflush]
root 13430 0.0 0.0 12284 8 pts/2 S Jan22 0:00 /bin/sh /usr/bin/mysqld_safe
mysql 13467 2.7 8.1 799308 329828 pts/2 Sl Jan22 115:08 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=330
6 --socket=/var/run/mysqld/mysqld.sock
root 13468 0.0 0.0 7124 8 pts/2 S Jan22 0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
root 14421 0.0 0.0 32816 108 pts/1 S+ 2010 5:09 mc
root 14423 0.0 0.0 21120 1056 pts/2 Ss 2010 0:02 bash -rcfile .bashrc
root 14557 0.0 0.0 27176 1168 ? Ss 2010 9:25 SCREEN
root 14559 0.0 0.0 20972 8 pts/1 Ss 2010 0:00 /bin/bash
root 15153 0.0 0.0 0 0 ? S 21:50 0:00 [pdflush]
root 16896 0.0 0.0 65932 3024 ? Ss 21:57 0:00 sshd: ovi [priv]
ovi 16946 0.0 0.0 66228 1936 ? S 21:58 0:00 sshd: ovi@pts/0
ovi 16949 0.0 0.0 22588 3388 pts/0 Ss 21:58 0:00 -sh
root 16963 0.0 0.0 34536 1180 pts/0 S 21:58 0:00 su
root 16964 0.0 0.0 20912 1900 pts/0 S 21:58 0:00 bash
root 16967 0.0 0.0 26768 1140 pts/0 S+ 21:58 0:00 screen -r
nobody 17875 0.0 0.0 29452 408 ? Ss Jan22 0:10 /usr/local/sbin/mydns -b
postfix 18613 0.0 0.0 52228 804 ? S 2010 3:14 qmgr -l -t fifo -u
web9 19832 1.4 3.1 328128 128976 ? S 22:11 0:11 /usr/bin/php-cgi
web9 19833 1.0 2.1 300460 85540 ? S 22:11 0:08 /usr/bin/php-cgi
web9 19834 2.1 3.2 318772 131088 ? S 22:11 0:17 /usr/bin/php-cgi
web9 19835 0.9 2.5 306676 101944 ? S 22:11 0:07 /usr/bin/php-cgi
web9 19838 1.0 3.1 314696 126660 ? S 22:11 0:08 /usr/bin/php-cgi
web9 19845 1.0 3.0 320108 122588 ? S 22:11 0:08 /usr/bin/php-cgi
web9 19849 1.1 2.6 303144 107640 ? S 22:11 0:09 /usr/bin/php-cgi
web9 19850 0.3 2.7 322692 111096 ? S 22:11 0:02 /usr/bin/php-cgi
root 19938 0.0 0.0 45324 2108 ? Ss Jan04 3:24 /usr/sbin/munin-node
root 20532 0.0 0.0 34072 192 ? Ss Jan23 0:00 pure-ftpd (SERVER)

postfix 21180 0.0 0.0 38904 2284 ? S 22:20 0:00 pickup -l -t fifo -u -c
root 21664 0.0 0.0 137548 2440 ? Ss Jan23 0:24 /usr/sbin/spamd --create-prefs --max-children 5 --helper-home-dir -d --pidfile=/var/run/spamd.pid
www-data 21859 1.0 0.1 275136 5996 ? S 22:23 0:01 /usr/sbin/apache2 -k start
www-data 21868 0.9 0.3 277744 12752 ? S 22:23 0:00 /usr/sbin/apache2 -k start
www-data 21880 0.9 0.3 277872 13004 ? S 22:23 0:00 /usr/sbin/apache2 -k start
www-data 21897 0.9 0.1 274564 5664 ? S 22:24 0:00 /usr/sbin/apache2 -k start
www-data 21910 0.3 0.3 277636 12800 ? S 22:24 0:00 /usr/sbin/apache2 -k start
root 21923 0.0 0.0 30464 956 ? S 22:25 0:00 /USR/SBIN/CRON
munin 21924 0.0 0.0 8832 1140 ? Ss 22:25 0:00 /bin/sh -c if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
munin 21926 0.0 0.0 8836 1148 ? S 22:25 0:00 /bin/sh /usr/bin/munin-cron
munin 21927 4.1 0.3 115948 15964 ? S 22:25 0:00 /usr/bin/perl /usr/share/munin/munin-update
munin 21947 0.0 0.3 116080 14284 ? S 22:25 0:00 /usr/share/munin/munin-update [Munin::Master::UpdateWorker<stratoserver.net;h1550 830.stratoserver.net>]
root 21948 0.0 0.0 45324 3888 ? S 22:25 0:00 /usr/sbin/munin-node [127.0.0.1]
amavis 21950 0.1 0.0 0 0 ? Zs 22:25 0:00 [amavis_cache] <defunct>
postfix 21964 0.1 0.1 104156 5260 ? S 22:25 0:00 smtpd -n smtp -t inet -u -c -o stress
postfix 21977 0.0 0.0 38904 2280 ? S 22:25 0:00 showq -t unix -u -c
web7 21988 118 2.4 287300 99980 ? R 22:25 0:01 /usr/bin/php-cgi
root 21989 0.0 0.0 18100 1108 pts/2 R+ 22:25 0:00 ps aux
root 21990 0.0 0.0 21120 464 pts/2 R+ 22:25 0:00 bash -rcfile .bashrc
clamav 24547 0.0 0.0 43120 960 ? Ss 2010 0:19 /usr/bin/freshclam -d --quiet
clamav 25101 0.0 3.3 341432 135800 ? Ssl 2010 22:59 /usr/sbin/clamd
amavis 25138 0.0 0.8 221232 33184 ? S 16:50 0:02 amavisd (ch9-accept)
root 27919 0.0 0.0 73560 1452 ? Ss 2010 5:52 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 29740 0.0 0.0 12384 8 pts/2 T 2010 0:00 /bin/bash /etc/init.d/mysql restart
root 29745 0.0 0.0 12396 8 pts/2 T 2010 0:00 /bin/bash /etc/init.d/mysql stop
root 29790 0.0 0.0 12396 4 pts/2 T 2010 0:00 /bin/bash /etc/init.d/mysql stop
root 29791 0.0 0.0 26728 8 pts/2 T 2010 0:00 /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf shutdown

[Ovidiu]

Ok, I removed nginx and am ready for the next steps. What else do you need to know?

[till]

You should switch all sites to fastcgi + suexec. Then make sure that you use the latest ispconfig (3.0.3.2) as older versions might use a outdated vhost template which has a higher resource usage.

Then make sure that you run a php binary cache, I recommend xcache. It is installed with:

apt-get install php5-xcache

afterwards restart apache.

[Ovidiu]

- I do use ISPCFG 3.0.3.2
- changed all sites to fastcgi + suexec
- installed xcache and restarted apache2

Lets give it a day or so and I'll report back with any findings.

what info would you need?