need socket specific tuning advice for nginx+php5-fpm

[Ovidiu]

Running Debian Squeeze on a server with 16GB of RAM. Server handles web server, mail server, DB server but I am currently busy setting it up so its almost not in use, so far only this test site has been put in place until its final configuration.

so far I have followed a lot of your tutorials and ended up with:

• MYSQL – tuned
• nginx + php5-fpm – tuned, using on_demand process manager + sockets not TCP/IP + APC + WP object caching to APC
• APC + fast_cgi caching to tmpfs

so to admire my work I tried running an ab test from another server:

ab -n 2000 -c 200 http://pacura.ru/
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking pacura.ru (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
apr_socket_recv: Connection reset by peer (104)
Total of 1087 requests completed

and it ends with: apr_socket_recv: Connection reset by peer (104)

so I googled that error, then checked my error logs on the tuned server and found LOADS of these:

2013/02/08 20:44:39 [crit] 22700#0: *536 open() "/var/www/pacura.ru/web/" failed (13: Permission denied), client: 108.162.231.27, server: pacura.ru, request: "GET / HTTP/1.1", host: "pacura.ru"

So what’s the story here? Clearly ab should be served from the fastcgi_cache so what is failing me here?

I have attempted to tune networking parameters via sysctl:

sysctl -p
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.proxy_arp = 0
kernel.sysrq = 1
net.ipv4.conf.eth0.proxy_arp = 1
net.core.somaxconn = 32768
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_tw_reuse = 1
fs.file-max = 1048576
kernel.pid_max = 1048576
kernel.shmmax = 8589934592
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.netdev_max_backlog = 16384
net.core.optmem_max = 25165824
net.ipv4.tcp_rmem = 4096 1048576 16777216
net.ipv4.tcp_wmem = 4096 1048576 16777216
net.ipv4.tcp_max_syn_backlog = 32768
vm.max_map_count = 131060

During my first test, the page was not in cache so when I first ran the ab test I saw a huge CPU spike but when I tried the test again, no cpu spike since the requests were being server from the fastcgi_cache. Also if I stop php5_fpm and open the page in a browser, I'm getting the cached version so the cache works just fine.

[Ovidiu]

I've been making progress:

ab -k -n 25000 -c 500 http://pacura.ru/
.
.
.
oncurrency Level: 500
Time taken for tests: 40.164 seconds
Complete requests: 25000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 1065375000 bytes
HTML transferred: 1051000000 bytes
Requests per second: 622.44 [#/sec] (mean)
Time per request: 803.286 [ms] (mean)
Time per request: 1.607 [ms] (mean, across all concurrent requests)
Transfer rate: 25903.74 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.4 0 86
Processing: 6 795 725.5 590 3741
Waiting: 6 794 725.5 589 3740
Total: 6 795 726.0 590 3741

Percentage of the requests served within a certain time (ms)
50% 590
66% 874
75% 1137
80% 1293
90% 1700
95% 2396
98% 2918
99% 3232
100% 3741 (longest request)