HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials - Wordpress admin on nginx+php5-fpm on VPS incredibly slow. Other apps/WP front end OK

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)

- General (http://www.howtoforge.com/forums/forumdisplay.php?f=25)

- - Wordpress admin on nginx+php5-fpm on VPS incredibly slow. Other apps/WP front end OK (http://www.howtoforge.com/forums/showthread.php?t=55383)

Wordpress admin on nginx+php5-fpm on VPS incredibly slow. Other apps/WP front end OK

I've got extremely slow Wordpress under ISPConfig 3.0.4.1, but it's the back end only. I updated to WP 3.3 this morning, but still the same.

The strange thing is, I'm running a personal WP install on a free "tiny" instance on Amazon S3 with a similar config and low memory, but not ispc and it's Amazon's own RHEL, and it flies along. Which is what makes it even more difficult to diagnose.

Anyway, here's a whole load of details:

Wordpress 3.3 on Ubuntu 11.10 + Nginx 1.0.10 + php5-fpm 5.3.8 + ISPconfig 3.0.4.1 + 256Mb VPS

I've got a 256Mb VPS running a Zen Cart store and phpbb3, both in different php-fpm pools. There's hardly anything running except the essentials and both those sites absolutely rocket along. As does the front end of the Wordpress site, when W3TC accelerated.

BUT.... the admin side takes 6-10 seconds to do anything.
There's nothing in the mysql slow log, or the php-fpm error log, the load doesn't spike, and the memory usage doesn't shoot up (but see below about memory).

The first time it loads, at wp-admin/options.php it shows a very long page that looks wrong, with line after line of stuff like...

active_plugins SERIALIZED DATA

Here's the main items from ps_mem.py

Code:

732.0 KiB +  87.5 KiB = 819.5 KiB       bash

  2.1 MiB + 369.0 KiB =   2.4 MiB       fail2ban-server

  1.8 MiB +   2.0 MiB =   3.9 MiB       nginx (5)

  5.1 MiB +  12.8 MiB =  17.9 MiB       php5-fpm (29)

 87.8 MiB + 149.0 KiB =  88.0 MiB       mysqld

---------------------------------

                        116.2 MiB

=================================

Here's the load average pretty much all the time:
load average: 0.48, 0.53, 0.51

And here's the output from free -m

Code:

             total       used       free     shared    buffers     cached

Mem:           241        202         38          0          3         49

-/+ buffers/cache:        149         92

Swap:          511         29        482

Here's the nginx.conf, including the cloudflare real_ip settings (tried without cloudflare too), and also the rewrite required to make permalinks work under nginx:

Code:

server {

        listen 31.172.x.x:80;

                

        

        server_name mysite.co.uk www.mysite.co.uk www.my-site.co.uk my-site.co.uk;



        root   /var/www/mysite.co.uk/web;

                                

        index index.html index.htm index.php index.cgi index.pl index.xhtml;

                

        error_page 400 /error/400.html;

        error_page 401 /error/401.html;

        error_page 403 /error/403.html;

        error_page 404 /error/404.html;

        error_page 405 /error/405.html;

        error_page 500 /error/500.html;

        error_page 502 /error/502.html;

        error_page 503 /error/503.html;

                

        error_log /var/log/ispconfig/httpd/mysite.co.uk/error.log;

        access_log /var/log/ispconfig/httpd/mysite.co.uk/access.log combined;



        ## Disable .htaccess and other hidden files

        location ~ /\. {

            deny all;

            access_log off;

            log_not_found off;

        }

                

        location = /favicon.ico {

            log_not_found off;

            access_log off;

        }



        location = /robots.txt {

            allow all;

            log_not_found off;

            access_log off;

        }

                

        location /stats {

            index index.html index.php;

            auth_basic "Members Only";

            auth_basic_user_file /var/www/clients/client3/web9/.htpasswd_stats;

        }



        location ~ \.php$ {

            try_files $uri =404;

            include /etc/nginx/fastcgi_params;

            fastcgi_pass unix:/var/lib/php5-fpm/web9.sock;

            fastcgi_index index.php;

            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

            fastcgi_param PATH_INFO $fastcgi_script_name;

            fastcgi_intercept_errors on;

        }

        

         set_real_ip_from   204.93.240.0/24;

          set_real_ip_from   204.93.177.0/24;

          set_real_ip_from   199.27.128.0/21;

          set_real_ip_from   173.245.48.0/20;

          set_real_ip_from   103.22.200.0/22;

          set_real_ip_from   141.101.64.0/18;

          set_real_ip_from   108.162.192.0/18;

          real_ip_header     CF-Connecting-IP;

        client_max_body_size 28M;

        client_body_buffer_size 128k;

        

        if (!-e $request_filename) {

                rewrite  ^(.*)$  /index.php?q=$1  last;

                break;

            }

        #include /var/www/mysite.co.uk/web/nginx.conf;             

}

Here's the php5-fpm pool conf for that site:

Code:

[web9]



listen = /var/lib/php5-fpm/web9.sock

listen.owner = web9

listen.group = client3

listen.mode = 0660



user = web9

group = client3



pm = dynamic

pm.max_children = 4

pm.start_servers = 1

pm.min_spare_servers = 1

pm.max_spare_servers = 2



chdir = /



php_admin_value[open_basedir] = /var/www/clients/client3/web9/web:/var/www/clients/client3/web9/tmp:/var/www/mysite.co.uk/web:/srv/www/mysite.co.uk/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin

php_admin_value[session.save_path] = /var/www/clients/client3/web9/tmp

php_admin_value[upload_tmp_dir] = /var/www/clients/client3/web9/tmp



php_admin_value[date.timezone] = "UTC"

php_admin_value[post_max_size] = 28M

php_admin_value[session.gc_maxlifetime] = 604800

php_admin_value[upload_max_filesize] = 28M

php_admin_flag[display_errors] = off

php_admin_flag[display_startup_errors] = off

php_admin_flag[log_errors] = off

php_admin_flag[ignore_repeated_errors] = off

php_admin_flag[ignore_repeated_source] = off

php_admin_value[memory_limit] = 32M

That last line about memory limit is new, but made no difference either.

I had to make changes to /etc/php5/conf.d/suhosin.ini as advised by phpmyadmin, and also upped the memory limit for WP as I was getting
"ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value".

Code:

; configuration for php suhosin module

extension=suhosin.so

suhosin.executor.include.whitelist="phar"

suhosin.request.max_vars=2048

suhosin.post.max_vars=2048

suhosin.request.max_array_index_length=256

suhosin.post.max_array_index_length=256

suhosin.request.max_totalname_length=8192

suhosin.post.max_totalname_length=8192

suhosin.get.max_value_length=1024

suhosin.memory_limit=128M

I reduced the memory limit in wp-config.php as shown below.

Code:

define('WP_MEMORY_LIMIT', '32M');

define('WP_MAX_MEMORY_LIMIT', '32M');

Although, I've changed these limits from 256 to 128 to 64 to 32 and it makes NO difference to the front or back end speeds.

I changed the theme to default, turned off all the plugins, changed all the mysql tables to to innodb and followed the recommendations of mysqltuner (even though, as I said, there's nothing about slow mysql in the logs I can see).

I've tried changing php-fpm from socket to port and back, and so on and so forth. Not really sure what else to do now - can anyone spot anything here or advise?

And yes, I might be running a bit tight on memory, but then why does Zen Cart and phpbb running with a big DB load pages sub 200ms?

I guess thats you have a problem with I/O Bandwidth or in other words, the harddisk speed is too slow. I've seen this problem several times with vps servers when too many vps are on the same host system. This can also affect mysql performance. I guess there is not much that you can do about this except of talking with your hosting provider.

What you can try to optimize mysql;

http://www.faqforge.com/linux/optimi...th-mysqltuner/

If your sites dont use innodb, then you should consider to switch innodb off in my.cnf file, so that mysql uses less memory.

Quote:

but then why does Zen Cart and phpbb running with a big DB load pages sub 200ms?

Most likely they cache the results better.

Quote:

Originally Posted by till (Post 269509)

As per my post above:

Quote:

Originally Posted by talkingnews (Post 269506)

I changed the theme to default, turned off all the plugins, changed all the mysql tables to to innodb and followed the recommendations of mysqltuner (even though, as I said, there's nothing about slow mysql in the logs I can see).

Quote:

Originally Posted by till (Post 269509)

Most likely they cache the results better.

Possibly, but here's the output from Zen Cart's diagnostics:

Code:

Database Data Size: 25,936 kB        Database Index Size: 6,496 kB

Parse Time: 0.119 - Number of Queries: 54 - Query Time: 0.008176764892578

So why would an old, non-optimized app manage to query a 26mb database and parse a complex page in 0.119s, while Wordpress admin takes 6-9 seconds to load a 2mb db?

Quote:

I changed the theme to default, turned off all the plugins, changed all the mysql tables to to innodb and followed the recommendations of mysqltuner (even though, as I said, there's nothing about slow mysql in the logs I can see).

Changing tables to innodb causes mysql to use more ram so that the performance can decrease on low memory systems.

Quote:

So why would an old, non-optimized app manage to query a 26mb database and parse a complex page in 0.119s, while Wordpress admin takes 6-9 seconds to load a 2mb db?

You compare apples with pies here. Not every mysql query is the same and the raw database size is only one of several parameters that affects mysql performance. If a query is slow or fast depends on a lot of things like joins, if it has good indexes set and if the temp table has to be created on the harddisk etc.

One indicator for low harddisk performance is e.g. when you run:

rkhunter -c

and the results are slow, rkhunter searches trough a lot of files, so if its slow, then your harddisk performance is low.

One other thing that you might want to check is if the nameservers in /etc/resolv.conf are rechable and responding fast.

I found the answer! Scroll down, but first, some answers...

Quote:

Originally Posted by till (Post 269520)

Changing tables to innodb causes mysql to use more ram so that the performance can decrease on low memory systems.

With Innodb being the standard for the current mysql, I thought I'd do some digging around here. Looking at speed/memory comparisons from 2008/2009, that might have been the case, but now, most tests put innodb as around 25% faster. So while it might use a little more memory, that memory is used for less time.

Quote:

Originally Posted by till (Post 269520)

.

Well, I think we can deal with the DB side of things quite quickly - here's the output from wordpress in debug mode:

[0] => SELECT wp_posts.* FROM wp_posts WHERE 1=1 AND wp_posts.ID = 4 AND wp_posts.post_type = 'page' ORDER BY wp_posts.post_date DESC
[1] => 0.032161951065063
[2] => require, wp, WP->main, WP->query_posts, WP_Query->query, WP_Query->get_posts, W3_Db->query

[0] => SELECT p.* FROM wp_posts AS p WHERE p.post_date < '2011-09-14 09:45:51' AND p.post_type = 'page' AND p.post_status = 'publish' ORDER BY p.post_date DESC LIMIT 1
[1] => 0.0060410499572754
[2] => require, require_once, include, get_header, locate_template, load_template, require_once, wp_head, do_action, call_user_func_array, adjacent_posts_rel_link_wp_head, adjacent_posts_rel_link, get_adjacent_post_rel_link, get_adjacent_post, W3_Db->query

[0] => SELECT p.* FROM wp_posts AS p WHERE p.post_date > '2011-09-14 09:45:51' AND p.post_type = 'page' AND p.post_status = 'publish' ORDER BY p.post_date ASC LIMIT 1
[1] => 0.00027585029602051
[2] => require, require_once, include, get_header, locate_template, load_template, require_once, wp_head, do_action, call_user_func_array, adjacent_posts_rel_link_wp_head, adjacent_posts_rel_link, get_adjacent_post_rel_link, get_adjacent_post, W3_Db->query

[0] => SELECT comment_approved, COUNT( * ) AS num_comments FROM wp_comments GROUP BY comment_approved
[1] => 0.00011110305786133
[2] => require, require_once, include, get_footer, locate_template, load_template, require_once, wp_footer, do_action, call_user_func_array, wp_admin_bar_render, do_action_ref_array, call_user_func_array, wp_admin_bar_comments_menu, wp_count_comments, W3_Db->query

So that doesn't even total 0.1 seconds for the db query part of things.

Quote:

Originally Posted by till (Post 269520)

One other thing that you might want to check is if the nameservers in /etc/resolv.conf are rechable and responding fast.

DNS is totally turned off, and handled by either Cloudflare, Amazon Route53 or ZoneEdit. The server doesn't run sendmail, smpt, pop3, dns... it just runs mysql, nginx, php5-fpm and ftp (oh, and a couple of basics like rkhunter!) but I only need websites, not other things. The front end runs fine, it was only the back end of WP that was slow. How would DNS affect this?

So, in fact, it was none of those things....

Two days of my life and a helluva lot of learning about things like xdebug, I can safely say you CANNOT now run WordPress on a VPS with less than 512Mb RAM. Which is a shame, because the 256Mb VPS had always been perfectly good before.

Previously, it had been OK. You remember in my original post, I noted the line about "ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value"?

Well, I also noted that I didn't see any memory spiking in the command line utility "top". But clearly, that doesn't refresh fast enough. With a bit of help from the VPS host who has graphical tools, we were seeing bursts of 228Mb usage when the ADMIN side of WP 3.3 was trying to load. But the front end? Well, as I said before, that zipped along and barely even registered on their graph.

I also mentioned that I tried limiting memory use in both php.ini and the wordpress config. Well, I stepped down through 256 to 128 to 64 to 32, although from 128Mb, it would only serve blank pages.

EDIT: Turns out this is confirmed by another blogger:
http://www.dev4press.com/2011/blog/b...vs-3-2-vs-3-3/

Quote:

I can’t end this benchmark on a positive note as I did back in July for WordPress 3.2. I can understand that WordPress is evolving and that new features can need more resources. But, in WordPress 3.2 six months ago, features and changes were followed by considerable optimization, and that resulted in faster WordPress. This time, we got no significant new features or improvements, apart from cosmetic ones (and some of them are questionable at best) and some changes that affected only some features (uploader and editor), and yet new WordPress is gone back a full year in terms of performance, and it is comparable to WP 3.1 or even WP 3.0 in some cases.

I've temporarily moved to a 512Mb RAM VPS while I decide what to do, and, of course, WP is fine again. But it's not an affordable long-term solution. So will either have to try and revert, or look at another CMS. Which is a shame, after 5 years of happy WordPressing.

Final note to pre-empt the inevitable "but you can get 1Gb RAM VPS very cheap these days" comment, yes, you can. And I've been badly burnt that way. Amazon EC2 looks incredibly tempting - but you try not being able to stop, recover, relaunch or save a backup or even a snapshot from a failed instance for the 6 days it takes support to reply. And as for HostEurope/Webfusion, hell will freeze over and I'll start listening to Justin Bieber before I even wish them on my worst enemy.

So I'm happy that I can just pick up the phone and immediately speak to someone, in London, in English.

Finally cracked it!

Finally, I've solved it!

The quick version:
Running php-fpm on ubuntu:

Code:

sudo nano /etc/php5/conf.d/apc.ini

You should see:

Code:

extension=apc.so

paste in the below lines:

Code:

apc.shm_size="196"

apc.ttl=0

apc.max_file_size="10M"

save and close.

If you've got limited memory, you could try:

Code:

apc.shm_size="96"

apc.ttl=0

apc.max_file_size="2M"

Code:

service php5-fpm restart.

Now go to your Wordpress admin and watch that sucker fly with near instant response.

The long version
I was so damn sick of this slow admin I just decided to keep going, trying stuff.

Finally, I came across this post:
http://stackoverflow.com/questions/3...e-memory-for-p
then this
http://groups.drupal.org/node/75583
which led me to this post:
http://www.litespeedtech.com/support...ead.php?t=4366
and finally this:
http://2bits.com/articles/high-php-e...rformance.html

The key to this was "download http://pecl.php.net/get/APC extract and run the apc.php, there you have a nice diagram how your cache usage look like"

It showed APC was completely using its tiny default 32Mb, and was 100% fragmented. I've been running it for an hour, and see
Hits: 70874 (98.6%) Misses: 991 (1.4%) Used: 124.4 MBytes (63.5%) Fragmentation: 0.00%

Before this tweak it was something like 70% misses and 100% fragmentation. No wonder Wordpress was running slow!

I've switched to Xcache which seems to be faster for me (haven't done any benchmarking, but that's how it feels).