Got a FASTCGI problem after my latestupdate

[Ovidiu]

The server (a small VPS) is set up according to this how-to: http://www.lowendbox.com/blog/wordpr...-lowendscript/

The problems started after the last update:

apticron report [Fri, 27 Jul 2012 00:41:08 -0300]
================================================== ======================

apticron has detected that some packages need upgrading on:

The following packages are currently pending an upgrade:

mysql-client-5.5 5.5.25a-1~dotdeb.1
mysql-client-core-5.5 5.5.25a-1~dotdeb.1
mysql-common 5.5.25a-1~dotdeb.1
mysql-server 5.5.25a-1~dotdeb.1
mysql-server-5.5 5.5.25a-1~dotdeb.1
mysql-server-core-5.5 5.5.25a-1~dotdeb.1
php5-cgi 5.3.15-1~dotdeb.0
php5-cli 5.3.15-1~dotdeb.0
php5-common 5.3.15-1~dotdeb.0
php5-curl 5.3.15-1~dotdeb.0
php5-gd 5.3.15-1~dotdeb.0
php5-mcrypt 5.3.15-1~dotdeb.0
php5-mysql 5.3.15-1~dotdeb.0
php5-suhosin 5.3.15-1~dotdeb.0

When I did these updates, I always chose to keep my configs and not to replace them with the distro ones. I am facing the problem that php-cgi seems to "freeze" to death, it doesn't seem to serve anything, it works for a few hours, then it freezes and only starts working again after I killall -9 php-cgi and then starting it again as restarting it doesn't work either.
All I get are these errors in nginx error log:

2012/07/29 09:26:21 [error] 18038#0: *1909 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 173.245.52.14, server: knightsenglish.com, request: "GET / HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "www.knightsenglish.com", referrer: "http://www.knightsenglish.com/"

Any ideas what could be wrong here? Any starting points to investigate? All this happened this weekend where there was almost zero traffic, I checked with netstat and there were maybe 10 connections active only :-(

As a first try, after googling this issue I added this value to my nginx configs: proxy_read_timeout 120; but that didn't change much.

[falko]

Do you use PHP-FPM? If not, I'd strongly suggest to set it up.

[Ovidiu]

Thanks Falko, I figured out the problem and will implement your suggestion after I fix the original problem.

After tinkering with the VPS I realized the problem went a lot deeper than that. This thread is basically closed as the problem is a totally different one, but if you ca nadd something to it it would be much appreciated.

I ran 2 sites for a friend on my server, one grew to big so I moved him to his own VPS that I also manage. The second one grew stale so I eventually de-activated it.

I now moved the stale one to his own server since we was going to update it and get it up and running again.

I see 2 possible reasons for my problems:

a) either I screwed up when transferring the old site to the new server
b) the site's files were infected with Timthumb and possibly other dangerous stuff

Facts: I moved the site, put it online, updated all plugins, ran a Timthumb vulnerability scanner over the entire wordpress installation (a WP plugin) and manually deleted/replaced all infected files, which were quite a lot.

My reasoning is that while doing that, I might have missed some infected file or simply have been to slow.

The symptoms are that a lot of processes stopped working, I checked quite a few log files and all are complaining about wrong ownerships, i.e. most of the problems I found is that www-data owns the folders/files now...

My explanation is that on the old server, everything was so secured and tied down, that the infection couldn't spread anywhere (I run those sites with FASTCGI and suexec and a lot of other security mechanisms) but the new one is "unprotected" from inside, meaning that me personally moving the infection onto the server, and the web server running no further protection, I basically spread the virus myself :-(

My plan now is to restore a backup of the new server, get it up and running again, then try and clean the infected site before moving it. The question is how do I detect/clean infected files within a wordpress site OFFLINE? All I could goggle, refers to how to clean/scan a live wordpress installation :-(

Any advice?

Please also feel free to comment if you think you see a flaw in my reasoning, this is not 100% proven (except for the mentioned FACTS), just my deductions.

oh, btw. I ran one of the many infected index.php files through an online scanner and here are the results:

http://r.virscan.org/report/994a8139...334871829.html
https://www.virustotal.com/file/4b66...is/1343601521/

Funnily enough, the Linux Malware tool I am using doesn't find anything wrong with this file :-( http://www.rfxn.com/projects/linux-malware-detect/
Need to post there for support too.

Any advice from someone who faced something like this before? Maybe some pointers about detecting MySQL injections once they have happened already?