HELP : Debian Squeeze perfect setup crashes Apache and MySql randomly
We have 2 servers (web and mail) running ISP Config 3.0.3.
All went great until last thursday at 6PM when Apache2 and MySql crashed on our web server.
Since then, this server randomly crashes (between 15 minutes and 3-4 hours), we found this in the kern.log :
CPU : 1 x 2.26 Ghz (Intel xeon E5507)
RAM : 4GB
Data partition : /dev/sda8 /var 800GB used 20%
Log partition : /dev/sda9 /var/log 10G used 75%
Swap : 2GB
OS : Debian Squeeze (apt upgraded)
Web server programs :
bind (primary DNS)
ISP Config as master (mail server uses ISP Config as slave)
Web server is hosting approx 200 domains and websites
Mail server programs :
bind (secondary DNS)
At first I thought it was a swapping problem (it seems our provider set a too small swap partition), so I try to reduce RAM use : I set Apache MPM Prefork MaxClients to 40 instead of default which I think is 150 and fcgid FcgidMaxProcesses to 40 instead of previously set 100, I removed some useless apache modules and disabled fail2ban and postgresql which we don't really use for now.
Unfortunately, this didn't solve anything, server crashed about an hour after...
I tried to fix with :
We also tried to fix mysql databases (as we first thought it was a mysql issue) :
Finally, we sent a ticket to our provider to tell them to try to fix the partition (our provider is theorically responsible for hardware issues)
On top of this, each time the web server (master) crashes, it output hundreds of MB of binary data in the /var/log/ispconfig/ispconfig.log and on the mail server, amavis also crashes (and messages get stuck in postqueue).
We commented the ISP Config server.sh cron task, so it "fixed" the huge logging issue, but the amavis crash still occur, I suppose it's related to the crash of the master MySql database...
Does this issue happened to any of you ? Does anyone have any idea about how we can fix that ?
We would really appreciate some help, because this is really a critical issue for us. Thank you for your help.
After hours of investigation during whole week-end we finally found the culprit : one website running Prestashop in production with a huge frequentation had enabled option "PS_SMARTY_FORCE_COMPILE" (since thurdsay 16PM). This option forces smarty to recompile templates on each request and made a huge consumption of CPU that lead to these repeated crashes.
We disabled this option this morning and since then server stopped crashing.
Now we would like to set a cpulimit to avoid this bug to come again. I've found this article : http://www.howtoforge.com/how-to-lim...-debian-ubuntu but I don't really see how to enable it on my configuration any hint please ?
On our way to try to fix these crashes, we made some improvements to reduce our RAM usage, like :
- Remove E_NOTICE from PHP logs, by replacing :
We made this because we had found 1000s of PHP Notice lines on a website right before crash so we first thought it might be a vlogger issue.
- Replacing mpm_prefork by mpm_worker and removing mod_php (useless because all our websites now use fcgid) : I just had seen several blog articles saying that mpm_worker is best for multi-core CPU and consumes less RAM than prefork, I'm not sure about this but removing mod_php from apache2 config reduced each apache2 process RES (RAM usage) by about 3-5MB each so I kept it even if it didn't fix my problem... Note that you have to enable fcgid on ALL sites (even phpmyadmin or default) and you won't be able to use mod_php again.
- We also removed apache2 modules we don't use, like : ruby, suphp, cgid and davfs, so each apache2 process consume now about 15MB instead of 25MB.
For next weeks/months we'd like to :
- Set cpulimit (if possible we'd like to have a "graceful limit behaviour" like processing by chunks instead of refusing to process, but dunno how it works...)
- Reduce PHP memory_limit : We set it to 128MB but it seems a lot, maybe 64MB ? Like CPU I dunno if Apache2 would just throw errors or if it could process chunks
- Increase RAM : 4GB -> 16GB
- Tune cpulimit, mpm worker and fcgid parameters (need to find simple formulas to fit each param to our config)
|All times are GMT +2. The time now is 20:28.|
Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.