On our way to try to fix these crashes, we made some improvements to reduce our RAM usage, like :
- Remove E_NOTICE from PHP logs, by replacing :
error_reporting = E_ALL & ~E_DEPRECATED
error_reporting = E_ALL & ~E_DEPRECATED & ~E_NOTICE
in /etc/php5/*/php.ini. Note that you have to restart apache2 and if you use php-apc pray for it to clean cache or else it'll keep PHP Notices.
We made this because we had found 1000s of PHP Notice lines on a website right before crash so we first thought it might be a vlogger issue.
- Replacing mpm_prefork by mpm_worker and removing mod_php (useless because all our websites now use fcgid) : I just had seen several blog articles saying that mpm_worker is best for multi-core CPU and consumes less RAM than prefork, I'm not sure about this but removing mod_php from apache2 config reduced each apache2 process RES (RAM usage) by about 3-5MB each so I kept it even if it didn't fix my problem... Note that you have to enable fcgid on ALL sites (even phpmyadmin or default) and you won't be able to use mod_php again.
- We also removed apache2 modules we don't use, like : ruby, suphp, cgid and davfs, so each apache2 process consume now about 15MB instead of 25MB.
For next weeks/months we'd like to :
- Set cpulimit (if possible we'd like to have a "graceful limit behaviour" like processing by chunks instead of refusing to process, but dunno how it works...)
- Reduce PHP memory_limit : We set it to 128MB but it seems a lot, maybe 64MB ? Like CPU I dunno if Apache2 would just throw errors or if it could process chunks
- Increase RAM : 4GB -> 16GB
- Tune cpulimit, mpm worker and fcgid parameters (need to find simple formulas to fit each param to our config)