Using Nice in cronjob mail_logs.php

[locosoft]

Hello,

Hope this is the right place to post this (can't post in Developer forum).

We have a small server running ISPconfig (192Mb ram, PII 350Mhz cpu).
The system is almost always on 100% cpu-usage.
It is only used as a mailserver and DNS-server.

It seems that most of the processortime (78%) is needed by a grep process.
I traced it back to a grep command inside ~/ispconfig/scripts/shell/mail_logs.php.

I edited this file and added "nice 10" before every grep command I could find. (lines 89 and 94).

I hope this will be a bit nicer on our server tomorrow :-)

Pieter van der Eems

[falko]

Can you post your changed file here (as a reference for other people)?

[lorello]

Hi, I have a similar problem with a Dell Poweredge 2850 (2 Xeon 3GHz, 2Mb cache, 2Gb ram, 2 Mirrored SCSI disks).

We have a big mail traffic, and our mail log file is 3/4Gb each day.
In these conditions mail_logs.php script is active 24/24h, taking 99.9% of each CPU.

I've modified crontab:

# ISPConfig
30 00 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php &> /dev/null
59 23 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/ftp_logs.php &> /dev/null
59 23 * * * /usr/bin/nice -n 16 /root/ispconfig/php/php /root/ispconfig/scripts/shell/mail_logs.php &> /dev/null
59 23 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/cleanup.php &> /dev/null
0 4 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/webalizer.php &> /dev/null
0,30 * * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services.php &> /dev/null
15 3,15 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/quota_msg.php &> /dev/null
40 00 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/traffic.php &> /dev/null
05 02 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/backup.php &> /dev/null
*/5 * * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/mysql_quota.php &> /dev/null
0 10 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_mx.php

but this is not a solution.
Reading the code of mail_logs i thought that I could modify it to analyze a domain each time, without running multiple processess.
Is there some other optimization that I can try to implement?

What about rewriteing the code in pure PHP, using some regexp, instead of external programs?
I suppose the script should make statistics about:
- traffic made from each domain and codomain
- add to it traffic of each user (?), stored in /home/admispconfig/mailstats/USERNAME
- store this value in some mysql table

Thanks for any info,
l.

[falko]

The easiest solution would be to disable mail_logs.php (if you don't need email traffic statistics).

[lorello]

I've optimized last year the script /root/ispconfig/scripts/shell/mail_logs.php, but I've never posted to the forum. I do it now, some other mailserver with high traffic (and so, big log files) could need it.

The optimization I've done is very simple: I've moved some grep outside the loop that cycle over all sites of ISPConfig, reducing the size of the file analyzed in the loop from 4Gb to less than 1Gb. This simple change reduce the exectution time on my server from more than 24h to less than one hour

[djtremors]

I have a script which processes lines like a "tail -f" command and splits up the line details as required for processing.
Originally this was going to be for webstats traffic graphs like MRTG but traffic per website, but after thinking about it I thought AWstats is doing this pretty much anyway even though this script would be 5 minute graphs and aborted the idea for now.

This script could also do the mail logs actively during the day saving the huge processing at night to do it during the day instead and process as it comes in and store in memory for XX given lines/minutes before logging it per site.