Using Nice in cronjob mail_logs.php

Discussion in 'Tips/Tricks/Mods' started by locosoft, Aug 23, 2006.

  1. locosoft

    locosoft New Member


    Hope this is the right place to post this (can't post in Developer forum).

    We have a small server running ISPconfig (192Mb ram, PII 350Mhz cpu).
    The system is almost always on 100% cpu-usage.
    It is only used as a mailserver and DNS-server.

    It seems that most of the processortime (78%) is needed by a grep process.
    I traced it back to a grep command inside ~/ispconfig/scripts/shell/mail_logs.php.

    I edited this file and added "nice 10" before every grep command I could find. (lines 89 and 94).

    I hope this will be a bit nicer on our server tomorrow :)

    Pieter van der Eems
  2. falko

    falko Super Moderator ISPConfig Developer

    Can you post your changed file here (as a reference for other people)? :)
  3. lorello

    lorello New Member

    Hi, I have a similar problem with a Dell Poweredge 2850 (2 Xeon 3GHz, 2Mb cache, 2Gb ram, 2 Mirrored SCSI disks).

    We have a big mail traffic, and our mail log file is 3/4Gb each day.
    In these conditions mail_logs.php script is active 24/24h, taking 99.9% of each CPU.

    I've modified crontab:

    # ISPConfig
    30 00 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php &> /dev/null
    59 23 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/ftp_logs.php &> /dev/null
    59 23 * * * /usr/bin/nice -n 16 /root/ispconfig/php/php /root/ispconfig/scripts/shell/mail_logs.php &> /dev/null
    59 23 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/cleanup.php &> /dev/null
    0 4 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/webalizer.php &> /dev/null
    0,30 * * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services.php &> /dev/null
    15 3,15 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/quota_msg.php &> /dev/null
    40 00 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/traffic.php &> /dev/null
    05 02 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/backup.php &> /dev/null
    */5 * * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/mysql_quota.php &> /dev/null
    0 10 * * * /usr/bin/nice -n 17 /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_mx.php
    but this is not a solution.
    Reading the code of mail_logs i thought that I could modify it to analyze a domain each time, without running multiple processess.
    Is there some other optimization that I can try to implement?

    What about rewriteing the code in pure PHP, using some regexp, instead of external programs?
    I suppose the script should make statistics about:
    - traffic made from each domain and codomain
    - add to it traffic of each user (?), stored in /home/admispconfig/mailstats/USERNAME
    - store this value in some mysql table

    Thanks for any info,
  4. falko

    falko Super Moderator ISPConfig Developer

    The easiest solution would be to disable mail_logs.php (if you don't need email traffic statistics).
  5. lorello

    lorello New Member

    mail_logs.php optimization

    I've optimized last year the script /root/ispconfig/scripts/shell/mail_logs.php, but I've never posted to the forum. I do it now, some other mailserver with high traffic (and so, big log files) could need it.

    The optimization I've done is very simple: I've moved some grep outside the loop that cycle over all sites of ISPConfig, reducing the size of the file analyzed in the loop from 4Gb to less than 1Gb. This simple change reduce the exectution time on my server from more than 24h to less than one hour :)

    Attached Files:

  6. djtremors

    djtremors ISPConfig Developer ISPConfig Developer

    I have a script which processes lines like a "tail -f" command and splits up the line details as required for processing.
    Originally this was going to be for webstats traffic graphs like MRTG but traffic per website, but after thinking about it I thought AWstats is doing this pretty much anyway even though this script would be 5 minute graphs and aborted the idea for now.

    This script could also do the mail logs actively during the day saving the huge processing at night to do it during the day instead and process as it comes in and store in memory for XX given lines/minutes before logging it per site.

Share This Page