ISPConfig 3, Dovecot 2 and traffic accounting

Discussion in 'Installation/Configuration' started by m.naumann, Sep 18, 2012.

  1. m.naumann

    m.naumann New Member

    Hi everyone at HowToForge,

    this is my first post here. :)

    I'd like to use ISPConfig 3 with Dovecot (preferrably v2.x) and do IMAP/POP3/submission traffic accounting. Preferrably on CentOS 6, but Ubuntu 12.04 LTS or Debian 6 would also work. According to a recent forum post by Till this should be possible with ISPConfig 3.0.4.6.

    I'd like to give this a try and set this up in a VM tomorrow. Before I do, is there something I should be aware of in this context? Are there any known issues with this setup? Should I prefer Debian 6 over Ubuntu 12.04? I will probably use this how-to unless you recommend otherwise.

    And most importantly, should I expect traffic accounting to work out of the box, or will I need to make any manual changes to make it work?

    Thanks in advance!
     
    Last edited: Sep 18, 2012
  2. falko

    falko Super Moderator Howtoforge Staff Moderator HowtoForge Supporter ISPConfig Developer

    No.

    Both Debian and Ubuntu are fine, but Debian is a bit more stable.

    It should work out of the box.
     
  3. pititis

    pititis Member

    "Should" or is working in 3.0.4.6? . This feature doesn't work for me in 12.04 LTS + dovecot 2

    Any way to check it? Is working for you?
     
  4. m.naumann

    m.naumann New Member

    So my installation is done, following Falko's great how-to to the point.

    I've created a mail domain, and a mail user on this domain. I've used an e-mail client on another computer to connect to this servers' IMAP and Submission ports, and have sent and received e-mail, amounting to several megabytes of mail traffic on both protocols / ports.

    Then I checked the mail traffic statistics for this user on ISPConfig's web interface. It listed 0. Which is understandable, since, most likely, these statistics are not updated real-time but by a regularly running cronjob.

    So I looked at the newly configured system and found (running
    Code:
    crontab -l
    as root) a daily (or rather nightly, running at 00:30 by default) ISPConfig cron job which runs this file:
    Code:
    /usr/local/ispconfig/server/cron_daily.php
    This PHP script imports the servers' ISPConfig related configuration and some helper classes (such as for structured databases access). After that, it contains multiple sections which each start with a comment. The first such section is
    Code:
    // store the mailbox statistics in the database
    The code in this section searches all mail users' maildir locations for a file called
    Code:
    ispconfig_mailsize
    Example for mail account someuser@domain.tld:
    Code:
    /var/vmail/domain.tld/someuser/ispconfig_mailsize
    So for each user, if ispconfig_mailsize exists, it renames it to "ispconfig_mailsize_save" then reads the file line by line, type casting each line to integer, and summing them up. It then add this value to any traffic previously recorded for this user (and stored in the mysql database, in table "mail_traffic") during the running month, if any, and stores this new value in the database

    For each user where ispconfig_mailsize does not exist, nothing is done.

    In my case, no such file exists (after sending large e-mails with file attachments in and out).

    There are other sections in this file, labelled as follows:
    Code:
    // Create webalizer statistics
    // Create awstats statistics
    // Make the web logfiles directories world readable to enable ftp access
    // Manage and compress web logfiles and create traffic statistics
    // Rotate the ispconfig.log file
    // Cleanup website tmp directories
    // Cleanup logs in master database (only the "master-server")
    // enforce traffic quota (run only on the "master-server")
    // deactivate virtual servers (run only on the "master-server")
    // Create website backups
    None of these seem to refer to mail traffic accounting.

    So what's the mechanism which creates the "ispconfig_mailsize" files in the first place?
    Code:
    # rgrep 'ispconfig_mailsize' /usr/local/
    /usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php:			// if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
    /usr/local/ispconfig/server/cron_daily.php:	if(@is_file($rec['maildir'].'/ispconfig_mailsize')) {
    /usr/local/ispconfig/server/cron_daily.php:		rename($rec['maildir'].'/ispconfig_mailsize',$rec['maildir'].'/ispconfig_mailsize_save');
    /usr/local/ispconfig/server/cron_daily.php:		$lines = file($rec['maildir'].'/ispconfig_mailsize_save');
    /usr/local/ispconfig/server/cron_daily.php:		if(@is_file($rec['maildir'].'/ispconfig_mailsize_save')) unlink($rec['maildir'].'/ispconfig_mailsize_save');
    /usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php:			// if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
    #
    /usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php is a symbolic link to /usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php which only contains one reference to this filename, but the line is commented out.
    /usr/local/ispconfig/server/cron_daily.php is the file we already discussed above, it does not create these files either.

    Maybe the file is created by Dovecot or some other daemon, based on modifications ISPConfig may have made to their configuration files during installation?
    Code:
    # rgrep 'ispconfig_mailsize' /etc/
    grep: /etc/fonts/conf.d/30-defoma.conf: No such file or directory
    #
    Apparently that's not the case either. Maybe it's somewhere next to the mail storage:
    Code:
    # rgrep 'ispconfig_mailsize' /var/vmail/
    /var/vmail/.mailfilter:`echo $SIZE >> /var/vmail/$HOST/$USER/ispconfig_mailsize`
    #
    Aha, so this is something. So some searching on the web will tell you that maildrop is a mail filter/mail delivery agent, which comes with the Courier MTA by default, but can also be used standalone, as well as in combination with Dovecot:
    http://www.courier-mta.org/maildrop/
    http://www.dovecot.org/list/dovecot/2011-February/057195.html

    Is maildrop packaged for Debian 6?
    Code:
    # apt-cache search maildrop
    courier-maildrop - Courier mail server - mail delivery agent
    getmail4 - mail retriever with support for POP3, IMAP4 and SDPS
    maildrop - mail delivery agent with filtering abilities
    rdeliver - A fully functioning mail filter with RubyFilter
    sensible-mda - Mail Delivery Agent wrapper
    spamprobe - Bayesian spam filter
    #
    OK, there are both the Courier one and a standalone maildrop package. Are any of these installed (and hopefully configured, too) now that I followed Falko's how-to?
    Code:
    # dpkg -l maildrop courier-maildrop
    No packages found matching maildrop.
    No packages found matching courier-maildrop.
    #
    Hmm, no. Maybe another installed package provides the "maildrop" command?
    Code:
    # which maildrop
    #
    Apparently not.

    So I'm not sure how mail (IMAP4r2, POP3, SMTP via Submission port) traffic accounting could work with this setup. Could you please elaborate, Falko? Did I miss something obvious (or not so obvious)?

    Thank you very much in advance.
     
    Last edited: Sep 20, 2012
  5. m.naumann

    m.naumann New Member

    It would be great to get a confirmation that my suspicion (traffic accounting with Dovecot 2 is not currently implemented) is correct. If it is, I could possibly spend some time on looking into how it could be implemented and provide some feedback.
     
  6. falko

    falko Super Moderator Howtoforge Staff Moderator HowtoForge Supporter ISPConfig Developer

    AFAIK this is correct. I can't tell if this is part of ISPConfig 3.0.5 or not. Will have to ask Till.
     
  7. m.naumann

    m.naumann New Member

    Thanks, Falko. Have you two had a chance to discuss this, yet?
     
  8. till

    till Super Moderator Howtoforge Staff HowtoForge Supporter ISPConfig Developer

    Your findings are partially right, but you mixed up two completely different setups. maildrop is part of the courier based setups only and not required or related to dovecot setups and the same is with the file ispconfig_mailsize which is for courier only too.

    Dovecot does not support traffic accounting yet and the sieve filtering language which is part of dovecot is not able to write files to the filesystem with traffic information as well. So its currently not possible to get the traffic information from dovecot as there is no such log file. As far as I know, the dovecot developers have planned to add a interface for that in future.
     
  9. m.naumann

    m.naumann New Member

    Thanks for your reply, Till.

    My understanding is that there is a standalone implementation available of maildrop which can also be integrated between MTA and dovecot LDA. It probably makes more sense to work with sieve filters, though.

    About ispconfig_mailsize, I was thinking of *somehow* making dovecot (or something added to dovecot) write out (properly formatted) ispconfig_mailsize files, too, so that the ISPConfig script doesn't need to be expanded/modified and run another 'find' on the file system, slowing down the system unnecessarily during the nightly cron job. Plus, that would mean less code to maintain on the ISPConfig side once it works. On the other hand it might require in a more complex sieve filter / maildrop setup (which could also increase processing overhead per e-mail, and that would be worse than a second 'find').

    My understanding here is that, while, as you're saying, standard sieve does not allow for writing files to the file system (nor executing external applications which could do so), there are extensions to the sieve filtering language, written and documented by dovecot developers as part of the Piegeonhole project, which allow you to pipe a copy of an e-mail into an external command. Now if this external command is a wrapper around 'wc -c' and writes its output to disk, you have a simplistic implementation for counting mail traffic with little overhead. If needed (such as for only counting the message body), one could use a more intelligent wrapper there, such as one written in PHP.

    http://wiki2.dovecot.org/Pigeonhole
    http://wiki2.dovecot.org/Pigeonhole/Sieve
    http://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/Extprograms

    So I think this should be possible already now. Of course, the plain size of e-mails transferred across IMAP/POP3 differs from the real traffic you're causing there, but I would assume it's close enough (and better than nothing). To get somewhat close to a proper implementation of mail traffic accounting you would also want/need to measure SMTP e-mail traffic. My current understanding is that this is not currently implemented in ISPConfig for any supported MTA (please correct me if wrong).

    So I also looked at how this could possibly be done with Postfix. While there can be 'size=...' values in your ESMTP log, this turns out to be an optional flag, i.e. clients/MTAs are not required to send it (and I think most clients actually don't). So the friendly people in #postfix on FreeNode recommended against relying on it and suggested to use a policy (with Postfix policy daemon) instead, because the 'size' parameter is always transferred there.

    Here's an incomplete and untested proof of concept shell script which should work for this purpose, logging the sizes of all e-mail it learns about to disk.
    Code:
    #!/bin/sh
    # http://www.postfix.org/SMTPD_POLICY_README.html
    IFSBACKUP=$IFS
    IFS=`echo -e "=\r\n"`
    while read param value morevalues
    do
      if [ "$morevalues" != '' ]
      then
        echo 'Error: More input found than expected' >&2
        exit 1
      fi
      case $param in
        sasl_username )
          domain=`echo $value | cut -d'@' -f2`
          user=`echo $value | cut -d'@' -f1`
          ;;
        size )
          size= $value
          ;;
      esac
    done
    IFS=$IFSBACKUP
    
    echo -e 'action=dunno\n\n'
    
    echo "$size" >> /var/vmail/$domain/$user/LOGFILE
    
    This policy should only be used for SMTP mail, and only be applied after successful client authentication has taken place via SASL. This should be possible by placing it at the proper position in the 'restrictions' section of postfix' main.cf

    So, while some of this is still theoretical, my general impression is that traffic accounting with all non legacy versions of Postfix and with Dovecot 2 should be achievable. Now "someone" "just" needs to find the time to actually implement and test this. ;-) I can't really promise to do it, though (but will try to find some time).
     
    Last edited: Oct 3, 2012
  10. Croydon

    Croydon Member Howtoforge Staff Moderator HowtoForge Supporter ISPConfig Developer

    I implemented a mail.log parsing fallback for ISPConfig 3.0.5 that fetches mail sizes from the amavis log lines in the mail.log.
    So we finally have a traffic accounting for dovecot in the daily cron.
     
  11. m.naumann

    m.naumann New Member

    Thank you Croydon!
    Have you tried to verify whether Amavis' mail log is a reliable source for this kind of information then? I.e. does it log the size of every e-mail passing through it (and not some other sizes), and do these values look correct?

    If this works, we'll only be lacking SMTP traffic accounting, that's quite some progress.
     
  12. Croydon

    Croydon Member Howtoforge Staff Moderator HowtoForge Supporter ISPConfig Developer

    The sizes given by amavis seem to be quite reliable and include attachments so it is a good thing to work with I think.
    Amavis checks inbound and outbound mails. So the current implementation should show all inbound and outbound traffic for each mailbox just (or nearly) like the traffic accounting in ispconfig for courier does.
     
  13. tahunasky

    tahunasky New Member

    How do i get mail traffic monitoring to work with the Amavis log - is there a how-to somewhere ?

    I have IPSConfig 3.4.6 and dovecot on a centOS 6.3 box.
     
  14. stevenlib

    stevenlib New Member

    Hi,

    Any progress on this? I have 0 mailbox traffic and I am trying to enable it.

    Regards,
     
  15. tahunasky

    tahunasky New Member

    It is working in my setup. I upgraded to version 3.0.5.2 earlier this year and mailbox traffic now works. I dont know how accurate it is though.

    My setup is:
    Ubuntu Server 13.04
    Postfix
    Dovecot
    ISPConfig 3.0.5.2
     

Share This Page