HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   Installation/Configuration (http://www.howtoforge.com/forums/forumdisplay.php?f=27)
-   -   ISPConfig 3, Dovecot 2 and traffic accounting (http://www.howtoforge.com/forums/showthread.php?t=58766)

m.naumann 18th September 2012 19:58

ISPConfig 3, Dovecot 2 and traffic accounting
 
Hi everyone at HowToForge,

this is my first post here. :)

I'd like to use ISPConfig 3 with Dovecot (preferrably v2.x) and do IMAP/POP3/submission traffic accounting. Preferrably on CentOS 6, but Ubuntu 12.04 LTS or Debian 6 would also work. According to a recent forum post by Till this should be possible with ISPConfig 3.0.4.6.

I'd like to give this a try and set this up in a VM tomorrow. Before I do, is there something I should be aware of in this context? Are there any known issues with this setup? Should I prefer Debian 6 over Ubuntu 12.04? I will probably use this how-to unless you recommend otherwise.

And most importantly, should I expect traffic accounting to work out of the box, or will I need to make any manual changes to make it work?

Thanks in advance!

falko 19th September 2012 14:51

Quote:

Originally Posted by m.naumann (Post 285606)
Are there any known issues with this setup?

No.

Quote:

Should I prefer Debian 6 over Ubuntu 12.04? I will probably use this how-to unless you recommend otherwise.
Both Debian and Ubuntu are fine, but Debian is a bit more stable.

Quote:

And most importantly, should I expect traffic accounting to work out of the box, or will I need to make any manual changes to make it work?
It should work out of the box.

pititis 19th September 2012 15:20

"Should" or is working in 3.0.4.6? . This feature doesn't work for me in 12.04 LTS + dovecot 2

Any way to check it? Is working for you?

m.naumann 19th September 2012 18:11

So my installation is done, following Falko's great how-to to the point.

I've created a mail domain, and a mail user on this domain. I've used an e-mail client on another computer to connect to this servers' IMAP and Submission ports, and have sent and received e-mail, amounting to several megabytes of mail traffic on both protocols / ports.

Then I checked the mail traffic statistics for this user on ISPConfig's web interface. It listed 0. Which is understandable, since, most likely, these statistics are not updated real-time but by a regularly running cronjob.

So I looked at the newly configured system and found (running
Code:

crontab -l
as root) a daily (or rather nightly, running at 00:30 by default) ISPConfig cron job which runs this file:
Code:

/usr/local/ispconfig/server/cron_daily.php
This PHP script imports the servers' ISPConfig related configuration and some helper classes (such as for structured databases access). After that, it contains multiple sections which each start with a comment. The first such section is
Code:

// store the mailbox statistics in the database
The code in this section searches all mail users' maildir locations for a file called
Code:

ispconfig_mailsize
Example for mail account someuser@domain.tld:
Code:

/var/vmail/domain.tld/someuser/ispconfig_mailsize
So for each user, if ispconfig_mailsize exists, it renames it to "ispconfig_mailsize_save" then reads the file line by line, type casting each line to integer, and summing them up. It then add this value to any traffic previously recorded for this user (and stored in the mysql database, in table "mail_traffic") during the running month, if any, and stores this new value in the database

For each user where ispconfig_mailsize does not exist, nothing is done.

In my case, no such file exists (after sending large e-mails with file attachments in and out).

There are other sections in this file, labelled as follows:
Code:

// Create webalizer statistics
// Create awstats statistics
// Make the web logfiles directories world readable to enable ftp access
// Manage and compress web logfiles and create traffic statistics
// Rotate the ispconfig.log file
// Cleanup website tmp directories
// Cleanup logs in master database (only the "master-server")
// enforce traffic quota (run only on the "master-server")
// deactivate virtual servers (run only on the "master-server")
// Create website backups

None of these seem to refer to mail traffic accounting.

So what's the mechanism which creates the "ispconfig_mailsize" files in the first place?
Code:

# rgrep 'ispconfig_mailsize' /usr/local/
/usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php:                        // if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
/usr/local/ispconfig/server/cron_daily.php:        if(@is_file($rec['maildir'].'/ispconfig_mailsize')) {
/usr/local/ispconfig/server/cron_daily.php:                rename($rec['maildir'].'/ispconfig_mailsize',$rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/cron_daily.php:                $lines = file($rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/cron_daily.php:                if(@is_file($rec['maildir'].'/ispconfig_mailsize_save')) unlink($rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php:                        // if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
#

/usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php is a symbolic link to /usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php which only contains one reference to this filename, but the line is commented out.
/usr/local/ispconfig/server/cron_daily.php is the file we already discussed above, it does not create these files either.

Maybe the file is created by Dovecot or some other daemon, based on modifications ISPConfig may have made to their configuration files during installation?
Code:

# rgrep 'ispconfig_mailsize' /etc/
grep: /etc/fonts/conf.d/30-defoma.conf: No such file or directory
#

Apparently that's not the case either. Maybe it's somewhere next to the mail storage:
Code:

# rgrep 'ispconfig_mailsize' /var/vmail/
/var/vmail/.mailfilter:`echo $SIZE >> /var/vmail/$HOST/$USER/ispconfig_mailsize`
#

Aha, so this is something. So some searching on the web will tell you that maildrop is a mail filter/mail delivery agent, which comes with the Courier MTA by default, but can also be used standalone, as well as in combination with Dovecot:
http://www.courier-mta.org/maildrop/
http://www.dovecot.org/list/dovecot/...ry/057195.html

Is maildrop packaged for Debian 6?
Code:

# apt-cache search maildrop
courier-maildrop - Courier mail server - mail delivery agent
getmail4 - mail retriever with support for POP3, IMAP4 and SDPS
maildrop - mail delivery agent with filtering abilities
rdeliver - A fully functioning mail filter with RubyFilter
sensible-mda - Mail Delivery Agent wrapper
spamprobe - Bayesian spam filter
#

OK, there are both the Courier one and a standalone maildrop package. Are any of these installed (and hopefully configured, too) now that I followed Falko's how-to?
Code:

# dpkg -l maildrop courier-maildrop
No packages found matching maildrop.
No packages found matching courier-maildrop.
#

Hmm, no. Maybe another installed package provides the "maildrop" command?
Code:

# which maildrop
#

Apparently not.

So I'm not sure how mail (IMAP4r2, POP3, SMTP via Submission port) traffic accounting could work with this setup. Could you please elaborate, Falko? Did I miss something obvious (or not so obvious)?

Thank you very much in advance.

m.naumann 24th September 2012 16:01

It would be great to get a confirmation that my suspicion (traffic accounting with Dovecot 2 is not currently implemented) is correct. If it is, I could possibly spend some time on looking into how it could be implemented and provide some feedback.

falko 25th September 2012 20:32

Quote:

Originally Posted by m.naumann (Post 285909)
It would be great to get a confirmation that my suspicion (traffic accounting with Dovecot 2 is not currently implemented) is correct.

AFAIK this is correct. I can't tell if this is part of ISPConfig 3.0.5 or not. Will have to ask Till.

m.naumann 2nd October 2012 17:51

Thanks, Falko. Have you two had a chance to discuss this, yet?

till 3rd October 2012 12:22

Your findings are partially right, but you mixed up two completely different setups. maildrop is part of the courier based setups only and not required or related to dovecot setups and the same is with the file ispconfig_mailsize which is for courier only too.

Dovecot does not support traffic accounting yet and the sieve filtering language which is part of dovecot is not able to write files to the filesystem with traffic information as well. So its currently not possible to get the traffic information from dovecot as there is no such log file. As far as I know, the dovecot developers have planned to add a interface for that in future.

m.naumann 3rd October 2012 15:25

Thanks for your reply, Till.

Quote:

Originally Posted by till (Post 286219)
Your findings are partially right, but you mixed up two completely different setups. maildrop is part of the courier based setups only and not required or related to dovecot setups and the same is with the file ispconfig_mailsize which is for courier only too.

My understanding is that there is a standalone implementation available of maildrop which can also be integrated between MTA and dovecot LDA. It probably makes more sense to work with sieve filters, though.

About ispconfig_mailsize, I was thinking of *somehow* making dovecot (or something added to dovecot) write out (properly formatted) ispconfig_mailsize files, too, so that the ISPConfig script doesn't need to be expanded/modified and run another 'find' on the file system, slowing down the system unnecessarily during the nightly cron job. Plus, that would mean less code to maintain on the ISPConfig side once it works. On the other hand it might require in a more complex sieve filter / maildrop setup (which could also increase processing overhead per e-mail, and that would be worse than a second 'find').

Quote:

Originally Posted by till (Post 286219)
Dovecot does not support traffic accounting yet and the sieve filtering language which is part of dovecot is not able to write files to the filesystem with traffic information as well. So its currently not possible to get the traffic information from dovecot as there is no such log file. As far as I know, the dovecot developers have planned to add a interface for that in future.

My understanding here is that, while, as you're saying, standard sieve does not allow for writing files to the file system (nor executing external applications which could do so), there are extensions to the sieve filtering language, written and documented by dovecot developers as part of the Piegeonhole project, which allow you to pipe a copy of an e-mail into an external command. Now if this external command is a wrapper around 'wc -c' and writes its output to disk, you have a simplistic implementation for counting mail traffic with little overhead. If needed (such as for only counting the message body), one could use a more intelligent wrapper there, such as one written in PHP.

http://wiki2.dovecot.org/Pigeonhole
http://wiki2.dovecot.org/Pigeonhole/Sieve
http://wiki2.dovecot.org/Pigeonhole/...ns/Extprograms

So I think this should be possible already now. Of course, the plain size of e-mails transferred across IMAP/POP3 differs from the real traffic you're causing there, but I would assume it's close enough (and better than nothing). To get somewhat close to a proper implementation of mail traffic accounting you would also want/need to measure SMTP e-mail traffic. My current understanding is that this is not currently implemented in ISPConfig for any supported MTA (please correct me if wrong).

So I also looked at how this could possibly be done with Postfix. While there can be 'size=...' values in your ESMTP log, this turns out to be an optional flag, i.e. clients/MTAs are not required to send it (and I think most clients actually don't). So the friendly people in #postfix on FreeNode recommended against relying on it and suggested to use a policy (with Postfix policy daemon) instead, because the 'size' parameter is always transferred there.

Here's an incomplete and untested proof of concept shell script which should work for this purpose, logging the sizes of all e-mail it learns about to disk.
Code:

#!/bin/sh
# http://www.postfix.org/SMTPD_POLICY_README.html
IFSBACKUP=$IFS
IFS=`echo -e "=\r\n"`
while read param value morevalues
do
  if [ "$morevalues" != '' ]
  then
    echo 'Error: More input found than expected' >&2
    exit 1
  fi
  case $param in
    sasl_username )
      domain=`echo $value | cut -d'@' -f2`
      user=`echo $value | cut -d'@' -f1`
      ;;
    size )
      size= $value
      ;;
  esac
done
IFS=$IFSBACKUP

echo -e 'action=dunno\n\n'

echo "$size" >> /var/vmail/$domain/$user/LOGFILE

This policy should only be used for SMTP mail, and only be applied after successful client authentication has taken place via SASL. This should be possible by placing it at the proper position in the 'restrictions' section of postfix' main.cf

So, while some of this is still theoretical, my general impression is that traffic accounting with all non legacy versions of Postfix and with Dovecot 2 should be achievable. Now "someone" "just" needs to find the time to actually implement and test this. ;-) I can't really promise to do it, though (but will try to find some time).

Croydon 1st November 2012 13:26

I implemented a mail.log parsing fallback for ISPConfig 3.0.5 that fetches mail sizes from the amavis log lines in the mail.log.
So we finally have a traffic accounting for dovecot in the daily cron.


All times are GMT +2. The time now is 01:04.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.