Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > ISPConfig 3 > Installation/Configuration

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 18th September 2012, 19:58
m.naumann m.naumann is offline
Junior Member
 
Join Date: Sep 2012
Posts: 11
Thanks: 1
Thanked 0 Times in 0 Posts
Default ISPConfig 3, Dovecot 2 and traffic accounting

Hi everyone at HowToForge,

this is my first post here.

I'd like to use ISPConfig 3 with Dovecot (preferrably v2.x) and do IMAP/POP3/submission traffic accounting. Preferrably on CentOS 6, but Ubuntu 12.04 LTS or Debian 6 would also work. According to a recent forum post by Till this should be possible with ISPConfig 3.0.4.6.

I'd like to give this a try and set this up in a VM tomorrow. Before I do, is there something I should be aware of in this context? Are there any known issues with this setup? Should I prefer Debian 6 over Ubuntu 12.04? I will probably use this how-to unless you recommend otherwise.

And most importantly, should I expect traffic accounting to work out of the box, or will I need to make any manual changes to make it work?

Thanks in advance!

Last edited by m.naumann; 18th September 2012 at 20:01.
Reply With Quote
Sponsored Links
  #2  
Old 19th September 2012, 14:51
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,744 Times in 2,577 Posts
Default

Quote:
Originally Posted by m.naumann View Post
Are there any known issues with this setup?
No.

Quote:
Should I prefer Debian 6 over Ubuntu 12.04? I will probably use this how-to unless you recommend otherwise.
Both Debian and Ubuntu are fine, but Debian is a bit more stable.

Quote:
And most importantly, should I expect traffic accounting to work out of the box, or will I need to make any manual changes to make it work?
It should work out of the box.
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #3  
Old 19th September 2012, 15:20
pititis pititis is offline
Senior Member
 
Join Date: Dec 2010
Location: München
Posts: 364
Thanks: 39
Thanked 90 Times in 68 Posts
Default

"Should" or is working in 3.0.4.6? . This feature doesn't work for me in 12.04 LTS + dovecot 2

Any way to check it? Is working for you?
Reply With Quote
  #4  
Old 19th September 2012, 18:11
m.naumann m.naumann is offline
Junior Member
 
Join Date: Sep 2012
Posts: 11
Thanks: 1
Thanked 0 Times in 0 Posts
Default

So my installation is done, following Falko's great how-to to the point.

I've created a mail domain, and a mail user on this domain. I've used an e-mail client on another computer to connect to this servers' IMAP and Submission ports, and have sent and received e-mail, amounting to several megabytes of mail traffic on both protocols / ports.

Then I checked the mail traffic statistics for this user on ISPConfig's web interface. It listed 0. Which is understandable, since, most likely, these statistics are not updated real-time but by a regularly running cronjob.

So I looked at the newly configured system and found (running
Code:
crontab -l
as root) a daily (or rather nightly, running at 00:30 by default) ISPConfig cron job which runs this file:
Code:
/usr/local/ispconfig/server/cron_daily.php
This PHP script imports the servers' ISPConfig related configuration and some helper classes (such as for structured databases access). After that, it contains multiple sections which each start with a comment. The first such section is
Code:
// store the mailbox statistics in the database
The code in this section searches all mail users' maildir locations for a file called
Code:
ispconfig_mailsize
Example for mail account someuser@domain.tld:
Code:
/var/vmail/domain.tld/someuser/ispconfig_mailsize
So for each user, if ispconfig_mailsize exists, it renames it to "ispconfig_mailsize_save" then reads the file line by line, type casting each line to integer, and summing them up. It then add this value to any traffic previously recorded for this user (and stored in the mysql database, in table "mail_traffic") during the running month, if any, and stores this new value in the database

For each user where ispconfig_mailsize does not exist, nothing is done.

In my case, no such file exists (after sending large e-mails with file attachments in and out).

There are other sections in this file, labelled as follows:
Code:
// Create webalizer statistics
// Create awstats statistics
// Make the web logfiles directories world readable to enable ftp access
// Manage and compress web logfiles and create traffic statistics
// Rotate the ispconfig.log file
// Cleanup website tmp directories
// Cleanup logs in master database (only the "master-server")
// enforce traffic quota (run only on the "master-server")
// deactivate virtual servers (run only on the "master-server")
// Create website backups
None of these seem to refer to mail traffic accounting.

So what's the mechanism which creates the "ispconfig_mailsize" files in the first place?
Code:
# rgrep 'ispconfig_mailsize' /usr/local/
/usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php:			// if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
/usr/local/ispconfig/server/cron_daily.php:	if(@is_file($rec['maildir'].'/ispconfig_mailsize')) {
/usr/local/ispconfig/server/cron_daily.php:		rename($rec['maildir'].'/ispconfig_mailsize',$rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/cron_daily.php:		$lines = file($rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/cron_daily.php:		if(@is_file($rec['maildir'].'/ispconfig_mailsize_save')) unlink($rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php:			// if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
#
/usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php is a symbolic link to /usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php which only contains one reference to this filename, but the line is commented out.
/usr/local/ispconfig/server/cron_daily.php is the file we already discussed above, it does not create these files either.

Maybe the file is created by Dovecot or some other daemon, based on modifications ISPConfig may have made to their configuration files during installation?
Code:
# rgrep 'ispconfig_mailsize' /etc/
grep: /etc/fonts/conf.d/30-defoma.conf: No such file or directory
#
Apparently that's not the case either. Maybe it's somewhere next to the mail storage:
Code:
# rgrep 'ispconfig_mailsize' /var/vmail/
/var/vmail/.mailfilter:`echo $SIZE >> /var/vmail/$HOST/$USER/ispconfig_mailsize`
#
Aha, so this is something. So some searching on the web will tell you that maildrop is a mail filter/mail delivery agent, which comes with the Courier MTA by default, but can also be used standalone, as well as in combination with Dovecot:
http://www.courier-mta.org/maildrop/
http://www.dovecot.org/list/dovecot/...ry/057195.html

Is maildrop packaged for Debian 6?
Code:
# apt-cache search maildrop
courier-maildrop - Courier mail server - mail delivery agent
getmail4 - mail retriever with support for POP3, IMAP4 and SDPS
maildrop - mail delivery agent with filtering abilities
rdeliver - A fully functioning mail filter with RubyFilter
sensible-mda - Mail Delivery Agent wrapper
spamprobe - Bayesian spam filter
#
OK, there are both the Courier one and a standalone maildrop package. Are any of these installed (and hopefully configured, too) now that I followed Falko's how-to?
Code:
# dpkg -l maildrop courier-maildrop
No packages found matching maildrop.
No packages found matching courier-maildrop.
#
Hmm, no. Maybe another installed package provides the "maildrop" command?
Code:
# which maildrop
#
Apparently not.

So I'm not sure how mail (IMAP4r2, POP3, SMTP via Submission port) traffic accounting could work with this setup. Could you please elaborate, Falko? Did I miss something obvious (or not so obvious)?

Thank you very much in advance.

Last edited by m.naumann; 20th September 2012 at 14:21. Reason: Re-reading my post, I realise it may have sounded a bit demanding or challenging - this wasn't intended - sorry about it!
Reply With Quote
  #5  
Old 24th September 2012, 16:01
m.naumann m.naumann is offline
Junior Member
 
Join Date: Sep 2012
Posts: 11
Thanks: 1
Thanked 0 Times in 0 Posts
Default

It would be great to get a confirmation that my suspicion (traffic accounting with Dovecot 2 is not currently implemented) is correct. If it is, I could possibly spend some time on looking into how it could be implemented and provide some feedback.
Reply With Quote
  #6  
Old 25th September 2012, 20:32
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,744 Times in 2,577 Posts
Default

Quote:
Originally Posted by m.naumann View Post
It would be great to get a confirmation that my suspicion (traffic accounting with Dovecot 2 is not currently implemented) is correct.
AFAIK this is correct. I can't tell if this is part of ISPConfig 3.0.5 or not. Will have to ask Till.
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #7  
Old 2nd October 2012, 17:51
m.naumann m.naumann is offline
Junior Member
 
Join Date: Sep 2012
Posts: 11
Thanks: 1
Thanked 0 Times in 0 Posts
Default

Thanks, Falko. Have you two had a chance to discuss this, yet?
Reply With Quote
  #8  
Old 3rd October 2012, 12:22
till till is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 36,421
Thanks: 834
Thanked 5,500 Times in 4,329 Posts
Default

Your findings are partially right, but you mixed up two completely different setups. maildrop is part of the courier based setups only and not required or related to dovecot setups and the same is with the file ispconfig_mailsize which is for courier only too.

Dovecot does not support traffic accounting yet and the sieve filtering language which is part of dovecot is not able to write files to the filesystem with traffic information as well. So its currently not possible to get the traffic information from dovecot as there is no such log file. As far as I know, the dovecot developers have planned to add a interface for that in future.
__________________
Till Brehm
--
Get ISPConfig support and the ISPConfig 3 manual from ispconfig.org.
Reply With Quote
  #9  
Old 3rd October 2012, 15:25
m.naumann m.naumann is offline
Junior Member
 
Join Date: Sep 2012
Posts: 11
Thanks: 1
Thanked 0 Times in 0 Posts
Default

Thanks for your reply, Till.

Quote:
Originally Posted by till View Post
Your findings are partially right, but you mixed up two completely different setups. maildrop is part of the courier based setups only and not required or related to dovecot setups and the same is with the file ispconfig_mailsize which is for courier only too.
My understanding is that there is a standalone implementation available of maildrop which can also be integrated between MTA and dovecot LDA. It probably makes more sense to work with sieve filters, though.

About ispconfig_mailsize, I was thinking of *somehow* making dovecot (or something added to dovecot) write out (properly formatted) ispconfig_mailsize files, too, so that the ISPConfig script doesn't need to be expanded/modified and run another 'find' on the file system, slowing down the system unnecessarily during the nightly cron job. Plus, that would mean less code to maintain on the ISPConfig side once it works. On the other hand it might require in a more complex sieve filter / maildrop setup (which could also increase processing overhead per e-mail, and that would be worse than a second 'find').

Quote:
Originally Posted by till View Post
Dovecot does not support traffic accounting yet and the sieve filtering language which is part of dovecot is not able to write files to the filesystem with traffic information as well. So its currently not possible to get the traffic information from dovecot as there is no such log file. As far as I know, the dovecot developers have planned to add a interface for that in future.
My understanding here is that, while, as you're saying, standard sieve does not allow for writing files to the file system (nor executing external applications which could do so), there are extensions to the sieve filtering language, written and documented by dovecot developers as part of the Piegeonhole project, which allow you to pipe a copy of an e-mail into an external command. Now if this external command is a wrapper around 'wc -c' and writes its output to disk, you have a simplistic implementation for counting mail traffic with little overhead. If needed (such as for only counting the message body), one could use a more intelligent wrapper there, such as one written in PHP.

http://wiki2.dovecot.org/Pigeonhole
http://wiki2.dovecot.org/Pigeonhole/Sieve
http://wiki2.dovecot.org/Pigeonhole/...ns/Extprograms

So I think this should be possible already now. Of course, the plain size of e-mails transferred across IMAP/POP3 differs from the real traffic you're causing there, but I would assume it's close enough (and better than nothing). To get somewhat close to a proper implementation of mail traffic accounting you would also want/need to measure SMTP e-mail traffic. My current understanding is that this is not currently implemented in ISPConfig for any supported MTA (please correct me if wrong).

So I also looked at how this could possibly be done with Postfix. While there can be 'size=...' values in your ESMTP log, this turns out to be an optional flag, i.e. clients/MTAs are not required to send it (and I think most clients actually don't). So the friendly people in #postfix on FreeNode recommended against relying on it and suggested to use a policy (with Postfix policy daemon) instead, because the 'size' parameter is always transferred there.

Here's an incomplete and untested proof of concept shell script which should work for this purpose, logging the sizes of all e-mail it learns about to disk.
Code:
#!/bin/sh
# http://www.postfix.org/SMTPD_POLICY_README.html
IFSBACKUP=$IFS
IFS=`echo -e "=\r\n"`
while read param value morevalues
do
  if [ "$morevalues" != '' ]
  then
    echo 'Error: More input found than expected' >&2
    exit 1
  fi
  case $param in
    sasl_username )
      domain=`echo $value | cut -d'@' -f2`
      user=`echo $value | cut -d'@' -f1`
      ;;
    size )
      size= $value
      ;;
  esac
done
IFS=$IFSBACKUP

echo -e 'action=dunno\n\n'

echo "$size" >> /var/vmail/$domain/$user/LOGFILE
This policy should only be used for SMTP mail, and only be applied after successful client authentication has taken place via SASL. This should be possible by placing it at the proper position in the 'restrictions' section of postfix' main.cf

So, while some of this is still theoretical, my general impression is that traffic accounting with all non legacy versions of Postfix and with Dovecot 2 should be achievable. Now "someone" "just" needs to find the time to actually implement and test this. ;-) I can't really promise to do it, though (but will try to find some time).

Last edited by m.naumann; 3rd October 2012 at 15:27.
Reply With Quote
  #10  
Old 1st November 2012, 13:26
Croydon Croydon is offline
ISPConfig Developer
 
Join Date: Jul 2007
Location: Koblenz, Germany
Posts: 932
Thanks: 16
Thanked 262 Times in 208 Posts
 
Default

I implemented a mail.log parsing fallback for ISPConfig 3.0.5 that fetches mail sizes from the amavis log lines in the mail.log.
So we finally have a traffic accounting for dovecot in the daily cron.
__________________
Marius Cramer

pixcept KG
Reply With Quote
The Following User Says Thank You to Croydon For This Useful Post:
m.naumann (1st November 2012)
Reply

Bookmarks

Tags
accounting, dovecot, imap, ispconfig 3, traffic

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Version 1.3 of the ISPConfig 3 Manual is finally available! falko General 44 2nd December 2011 12:04
How does email traffic accounting work? ham-ster General 1 9th June 2011 14:54
Munin IP Traffic stops on ISPConfig 3 updates.... BorderAmigos General 3 17th January 2011 14:35
ISPconfig is not processing apache logs: the web Traffic overview and stats are gone KingBeto General 12 15th January 2010 18:00
Since install of ISPConfig 3 out-bound traffic fails (eventually) peterwbowey Installation/Configuration 4 14th July 2009 11:11


All times are GMT +2. The time now is 06:21.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.