Postfix Monitoring With Mailgraph And pflogsumm On Debian Lenny

Want to support HowtoForge? Become a subscriber!
 
Submitted by falko (Contact Author) (Forums) on Sun, 2010-06-13 16:25. :: Debian | Monitoring | Postfix

Postfix Monitoring With Mailgraph And pflogsumm On Debian Lenny

Version 1.0
Author: Falko Timme <ft [at] falkotimme [dot] com>
Follow me on Twitter
Last edited 04/27/2010

This article describes how you can monitor your Postfix mailserver with the tools Mailgraph and pflogsumm. Mailgraph creates daily, weekly, monthly, and yearly graphs of sent, received, bounced, and rejected emails and also of spam and viruses, if SpamAssassin and ClamAV are integrated into Postfix (e.g. using amavisd-new). These graphs can be accessed with a browser, whereas pflogsumm ("Postfix Log Entry Summarizer") can be used to send reports of Postfix activity per email.

In the following I will describe how to install and configure Mailgraph and pflogsumm on Debian Lenny.

I do not issue any guarantee that this will work for you!

 

1 Preliminary Note

In this tutorial my Linux system has the IP address 192.168.0.100 and hosts the web site http://www.example.com with the document root /var/www/www.example.com/web and a cgi-bin directory of /var/www/www.example.com/cgi-bin, and I will send the pflogsumm reports to the email address postmaster@example.com.

 

2 Mailgraph

Debian Lenny has packages for Mailgraph and pflogsumm, so we simply install these. We also install rrdtool that stores the data which is needed by Mailgraph to draw the graphs:

aptitude install rrdtool mailgraph

Now we configure the mailgraph package like this:

dpkg-reconfigure mailgraph

You will be asked a few questions:

Should Mailgraph start on boot? <-- Yes
Logfile used by mailgraph: <-- /var/log/mail.log

Then there's also this question:

Count incoming mail as outgoing mail?

If you have integrated a content filter like amavisd (for spam and virus scanning) into Postfix (like in this tutorial: Integrating amavisd-new Into Postfix For Spam- And Virus-Scanning), then answer No to avoid that Mailgraph counts your emails twice (because Postfix delivers emails to amavisd which then - after successful scanning - delivers the mails back to Postfix). If you don't use a content filter, then answer Yes.

During the installation, the system startup links for Mailgraph are created automatically, and Mailgraph also gets started automatically, so we don't need to start it manually.

Now we must copy the mailgraph.cgi script (which draws the graphs and creates the output for our web browsers) to the cgi-bin directory of our www.example.com web site:

cp -p /usr/lib/cgi-bin/mailgraph.cgi /var/www/www.example.com/cgi-bin

The script is already executable, so we don't need to chmod it. If you use suExec for the www.example.com web site, you must chown mailgraph.cgi to the appropriate owner and group.

Now direct your browser to http://www.example.com/cgi-bin/mailgraph.cgi, and you should see some graphs. Of course, there must be some emails going through your system before you see the first results, so be patient.

After some time your graphs could look like this (the following output is customized, so it doesn't look exactly like yours):

Daily Statistics.

Weekly Statistics.

Monthly Statistics.

Yearly Statistics.

Please note: Mailgraph will report spam and viruses only if you have integrated a content filter like amavisd-new into Postfix which is configured to use SpamAssassin and ClamAV to tag spam and virus mails. If you don't do this, you will still see graphs, but without the spam and virus report.

 

3 pflogsumm

To install pflogsumm, we run

aptitude install pflogsumm

We want pflogsumm to be run by a cron job each day and send the report to postmaster@example.com. Therefore we must configure our system that it writes one mail log file for 24 hours, and afterwards starts the next mail log so that we can feed the old mail log to pflogsumm. Therefore we configure logrotate (that's the program that rotates our system's log files) like this: open /etc/logrotate.conf and append the following stanza to it, after the line # system-specific logs may be configured here:

vi /etc/logrotate.conf

[...]
# system-specific logs may be configured here
/var/log/mail.log {
    missingok
    daily
    rotate 7
    create
    compress
    start 0
}

There's a logrotate script in /etc/cron.daily. This script is called every day between 06:00h and 07:00h. With the configuration we just made, it will copy the current Postfix log /var/log/mail.log to /var/log/mail.log.0 and compress it, and the compressed file will be /var/log/mail.log.0.gz. It will also create a new, empty /var/log/mail.log to which Postfix can log for the next 24 hours.

Now we create the script /usr/local/sbin/postfix_report.sh which invokes pflogsumm and makes it send the report to postmaster@example.com:

vi /usr/local/sbin/postfix_report.sh

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
gunzip /var/log/mail.log.0.gz

pflogsumm /var/log/mail.log.0 | formail -c -I"Subject: Mail Statistics" -I"From: pflogsumm@localhost" -I"To: postmaster@example.com" -I"Received: from www.example.com ([192.168.0.100])" | sendmail postmaster@example.com

gzip /var/log/mail.log.0
exit 0

We must make this script executable:

chmod 755 /usr/local/sbin/postfix_report.sh

Then we create a cron job which calls the script everyday at 07:00h:

crontab -e

0 7 * * * /usr/local/sbin/postfix_report.sh &> /dev/null

This will send the report to postmaster@example.com. It looks like this in an email client:

 

4 Links


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by theWoosh (not registered) on Wed, 2014-05-21 10:34.
Hi - it's a fair bit later on and I noticed while getting mailgraph (1.14) to work on my debian/plesk installation, that a few things have changed. As it sure didn't work out of the box for me and there is little other documentation, I thought I would share my findings for anyone else struggling...

Changes

Two main things are new - one that some of the variables have been separated out to a conf file: /etc/default/mailgraph (that is automatically written to during package install), so for instance, the way to ignore localhost is now just to change the line in that file to read:

IGNORE_LOCALHOST=true

& for Plesk users:

MAIL_LOG=/usr/local/psa/var/log/maillog

I guess BOOT_START should also be set to  true...

...the other thing that has changed is that there is now an external css file that the mailgraph.cgi code uses to format output. I found that this didn't automatically get installed from the package and I had to get it from the tarball. I then found that it was broke if placed in the cgi-bin directory alongside mailgraph.cgi (apparently it tries to execute it as a cgi), so instead moved it to the httpdocs directory and modified the code to read:

 <link rel="stylesheet" href="../mailgraph.css" type="text/css" />

Missing Perl Modules 

Anyway that was later, since none of it worked when I installed it from the debian repository, so from here: https://github.com/DamianZaremba/mailgraph I determined I was missing perl modules (check like so: http://www.cyberciti.biz/faq/how-do-i-find-out-what-perl-modules-already-installed-on-my-system/ ) as I don't really use any perl stuff on this server.

Installed cpanimus from here: http://www.cpan.org/modules/INSTALL.html. FILE:Tail got installed in the process...but had to manually install Time::HiRes:

cpanm Time::HiRes

Permissions & image files 

 I still had errors as it was trying to save temporary image files to a directory it didn't have permissions on, so I modified mailgraph.cgi to read:

 my $tmp_dir = '/var/www/vhosts/domain.com/httpdocs/temp';

...which was a directory already with write permission for the apache user.

Bigger Pictures

It now worked , but the charts were a bit small, so modified overall width in the css file and changed the mailgraph.cgi script so it read :

 my $xpoints = 930;

&

my $ypoints = 250;

.... Now it's working fine. Not an automatic setup by a long chalk, but I got there in the end and it looks fine!

I think it would be great to have some more documentation for this... from someone who undrstands it better than me! like I'm not even exactly clear what the legends represent on the graphs...! and is there any way you can get it to import old logs to get a view of the time before it was installed?? that would be awesome! How about getting a daily image emailed??

Submitted by Birta Levente (not registered) on Thu, 2012-08-09 08:49.
If someone is interested, take a look to this mailgraph patch ... add postscreen rejects to the errors graph:

 http://www.birkosan.com/2012/05/mailgraph-with-postfixpostscreen.html

  

 

Submitted by Andri (not registered) on Wed, 2011-06-08 09:35.
hi.. i want to ask something.. I know that mailgraph can log spam/virus but dont know who send it... and pflogsumm can log message, recepient... so is pflogsumm can log virus/spam too ?
Submitted by Jasper Tepper (not registered) on Wed, 2011-01-05 12:05.

I modified the configuration, maybe it is usefull for someone:

postfix_report.sh (no zip and unzip):

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

pflogsumm /var/log/mail.log | formail -c -I"Subject: Mail Statistics" -I"From: pflogsumm@localhost" -I"To: postmaster@example.com" -I"Received: from www.example.com ([192.168.0.100])" | sendmail postmaster@example.com

exit 0

And I use the rsyslog:

/var/log/mail.log {
    missingok
    daily
    rotate 7
    create
    compress
    start 0
prerotate
              /usr/local/sbin/postfix_report.sh
endscript
postrotate
             invoke-rc.d rsyslog reload > /dev/null
endscript
}

 

Greetings,

 

Jasper Tepper

 

Submitted by roche (not registered) on Mon, 2010-12-27 21:09.

Maybe you are using another syslog version but I had to restart the syslog deamon in order to get log in the mail.log file after the rotations.

 

    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
 

Submitted by sjwest (not registered) on Wed, 2010-06-23 14:14.

Logwatch even handles spf reporting,  it should be also said that mailgraph uses all postfix stats, and if you have a computer with multiple instances of postfix on different ip addresses then mailgraph only gives a general idea of all traffic, not specific traffic to that ip/site.

 Nitpick ? maybe but one computer one website is a bit simple.  The mailgraph charts are nice. and can be improved with a bit of a simple stylesheet.

 

Submitted by ppuk (registered user) on Mon, 2010-06-21 02:32.

Thanks for this (an many other) article(s)

I optimised my cron job for pfloggsumm as follows:

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

 

zcat /var/log/mail.log.0.gz |pflogsumm | formail -c -I"Subject: Mail Statistics" -I"From: pflogsumm@localhost" -I"To: postmaster@example.com" -I"Received: from www.example.com ([192.168.0.100])" | sendmail postmaster@example.com

exit 0

Which i think saves some cpu/disk cycles.

Do you see a drawback to this method ?

Kind regards,

Piet

Submitted by Sebastian Bremicker (not registered) on Sun, 2010-06-20 18:10.

Hi,

please note that users of rsyslog will already have an entry for /var/log/mail.log in /etc/logrotate.d/rsyslog (other syslog daemons might have a similar file), so this one should be modified instead of /etc/logrotate.conf.

Please also take into consideration to use the "prerotate" feature of logrotate instead of a cron job that unzips and rezips the logfile. In your example you just need to add

prerotate
              /usr/local/sbin/postfix_report.sh
endscript

to your logrotate configuration to have it executed right before the rotation (and remove the gunzip/gzip lines and use /var/log/mail.log for pflogsumm).

But apart from my comments about logrotate I really appreciate your Howto as I always wanted to have statistics but never thought about those two packages. 

Kind regards

Sebastian

Submitted by Ovidiu (registered user) on Mon, 2010-09-13 13:16.

@sebastian:

 Please elaborate. I am stuck with this problem of conflicting logrotate.conf and rsyslog rotation :-( your solution sounds just like what I need, see  thread: http://www.howtoforge.com/forums/showthread.php?t=25175

Submitted by Guest1 (not registered) on Thu, 2010-06-17 18:24.
Can this be used on a system with ISPconfig installed?