Postfix Monitoring With Mailgraph And pflogsumm

Version 1.0
Author: Falko Timme
Last edited 06/23/2006

This article describes how you can monitor your Postfix mailserver with the tools Mailgraph and pflogsumm. Mailgraph creates daily, weekly, monthly, and yearly graphs of sent, received, bounced, and rejected emails and also of spam and viruses, if SpamAssassin and ClamAV are integrated into Postfix. These graphs can be accessed with a browser, whereas pflogsumm ("Postfix Log Entry Summarizer") can be used to send reports of Postfix activity per email.

In the following I will describe how to install and configure Mailgraph and pflogsumm on Debian Sarge, Ubuntu Dapper Drake (6.06 LTS), and Fedora Core 5.

I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!

 

1 Preliminary Note

In this tutorial my Linux system has the IP address 192.168.0.100 and hosts the web site http://www.example.com with the document root /var/www/www.example.com/web and a cgi-bin directory of /var/www/www.example.com/cgi-bin, and I will send the pflogsumm reports to the email address postmaster@example.com.

 

2 Debian Sarge

 

2.1 Mailgraph

Debian Sarge has packages for Mailgraph and pflogsumm, so we simply install these. We also install rrdtool that stores the data which is needed by Mailgraph to draw the graphs:

apt-get install rrdtool mailgraph

You will be asked a few questions:

Should Mailgraph start on boot? <-- Yes
Which logfile should be used by mailgraph? <-- /var/log/mail.log
Remove RRD files on purge? <-- Yes

Then there's also this question:

Count incoming mail as outgoing mail?

If you have integrated a content filter like amavisd (for spam and virus scanning) into Postfix (like in this tutorial: Virtual Users And Domains With Postfix, Courier And MySQL (+ SMTP-AUTH, Quota, SpamAssassin, ClamAV)), then answer No to avoid that Mailgraph counts your emails twice (because Postfix delivers emails to amavisd which then - after successful scanning - delivers the mails back to Postfix). If you don't use a content filter, then answer Yes.

During the installation, the system startup links for Mailgraph are created automatically, and Mailgraph also gets started automatically, so we don't need to start it manually.

Now we must copy the mailgraph.cgi script (which draws the graphs and creates the output for our web browsers) to the cgi-bin directory of our www.example.com web site:

cp -p /usr/lib/cgi-bin/mailgraph.cgi /var/www/www.example.com/cgi-bin

The script is already executable, so we don't need to chmod it. If you use suExec for the www.example.com web site, you must chown mailgraph.cgi to the appropriate owner and group.

Now direct your browser to http://www.example.com/cgi-bin/mailgraph.cgi, and you should see some graphs. Of course, there must be some emails going through your system before you see the first results, so be patient.

After some time your graphs could look like this (the following output is customized, so it doesn't look exactly like yours):

Daily Statistics.

Weekly Statistics.

Monthly Statistics.

Yearly Statistics.

Please note: Mailgraph will report spam and viruses only if you have integrated a content filter like amavisd into Postfix which is configured to use SpamAssassin and ClamAV to tag spam and virus mails. If you don't do this, you will still see graphs, but without the spam and virus report.

 

2.2 pflogsumm

To install pflogsumm, we run

apt-get install pflogsumm

We want pflogsumm to be run by a cron job each day and send the report to postmaster@example.com. Therefore we must configure our system that it writes one mail log file for 24 hours, and afterwards starts the next mail log so that we can feed the old mail log to pflogsumm. Therefore we configure logrotate (that's the program that rotates our system's log files) like this: open /etc/logrotate.conf and append the following stanza to it, after the line # system-specific logs may be configured here:

vi /etc/logrotate.conf

/var/log/mail.log {
    missingok
    daily
    rotate 7
    create
    compress
    start 0
}

There's a logrotate script in /etc/cron.daily. This script is called everyday between 06:00h and 07:00h. With the configuration we just made, it will copy the current Postfix log /var/log/mail.log to /var/log/mail.log.0 and compress it, and the compressed file will be /var/log/mail.log.0.gz. It will also create a new, empty /var/log/mail.log to which Postfix can log for the next 24 hours.

Now we create the script /usr/local/sbin/postfix_report.sh which invokes pflogsumm and makes it send the report to postmaster@example.com:

vi /usr/local/sbin/postfix_report.sh

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
gunzip /var/log/mail.log.0.gz

pflogsumm /var/log/mail.log.0 | formail -c -I"Subject: Mail Statistics" -I"From: pflogsumm@localhost" -I"To: postmaster@example.com" -I"Received: from www.example.com ([192.168.0.100])" | sendmail postmaster@example.com

gzip /var/log/mail.log.0
exit 0

We must make this script executable:

chmod 755 /usr/local/sbin/postfix_report.sh

Then we create a cron job which calls the script everyday at 07:00h:

crontab -e

0 7 * * * /usr/local/sbin/postfix_report.sh &> /dev/null

This will send the report to postmaster@example.com. It looks like this in an email client:

Share this page:

14 Comment(s)

Add comment

Comments

From: Anonymous at: 2006-07-13 15:07:02

Falko, you're the man! just a perfect howto!

From: Doug at: 2009-02-23 14:19:22

If you do a pflogsumm --help, you'll note the option of -d yesterday.  This means you can get away from the logrotate entries entirely.

 

Doug 

From: venol at: 2011-01-09 14:54:02

how to make it mailgraph and pflogsum to monitoring email one by one for user?

From: at: 2012-09-12 07:46:28

If someone is interested, take a look to this mailgraph patch ... add postscreen rejects to the errors graph:

http://www.birkosan.com/2012/05/mailgraph-with-postfixpostscreen.html

From: at: 2006-12-28 01:50:06

wouldn't this script be more logical?

Saves a lot on cpu/disk io

 

#!/bin/sh 
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 
zcat /var/log/mail.log.0.gz | pflogsumm | formail -c -I"Subject: Mail Statistics" -I"From: pflogsumm@localhost" -I"To: postmaster@example.com" -I"Received: from www.example.com ([192.168.0.100])" | sendmail postmaster@example.com; 
exit 0

From: at: 2007-12-10 01:26:07

Hi,

Thanks for the article it was a very good starting point for me and I'd also like to contribute with my two comments:

- pflogsumm is capable of sending reports for yesterday and today so there's no need to change the rotation of the mail log files. You can just simply execute:

cat /var/log/mail.log.0 /var/log/mail.log | pflogsumm -d yesterday --problems_first

This is going to give you statistics for yesterday.

I personally created a file in /etc/cron.daily/ with this content:

#!/bin/sh
echo -e "From: root@example.com\nSubject: Daily Mail Statistics on `hostname --fqdn`\nTo: <postmaster@example.com>\n\n`cat /var/log/mail.log.0 /var/log/mail.log | pflogsumm -d yesterday --problems_first`\n\n\n\n`cat /var/log/mail.log.0 /var/log/mail.log | pflogsumm -d today --mailq --problems_first`\n"|sendmail -t
exit 0

This gives me statistics for yesterday 0-24 and also for today 0-6:47 am and also lists the content of the queues.

- If for some reason this was not working for you than it is very important to use the right tool for the log rotation. /var/log/mail* files are rotated by a script which comes with the sysklogd package. In your solution /var/log/mail.log is rotated twice, once by the sysklogd script on Sunday and once every day at 6:25. Which will result in a strange situation and you have only 4 days of history, because the sysklogd keeps only 4 versions.

So you can either disable the weekly rotation and use the rotation you described or do it the "proper way". I'm saying proper because the sysklogd rotation is doing other things as well. 

You need to edit two files in order to change the default (weekly rotation with 4 weeks of history) behavior. Add mail to the daily rotation script:

/etc/cron.daily/sysklogd

Add these lines: (-c 14 means keep 14 days of history)

# Non default logrotate for mail logs
for LOG in `syslogd-listfiles -a | grep mail`
do
  if [ -s $LOG ]; then
    savelog -g adm -m 640 -u root -c 14 $LOG >/dev/null
  fi
done

Disable mail logs in the weekly rotation file:

/etc/cron.weekly/sysklogd

modify this line: 

for LOG in `syslogd-listfiles --weekly`

to look like this: 

for LOG in `syslogd-listfiles --weekly -s mail`


From: Anonymous at: 2006-07-06 10:08:38

This is quite neat. Just one minor gripe - after the first log rotation postfix wouldn't write to the new maillog - turned out I had to restart syslog. Maybe you could add this to your howto.

 

From: Anonymous at: 2006-07-06 18:28:41

previous poster -- syslog should be restarted as per logrotate.conf, check you haven't typo'ed.

 

 I only speed-read this so forgive me if I missed it, but if you are using amavisd-new or something similar you'll still end up with double-reporting of emails as pflogsumm doesn't handle this (see http://jimsun.linxnet.com/downloads/pflogsumm-faq.txt  Q.14) 

 

Casper. 

From: at: 2008-03-30 12:11:41

Many users have reported that syslog needs restarting otherwise the maillog file doesn't fill up and the pflogsumm emails just report zero emails. This seems to happen on RedHat based systems (Fedora, RHEL, CentOS).

The fix for this is to change the appropriate section of /etc/logrotate.conf to look like this:

 /var/log/maillog {
    missingok
    daily
    rotate 7
    create
    compress
    start 0
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Additionally, to the previous poster, there is no double-reporting of emails when used with amavisd - this problem is overcome within the tutorial. 

From: at: 2011-01-27 10:13:30

When I run this command to check the output..

pflogsumm /var/log/maillog

I get all zeros, 

maillog file is okay, having lots of data.

 

I am using centos, ispconfig3.

 

Richard

From: Anonymous at: 2006-07-12 17:05:38

I agree... it seems to have taken some time, possibly two days, but it wasn't apparent that the logs broke.  I figured it is due to the edit of /etc/logrotate.d/syslog; removenig /var/log/maillog.  Since this is being excluded now, NO maillog logging is happening.  What is up with that suggestion?

From: Birta Levente at: 2012-05-08 10:53:03

If someone interested in I made a patch for mailgraph to show postfix/postscreen rejects: http://www.birkosan.com/2012/05/mailgraph-with-postfixpostscreen.html

From: theWoosh at: 2014-05-21 09:27:05

Hi - it's a fair bit later on and I noticed while getting mailgraph (1.14) to work on my debian/plesk installation, that a few things have changed. As it sure didn't work out of the box for me and there is little other documentation, I thought I would share my findings for anyone else struggling...

Changes

Two main things are new - one that some of the variables have been separated out to a conf file: /etc/default/mailgraph (that is automatically written to during package install), so for instance, the way to ignore localhost is now just to change the line in that file to read:

IGNORE_LOCALHOST=true

& for Plesk users:

MAIL_LOG=/usr/local/psa/var/log/maillog

I guess BOOT_START should also be set to  true...

...the other thing that has changed is that there is now an external css file that the mailgraph.cgi code uses to format output. I found that this didn't automatically get installed from the package and I had to get it from the tarball. I then found that it was broke if placed in the cgi-bin directory alongside mailgraph.cgi (apparently it tries to execute it as a cgi), so instead moved it to the httpdocs directory and modified the code to read:

 <link rel="stylesheet" href="../mailgraph.css" type="text/css" />

Missing Perl Modules 

Anyway that was later, since none of it worked when I installed it from the debian repository, so from here: https://github.com/DamianZaremba/mailgraph I determined I was missing perl modules (check like so: http://www.cyberciti.biz/faq/how-do-i-find-out-what-perl-modules-already-installed-on-my-system/ ) as I don't really use any perl stuff on this server.

Installed cpanimus from here: http://www.cpan.org/modules/INSTALL.html. FILE:Tail got installed in the process...but had to manually install Time::HiRes:

cpanm Time::HiRes

Permissions & image files 

 I still had errors as it was trying to save temporary image files to a directory it didn't have permissions on, so I modified mailgraph.cgi to read:

 my $tmp_dir = '/var/www/vhosts/domain.com/httpdocs/temp';

...which was a directory already with write permission for the apache user.

Bigger Pictures

It now worked , but the charts were a bit small, so modified overall width in the css file and changed the mailgraph.cgi script so it read :

 my $xpoints = 930;

&

my $ypoints = 250;

.... Now it's working fine. Not an automatic setup by a long chalk, but I got there in the end and it looks fine!

I think it would be great to have some more documentation for this... from someone who undrstands it better than me! like I'm not even exactly clear what the legends represent on the graphs...! and is there any way you can get it to import old logs to get a view of the time before it was installed?? that would be awesome!  ...or get it to email a daily chart - bit more proactive?! Anyone.....? :-)

From: theWoosh at: 2014-05-21 09:31:30

sorry that last comment was meant to go on the Debian Lenny install not this one....