Postfix Monitoring With Mailgraph And pflogsumm On Debian Etch

[klonos]

Hey falko does this Howto work with "Virtual Users And Domains With Postfix, Courier And MySQL (Ubuntu 6.10 Edgy Eft)"???

I did try it and I am having the following issues (sorry for the long story):

1. I executed:

pflogsumm /var/log/mail.log.0 | formail -c -I"Subject: Mail Statistics" -I"From: stats@domain.net" -I"To: postmaster@domain.net" -I"Received: from mail.domain.net ([10.0.0.11])" | sendmail postmaster@domain.net
from the command prompt and I did receive one or two stat reports. So, I thought it would work fine as a cronjob too. After a day or two I stoped receiving them and I got a "Considered UNSOLICITED BULK EMAIL, apparently from you" email instead:

Content-filter at mail.domain.net [postmaster@mail.domain.net]

A message from <root@mail.domain.net> to:
-> postmaster@domain.net

was considered unsolicited bulk e-mail (UBE).

Our internal reference code for your message is 21246-02/J26IUlFdPLY3

The message carried your return address, so it was either a genuine mail from you, or a sender address was faked and your e-mail address abused by third party, in which case we apologize for undesired notification.

We do try to minimize backscatter for more prominent cases of UBE and for infected mail, but for less obvious cases of UBE some balance between losing genuine mail and sending undesired backscatter is sought, and there can be some collateral damage on both sides.

Return-Path: <root@mail.domain.net>
Message-ID: <20070817220001.180FE148130@mail.domain.net>
Subject: Mail Statistics

Delivery of the email was stopped!
So I think that the stats are blocked as spam. Am I right on this one?

So after some research/reading/googling I found out that I need to use the amavid-release command in order to get quarantined (false positive) mail delivered to its final receipient.

So I did:

root@mail:~# amavisd-release
Not enough arguments
amavisd-release version 1.1
Usage: $ amavisd-release mail_file [secret_id [alt_recip1 alt_recip2 ...]]
or to read request lines from stdin: $ amavisd-release -
root@mail:~# updatedb
root@mail:~# locate 21246-02/J26IUlFdPLY3
root@mail:~# locate J26IUlFdPLY3
/var/lib/amavis/virusmails/spam-J26IUlFdPLY3.gz
root@mail:~# amavisd-release J26IUlFdPLY3
Can't connect to UNIX socket /var/run/amavis/amavisd.sock: No such file or directory at /usr/sbin/amavisd-release line 191.
root@mail:~# amavisd-release 21246-02/J26IUlFdPLY3
Can't connect to UNIX socket /var/run/amavis/amavisd.sock: No such file or directory at /usr/sbin/amavisd-release line 191.
root@mail:~# amavisd-release spam-J26IUlFdPLY3
Can't connect to UNIX socket /var/run/amavis/amavisd.sock: No such file or directory at /usr/sbin/amavisd-release line 191.
root@mail:~# amavisd-release spam-J26IUlFdPLY3.gz
Can't connect to UNIX socket /var/run/amavis/amavisd.sock: No such file or directory at /usr/sbin/amavisd-release line 191.
root@mail:~# amavisd-release /var/lib/amavis/virusmails/spam-J26IUlFdPLY3.gz
Can't connect to UNIX socket /var/run/amavis/amavisd.sock: No such file or directory at /usr/sbin/amavisd-release line 191.
So it didn't take me log to realise that there was nothing wrong with the way I was typing the filename of the mail I wanted to release. Tried to release some other spam and had the same error. Can you please tell me how to have this email (or any email) released?

2. I also received a "Cron <root@mail> /usr/local/sbin/postfix_report.sh &> /dev/null" mail.

First it said:

gunzip: /var/log/mail.log.0.gz: No such file or directory
did an ls in /var/log and saw that there were only the files:

- mail.log
- mail.log.0
- mail.log.1.gz
- mail.log.2.gz
- mail.log.3.gz
- mail.log.4.gz
- mail.log.5.gz
- mail.log.6.gz

but no mail.log.0.gz

So, I edited /usr/local/sbin/postfix_report.sh to read:

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#gunzip /var/log/mail.log.0.gz

pflogsumm /var/log/mail.log.0 | formail -c -I"Subject: Mail Statistics" -I"From: stats@gnosis.gr" -I"To: postmaster@gnosis.gr" -I"........

#gzip /var/log/mail.log.0
Because I thought that since the mail.log.0 file is already gunziped it did not need to be done twice. In fact I believe that this was what caused the error. I also commented the gzip line since no gzipping was needed either.

After a day I got a mail with the same subject but this time it said:

Can't open /var/log/mail.log.0: No such file or directory at /usr/sbin/pflogsumm line 513.
How do I resolve this one? What am I doing wrong?

3. The Mailgraph is not working either. I gave it some time to collect data for a day or two, but instead of graphs I see a "mailgraph" string when I browse http://mail.domain.net/cgi-bin/mailgraph.cgi

[falko]

Hey falko does this Howto work with "Virtual Users And Domains With Postfix, Courier And MySQL (Ubuntu 6.10 Edgy Eft)"???

I did try it and I am having the following issues (sorry for the long story):

1. I executed:


from the command prompt and I did receive one or two stat reports. So, I thought it would work fine as a cronjob too. After a day or two I stoped receiving them and I got a "Considered UNSOLICITED BULK EMAIL, apparently from you" email instead:


So I think that the stats are blocked as spam. Am I right on this one?

So after some research/reading/googling I found out that I need to use the amavid-release command in order to get quarantined (false positive) mail delivered to its final receipient.

So I did:


So it didn't take me log to realise that there was nothing wrong with the way I was typing the filename of the mail I wanted to release. Tried to release some other spam and had the same error. Can you please tell me how to have this email (or any email) released?I think you can specify your local domain somewhere in the amavisd configuration so that mails from this domain won't be considered as spam.

2. I also received a "Cron <root@mail> /usr/local/sbin/postfix_report.sh &> /dev/null" mail.

First it said:


did an ls in /var/log and saw that there were only the files:

- mail.log
- mail.log.0
- mail.log.1.gz
- mail.log.2.gz
- mail.log.3.gz
- mail.log.4.gz
- mail.log.5.gz
- mail.log.6.gz

but no mail.log.0.gz

So, I edited /usr/local/sbin/postfix_report.sh to read:


Because I thought that since the mail.log.0 file is already gunziped it did not need to be done twice. In fact I believe that this was what caused the error. I also commented the gzip line since no gzipping was needed either.

After a day I got a mail with the same subject but this time it said:


How do I resolve this one? What am I doing wrong?Did you configure logrotate as shown in the tutorial?

3. The Mailgraph is not working either. I gave it some time to collect data for a day or two, but instead of graphs I see a "mailgraph" string when I browse http://mail.domain.net/cgi-bin/mailgraph.cgiI think the directory where the graphs should be created isn't writable. What's in your mailgraph.cgi (in the beginning)?

[klonos]

I think you can specify your local domain somewhere in the amavisd configuration so that mails from this domain won't be considered as spam.
Yes, I will look into this, but what I really want is a way to release other mail as well. amavisd-release doesn't seem to work and keeps spitting:

Can't connect to UNIX socket /var/run/amavis/amavisd.sock: No such file or directory at /usr/sbin/amavisd-release line 191.
no matter what spam false positive I try to release.

Did you configure logrotate as shown in the tutorial?
I double-checked it. here it is:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

/var/log/btmp {
missingok
monthly
create 0664 root utmp
rotate 1
}

# system-specific logs may be configured here
/var/log/mail.log {
missingok
daily
rotate 7
create
compress
start 0
}
I think the directory where the graphs should be created isn't writable. What's in your mailgraph.cgi (in the beginning)?
Here is what is in the begging of my mailgraph.cgi:

#!/usr/bin/perl -w

# mailgraph -- a postfix statistics rrdtool frontend
# copyright (c) 2000-2005 David Schweikert <dws@ee.ethz.ch>
# released under the GNU General Public License

use RRDs;
use POSIX qw(uname);

my $VERSION = "1.12";

my $host = (POSIX::uname())[1];
my $scriptname = 'mailgraph.cgi';
my $xpoints = 540;
my $points_per_sample = 3;
my $ypoints = 160;
my $ypoints_err = 96;
my $rrd = '/var/lib/mailgraph/mailgraph.rrd'; # path to where the RRD database is
my $rrd_virus = '/var/lib/mailgraph/mailgraph_virus.rrd'; # path to where the Virus RRD database is
my $tmp_dir = '/var/lib/mailgraph'; # temporary directory where to store the images
I've checked my /var/lib/mailgraph directory and it only contains a directory ,cgi-bin. The /var/lib/mailgraph/,cgi-bin directory is also empty.

Here is a permissions listing:

root@mail:/var/lib# ls -l
total 112
drwxr-xr-x 2 root root 4096 2007-08-01 12:15 alsa
drwxr-xr-x 7 amavis amavis 4096 2007-08-02 02:22 amavis
drwxr-xr-x 2 root root 4096 2006-09-27 19:54 apache2
drwxr-xr-x 4 root root 4096 2007-08-16 02:04 apt
drwxr-xr-x 2 root root 4096 2006-09-12 00:19 aptitude
drwxr-xr-x 2 root root 4096 2007-08-01 15:00 belocs
drwxr-xr-x 2 clamav clamav 4096 2007-08-20 19:27 clamav
drwxr-xr-x 2 bin bin 4096 2006-07-20 12:17 courier
drwxr-xr-x 2 dcc dcc 4096 2007-08-16 02:05 dcc
drwxr-xr-x 3 root root 4096 2007-08-10 00:56 defoma
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 dhcp3
drwxr-xr-x 7 root root 4096 2007-08-16 02:04 dpkg
drwxr-xr-x 2 root root 4096 2007-08-01 15:00 initramfs-tools
drwxr-xr-x 2 root root 4096 2006-10-06 14:34 initscripts
drwxr-xr-x 3 root root 4096 2007-08-01 14:59 locales
drwxr-xr-x 2 root root 4096 2007-08-02 06:25 logrotate
drwxr-xr-x 3 www-data www-data 4096 2007-08-16 01:51 mailgraph
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 mdadm
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 misc
drwxr-xr-x 4 mysql mysql 4096 2007-08-16 02:05 mysql
drwxr-xr-x 2 root root 4096 2007-04-04 12:53 mysql-cluster
drwx-wx-wt 2 root root 4096 2007-08-20 13:09 php5
drwxr-xr-x 2 root root 4096 2007-08-02 01:11 phpmyadmin
drwxr-xr-x 3 root root 4096 2007-08-02 02:21 python-support
drwxr-xr-x 3 root root 4096 2007-08-10 00:56 ucf
drwxr-xr-x 2 root root 4096 2007-08-16 02:06 urandom
drwxr-xr-x 3 root root 4096 2007-08-01 14:59 vim
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 x11
root@mail:/var/lib# cd mail*
root@mail:/var/lib/mailgraph# ls -l
total 4
drwxr-xr-x 2 www-data www-data 4096 2007-08-16 01:51 ,cgi-bin
root@mail:/var/lib/mailgraph#

[falko]

Can you try
chmod -R 777 /var/lib/mailgraph?

What's the output of updatedb
locate amavisd.sock?

[klonos]

Can you try
chmod -R 777 /var/lib/mailgraph?

Ok, changed perimisions and verified:

root@mail:~# ls -l /var/lib/
total 112
drwxr-xr-x 2 root root 4096 2007-08-01 12:15 alsa
drwxr-xr-x 7 amavis amavis 4096 2007-08-02 02:22 amavis
drwxr-xr-x 2 root root 4096 2006-09-27 19:54 apache2
drwxr-xr-x 4 root root 4096 2007-08-16 02:04 apt
drwxr-xr-x 2 root root 4096 2006-09-12 00:19 aptitude
drwxr-xr-x 2 root root 4096 2007-08-01 15:00 belocs
drwxr-xr-x 2 clamav clamav 4096 2007-08-21 19:28 clamav
drwxr-xr-x 2 bin bin 4096 2006-07-20 12:17 courier
drwxr-xr-x 2 dcc dcc 4096 2007-08-16 02:05 dcc
drwxr-xr-x 3 root root 4096 2007-08-10 00:56 defoma
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 dhcp3
drwxr-xr-x 7 root root 4096 2007-08-16 02:04 dpkg
drwxr-xr-x 2 root root 4096 2007-08-01 15:00 initramfs-tools
drwxr-xr-x 2 root root 4096 2006-10-06 14:34 initscripts
drwxr-xr-x 3 root root 4096 2007-08-01 14:59 locales
drwxr-xr-x 2 root root 4096 2007-08-02 06:25 logrotate
drwxrwxrwx 3 www-data www-data 4096 2007-08-16 01:51 mailgraph
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 mdadm
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 misc
drwxr-xr-x 4 mysql mysql 4096 2007-08-16 02:05 mysql
drwxr-xr-x 2 root root 4096 2007-04-04 12:53 mysql-cluster
drwx-wx-wt 2 root root 4096 2007-08-21 16:09 php5
drwxr-xr-x 2 root root 4096 2007-08-02 01:11 phpmyadmin
drwxr-xr-x 3 root root 4096 2007-08-02 02:21 python-support
drwxr-xr-x 3 root root 4096 2007-08-10 00:56 ucf
drwxr-xr-x 2 root root 4096 2007-08-16 02:06 urandom
drwxr-xr-x 3 root root 4096 2007-08-01 14:59 vim
drwxr-xr-x 2 root root 4096 2007-08-01 14:59 x11
root@mail:~# ls -l /var/lib/mailgraph
total 4
drwxrwxrwx 2 www-data www-data 4096 2007-08-16 01:51 ,cgi-bin
root@mail:~#

Lets let it collect some data for a day or two and I'll let you know. By the way... is it normal for that cgi-bin directory in /var/lib/mailgraph to have the comma in the beggining of its filename?? ...Just a thought.

What's the output of updatedb
locate amavisd.sock?

There is no file or directory by that name on the drive:

root@mail:~# updatedb
root@mail:~# locate amavisd.sock

Here is a list of all amavisd files found though. Hope it helps:

root@mail:~# locate amavisd
/etc/cron.daily/amavisd-new
/etc/cron.d/amavisd-new
/etc/logcheck/ignore.d.server/amavisd-new
/etc/logcheck/ignore.d.workstation/amavisd-new
/usr/sbin/amavisd-agent
/usr/sbin/amavisd-nanny
/usr/sbin/amavisd-new
/usr/sbin/amavisd-release
/usr/share/doc/amavisd-new
/usr/share/doc/amavisd-new/AAAREADME.first
/usr/share/doc/amavisd-new/amavisd-new-docs.html
/usr/share/doc/amavisd-new/changelog.Debian.gz
/usr/share/doc/amavisd-new/changelog.gz
/usr/share/doc/amavisd-new/copyright
/usr/share/doc/amavisd-new/examples
/usr/share/doc/amavisd-new/examples/amavisd-agent.gz
/usr/share/doc/amavisd-new/examples/amavisd.conf-default.gz
/usr/share/doc/amavisd-new/examples/amavisd.conf-sample.gz
/usr/share/doc/amavisd-new/NEWS.Debian.gz
/usr/share/doc/amavisd-new/README.banned.gz
/usr/share/doc/amavisd-new/README.chroot.gz
/usr/share/doc/amavisd-new/README.contributed
/usr/share/doc/amavisd-new/README.courier.gz
/usr/share/doc/amavisd-new/README.courier-old.gz
/usr/share/doc/amavisd-new/README.customize.gz
/usr/share/doc/amavisd-new/README.Debian
/usr/share/doc/amavisd-new/README.exim_v3_app.gz
/usr/share/doc/amavisd-new/README.exim_v3.gz
/usr/share/doc/amavisd-new/README.exim_v4
/usr/share/doc/amavisd-new/README.exim_v4_app2.gz
/usr/share/doc/amavisd-new/README.exim_v4_app.gz
/usr/share/doc/amavisd-new/README.ldap.gz
/usr/share/doc/amavisd-new/README.lookups.gz
/usr/share/doc/amavisd-new/README.milter.gz
/usr/share/doc/amavisd-new/README.old.scanners.gz
/usr/share/doc/amavisd-new/README.performance.gz
/usr/share/doc/amavisd-new/README.policy-on-notifications.gz
/usr/share/doc/amavisd-new/README.postfix.gz
/usr/share/doc/amavisd-new/README.protocol.gz
/usr/share/doc/amavisd-new/README.sendmail-dual.gz
/usr/share/doc/amavisd-new/README.sendmail-dual.old.gz
/usr/share/doc/amavisd-new/README.sendmail.gz
/usr/share/doc/amavisd-new/README.sql.gz
/usr/share/doc/amavisd-new/RELEASE_NOTES.gz
/usr/share/doc/amavisd-new/TODO.Debian
/usr/share/doc/amavisd-new/TODO-SNMP-AGENT.gz
/usr/share/lintian/overrides/amavisd-new
/usr/share/man/man8/amavisd-new.8.gz
/var/cache/apt/archives/amavisd-new_1%3a2.4.1-1_all.deb
/var/lib/dpkg/info/amavisd-new.conffiles
/var/lib/dpkg/info/amavisd-new.list
/var/lib/dpkg/info/amavisd-new.md5sums
/var/lib/dpkg/info/amavisd-new.postinst
/var/lib/dpkg/info/amavisd-new.postrm
/var/lib/dpkg/info/amavisd-new.prerm
/var/lib/dpkg/info/amavisd-new.templates
/var/run/amavis/amavisd.lock
/var/run/amavis/amavisd.pid
root@mail:~#

[klonos]

Ok, I've had some progress regarding the issue with the amavisd-release giving:

Can't connect to UNIX socket /var/run/amavis/amavisd.sock: No such file or directory at /usr/sbin/amavisd-release line 191.

error. There is no /var/run/amavis/amavisd.sock file, but there is a /var/run/amavis/amavis.sock.

So, I changed /usr/sbin/amavisd-release from:

[...]

$log_level = 1;
# $socketname = '127.0.0.1:9998';
$socketname = '/var/run/amavis/amavisd.sock';

[...]

to

[...]

$log_level = 1;
# $socketname = '127.0.0.1:9998';
# $socketname = '/var/run/amavis/amavisd.sock';
$socketname = '/var/run/amavis/amavis.sock';

[...]

Now everything seems to be working ok!

Also playing around with the mail.log.[x].gz files (renaming a few of them) and re-enabling the gunzip/gzip lines in /usr/local/sbin/postfix_report.sh got my mail stats working!!

In other words, 2 issues solved, 1 remains. That seems to be going pretty well. Lets see if we will get mailgraph working.

[falko]

I've just read through this thread again. It seems that /var/lib/mailgraph/mailgraph.rrd and /var/lib/mailgraph/mailgraph_virus.rrd aren't existing.
What's the output of
locate mailgraph.rrd
locate mailgraph_virus.rrd?

[klonos]

Yes, I was sure something like this would happen since I posted 3 issues at one thread. Sorry.

Anyways... as I've already mentioned, my mailgraph dir contains nothing but a ,cgi-bin directory that is empty itself. (by the way should that comma be there in the first place??? or is it a typo)

Here is what mailgraph files/dirs I have plus a listing of both /var/lib/mailgraph and /var/lib/mailgraph/,cgi-bin :

root@mail:~# locate mailgraph
/etc/default/mailgraph
/etc/init.d/mailgraph
/usr/lib/cgi-bin/mailgraph.cgi
/usr/sbin/mailgraph.pl
/usr/share/doc/mailgraph
/usr/share/doc/mailgraph/changelog.Debian.gz
/usr/share/doc/mailgraph/changelog.gz
/usr/share/doc/mailgraph/copyright
/usr/share/doc/mailgraph/README
/usr/share/doc/mailgraph/README.Debian
/usr/share/doc/mailgraph/README.fetchmail
/var/cache/apt/archives/mailgraph_1.12-1_all.deb
/var/lib/dpkg/info/mailgraph.conffiles
/var/lib/dpkg/info/mailgraph.config
/var/lib/dpkg/info/mailgraph.list
/var/lib/dpkg/info/mailgraph.md5sums
/var/lib/dpkg/info/mailgraph.postinst
/var/lib/dpkg/info/mailgraph.postrm
/var/lib/dpkg/info/mailgraph.preinst
/var/lib/dpkg/info/mailgraph.prerm
/var/lib/dpkg/info/mailgraph.templates
/var/lib/mailgraph
/var/lib/mailgraph/,cgi-bin
root@mail:~# ls /var/lib/mailgraph
,cgi-bin
root@mail:~# ls /var/lib/mailgraph/,cgi-bin
root@mail:~#

as you can see, there are no rrd files. Also, those two dirs have full permisions since I've changed that during a previous troubleshooting step:

root@mail:~# ls -l /var/lib/
total 112
...
drwxrwxrwx 3 www-data www-data 4096 2007-08-16 01:51 mailgraph
...
root@mail:~# ls -l /var/lib/mailgraph
total 4
drwxrwxrwx 2 www-data www-data 4096 2007-08-16 01:51 ,cgi-bin
root@mail:~#

[falko]

(by the way should that comma be there in the first place??? or is it a typo)I *think* the comma is correct, but I'm not sure.

The lack of the two .rrd files makes me think the mailgraph isn't running at all. Can you see it in the output of ps aux? Do you get any errors when you restart mailgraph?

[klonos]

Yes, once again you were right. A simple /etc/init.d/mailgraph start solved it. Now the two .rrd files are there as they should.

It seems that each time I restart the server mailgraph doesn't auto-start though. So, how can I get it to start automatically everytime?

[falko]

Try
update-rc.d mailgraph defaults

[klonos]

That did the trick! Thank you.

One last thing though... I tried upgrading the mailgraph from 1.12 to 1.13 (from source) by replacing the /usr/lib/cgi-bin/mailgraph.cgi and /usr/sbin/mailgraph.pl and after I have updated the cgi code to point to the correct .rrd files I got the first graph (the one showing sent & received) for each day, week, month and year working, but the second graph (the one with spam, virus etc stats) could not show. All I am getting is a "mailgraph" placeholder instead. Any ideas? Are there any mailgraph logs I failed to locate in my drive?

By the way (irrelevant to this thread), I now have another 2 servers to "play with", so if you were thinking of updating the "Virtual Users And Domains With Postfix, Courier And MySQL (Ubuntu 6.10 Edgy Eft)" for 7.04 or 7.10 once it is out, let me know if I can help by testing, ok? You could also include things like this stats howto and a webmail (squirrelmail perhaps) installation as well to make it more complete. Also remember that I've found out during this troubleshooting that amavisd-release doesn't work out of the box and maybe a gui solution like MailZu would also be a plus. As I said, I am more than willing to help with testing.

[falko]

That did the trick! Thank you.

One last thing though... I tried upgrading the mailgraph from 1.12 to 1.13 (from source) by replacing the /usr/lib/cgi-bin/mailgraph.cgi and /usr/sbin/mailgraph.pl and after I have updated the cgi code to point to the correct .rrd files I got the first graph (the one showing sent & received) for each day, week, month and year working, but the second graph (the one with spam, virus etc stats) could not show. All I am getting is a "mailgraph" placeholder instead. Any ideas? Are there any mailgraph logs I failed to locate in my drive?

Did it work prior to upgrading?

[klonos]

Yes it did and it still does. I mean this is how I worked around it. I revertted back to 1.12. I am just trying to upgrade to the newest version.

PS. sorry for te late reply.