Postfix Spam Filter using Ubuntu Dapper, MailScanner, SpamAssassin, Razor, Pyzor, DCC and ClamAV - Page 4

3 Pyzor, Razor, DCC, SpamAssassin and MailScanner Configuration

3.1 Pyzor Configuration

We need to change some permissions on pyzor first:

chmod -R a+rX /usr/share/doc/pyzor /usr/bin/pyzor /usr/bin/pyzord


This next command will have to be modified if you have a different version of python. Try locate pyzor.

chmod -R a+rX /usr/lib/site-python/pyzor


Here we supply the IP address of the Pyzor server to Pyzor. This will create a .pyzor directory in both user's home directories, and place the server's IP address in a servers file therein. Then it will test the connection. If you are behind a firewall, open port 24441/udp in and out to your server. While you're at it also open up 6277/udp for DCC, 2703/tcp for Razor and 783/tcp for SpamAssassin:

pyzor ping


Usually you'll get a timeout from the pyzor ping so don't worry about it. We'll test again later.

If in the future the IP address of the server changes, you will need to run through this section again. You can find the address of the current Pyzor server here http://pyzor.sourceforge.net/cgi-bin/inform-servers-0-3-x.

If Pyzor is working, you will see "Pyzor: got response:" Pyzor queries a Pyzor server in much the same way your computer queries a DNS server. The only practical difference is the port number that is used. If Pyzor is not working, you might need to open up the port on your firewall or the Pyzor server is busy.

3.2 Razor Configuration

Create a default .razor configuration under root home directory:

cd
rm /etc/razor/razor-agent.conf
razor-admin -create
razor-admin -create


Razor v2 requires reporters to be registered so their reputations can be computed over time and they can participate in the revocation mechanism. Registration is done with razor-admin -register. When razor-admin -register is invoked as root, it negotiates a registration with the Nomination Server and writes the identity information in /root/.razor/identity-username. Manually invoked it in one of the following ways:

1. To register user:foo and password:s1kret (foo and s1kret are examples) :

razor-admin -register -user=foo -pass=s1kr3t


2. To register with an email address and have the password assigned:

razor-admin -register -user=foo@bar.com


3. To have both (random) username and password assigned:

razor-admin -register


I usually just do number 3. Make the following changes to /root/.razor/razor-agent.conf:

vi /root/.razor/razor-agent.conf


Change debuglevel = 3 to debuglevel = 0 (yes zero not "o"). This will prevent Razor from filling up your drive with debug information. Also we will move these configs someplace that the Postfix user can read them so add the razorhome line to the end of the file. Those two lines should look like this when done:

debuglevel             = 0
razorhome             = /var/lib/MailScanner/.razor/

We will test Razor later. man razor-agent.conf or go to http://razor.sourceforge.net/docs/razor-agent.conf.php for more information on Razor.

3.3 DCC Setup and Configuration

Install DCC:

apt-get install dcc-client


We are not running a DCC server, so we don't need to waste time checking ourselves: If you are a large organization (100,000 messages per day), you should investigate running your own server.

Once the installation is done run:

cdcc "delete 127.0.0.1"
cdcc "delete 127.0.0.1 Greylist"


Test our installation with:

cdcc info


You should get 'requests ok' from the servers.

4 MailScanner, ClamAV and SpamAssassin Configuration

4.1 MailScanner and ClamAV

Stop Postfix:

postfix stop


Install the packages:

apt-get install mailscanner clamav


Update ClamAV virus defenitions:

freshclam


Let's start with MailScanner. The MailScanner that was just installed from the repositories is a very old version so we will now remove it and install the MailScanner package from source

Download the tarball from http://www.mailscanner.info/downloads.html ... At the time of this writing it is at version 4.56.8-1 and the tarball link is http://www.mailscanner.info/files/4/tar/MailScanner-install-4.56.8-1.tar.gz. Then install MailScanner using the install.sh script.

cd
apt-get remove mailscanner
wget http://www.mailscanner.info/files/4/tar/MailScanner-install-4.56.8-1.tar.gz
tar zxvf MailScanner-install-4.56.8-1.tar.gz
cd MailScanner-install-4.56.8
./install.sh


Ignore the message about the cron lines that we need to add to cron for now.

Once that is done, we need to make a directory for SpamAssassin in the spool and give postfix permissions to it, if you run sa-learn --force as root, bayes databese that is stored in these directories will change to root:root and spamassassin will error looking at the db. Just keep an eye on the mail.log and you'll remember to change the permissions back. Also disable the MailScanner default configs:

mkdir /var/spool/MailScanner/spamassassin
mv /etc/MailScanner /etc/MailScanner.dist


Backup your MailScanner.conf file:

cp /opt/MailScanner/etc/MailScanner.conf /opt/MailScanner/etc/MailScanner.conf.back


Edit MailScanner.conf:

vi /opt/MailScanner/etc/MailScanner.conf


Change the following parameters in MailScanner.conf:

%org-name% = YOURDOMAIN-COM
%org-long-name% = Your Company Long Name INC
%web-site% = www.yourdomain.com
Run As User = postfix
Run As Group = postfix
Incoming Queue Dir = /var/spool/postfix/hold
Outgoing Queue Dir = /var/spool/postfix/incoming
MTA = postfix
Virus Scanners = clamav
Spam Subject Text = [SPAM]
Send Notices = no
Spam List = ORDB-RBL SBL+XBL
Required SpamAssassin Score = 6
High SpamAssassin Score = 10
Spam Actions = deliver striphtml
High Scoring Spam Actions = delete
Rebuild Bayes Every = 86400
Wait During Bayes Rebuild = yes
SpamAssassin User State Dir = /var/spool/MailScanner/spamassassin

The first 9 lines are basically required in order for everything to work, the rest are recommended. The MailScanner.conf is well documented so please read the notes there if you have any questions about the rest of the options we changed. Poke around this file from top to bottom.

Also take a look at the section "Removing/Logging dangerous or potentially offensive content" in the MailScanner.conf file. I had to disable most of these because clients were complaining about '{Disarmed}' messages.

4.2 SpamAssassin

First we need to disable the default SpamAssassin configuration file:

mv /etc/spamassassin/local.cf /etc/spamassassin/local.cf.disabled


Now lets backup the SpamAssassin configuration file in MailScanner then edit:

cp /opt/MailScanner/etc/spam.assassin.prefs.conf /opt/MailScanner/etc/spam.assassin.prefs.conf.back


vi /opt/MailScanner/etc/spam.assassin.prefs.conf


Add these two lines to the top of spam.assassin.prefs.conf:

pyzor_options --homedir /var/lib/MailScanner/
razor_config /var/lib/MailScanner/.razor/razor-agent.conf

Change where SpamAssassing looks for the Bayes database, comment out the default bayes_path or change it accordingly:

#bayes_path /var/lib/MailScanner/bayes
bayes_path /var/spool/MailScanner/spamassassin/bayes

Look for these lines and change them accordingly:

bayes_ignore_header X-YOURDOMAIN-COM-MailScanner
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamCheck
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamScore
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-Information

"YOURDOMAIN-COM" should be replaced with whatever you used for "%org-name%" in the MailScanner.conf file. Leave the "X-" in place.

Make sure that "bayes_auto_expire 0" is not commented out:

bayes_auto_expire 0

Edit the SpamAssassin v310.pre to enable Razor and DCC

vi /etc/spamassassin/v310.pre


Uncomment the following lines:

loadplugin Mail::SpamAssassin::Plugin::DCC
loadplugin Mail::SpamAssassin::Plugin::Razor2

5 Bring it all Together

Copy over the Pyzor and Razor configs to someplace that the Postfix user will be able to read them:

cp -R /root/.pyzor /var/lib/MailScanner
cp -R /root/.razor /var/lib/MailScanner


Now that we have everything in there, set the correct permissions:

chown -R postfix.postfix /var/spool/MailScanner/
chown -R postfix.postfix /var/lib/MailScanner/


Let's see if SpamAssassin is happy:

su postfix -p -c 'spamassassin -x -D -C /opt/MailScanner/etc/spam.assassin.prefs.conf --lint'


You should see lines come up with DCC, Pyzor and Razor that say loading plugin and hopefully no errors.

NOTE: If your ever run the sa-learn, remember to run it like this, su postfix -p -c 'sa-learn --sync --force-expire -C /opt/MailScanner/etc/spam.assassin.prefs.conf' otherwise when SpamAssassin rebuilds the bayes databese it will not be able to read it.

If everything is looking dandy, continue, if not, troubleshoot and then continue.

Finishing up this part we need to add cron jobs that will clean/update/run Mailscanner, you probably saw the message about this after the MailScanner install script finished. The reason we do it now is because we don't want MailScanner starting while we finish the SpamAssassin configuration.

crontab -e


Add these lines:

37      5 * * * /opt/MailScanner/bin/update_phishing_sites
58     23 * * * /opt/MailScanner/bin/clean.quarantine
42      * * * * /opt/MailScanner/bin/update_virus_scanners
3,23,43 * * * * /opt/MailScanner/bin/check_mailscanner

We need to add a line to rc.local so that MailScanner starts on a reboot:

vi /etc/rc.local


Before the "Exit 0" line add:

/opt/MailScanner/bin/check_mailscanner

Might as well link the "check_mailscanner" file into the bin directory. This way you can run it whenever you need to restart MailScanner:

cd /usr/bin
ln -s /opt/MailScanner/bin/check_mailscanner check_mailscanner


Just to give you a clean log to look at and reboot:

rm /var/log/mail.log
reboot


If you see some errors on reboot when starting MailScanner about the Perl Sys/Hostname/Long.pm you need to install it like so:

perl -MCPAN -e shell


If it asks you to configure it now say no, unless you know what you're doing. Now run the following to install the module:

install Sys::Hostname::Long

When that's done type "quit" to go back to the console. Reboot or start MailScanner with the "check_mailscanner" script.

At this point you should have a fully functional spamfilter. Take a look at tail -f /var/log/mail.log, it should be pretty much clear of errors.

This README should come in handy for future use. Copy it to your spamfilter for a quick reference.

POSTFIX, UBUNTU, MAILSCANNER README
**************************************************
ADD DOMAIN
**************************************************
- Edit '/etc/postfix/relay_recipients', 'relay_domains' and 'transport'.
- Run 'postmap /etc/postfix/relay_recipients'. Same for 'relay_domains' and 'transport' after edit to add domains into db file.
- 'postfix reload' for postfix to read new db files.
- To add users to domains edit the 'relay_recipients' and 'postmap' it.
**************************************************
CONTROLLING BLIST WLIST
**************************************************
- Edit /etc/postfix/sender_access
- Run 'postmap /etc/postfix/sender_access'
- Run 'postfix reload'
**************************************************
OTHER CONFIG FILES
**************************************************
- To edit MailScanner Settings "/opt/MailScanner/etc/MailScanner.conf"
- To edit spamassassin Settings "/opt/MailScanner/etc/spam.assassin.prefs.conf"
- To edit clamav Settings "/etc/clamav/clamd.conf"
**************************************************
MISC
**************************************************
- Run 'LINUX2' if postfix errors appear about files not matching in the jailroot.
- Run 'postfix check' to see if postfix is synched with jailroot.
- Run newaliases to refresh the /etc/postfix/aliases database if any changes are made on that file..
- 'mailq' and 'qshape' to check queue.
- 'check_mailscanner' to restart MailScanner.
Share this page:

18 Comment(s)

Add comment

Comments

From: Anonymous at: 2006-07-27 20:16:09

Just working my way through this great looking howto and have run into the following issues on page 1. 

1.) Your sources.list references breezy repeatedly despite this being a dapper howto.  I assumed I could just uncomment my dapper repos and install away.

 2.) When running the big apt-get install the following packages could not be found: unarj, unrar, and lha.  I just skipped them so hopefully it does not matter too much.

3.) There is no pcmcia under init.d in my install but there is a pcmciautils.  Should I be shutting down and removing that instead?

4.) There is no inetd in /etc/init.d on my system for me to restart.

Everything seems to be working ok so far though so I am forging ahead with the install.

From: Anonymous at: 2006-08-14 22:06:43

Ok thanks. Any help is appreciated, when I have a chance I will update.

From: Anonymous at: 2006-08-18 15:06:35

When trying to install the long list of software packages in section 1.5, if there's a problem with one package, none will install.  I found that it was easier to install 4 or 5 packages at a time.

From: fdalmoro at: 2006-08-25 20:36:41

Good point. I need to do some more testing because I know some of the packages don't install with the repositories. That's why I made sure to put 'BETA' in the title :) ...

From: fdalmoro at: 2006-08-25 20:44:00

For those that are trying the install... I will be updating the documentation next week. I finally got all of the kinks worked out of my system and it's working fine. I have seen many posts regarding MailScanner+Postfix comments that say it is not recommended. I have not had any problems but like some posts say, updating either MailScanner or Postfix could be risky affair so fair warning. I'm willing to take the gamble because Postfix + MailScanner are the best in their respective fields I think (especially once MailScanner-MRTG is working). The graphs make it all worth it.

 

In any case if anyone runs into any snags just let me know through here or the forums and I will try to help out the best I can. Speaking of forums I'll have to hit those today just in case. Have not looked at them yet.

From: jtkooch at: 2006-09-07 15:04:55

Excellent guide for the most part but there are some things that have me confused. You mention this will use mailscanner instead of Amavis, but page 4 references the amavis user accounts.

Also, there doesn't seem to be anypoint where either of those programs actually get installed.

Am I missing it?

From: fdalmoro at: 2006-09-18 16:27:59

Been busy, have not finished this howto. I should have posted it when it was finished sorry.

From: till at: 2006-09-13 08:11:19

The title mentions that this howto installs ClamAV, in which step is it installed or is this part missing yet?

From: fdalmoro at: 2006-09-26 18:22:15

Page 4 has it.

From: at: 2006-11-06 11:33:30

I've set this up on dapper and now on eft. I've found on both that unrar, and lha are no longer on the reps for download. As alternatives, I have used unp & unrar-free which seem to work fine.

From: Anonymous at: 2011-06-21 14:20:25

I just tried to install on Ubuntu 11.04 and the installation fails on the line:

apt-get install libc6-dev dpkg-dev db4.3-util libdb4.3-dev vim lynx bzip2 unzip perl-doc libwww-perl ntp-simple

Result:

The following packages have unmet dependencies:
 libc6-dev : Depends: libc6 (= 2.3.6-0ubuntu20.6) but 2.13-0ubuntu13 is to be installed
E: Broken packages

From: Anonymous at: 2011-09-04 04:39:52

same here. : (

 e: unable to locate package link.

From: Anonymous at: 2006-08-18 15:02:38

If you're trying to use the "getadsmtp.pl" script to grab valid email addresses from exchange, as mentioned in section 2.2.11, you may have to do something extra.  It told me "permission denied" when I tried to execute the script.  The trick was to run "chmod +x getadsmtp.pl" before trying to execute the script.  More experienced users might scoff, but being new to Linux, it was a real head-scratcher for me.  Hope it helps someone!

From: at: 2008-03-30 10:39:46

the error traced back to problem in /etc/postfix/master.cf

line for local loopback was like this

"smtp      unix  -       -       n       -       -       smtp"

when cahnged to

"smtp      unix  -       -       y       -       -       smtp"

problem resolved.

 

From: Anonymous at: 2006-08-30 06:24:13

shouldn't

postfix start

read

/etc/init.d/postfix start ?

From: fdalmoro at: 2006-09-26 18:26:06

just 'postfix start' works fine too without having to put in the /etc/init.d/postfix in it.

From: at: 2008-03-16 13:15:58

when regisrering with razor you can recieve a

Error 202 while performing register, aborting.

Turns-out you need to run the razor-admin with a -discover first if you come across this error. In general it’s some sort of network error.

razor-admin -discover
razor-admin -create
razor-admin -register

thanks to
http://devnulled.com/content/2005/03/razor-error-202-while-performing-register-aborting/

for the solution

From: Damien at: 2009-11-24 11:22:45

Thanks for this very usefull HowTo.

I used it, technicaly, it works (the linux box relay the messages to the exchange server), but every message received by the exchange server is recognized as SPAM :(

Does anybody get this kind of troubles?

Thx,

Damien