The Perfect SpamSnake - Ubuntu 8.04 LTS - Page 5

9 MailWatch Installation Instructions

This setup assumes you are using Apache v2.x and not Apache v1.x.

9.1 Before Starting

Make sure that MailScanner is working before you continue with the MailWatch install!

Notes for Ubuntu:

You must have a working MailScanner set-up and running copies of MySQL, Apache, and PHP. You must also have the Perl DBD-MySQL package installed for the Perl portions of MailScanner to utilize the MySQL database.
The default php.ini set should have the following set correctly, you may want to check this:

short_open_tag = On
safe_mode = Off
register_globals = Off
magic_quotes_gpc = On
magic_quotes_runtime = Off
session.auto_start = 0

These will be commented out you must remove the "#" to activate them:

extension=mysql.so
extension=gd.so

9.2 Installation

All commands below should be run as root.

9.3 Download the latest MailWatch release

wget http://downloads.sourceforge.net/mailwatch/mailwatch-1.0.4.tar.gz?modtime=1178902008&big_mirror=0
tar xzvf mailwatch-1.0.4.tar.gz
cd mailwatch-1.0.4

9.4 Create the database

mysql -p < create.sql

NOTE: you will need to modify the above as necessary for your system if you have a root password for your MySQL database (recommended!) - Debian will ask for one.

9.5 Create a MySQL user and password & Set-up MailScanner for SQL logging

mysql -p
mysql> GRANT ALL ON mailscanner.* TO mailwatch@localhost IDENTIFIED BY 'password';

Remember the password! You need the single quotes ' to surround your password.

9.6 Edit and copy MailWatch.pm

Edit MailWatch.pm and change the $db_user and $db_pass values accordingly and move MailWatch.pm.

mv MailWatch.pm /etc/MailScanner/CustomFunctions/

9.7 Create a MailWatch Web User

mysql mailscanner -u mailwatch -p

Enter password: ******

mysql> INSERT INTO users VALUES ('username',md5('password'),'mailscanner','A','0','0','0','0','0');

9.8 Install & Configure MailWatch

From within the unpacked mailwatch directory move the directory called 'mailscanner' to the web server's root.

mv mailscanner/ /var/www/
cd /var/www/mailscanner

Make a temp directory:

mkdir temp
chgrp www-data temp
chmod g+w temp

Check the permissions of /var/www/mailscanner/images and /var/www/images/cache - they should be ug+rwx and owned by root and in the same group as the web server user.

chown root:www-data images
chmod ug+rwx images
chown root:www-data images/cache
chmod ug+rwx images/cache

Create conf.php by copying conf.php.example and edit the values to suit, you will need to set DB_USER and DB_PASS to the MySQL user and password that you created earlier.

Change these values as shown below:

define(DB_USER, 'mailwatch');
define(DB_PASS, 'password');
define(MAILWATCH_HOME, '/var/www/mailscanner');
define(MS_LIB_DIR, '/usr/share/MailScanner/');
define(QUARANTINE_USE_FLAG, true);

9.9 Set-up MailScanner

Next edit /etc/MailScanner/MailScanner.conf.

vi /etc/MailScanner/MailScanner.conf

You need to make sure that the following options are set:

Quarantine User = root
Quarantine Group = www-data
Quarantine Permissions = 0660
Quarantine Whole Message = yes
Always Looked Up Last = &MailWatchLogging

And check these as well:

Quarantine Whole Message As Queue Files = no
Detailed Spam Report = yes
Include Scores In SpamAssassin Report = yes

Spam Actions, High Scoring Spam Actions and No Spam Actions should also have 'store' as one of the keywords if you want to quarantine those items for bayes learning or viewing from within MailWatch.

9.10 Integrate SQL Blacklist/Whitelist (optional)

If you would like to manage the MailScanner whitelist and blacklist from within the MailWatch web interface perform the following steps.

1. Edit the MySQL connection values within the CreateList subroutine of SQLBlackWhiteList.pm to match the values you entered previous into MailWatch.pm. Both files should contain the same values. (Look for the following lines in SQLBlackWhiteList.pm and enter your own data.)

my($db_user) = 'mailwatch';
my($db_pass) = 'password';

2. Copy SQLBlackWhiteList.pm to /etc/MailScanner/CustomFunctions/.

3. Edit MailScanner.conf and set:

Is Definitely Not Spam = &SQLWhitelist
Is Definitely Spam = &SQLBlacklist

9.11 Fix to allow MailWatch to work with Postfix Inbound/Outbound Queue

Download the patch from http://www.gbnetwork.co.uk/mailscanner/postfixmail.tar.gz

cd /usr/src
wget http://www.gbnetwork.co.uk/mailscanner/files/postfixmail.tar.gz
tar xvfz postfixmail.tar.gz
cd postfixmail
cp postfix* /var/www/mailscanner
patch /var/www/mailscanner/functions.php functions.php.diff

9.12 SpamAssassin

First we need to disable the default SpamAssassin configuration file:

mv /etc/spamassassin/local.cf /etc/spamassassin/local.cf.disabled

Now let's backup the SpamAssassin configuration file in MailScanner then edit:

cp /etc/MailScanner/spam.assassin.prefs.conf /etc/MailScanner/spam.assassin.prefs.conf.back

Add pyzor and razor paths:

vi /etc/MailScanner/spam.assassin.prefs.conf

Add these lines to the top of spam.assassin.prefs.conf:

pyzor_options --homedir /var/lib/MailScanner/
razor_config /var/lib/MailScanner/.razor/razor-agent.conf

Comment out the following:
#bayes_auto_expire 0

9.13 Move the Bayesian Databases and set-up permissions (skip this if you don't use bayes)

Edit /etc/MailScanner/spam.assassin.prefs.conf and set:

vi /etc/MailScanner/spam.assassin.prefs.conf

bayes_path /etc/MailScanner/bayes/bayes
bayes_file_mode 0660

Look for these lines and change them accordingly:

bayes_ignore_header X-YOURDOMAIN-COM-MailScanner
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamCheck
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamScore
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-Information

"YOURDOMAIN-COM" should be replaced with whatever you used for "%org-name%" in the MailScanner.conf file. Leave the "X-" in place.
This is the same orgname used in the MailScanner.conf above.

Create the 'new' bayes directory, make the directory owned by the same group as the web server user and make the directory setgid:

mkdir /etc/MailScanner/bayes
chown -R root:www-data /etc/MailScanner/bayes
chmod -R ug+rw /etc/MailScanner/bayes
chmod g+s /etc/MailScanner/bayes

Copy the existing bayes databases and set the permissions (Note: This part can be skipped if bayes was not previously enabled because the bayes directory would not have been created):

cp /var/lib/MailScanner/bayes_* /etc/MailScanner/bayes
chown root:www-data /etc/MailScanner/bayes/bayes_*
chmod g+rw /etc/MailScanner/bayes/bayes_*

Make sure that "bayes_auto_expire 0" is not commented out in spam.assassin.prefs.conf:

bayes_auto_expire 0

Edit the SpamAssassin v310.pre to enable Razor and DCC:

vi /etc/spamassassin/v310.pre

Uncomment the following lines:

loadplugin Mail::SpamAssassin::Plugin::DCC
loadplugin Mail::SpamAssassin::Plugin::Razor2

If you want then you can test SpamAssassin to make sure that it is using the new databases correctly:

spamassassin -D -p /etc/MailScanner/spam.assassin.prefs.conf --lint

and you should see something like:

debug: using "/etc/MailScanner/spam.assassin.prefs.conf" for user prefs file
debug: bayes: 28821 tie-ing to DB file R/O /etc/MailScanner/bayes/bayes_toks
debug: bayes: 28821 tie-ing to DB file R/O /etc/MailScanner/bayes/bayes_seen
debug: bayes: found bayes db version 2
debug: Score set 3 chosen.

9.13.1 SpamAssassin Bayes Database to SQL Conversion

Pre-requisities

a. You'll need the perl-DBI and perl-DBD-MySQL modules installed.

Assumptions and Variables:

SpamAssassin Bayes Database Name: sa_bayes
SpamAssassin Bayes Database UserName: sa_user
SpamAssassin Bayes Database Password: sa_password

Create the MySQL database:

First of all, create a database on the server where you intend on storing the bayesian information.

mysql -u root -p

mysql> create database sa_bayes;
mysql> GRANT ALL ON sa_bayes.* TO sa_user@localhost IDENTIFIED BY 'sa_password';
mysql> flush privileges;

Locate the bayes_mysql.sql file:

find / -name bayes_mysql.sql
mysql -u sa_user -p sa_bayes < /path/to/bayes_mysql.sql

Backup your current bayes database:

sa-learn -p /etc/MailScanner/spam.assassin.prefs.conf --backup > sa_bayes_backup.txt

Warning: The next command can completely wipe out your bayes database!

sa-learn -p /path/to/spam.assassin.prefs.conf --clear #(entirely optional, incase you want to rollback)

Make some changes to your spam.assassin.prefs.conf:

bayes_store_module Mail::SpamAssassin::BayesStore::SQL
bayes_sql_dsn DBI:mysql:sa_bayes:localhost
bayes_sql_username sa_user
bayes_sql_password sa_password
bayes_sql_override_username root

and comment out the following lines:

#bayes_path /etc/MailScanner/bayes/bayes
#bayes_file_mode 0660

Populate the Bayes SQL database.

Now for recovering the bayes_dbm to bayes_sql.

sa-learn -p /etc/MailScanner/spam.assassin.prefs.conf --restore sa_bayes_backup.txt

This process may take some time depending on the size of your bayes database.

Also add this to your crontab:

crontab -e

30 01 * * * /path/to/sa-learn --force-expire --sync -p /etc/MailScanner/spam.assassin.prefs.conf

9.14 Bring it all Together

Now that we have everything in there, set the correct permissions:

chown -R postfix:www-data /var/spool/MailScanner
chown -R postfix:www-data /var/lib/MailScanner
chown -R postfix:www-data /var/run/MailScanner
chown -R postfix:www-data /var/lock/subsys/MailScanner
chown -R postfix:www-data /var/spool/postfix/hold
chmod -R ug+rwx /var/spool/postfix/hold

chmod -R u+rwx,g+rx /var/spool/MailScanner/quarantine

Finally make sure you restart MailScanner.

/etc/init.d/mailscanner restart

Test out the setup:

spamassassin -x -D -p /etc/MailScanner/spam.assassin.prefs.conf --lint

Check for lines like:

debug: bayes: Database connection established
debug: bayes: found bayes db version 3
debug: bayes: Using userid: 2

and some more like

debug: bayes: tok_get_all: Token Count: 20
debug: bayes token 'somewhat' ? 0.978
debug: bayes: score = 0.845189622547555

You should see lines come up with DCC, Pyzor and Razor that say loading plugin and hopefully no errors.

Finishing up this part we need to add cron jobs that will clean/update, you probably saw the message about this after the MailScanner install script finished.

First edit conf.php and set 'QUARANTINE_DAYS_TO_KEEP' in conf.php and change the following line in db_clean.

#!/usr/bin/php -qn

#!/usr/bin/php -q

Install quarantine clean up script:

cp /usr/src/mailwatch-1.0.4/tools/quarantine_maint.php /usr/bin/quarantine_maint.php
cp /usr/src/mailwatch-1.0.4/tools/db_clean.php /usr/bin/db_clean.php
chmod +x /usr/bin/quarantine_maint.php
chmod +x /usr/bin/db_clean.php

Run

crontab -e

and add the following:

15 10 * * 2 /usr/bin/quarantine_maint.php --clean &> /dev/null
58 23 * * * /usr/bin/db_clean.php &> /dev/null

Disable the mailscanner installed cron script /etc/cron.daily/clean.quarantine (Note: Do this only if the clean.quarantine script exists).

$disabled = 1;

9.15 Reboot

reboot

Check your mail.log again:

tail -f /var/log/mail.log

At this point you should have a functional spamfilter and should see something like:

Jun 13 12:18:23 hoshi MailScanner[26388]: MailScanner E-Mail Virus Scanner version 4.20-3 starting...
Jun 13 12:18:24 hoshi MailScanner[26388]: Config: calling custom init function MailWatchLogging
Jun 13 12:18:24 hoshi MailScanner[26388]: Initialising database connection
Jun 13 12:18:24 hoshi MailScanner[26388]: Finished initialising database connection

Congratulations - you now have MailScanner logging to MySQL.

9.16 Test the MailWatch interface

Point your browser to http://<hostname>/mailscanner/ - you should be prompted for a username and password - enter the details of the MailWatch web user that you created earlier, and you should see a list of the last 50 messages processed by MailScanner.

If you're not able to see the mails, then you may have to set the following persmissions:

chgrp -R www-data /var/spool/MailScanner

You may have to create the following to prevent an error in a lint test:

mkdir /var/www/.spamassassin

9.17 Fix for Ubuntu 8.04 (kept removing directories upon reboot)

Edit /etc/rc.local and add the following before the exit line:

mkdir /var/run/MailScanner
mkdir /var/lock/subsys
mkdir /var/lock/subsys/MailScanner
chown -R postfix:www-data /var/run/MailScanner
chown -R postfix:www-data /var/lock/subsys/MailScanner
/etc/init.d/postfix restart
/etc/init.d/mailscanner restart

9.18 Update the SpamAssassin Rules table

MailWatch keeps a list of all the SpamAssassin rules and descriptions which are displayed on the 'Message Detail' page - to show the descriptions, you need to run the updater every time you add new rules or upgrade SpamAssassin. Click on the 'Tools/Links' menu and select 'Update SpamAssassin Rule Descriptions' and click 'Run Now'.

9.19 Update the GeoIP database

Change /var/www/mailscanner/geoip_update.php:

vi /var/www/mailscanner/geoip_update.php

dbquery("LOAD DATA INFILE

dbquery("LOAD DATA LOCAL INFILE

Make sure you have allow_url_fopen = On in your php.ini set.

Click on the 'Tools/Links' menu and select 'Update GeoIP database' and click 'Run Now'.

9.20 Setup the Mail Queue watcher (optional)

You can get MailWatch to watch and display your sendmail or exim queue directories - all you need to do is copy mailq.php (from the root of the mailwatch tarball - not from the mailscanner directory - they are different!) to /usr/local/bin and set-up a cron-job to run it.

Edit mailq.php first to change the require line to point to the location of functions.php, then:

cp mailq.php /usr/local/bin
crontab -e

0-59 * * * * /usr/local/bin/mailq.php

Note: mailq.php re-creates all entries on each run, so for busy sites you will probably want to change this to run every 5 minutes or greater.

9.21 Setup the Sendmail Relay Log watcher (optional)

You can get MailWatch to watch your sendmail logs and store all message relay information which is then displayed on the 'Message Detail' page which helps debugging and makes it easy for a Helpdesk to actually see where a message was delivered to by the MTA and what the response back was (e.g. the remote queue id etc.).

cp tools/sendmail_relay.php /usr/local/bin
nohup /usr/local/bin/sendmail_relay.php 2>&1 > /dev/null &

9.22 Fix to allow wildcards in Whitelist/Blacklist

Add the following to the bottom of the return 1 section in your SQLBlackWhiteList.pm:

return 1 if $BlackWhite->{$to}{'*@'.$fromdomain};
return 1 if $BlackWhite->{$to}{'*@*.'.$fromdomain};
return 1 if $BlackWhite->{$todomain}{'*@'.$fromdomain};
return 1 if $BlackWhite->{$todomain}{'*@*.'.$fromdomain};
return 1 if $BlackWhite->{'default'}{'*@'.$fromdomain};
return 1 if $BlackWhite->{'default'}{'*@*.'.$fromdomain};

9.23 Fix for the Reporting Function in Message Operations

Change the following in /var/www/mailscanner/do_message_ops.php file:

vi /var/www/mailscanner/do_message_ops.php

$id = $Regs[1];

$id = str_replace("_", ".",$Regs[1]);

9.24 Fix to Allow Quarantine Release of Messages

Change the following in /var/www/mailscanner/conf.php:

define(QUARANTINE_FROM_ADDR, '[email protected]');

*You need to put the full email address or this will not work.

Also make sure the following string is set to true:
define(QUARANTINE_USE_FLAG, true);

If you'd like the message to be released in it's original form and not as an attachment, set the following line to true:

define(QUARANTINE_USE_SENDMAIL, true);

9.24.1 Dangerous Content:
Open /etc/MailScanner/MailScanner.conf and change the following:

Dangerous Content Scanning = yes

Dangerous Content Scanning = %rules-dir%/content.scanning.rules

Create /etc/MailScanner/rules/content.scanning.rules and add the following:
From: 127.0.0.1 no
FromOrTo: default yes

9.24.2 Filename and Filetype Release:
Modify /etc/MailScanner/MailScanner.conf and set the following:

Filename Rules = %etc-dir%/filename.rules
Filetype Rules = %etc-dir%/filetype.rules

Then create the following files as shown in /etc/MailScanner:

/etc/MailScanner/filename.rules:

From: 127.0.0.1 /etc/MailScanner/filename.rules.allowall.conf
FromOrTo: default /etc/MailScanner/filename.rules.conf

/etc/MailScanner/filetype.rules:

From: 127.0.0.1 /etc/MailScanner/filetype.rules.allowall.conf
FromOrTo: default /etc/MailScanner/filetype.rules.conf

/etc/MailScanner/filename.rules.allowall.conf:
allow .* - -

/etc/MailScanner/filetype.rules.allowall.conf:
allow .* - -

9.24.3 Releasing Spam Messages

To allow MailWatch to release Spam messages without them being processed again, add 127.0.0.1 as a whitelist item in MailWatch/List interface. Make sure to restart MailScanner after configuring these options. Below is what my entry looks like.

127.0.0.1

default

Delete

9.25 Fix to Allow Multiple Release of Messages in Message Operations

Edit /var/www/mailscanner/do_message_ops.php and make the following changes:

case 'F':

$type='forget';

break;

case 'R':

$type='release';

break;

default:

continue;

break;

Then, find the following section and change it to look like this:

$itemnum = array($num);

if ($type == 'release'){

if($quarantined = quarantine_list_items($id,RPC_ONLY)) {

$to = $quarantined[0]['to'];

}

echo "<tr><td><a href=\"detail.php?id=$id\">$id</a></td><td>$type</td><td>" . quarantine_release($quarantined, $itemnum, $to, RPC_ONLY) . "</td></tr>\n";

} else {

echo "<tr><td><a href=\"detail.php?id=$id\">$id</a></td><td>$type</td><td>" . quarantine_learn($items, $itemnum, $type, RPC_ONLY) . "</td></tr>\n";

}

echo "</TABLE>\n";

}

echo " </TD>\n";

Next we edit the /var/www/mailscanner/functions.php file and change:

$fieldname[$f] = "Ops<br>S  H  F";

$fieldname[$f] = "Ops<br>S  H  F  R";

Next change:

array_unshift($row, "<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"S\"> <INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"H\"> <INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"F\">");

To:

array_unshift($row, "<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"S\"> <INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"H\"> <INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"F\"> <INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"R\"> ");

Next find the block with the javascript function to handle radio buttons. Add a third value like so:

echo "function SetRadios(p) {\n";

echo " var val;\n";

echo " if (p == 'S') {\n";

echo " val = 0;\n";

echo " } else if (p == 'H') {\n";

echo " val = 1;\n";

echo " } else if (p == 'F') {\n";

echo " val = 2;\n";

echo " } else if (p == 'R') {\n";

echo " val = 3;\n";

echo " } else if (p == 'C') {\n";

echo " ClearRadios();\n";

Now, add the text for the radios:

echo "  <a href=\"javascript:SetRadios('S')\">S</a>";

echo "  <a href=\"javascript:SetRadios('H')\">H</a>";

echo "  <a href=\"javascript:SetRadios('F')\">F</a>";

echo "  <a href=\"javascript:SetRadios('R')\">R</a>";

Finally, change:

echo "<P><b>S</b> = Spam   <b>H</b> = Ham   <b>F</b> = Forget\n";

To:

echo "<P><b>S</b> = Spam   <b>H</b> = Ham   <b>F</b> = Forget   <b>R</b> = Release\n";

9.26 Fix to Allow Correct ClamAV Status

Change the following in /var/www/mailscanner/clamav_status.php file:

<?passthru(get_virus_conf('clamav')." -V | awk -f ./clamav.awk");?>

<?passthru('/usr/sbin/clamd -V | awk -f ./clamav.awk');?>

Log in or Sign up