The Perfect SpamSnake - Ubuntu Jeos 9.10 - Page 3

7. MailWatch Installation Instructions

This setup assumes you are using Apache v2.x and not Apache v1.x.

Before Starting, make sure that MailScanner is working before you continue with the MailWatch install!

*Notes for Ubuntu:

You must have a working MailScanner set-up and running copies of MySQL, Apache, and PHP. You must also have the Perl DBD-MySQL package installed for the Perl portions of MailScanner to utilize the MySQL database. The default php.ini set should have the following set correctly, you may want to check this:

short_open_tag = On 
safe_mode = Off 
register_globals = Off 
magic_quotes_gpc = On 
magic_quotes_runtime = Off 
session.auto_start = 0

These will be commented out you must remove the "#" to activate them:

extension=mysql.so 
extension=gd.so

Installation

Download the latest MailWatch release:

cd /usr/src
wget http://downloads.sourceforge.net/mailwatch/mailwatch-1.0.4.tar.gz?modtime=1178902008&big_mirror=0
tar xzvf mailwatch-1.0.4.tar.gz
cd mailwatch-1.0.4

Create the database:

mysql -p < create.sql

*Note:

You will need to modify the above as necessary for your system if you have a root password for your MySQL database (recommended!) - Debian will ask for one.

Create a MySQL user and password & set up MailScanner for SQL logging:

mysql -p
mysql> GRANT ALL ON mailscanner.* TO username@localhost IDENTIFIED BY 'password';

Remember the password! You need the single quotes ' to surround your password.

Edit MailWatch.pm and change the $db_user and $db_pass values accordingly and move MailWatch.pm:

cp MailWatch.pm /opt/MailScanner/lib/MailScanner/CustomFunctions/

Create a MailWatch Web User:

mysql mailscanner -u username -p

Enter password: password

mysql> INSERT INTO users VALUES ('admin',md5('password'),'mailscanner','A','0','0','0','0','0');

Install & Configure MailWatch

From within the unpacked mailwatch directory move the directory called 'mailscanner' to /var/www/.

mv mailscanner/ /var/www/
cd /var/www/mailscanner

Make a temp directory:

mkdir temp
chmod g+w temp

Check the permissions of /var/www/mailscanner/images and /var/www/images/cache - they should be ug+rwx and owned by root and in the same group as the web server user.

chmod ug+rwx images
chmod ug+rwx images/cache
chown –R root:www-data *

Create conf.php by copying conf.php.example and edit the values to suit, you will need to set DB_USER and DB_PASS to the MySQL user and password that you created earlier.

define(DB_USER, 'username');
define(DB_PASS, 'password');
define(MAILWATCH_HOME, '/var/www/mailscanner/');
define(MS_CONFIG_DIR, '/opt/MailScanner/etc/');
define(MS_LIB_DIR, '/opt/MailScanner/lib/MailScanner/'); 
define(QUARANTINE_USE_FLAG, true);
define(QUARANTINE_FROM_ADDR, '[email protected]');
define(QUARANTINE_REPORT_FROM_NAME, 'SpamSnake');
define(QUARANTINE_USE_SENDMAIL, true);

Set-up MailScanner

Edit /opt/MailScanner/etc/MailScanner.conf and make sure the following are set:

Quarantine User = root
Quarantine Group = www-data 
Quarantine Permissions = 0660
Quarantine Whole Message = yes 
Always Looked Up Last = &MailWatchLogging

*Important: Spam Actions, High Scoring Spam Actions and No Spam Actions should also have 'store' as one of the keywords if you want to quarantine those items for bayes learning or viewing from within MailWatch.

Integrate SQL Blacklist/Whitelist (optional)

If you would like to manage the MailScanner whitelist and blacklist from within the MailWatch web interface perform the following steps.

1. Edit the MySQL connection values within the CreateList subroutine of SQLBlackWhiteList.pm to match the values you entered previous into MailWatch.pm. Both files should contain the same values. (Look for the following lines in SQLBlackWhiteList.pm and enter your own data.)

my($db_user) = 'username';
my($db_pass) = 'password';

2. Copy SQLBlackWhiteList.pm to /opt/MailScanner/lib/MailScanner/CustomFunctions/.

3. Edit MailScanner.conf and set:

Is Definitely Not Spam = &SQLWhitelist 
Is Definitely Spam = &SQLBlacklist

Fix to allow MailWatch to work with Postfix Inbound/Outbound Queue

Download the patch from http://www.gbnetwork.co.uk/mailscanner/postfixmail.tar.gz:

cd /usr/src
wget http://www.gbnetwork.co.uk/mailscanner/files/postfixmail.tar.gz
tar xvfz postfixmail.tar.gz
cd postfixmail
cp postfix* /var/www/mailscanner/
patch /var/www/mailscanner/functions.php functions.php.diff

SpamAssassin

First we need to disable the default SpamAssassin configuration file:

mv /etc/spamassassin/local.cf /etc/spamassassin/local.cf.disabled

Now let's backup the SpamAssassin configuration file in MailScanner then edit:

cp /opt/MailScanner/etc/spam.assassin.prefs.conf /opt/MailScanner/etc/spam.assassin.prefs.conf.back

Add pyzor and razor paths to the top of /opt/MailScanner/etc/spam.assassin.prefs.conf:

pyzor_options --homedir /var/lib/MailScanner/
razor_config /var/lib/MailScanner/.razor/razor-agent.conf
dcc_path /usr/bin/dccproc

Look for these lines and change them accordingly:

bayes_ignore_header X-YOURDOMAIN-COM-MailScanner
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamCheck
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamScore
bayes_ignore_header X-YOURDOMAIN-COM-MailScanner-Information

"YOURDOMAIN-COM" should be replaced with whatever you used for "%org-name%" in the MailScanner.conf file. Leave the "X-" in place. This is the same orgname used in the MailScanner.conf above.

Edit the SpamAssassin /etc/spamassassin/v310.pre to enable DCC:

loadplugin Mail::SpamAssassin::Plugin::DCC

SpamAssassin SQL Bayes

Pre-requisities:
You'll need the perl-DBI and perl-DBD-MySQL modules installed.

Assumptions and Variables:

SpamAssassin Bayes Database Name: sa_bayes
SpamAssassin Bayes Database UserName: sa_user
SpamAssassin Bayes Database Password: sa_password

Create the MySQL database: First of all, create a database on the server where you intend on storing the bayesian information.

mysql -u root -p
mysql> create database sa_bayes;
mysql> GRANT ALL ON sa_bayes.* TO sa_user@localhost IDENTIFIED BY 'sa_password';
mysql> flush privileges;

Locate the bayes_mysql.sql file:

find / -name bayes_mysql.sql
mysql -u sa_user -p sa_bayes < /path/to/bayes_mysql.sql

Make some changes to your spam.assassin.prefs.conf:

bayes_store_module Mail::SpamAssassin::BayesStore::SQL
bayes_sql_dsn DBI:mysql:sa_bayes:localhost
bayes_sql_username sa_user
bayes_sql_password sa_password
bayes_sql_override_username root

You may have to create the following to prevent an error in a lint test:

mkdir /var/www/.spamassassin

Also add this to your crontab:

30 01 * * * /usr/bin/sa-learn --force-expire --sync -p /opt/MailScanner/etc/spam.assassin.prefs.conf

Set permissions to bring it all together:

chown -R postfix:www-data /var/spool/postfix/hold
chmod -R ug+rwx /var/spool/postfix/hold

Finally make sure you restart MailScanner.

/etc/init.d/mailscanner restart

Test out the setup:

spamassassin -x -D -p /opt/MailScanner/etc/spam.assassin.prefs.conf --lint

Check for lines like:

debug: bayes: Database connection established
debug: bayes: found bayes db version 3
debug: bayes: Using userid: 2

You should see lines come up with DCC, Pyzor and Razor that say loading plugin and hopefully no errors.

Finishing up this part we need to add cron jobs that will clean/update, you probably saw the message about this after the MailScanner install script finished.

First edit conf.php and set 'QUARANTINE_DAYS_TO_KEEP' and change the following line in /usr/src/mailwatch-1.0.4/tools/db_clean.php and quarantine_maint.php to:

#!/usr/bin/php –q
require('/var/www/mailscanner/functions.php');

Install quarantine clean up script:

cp /usr/src/mailwatch-1.0.4/tools/quarantine_maint.php /usr/bin/quarantine_maint.php
cp /usr/src/mailwatch-1.0.4/tools/db_clean.php /usr/bin/db_clean.php
chmod +x /usr/bin/quarantine_maint.php
chmod +x /usr/bin/db_clean.php

Run

crontab -e

and add the following:

15 10 * * 2 /usr/bin/quarantine_maint.php --clean &> /dev/null
58 23 * * * /usr/bin/db_clean.php --clean &> /dev/null

Reboot the system:

reboot

Check your mail.log:

tail –f /var/log/mail.log

and you should see the following:

Jun 13 12:18:23 hoshi MailScanner[26388]: MailScanner E-Mail Virus Scanner version 4.20-3 starting...
Jun 13 12:18:24 hoshi MailScanner[26388]: Config: calling custom init function MailWatchLogging
Jun 13 12:18:24 hoshi MailScanner[26388]: Initialising database connection
Jun 13 12:18:24 hoshi MailScanner[26388]: Finished initialising database connection

Congratulations - you now have MailScanner logging to MySQL.

Test the MailWatch interface

Point your browser to http://<hostname>/mailscanner/ - you should be prompted for a username and password - enter the details of the MailWatch web user that you created earlier, and you should see a list of the last 50 messages processed by MailScanner.

Update the SpamAssassin Rules table

MailWatch keeps a list of all the SpamAssassin rules and descriptions which are displayed on the 'Message Detail' page - to show the descriptions, you need to run the updater every time you add new rules or upgrade SpamAssassin. Click on the 'Tools/Links' menu and select 'Update SpamAssassin Rule Descriptions' and click 'Run Now'.

Update the GeoIP database

Change this line in /var/www/mailscanner/geoip_update.php to look like:

dbquery("LOAD DATA LOCAL INFILE

*Note: Make sure you have allow_url_fopen = On in your php.ini set.

Click on the 'Tools/Links' menu and select 'Update GeoIP database' and click 'Run Now'.

Fix to allow wildcards in Whitelist/Blacklist

Add the following to the bottom of the return 1 section in your /opt/MailScanner/lib/MailScanner/CustomFunctions/SQLBlackWhiteList.pm:

return 1 if $BlackWhite->{$to}{'*@'.$fromdomain};
return 1 if $BlackWhite->{$to}{'*@*.'.$fromdomain};
return 1 if $BlackWhite->{$todomain}{'*@'.$fromdomain};
return 1 if $BlackWhite->{$todomain}{'*@*.'.$fromdomain};
return 1 if $BlackWhite->{'default'}{'*@'.$fromdomain};
return 1 if $BlackWhite->{'default'}{'*@*.'.$fromdomain}; 

Fix for Message Operations Not Finding Messages

Change the following in /var/www/mailscanner/do_message_ops.php file:

$id = $Regs[1];

to

$id = str_replace("_", ".",$Regs[1]);

Releasing Spam Messages

To allow MailWatch to release Spam messages without them being processed again, add 127.0.0.1 as a whitelist item in MailWatch/List interface. Make sure to restart MailScanner after configuring these options. Below is what my entry looks like.

127.0.0.1 Default Delete

Fix to Allow Multiple Release of Messages in Message Operations

Edit /var/www/mailscanner/do_message_ops.php and make the following changes:

case 'F':
$type='forget';
break;
case 'R':
$type='release';
break;
default:
continue;
break;

Then, find the following section and change it to look like this:

if ($type == 'release'){
if($quarantined = quarantine_list_items($id,RPC_ONLY)) {
$to = $quarantined[0]['to'];
}
echo "<tr><td><a href=\"detail.php?id=$id\">$id</a></td><td>$type</td><td>" . quarantine_release($quarantined, $itemnum, $to, RPC_ONLY) . "</td></tr>\n";
} else {
echo "<tr><td><a href=\"detail.php?id=$id\">$id</a></td><td>$type</td><td>" . quarantine_learn($items, $itemnum, $type, RPC_ONLY) . "</td></tr>\n";
}
}
echo "</TABLE>\n";
}
}
}
}
echo "  </TD>\n";

Next we edit the /var/www/mailscanner/functions.php file and change:

$fieldname[$f] = "Ops<br>S&nbsp;&nbsp;H&nbsp;&nbsp;F";

to:

$fieldname[$f] = "Ops<br>S&nbsp;&nbsp;H&nbsp;&nbsp;F&nbsp;&nbsp;R";

Next change:

array_unshift($row, "<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"S\">&nbsp;<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"H\">&nbsp;<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"F\">");

to:

array_unshift($row, "<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"S\">&nbsp;<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"H\">&nbsp;<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"F\">&nbsp;<INPUT NAME=\"OPT-REPLACEME\" TYPE=RADIO VALUE=\"R\"> ");

Next find the block with the javascript function to handle radio buttons. Add a third value like so:

echo "function SetRadios(p) {\n";
echo " var val;\n";
echo " if (p == 'S') {\n";
echo "  val = 0;\n";
echo " } else if (p == 'H') {\n";
echo "  val = 1;\n";
echo " } else if (p == 'F') {\n";
echo "  val = 2;\n";
echo " } else if (p == 'R') {\n";
echo "  val = 3;\n";
echo " } else if (p == 'C') {\n";
echo "  ClearRadios();\n";

Now, add the text for the radios:

echo "&nbsp; <a href=\"javascript:SetRadios('S')\">S</a>";
echo "&nbsp; <a href=\"javascript:SetRadios('H')\">H</a>";
echo "&nbsp; <a href=\"javascript:SetRadios('F')\">F</a>";
echo "&nbsp; <a href=\"javascript:SetRadios('R')\">R</a>";

Finally, change:

echo "<P><b>S</b> = Spam &nbsp; <b>H</b> = Ham &nbsp; <b>F</b> = Forget\n";

to:

echo "<P><b>S</b> = Spam &nbsp; <b>H</b> = Ham &nbsp; <b>F</b> = Forget &nbsp; <b>R</b> = Release\n";

Patch to fix autocommit error when stopping MailScanner

Edit line 80 of MailWatch.pm to look like this:

$dbh = DBI->connect("DBI:mysql:database=$db_name;host=$db_host", $db_user, $db_pass, {PrintError => 0, AutoCommit => 0});

*Note: This turns off autocommit when it connects so the commit line when it closes does not throw an error.

Clamd Error Fix

If you see the following error in mail.log, you have to add user clamav to the www-data group.

Dec 3 08:29:51 server1 MailScanner[3728]: Clamd::ERROR:: UNKNOWN CLAMD RETURN ./lstat() failed: Permission denied. ERROR :: /var/spool/MailScanner/incoming/3728

usermod -a -G www-data clamav