The Perfect SpamSnake - Ubuntu Jeos 12.04 LTS Precise Pangolin

Author: Rocky
Version: 5

Postfix w/Bayesian Filtering, Postscreen, Postfix Recipient Callout (Relay Recipients via look-ahead Optional), Nginx/Uwsgi, Mysql, Dnsmasq, MailScanner (Spamassassin, ClamAV, Pyzor, Razor, DCC-Client), Baruwa, SPF Checks, FuzzyOcr, Sanesecurity Signatures, Greyfix, KAM, Scamnailer, FireHOL (Iptables Firewall), Relay Recipients Script (Optional), Webmin (Optional), Outgoing Disclaimer with alterMIME (Optional)

This tutorial shows how to set up an Ubuntu Jeos based server as a spamfilter in Gateway mode. In the end, you will have a SpamSnake Gateway which will relay clean emails to your MTA. You will also be able to view your incoming queue, train your SpamSnake and carry out a few more advanced operations via Baruwa.

I cannot offer any guarantees that this will work for you, the same way it’s working for me.

I will use the following software:
• Web Server: Nginx v1.1.19/Uwsgi v1.0.3
• Database Server: MySQL v5.5.28
• Mail Server: Postfix v2.9.3
• Caching DNS Server: Dnsmasq 2.59
• Filter: MailScanner v4.84.5-3
• Frontend: Baruwa v1.1.2-4sn

Credit goes to the guys at HowToForge and the developers of MailScanner, Baruwa, Clamav, Nginx/Uwsgi, Mysql, Postfix, Spamassassin, Razor/Pyzor/DCC and Firehol.

 

BASE INSTALL

1. Install minimum vm option
    Set hostname to server1
2. Default guided partition method
3. Setup user:
    u: administrator
    p: password
    No encryption
4. No auto-updates
5. Install OpenSSH

 

POST INSTALLATION

1. Get root Privileges

Enable the root login by running the following and giving root a password. You can then directly log in as root:

sudo passwd root

 

2. Configure The Network

Because the Ubuntu installer has configured our system to get its network settings via DHCP, we have to change that now because a server should have a static IP address. Edit /etc/network/interfaces and adjust it to your needs (in this example setup I will use the IP address 192.168.0.100):

vi /etc/network/interfaces

and make it look like the following:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.0.100
        netmask 255.255.255.0
        network 192.168.0.0
        broadcast 192.168.0.255
        gateway 192.168.0.1
        dns-nameservers 192.168.0.1

Then restart your network:

/etc/init.d/networking restart

vi /etc/hosts

and make it look like this:

127.0.0.1       localhost.localdomain   localhost
192.168.0.100   server1.example.com     server1
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Now run:

echo server1.example.com > /etc/hostname
reboot now

Afterwards, run:

hostname
hostname -f

Both should show server1.example.com now.

 

3. Change The Default Shell

/bin/sh is a symlink to /bin/dash, however we need /bin/bash, not /bin/dash. Therefore we do this:

dpkg-reconfigure dash

Install dash as /bin/sh? <-- No

Install a few packages and requirements that are needed later on:

apt-get install binutils cpp fetchmail flex gcc libarchive-zip-perl libc6-dev libcompress-raw-zlib-perl libdb4.8-dev libpcre3 libpopt-dev lynx m4 make ncftp nmap openssl perl perl-modules unzip zip zlib1g-dev autoconf automake1.9 libtool bison autotools-dev g++ build-essential telnet wget gawk -y

 

4. Caching Dnsmasq

apt-get install dnsmasq -y

vi /etc/dnsmasq.conf

and make Dnsmasq listen on localhost:

listen-address=127.0.0.1

 

5. Install Mysql

apt-get install mysql-client mysql-server libdbd-mysql-perl -y

You will be asked to provide a password for the MySQL root user - this password is valid for the user root@localhost as well as root@server1.example.com, so we don't have to specify a MySQL root password manually later on:

New password for the MySQL "root" user: <-- yourrootsqlpassword
Repeat password for the MySQL "root" user: <-- yourrootsqlpassword

 

6. Install Postfix:

apt-get install postfix postfix-mysql postfix-doc procmail -y

You will be asked two questions. Answer as follows:

General type of mail configuration: --> Internet Site
System mail name: --> server1.example.com

Stop Postfix:

postfix stop

vi /etc/postfix/master.cf

and make it look like the following:

pickup    fifo  n       -       -       60      1       pickup
         -o content_filter=
         -o receive_override_options=no_header_body_checks

Edit main.cf:

vi /usr/src/postfix.sh

with the following:

#!/bin/sh
postconf -e "alias_maps = hash:/etc/aliases"
newaliases
postconf -e "myorigin = domain.tld"
postconf -e "myhostname = server1.domain.tld"
postconf -e "mynetworks = 127.0.0.0/8, 192.168.0.0/24"
postconf -e "message_size_limit = 10485760"
postconf -e "local_transport = error:No local mail delivery"
postconf -e "mydestination = "
postconf -e "local_recipient_maps = "
postconf -e "relay_domains = mysql:/etc/postfix/mysql-relay_domains.cf"
postconf -e "relay_recipient_maps = mysql:/etc/postfix/mysql-relay_recipients.cf"
postconf -e "transport_maps = mysql:/etc/postfix/mysql-transports.cf"
postconf -e "virtual_alias_maps = hash:/etc/postfix/virtual"
postconf -e "disable_vrfy_command = yes"
postconf -e "strict_rfc821_envelopes = no"
postconf -e "smtpd_banner = $myhostname ESMTP SpamSnake"
postconf -e "smtpd_delay_reject = yes"
postconf -e "smtpd_recipient_limit = 100"
postconf -e "smtpd_helo_required = yes"
postconf -e "smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, permit"
postconf -e "smtpd_helo_restrictions = permit_sasl_authenticated, permit_mynetworks, permit"
postconf -e "smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_non_fqdn_sender, reject_unknown_sender_domain, permit"
postconf -e "smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unknown_recipient_domain, reject_unauth_destination, whitelist_policy, grey_policy, rbl_policy, spf_policy, permit"
postconf -e "smtpd_data_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_pipelining"
postconf -e "smtpd_restriction_classes = spf_policy, grey_policy, whitelist_policy"
postconf -e "spf_policy = check_policy_service unix:private/policy-spf"
postconf –e "policy-spf_time_limit = 3600s"
postconf -e "rbl_policy = reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net"
postconf -e "grey_policy = check_policy_service unix:private/greyfix"
postconf -e "whitelist_policy = check_client_access mysql:/etc/postfix/mysql-global_whitelist.cf, check_sender_access mysql:/etc/postfix/mysql-global_whitelist.cf"
postconf -e "header_checks = regexp:/etc/postfix/header_checks"
touch /etc/postfix/virtual
echo "root administrator@example.com" >> /etc/postfix/virtual && echo "abuse administrator@example.com" >> /etc/postfix/virtual && echo "postmaster administrator@example.com" >> /etc/postfix/virtual
postmap /etc/postfix/virtual
touch /etc/postfix/header_checks
echo "/^Received:/ HOLD" >> /etc/postfix/header_checks
postmap /etc/postfix/header_checks
cat > /etc/postfix/mysql-global_whitelist.cf <<EOF
#mysql-global_whitelist
user = baruwa
password =
password
dbname = baruwa
query = select concat('PERMIT') 'action' from lists where from_address='%s' AND list_type='1';
hosts = 127.0.0.1
EOF
cat > /etc/postfix/mysql-relay_domains.cf <<EOF
#mysql-relay_domains
user = baruwa
password =
password
dbname = baruwa
query = select concat(address, ' ', 'OK') 'domain' from user_addresses where user_addresses.address='%s' and user_addresses.enabled='1';
hosts = 127.0.0.1
EOF

cat > /etc/postfix/mysql-relay_recipients.cf <<EOF
#mysql-relay_recipients
user = baruwa
password = password
dbname = baruwa
query = select concat('@', address, 'OK') 'email' from user_addresses where user_addresses.address='%d';
hosts = 127.0.0.1
EOF
cat > /etc/postfix/mysql-transports.cf <<EOF
#mysql-transports
user = baruwa
password = password
dbname = baruwa
query = select concat('smtp:[', mail_hosts.address, ']', ':', port) 'transport' from mail_hosts, user_addresses where user_addresses.address = '%s' AND user_addresses.id = mail_hosts.useraddress_id;
hosts = 127.0.0.1
EOF

Note: For this step, make sure to replace anything@example.com, example.com and @example.com with real values that matches your setup.

Make it executable and run it:

chmod +x /usr/src/postfix.sh
./usr/src/postfix.sh

*Note: The user/password for the cf files needs to be the same as the user/password you'll use with your Baruwa DB setup later on.  Make sure to change everything in red before running the script.

 

Postfix Recipient Callout(Optional)

This feature queries the recipient server to see if the recipient exists. If not, it replies with a 550 error to the sending server and drops the connection. If the user does exist, the SpamSnake will continue processing the email. This is just another method to prevent backscatter, but comes at a price. Read up on it at http://www.postfix.org/ADDRESS_VERIFICATION_README.html. You can skip this method and use the script method (later on in this guide) if you decide it will bog down your server.

vi /etc/postfix/main.cf and add the following:

verify_recipient = reject_unknown_recipient_domain, reject_unverified_recipient
look_ahead = check_recipient_access hash:/etc/postfix/access
unverified_recipient_reject_code = 550
address_verify_map = btree:/var/lib/postfix/verify

Add this to your smtpd_restriction_classes:

verify_recipient, look_ahead

Add this to smptd_recipient_restrictions:

smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, look_ahead, whitelist_policy, grey_policy, rbl_policy, spf_policy, permit

Create the access file:

touch /etc/postfix/access

Add your domains:

cat > /etc/postfix/access <<EOF
#mysql-transports
example.com verify_recipient
example2.com verify_recipient
EOF

*Note: Make sure to add valid domains you're filtering for.

Postmap it:

postmap /etc/postfix/access

Final look at the Postfix install:

less /etc/postfix/main.cf

Check the contents of the file for errors and repair if needed. Fire up Postfix:

postfix start

Check that Postfix responds:

telnet 127.0.0.1 25

You should see:

220 [yourFQDNhere] ESMTP Postfix (Ubuntu)

Share this page:

43 Comment(s)

Add comment

Comments

From: Marco at: 2013-03-01 17:38:32

don't you need to add spf to main.cf?  I got errors on policy-spf_time_limit until i added

policy-spf    unix  -     n     n     -     -   spawn

user=nobody argv=/usr/bin/policyd-spf

to /etc/postfix/main.cf

 

From: at: 2013-03-19 20:26:59

spf is added to master.cf later in the tutorial, however the line in the script to setup main.cf has an error. It says:

postconf -e "spf_policy = check_policy_service unix:private/policy" 

it should be:

postconf -e "spf_policy = check_policy_service unix:private/policy-spf"

 also the line about rbl policy is commented out with #, which postfix didn't like when I ran the script.  I just added it manually when i set up rbl.

From: Anonymous at: 2013-09-18 03:20:37

In the edit main.cf for postfix, the next line says:

 vi /usr/src/postfix.sh 

which makes no sense whatsoever ??

 Also, the username and password for mysql are not referenced elsewhere - I assume the user should be root, and the associated password?

 Thanks

From: at: 2014-01-24 16:15:31

You are creating a script to edit main.cf, so the command "vi /usr/src/postfix.sh" creates an empty file then you add the text indicated to the file.  Make sure to change the red text to match your setup.

 There is no need for your mysql password here.  You are simply adding the settings to postfix.  Make sure you remember the password you select for the baruwa db.  You will need to make sure it matches the password you set when you create that db.

From: at: 2013-11-25 21:58:53

I have been trying to set this up, and having problems. Even opened a thread up in forums: http://www.howtoforge.com/forums/showthread.php?p=306340, however no reply so far.

 Is there anyone who can help to complete the setup ?

From: at: 2014-01-24 16:06:38

Rocky changed jobs so isn't as available as he once was.  What problems are you having?

From: at: 2014-02-09 19:38:27

Hi, 

Since i didn't get a notice on a reply, I didn't notice your post even. I had posted by issue on the forums -> http://www.howtoforge.com/forums/showthread.php?p=306340, to which Rocky did reply. I can understand his unavailability. Is there an updated guide which i can use ?

I wish to do inbound as well as outbound spam filtering, separate ofcourse.

Thanks

From: JR at: 2012-12-21 20:40:34

In addition to the instructions provided here, it was also necessary to create the MailScanner incoming directory:

mkdir /var/spool/MailScanner/incoming
chown postfix:www-data /var/spool/MailScanner/incoming

Mailscanner started just fine after this...  

 

 

From: Anonymous at: 2013-03-18 19:09:08

dcc package dcc-common_1.3.130-0ubuntu1~ppa2~quantal1 as well as client seems to be unavailable.  I installed with 1.3.144-0ubuntu1~ppa1~precise1 which seems to be working fine. So to install dcc try this:

 wget http://ppa.launchpad.net/jonasped/ppa/ubuntu/pool/main/d/dcc/dcc-common_1.3.144-0ubuntu1~ppa1~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb && dpkg -i dcc-common_1.3.144-0ubuntu1~ppa1~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb
wget http://ppa.launchpad.net/jonasped/ppa/ubuntu/pool/main/d/dcc/dcc-client_1.3.144-0ubuntu1~ppa1~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb && dpkg -i dcc-client_1.3.144-0ubuntu1~ppa1~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb

NB I didn't run this by Rocky...  but it is working fine for me.

 

 

From: Andy at: 2013-04-08 02:56:55

Another question on this...  Is there any particular reason to install the "quantal" version on "precise", and why are we doing it this way instead of adding the ppa and installing it via apt?  Not criticising, just asking the question, because I'm sure Rocky has a good reason, and I want to know :)

Cheers.

Andy.

From: newbie at: 2013-04-15 04:17:06

Hi,

DCC still failed to start based on your sources;

I got this unavailable error:

root@unknown:/tmp#  wget http://ppa.launchpad.net/jonasped/ppa/ubuntu/pool/main/d/dcc/dcc-common_1.3.144-0ubuntu1~ppa1~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb && dpkg -i dcc-common_1.3.144-0ubuntu1~ppa1~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb
--2013-04-15 12:06:21--  http://ppa.launchpad.net/jonasped/ppa/ubuntu/pool/main/d/dcc/dcc-common_1.3.144-0ubuntu1~ppa1~precise1_amd64.deb
Resolving ppa.launchpad.net (ppa.launchpad.net)... 91.189.95.83
Connecting to ppa.launchpad.net (ppa.launchpad.net)|91.189.95.83|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-04-15 12:06:21 ERROR 404: Not Found.

Please help , many thanks:)

From: at: 2013-09-27 15:52:34

ya they changed the file name again.  In this case you just need to change the instances of ppa1 in the name to ppa2.  You can always go to http://ppa.launchpad.net/jonasped/ppa/ubuntu/pool/main/d/dcc/ to see which version of the files are currently available.

From: jamesloker at: 2013-07-16 15:21:48

If you receive this error when running the spamassassin test run the command:

#sa-learn --sync

Then try the spamassassin test again 

From: Anonymous at: 2013-11-22 21:05:50

you might need to run this  

 apt-get install libssl-dev 

 before you can install Crypt::OpenSSL::RSA

From: at: 2014-01-24 16:35:13

I had the same issue.  I also had to install libmysqlclient-dev before I could install DBD::mysql

From: at: 2014-02-14 07:16:25

Create the following to prevent an error in a lint test:

mkdir /var/www/.spamassassin

But not have /var/www now.

 

 

From: at: 2014-03-10 15:26:24

For this part don't forget to run  

apt-get install libmysqlclient-dev

 

From: Real at: 2012-12-19 15:10:55

Same probléme with the FTP auth 

 But very good How to, more details, nice!

From: Dan at: 2012-12-19 05:09:16

ftp://www.mcrirents.com/pub      

 

The above (or any variant) is not public.  If it is your ftp, please set it to anonymous ftp or provide another way to download the file.  Thanks for all the effort to put this together.

From: JhonKa at: 2012-12-27 23:30:08

Rocky,

 I followed all your instructions to a T and i'm having an error when I start up nginx I get this error

 /etc/init.d/uwsgi restart && /etc/init.d/nginx restart

* Restarting app server(s) uwsgi [ OK ] 

Restarting nginx: nginx: [emerg] unexpected end of file, expecting ";" or "}" in /etc/nginx/sites-enabled/baruwa.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed

I even used your SN packages from your google docs account.  Could you help point me in the right direction?

From: at: 2012-12-28 08:40:20

What is on line 7 of /etc/nginx/sites-enabled/baruwa.conf

From: at: 2012-12-28 15:50:47

You can ignore this!  I found out that the baruwa.ini and .conf were switched in the uwsgi and nginx files.

From: at: 2012-12-28 19:41:11

Fixed, nginx and uwsgi config files were mixed up.

From: Doug Thomas at: 2013-01-06 21:34:49

Can't use an undefined value as an ARRAY reference at /opt/MailScanner/lib/MailScanner/Config.pm line 2588, <DATA> line 500.

 
Is anyone else getting this error? 

From: at: 2013-04-09 09:48:39

 I do.

 

Did you find the solution?

From: Andy at: 2013-06-06 00:52:19

I have this too. My guess is it's something in the MailScanner.conf, but line 500 is just a comment, and if you grep out all the comments there's less than 500 lines.  I'm still looking into it, but if anyone could shed light it would be appreciated :)

From: Andy at: 2013-06-06 01:25:37

I found this occurred because I hadn't commented out the following lines:

#Inline HTML Signature = htmlsigs.customize
#Inline Text Signature = textsigs.customize
#Signature Image Filename = sigimgfiles.customize
#Signature Image Filename = sigimgs.customize

as described above, AND I hadn't set up signatures properly (I'm not really sure how to do this at this stage).  So I commented out the lines, ran /opt/MailScanner/bin/MailScanner --lint, and things seem to be working a lot better.

Hope that helps someone.

Andy.

From: SHL at: 2013-01-27 16:08:17

Hi

 I believe i've followed to guide but my SpamSnake isn't working when rbl_policy and spf_policy is added here: smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unknown_recipient_domain, reject_unauth_destination, whitelist_policy, grey_policy, permit

I'm getting these errors:

postfix/smtpd[6073]: warning: connect to private/policy: No such file or directory

warning: unknown smtpd restriction: "rbl_policy"

Could someone point me in the right direction? :) 

From: monopati at: 2013-03-24 18:10:07

I get 

 # baruwa-admin migrate
Unknown command: 'migrate'
Type 'baruwa-admin help' for usage.

 and

 # baruwa-admin createsuperuser
Unknown command: 'createsuperuser'
Type 'baruwa-admin help' for usage.

Is there any replacement for these commands?

 Thanks

From: kup at: 2013-04-23 07:25:25

Rocky, thank you for this great howto. Please let us know, how to migrate Baruwa frontend to next version (2.0), if you have some way to do it.

Many thanks!

From: at: 2013-08-06 17:39:07

There is no direct upgrade path to 2.0.  Also, 2.0 only currently supports Exim for an MTA.   I did get it working rather poorly with Postfix on a test server, but I wouldn't put it into production.  I would wait until Postfix is supported.

From: kec at: 2013-04-23 07:34:22

For the first Rocky, I have to say - thank you for this great guide. I also want to ask you, if you have a way, how to migrate Baruwa frontend to next version (2.0)?

From: kup at: 2013-04-23 10:30:21

Just one hint for those who want to see Baruwa translated:

#apt-get install gettext
#baruwa-admin compilemessages

Event. edit main language of Baruwa:

#vi /etc/baruwa/settings.py

LANGUAGE_CODE = 'cs'

 

 

From: Mikacom at: 2013-11-21 21:20:10

Populating the database has changed to


#baruwa-admin migrate djcelery

From: Anonymous at: 2013-04-19 09:49:15

Nice guide, some minor problems at start but works perfect now !
Thanks alot ! ! !

 

From: Anonymous at: 2013-05-20 15:22:41

Hoi,

do not plan on virtual images to be issued?

 

From: e3fi389 at: 2013-05-27 11:02:40

Little (but interest!) error in getadsmtp.pl script, correct line 68:

  attrs  => ["proxyAddresses"],

 

From: at: 2014-05-04 06:50:09

Hi,

Great tutorial.

in the update-relay-recipients.sh, i was getting an error message regarding postman and postfix. There are 2 ways around it.

1) Put update-relay-recipients.sh and getadsmtp in the /usr/sbin directory and update the cron accordingly using crontab -e
2) in the update-relay-recipients.sh change the following lines:
postmap /etc/postfix/relay_recipients
postfix reload

to
/usr/sbin/postmap /etc/postfix/relay_recipients
/usr/sbin/postfix reload
 

From: Kevin Traas at: 2014-06-12 19:01:00

* Line 141:  replace 'mailscanner.tv' with 'mailscanner.eu'.
* Line 257:  replace the (invalid) less-than character.

From: Anonymous at: 2014-06-12 19:05:46

The suggested cron change to run update_scamnailer on a daily basis has the wrong path, and is a bit ambiguous.  

 Instead, add the following to /etc/crontab

 53      3 * * * /opt/MailScanner/bin/update_scamnailer

From: kecup at: 2014-11-03 15:45:08

Thanks for great howto, but let me ask you, do you plan an update of this all? I mean upversion this howto to Ubuntu 14.04? Thank you.

From: Michael at: 2015-02-24 21:32:36

just having a small issue with the setup. im new to linux but am trying to learn. so the question i have might be extremely basic but im not sure.

 

im getting the following error when trying to start maillscanner via /etc/init.d/mailscanner start

Can't use an undefined value as an ARRAY reference at /opt/MailScanner/lib/MailScanner/Config.pm line 2588, <DATA> line 500.

so i commented out the following in the file /etc/MailScanner/conf.d/baruwa.conf

#Inline HTML Signature = htmlsigs.customize#Inline Text Signature = textsigs.customize#Signature Image Filename = sigimgfiles.customize#Signature Image Filename = sigimgs.customize

and it appears that everything is working fine. until i click the connect button in baruwa interface to test if the connection to the exchange server is valid.

 

it takes me to http://localhost/settings/hosts/2/test/

and says page unavailable

sorry the requested page is unavailable due to a server malfunction.

 

anyone got any ideas

From: Alexandro at: 2015-03-18 15:14:06

as of 2015 installation just have to fix (like said) most of wget instructions to use up2date releases of downloaded softwares, but that's a mino issue, big problem seem that djcelery need to be migrated with command:

$baruwa-admin migrate djcelery

after the the baruwa sync procedure, otherwise it wouldn't create djcelery related tables in baruwa db (preventing message preview/train/delete, smtp test connection, and few other things via web interface) beside that.. damn great guide <3

Alex