The Perfect SpamSnake - Ubuntu Jeos 9.10 - Page 4

Want to support HowtoForge? Become a subscriber!
 
Submitted by Rocky (Contact Author) (Forums) on Fri, 2010-04-16 12:27. ::

8. Install and Configure SPF

The postfix-policyd-spf-perl package depends on the Mail::SPF and the NetAddr::IP Perl modules.

We need to download postfix-policyd-spf-perl from http://www.openspf.org/Software to the /usr/src/ directory and install it to the /usr/lib/postfix/ directory like this:

cd /usr/src
wget http://www.openspf.org/blobs/postfix-policyd-spf-perl-2.007.tar.gz
tar xvfz postfix-policyd-spf-perl-2.007.tar.gz
cd postfix-policyd-spf-perl-2.007
cp postfix-policyd-spf-perl /usr/lib/postfix/policyd-spf-perl

Then we edit /etc/postfix/master.cf and add the following stanza at the end:

vi /etc/postfix/master.cf

policy unix - n n - - spawn
  user=nobody argv=/usr/bin/perl /usr/lib/postfix/policyd-spf-perl

(The leading spaces before user=nobody are important so that Postfix knows that this line belongs to the previous one!)

Then open /etc/postfix/main.cf and search for the smtpd_recipient_restrictions directive. You should have reject_unauth_destination in that directive, and right after reject_unauth_destination you add check_policy_service unix:private/policy like this:

vi /etc/postfix/main.cf

[...]
smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination,check_policy_service unix:private/policy
[...]

or like this:

[...]
smtpd_recipient_restrictions = reject_unauth_destination,check_policy_service unix:private/policy
[...]

It is important that you specify check_policy_service AFTER reject_unauth_destination or else your system can become an open relay!

Then restart Postfix:

/etc/init.d/postfix restart

That's it already.

 

9. Install and Configure FuzzyOcr

apt-get install netpbm gifsicle libungif-bin gocr ocrad libstring-approx-perl libmldbm-sync-perl imagemagick tesseract-ocr

Download and install the latest FuzzyOCR devel version from http://fuzzyocr.own-hero.net/wiki/Downloads:

cd /usr/src/
wget http://users.own-hero.net/~decoder/fuzzyocr/fuzzyocr-3.5.1-devel.tar.gz

Unpack FuzzyOCR and move all FuzzyOcr* files and the FuzzyOcr directory (they are all in the FuzzyOcr-3.5.1/ directory) to /etc/mail/spamassassin:

tar xvfz fuzzyocr-3.5.1-devel.tar.gz
cd FuzzyOcr-3.5.1/
mv FuzzyOcr* /etc/mail/spamassassin/
wget http://www.gbnetwork.co.uk/mailscanner/FuzzyOcr.words -O /etc/mail/spamassassin/FuzzyOcr.words

We will be storing the image hashes in a mysql database to improve on performance such that images that we have already scanned do not get scanned again as OCR is a resource intense activity.

 

Create MySQL Database

:

mysql -p < /etc/mail/spamassassin/FuzzyOcr.mysql

Setup the password:

mysql -u root -p
mysql> GRANT ALL ON FuzzyOcr.* TO fuzzyocr@localhost IDENTIFIED BY 'password';

 

MailWatch Fix

Do the following to prevent an error in MailWatch:

vi /etc/mail/spamassassin/FuzzyOcr.pm

Change

'use POSIX;'

to

'use POSIX qw(SIGTERM);'

 

FuzzyOcr Configuration

FuzzyOCR's configuration file is /etc/mail/spamassassin/FuzzyOcr.cf. In that file almost everything is commented out. We open that file now and make some modifications:

vi /etc/mail/spamassassin/FuzzyOcr.cf

Put the following line into it to define the location of FuzzyOCR's spam words file:

focr_global_wordlist /etc/mail/spamassassin/FuzzyOcr.words

/etc/mail/spamassassin/FuzzyOcr.words is a predefined word list that comes with FuzzyOCR. You can adjust it to your needs. Next change:

# Include additional scanner/preprocessor commands here:
#
focr_bin_helper pnmnorm, pnminvert, pamthreshold, ppmtopgm, pamtopnm
focr_bin_helper tesseract

to

# Include additional scanner/preprocessor commands here:
#
focr_bin_helper pnmnorm, pnminvert, convert, ppmtopgm, tesseract

Finally add/enable the following lines:

# Search path for locating helper applications
focr_path_bin /usr/local/netpbm/bin:/usr/local/bin:/usr/bin
focr_preprocessor_file /etc/mail/spamassassin/FuzzyOcr.preps
focr_scanset_file /etc/mail/spamassassin/FuzzyOcr.scansets
focr_digest_db /etc/mail/spamassassin/FuzzyOcr.hashdb
focr_db_hash /etc/mail/spamassassin/FuzzyOcr.db
focr_db_safe /etc/mail/spamassassin/FuzzyOcr.safe.db
focr_minimal_scanset 1
focr_autosort_scanset 1
focr_enable_image_hashing 3
focr_logfile /var/log/FuzzyOcr.log
#Mysql Connection#
focr_mysql_db FuzzyOcr
focr_mysql_hash Hash
focr_mysql_safe Safe
focr_mysql_user fuzzyocr
focr_mysql_pass password
focr_mysql_host localhost
focr_mysql_port 3306
focr_mysql_socket /var/run/mysqld/mysqld.sock

This is what the FuzzyOCR developers say about image hashing:

"The Image hashing database feature allows the plugin to store a vector of image features to a database, so it knows this image when it arrives a second time (and therefore does not need to scan it again). The special thing about this function is that it also recognizes the image again if it was changed slightly (which is done by spammers)."

 

Test FuzzyOCR

cd /usr/src/FuzzyOcr-3.5.1/samples
spamassassin --debug FuzzyOcr < ocr-animated.eml > /dev/null

You see the following:

[14808] info: FuzzyOcr: Found Score <9.000> for Exact Image Hash
[14808] info: FuzzyOcr: Matched [1] time(s). Prev match: 16 sec. ago
[14808] info: FuzzyOcr: Message is SPAM. Words found:
[14808] info: FuzzyOcr: "price" in 1 lines
[14808] info: FuzzyOcr: "company" in 1 lines
[14808] info: FuzzyOcr: "alert" in 1 lines
[14808] info: FuzzyOcr: "news" in 1 lines
[14808] info: FuzzyOcr: (6 word occurrences found)
[14808] dbg: FuzzyOcr: Remove DIR: /tmp/.spamassassin14808JZSvHBtmp
[14808] dbg: FuzzyOcr: Processed in 0.104555 sec.

 

10. Filtering PDF, XLS and Phishing Spam with ClamAV (Sanesecurity Signatures)

There is currently a lot of spam where the spam "information" is attached as .pdf or .xls files, sometimes also hidden inside a .zip file. While these spam mails are not easy to catch with e.g. SpamAssassin or a Bayes filter, the ClamAV virus scanner can catch them easily when it is fed with the correct signatures as ClamAV is built to scan mail attachments.

Create a folder for sanesecurity and download and give the script the proper permission.

apt-get install curl rsync
mkdir /usr/src/sanesecurity && cd /usr/src/sanesecurity
wget http://www.inetmsg.com/pub/clamav-unofficial-sigs.tar.gz
tar -zxf clamav-unofficial-sigs.tar.gz && cd clamav-unofficial-sigs-3.7
mv clamav-unofficial-sigs.sh /usr/bin
mv clamav-unofficial-sigs.conf /etc/
chmod +x /usr/bin/clamav-unofficial-sigs.sh

Edit clamav-unofficial-sigs.conf and change the following variables to match your installation:

clam_dbs="/var/lib/clamav"

The variable clamav_dbs contains the path to the directory where your ClamAV signatures are stored.

Path to clamd.pid:

clamd_pid="/var/run/clamav/clamd.pid"

Reload after update:

reload_dbs="yes"
reload_opt="kill -USR2 `cat $clamd_pid`" #Signals PID to reload dbs

Work Directory:

work_dir="/var/lib/clamav"

And once you're done with the configuration, set the following to yes:

user_configuration_complete="yes" 

Now we run the update script to check if the download works:

clamav-unofficial-sigs.sh

Now we a add the script to the root crontab to be run once a day:

crontab -e

Add the following line at the end of the root crontab:

00 04 * * * /usr/bin/clamav-unofficial-sigs.sh –c /etc/clamav-unofficial-sigs.conf &> /dev/null

 

11. Greylisting with Sqlgrey

Install sqlgrey:

apt-get install sqlgrey
touch clients_fqdn_whitelist.local && touch clients_ip_whitelist.local

Edit sqlgrey.conf to look like:

#########################
## SQLgrey config file ##
#########################
# Notes:
# - Unless specified otherwise commented settings are SQLgrey's defaults
# - SQLgrey uses a specific config file when called with -f <conf_file>
## Configuration files
conf_dir = /etc/sqlgrey
## Log level
# Uncomment to change the log level (default is normal: 2)
# nothing: O, errors only: 0, warnings: 1, normal: 2, verbose: 3, debug: 4
# loglevel = 2
## log categories can be fine-tuned,
# here are the log messages sorted by types and levels,
# (anything over the loglevel is discarded):
#
# grey     : (0) internal errors,
#            (2) initial connections, early reconnections,
#                awl matches, successful reconnections, AWL additions,
#            (3) smart decision process debug,
# whitelist: (2) whitelisted connections,
#            (3) actual whitelist hit,
#            (4) whitelists reloads,
# optin:     (3) optin/optout global result
#            (4) optin/optout SQL query results
# spam     : (2) attempts never retried,
# mail     : (1) error sending mails,
#            (4) rate-limiter debug,
# dbaccess : (0) DB errors,
#            (1) DB upgrade,
#            (2) DB upgrade details,
# martians : (2) invalid e-mail addresses,
# perf     : (2) cleanup time,
# system   : (0) error forking,
#            (3) forked children PIDs, children exits,
# conf     : (0) errors in config files, missing required file,
#            (1) warnings in config files,
#                missing optional configuration files,
#            (2) reloading configuration files,
# other    : (4) Startup cleanup
# you can set a level to O (capital o) to disable logs completely,
# but be aware that then SQLgrey can come back to haunt you...
# Provide a coma-separated "logtype:loglevel" string
# For example if you set the loglevel to 3 (verbose) but want SQLgrey to be:
# . quiet for whitelists
# . normal for greylisting
# uncomment the following line.
# log_override = whitelist:1,grey:2
# By default, log_override is empty
## Log identification
# by default this is the process name. If you define the following variable
# SQLgrey will use whatever you set it to
#log_ident = sqlgrey
## username and groupname the daemon runs as
user = sqlgrey
group = sqlgrey
## Socket
# On which socket do SQLgrey wait for queries
# use the following if you need to bind on a public IP address
# inet =
<public_ip>:2501
# default :
# inet = 2501    # bind to localhost:2501
## PID
# where to store the process PID
# pidfile = /var/run/sqlgrey.pid
## Config directory
# where to look for other configuration files (whitelists)
confdir = /etc/sqlgrey
## Greylisting delays
# If you want to be really strict (RFC-wise) use these
# This is *not* recommended, you'll have false positives
# reconnect_delay = 15    # don't allow a reconnection before 15 minutes
# max_connect_age = 2     # don't allow a reconnection after 2 hours
# default: (based on real-life experience)
# reconnect_delay = 5
# max_connect_age = 24
connect_src_throttle = 5
## Auto whitelists settings
# default is tailored for small sites
awl_age = 60
group_domain_level = 2
# For bigger sites you may want
# a smaller awl_age and a bigger group_domain_level
# awl_age = 32            # AWL must be renewed at least once a month
                          # 32 > 31 (max delay between monthly newsletters)
# group_domain_level = 10 # wait for 10 validated adresses to add a whole
                          # domain in AWL
## Database settings
# instead of Pg below use "mysql" for MySQL, "SQLite" for SQLite
# any DBD driver is allowed, but only the previous 3 have been tested
db_type = mysql
db_name = sqlgrey
# Note: the following are not used with SQLite
db_host = localhost
db_port = default
db_user = sqlgrey
db_pass = password
db_cleandelay = 1800 # in seconds, how much time between database cleanups
clean_method = sync # sync : cleanup is done in the main process,
                      #        delaying other operations
                      # async: cleanup is done in a forked process,
                      #        it won't delay mail processing
                      #        BEWARE: lockups have been reported
                      #        and are still investigated
## X-Greylist header added?
# This adds delay, whitelist and autowhitelist information in the headers
prepend = 1
## Greylisting method:
# - full   : greylist by IP address
# - classc : greylist by class C network. eg:
#            2.3.4.6 connection accepted if 2.3.4.145 did connect earlier
# - smart  : greylist by class C network unless there is no reverse lookup
#            or it looks like a home-user address
# Default is smart
# greymethod = smart
## Optin/Optout (see README.OPTINOUT for details)
# - none   : everyone is greylisted (default)
# - optin  : one must optin to have its (incoming) messages being greylisted
# - optout : one must optout to not have its messages being greylisted
optmethod = out
## SQLgrey return value.
# SQLgrey can tell Postfix to:
# - immediately reject a message with a temporary reject code
# - only do so if following rules would allow the message to pass
# The first choice will prevent Postfix from spending time evaluating
# potentially expensive rules.
# In some cases you may want following rules to be aware of the connection
# this.
#
# We can specify a different rejection strategy for the first connection
# attempt, and for early reconnections. 'immed' chooses immediate rejection
# 'delay' choose delayed rejection
#
# By default we use delay on first attempt
reject_first_attempt = immed
# Default for early reconnection is the value affected to reject_first_attempt
reject_early_reconnect = immed
## Update server
# where to get updates for whitelists
whitelists_host = sqlgrey.bouton.name
## Postmaster address
# who gets urgent notifications (DB is down for example)
# default or empty: don't send mail notifications
# admin_mail = postmaster@yourdomain.net

Create the database:

mysql -u root -p
CREATE DATABASE /*!32312 IF NOT EXISTS*/ sglgrey;
GRANT ALL ON sqlgrey.* TO sqlgrey@localhost IDENTIFIED BY 'password';
FLUSH PRIVILEDGES;

Edit /etc/sqlgrey/sqlgrey.conf and update the database settings sections accordingly

Add the following to smtpd_recipient_restrictions:

check_policy_service inet:127.0.0.1:2501, permit

Create

vi /usr/sbin/sqlgrey

(You can download the file from http://downloads.howtoforge.com/spamsnake_ubuntu_9.10/sqlgrey.)

chmod +x /usr/sbin/sqlgrey


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Patrick (not registered) on Wed, 2011-08-17 20:07.

In Step 11, shouldnt touch be:

 touch /etc/sqlgrey/clients_fqdn_whitelist.local && touch /etc/sqlgrey/clients_ip_whitelist.local

Submitted by Alvaro Soto (not registered) on Sat, 2010-11-13 06:16.

Hello... I had the same problem... here is the solution...

http://headup.sytes.net/?q=node/5

Rocky.... great guide..!!!

Submitted by sergrok (not registered) on Thu, 2010-08-26 04:58.

CREATE DATABASE /*!32312 IF NOT EXISTS*/ sglgrey;

 database name should be 'sqlgrey', not 'sglgrey'

Submitted by sergrok (not registered) on Thu, 2010-08-26 04:47.

mistake in line

 FLUSH PRIVILEDGES;

 it shoud be

 FLUSH PRIVILEGES;

Submitted by sergrok (not registered) on Thu, 2010-08-26 04:42.
in sqlgrey.conf you used a line
optmethod = out
but allowed values for optmethod is 'none', 'optin', 'optout'.  in your case value 'out' is equal to 'none'
Submitted by Francisco Maia (not registered) on Thu, 2010-06-24 11:01.

Hi,

 I keep getting these errors when testing FuzzyOCR:

Jun 24 10:50:11.147 [14957] dbg: FuzzyOcr: Exec : /usr/bin/giftext /tmp/.spamassassin149534oFg6Btmp/CIMG0980.gif
Jun 24 10:50:11.147 [14957] dbg: FuzzyOcr: Stdout: >/tmp/.spamassassin149534oFg6Btmp/giftext.info
Jun 24 10:50:11.147 [14957] dbg: FuzzyOcr: Stderr: >>/tmp/.spamassassin149534oFg6Btmp/giftext.err
save_execute: Insecure dependency in open while running with -T switch at ../FuzzyOcr/Misc.pm line 92.
save_execute: Insecure dependency in open while running with -T switch at ../FuzzyOcr/Misc.pm line 92.
Jun 24 10:50:11.155 [14953] dbg: FuzzyOcr: Elapsed [14957]: 0.025447 sec. (/usr/bin/giftext: exit 8)
Jun 24 10:50:11.156 [14953] warn: readline() on closed filehandle INFILE at ../FuzzyOcr/Misc.pm line 205.
Jun 24 10:50:11.156 [14953] info: FuzzyOcr: Image is single non-interlaced...
Jun 24 10:50:11.156 [14953] warn: rules: failed to run FUZZY_OCR test, skipping:
Jun 24 10:50:11.156 [14953] warn:  (Insecure dependency in printf while running with -T switch at /etc/spamassassin/FuzzyOcr.pm line 471.
Jun 24 10:50:11.156 [14953] warn: )

I am using the newest version of Ubuntu to install this, so I think this must be related with some incompatibility with a newer Perl version or something like this.

 I tried to install the latest version of FuzzyOCR (3.6.0) and the problem persists.

 Do you have any ideia how to overcome this?

 

 Thanks!

Submitted by Nathan (not registered) on Wed, 2010-09-29 18:00.

I too had this problem and was able to resolve it by performing a reinstall of fuzzyocr and netpbm:

sudo apt-get install fuzzyocr3

Although fuzzyocr was already installed, when I went to install this package it acted like both fuzzyocr and netpbm were not installed.

Submitted by iser0073 (not registered) on Mon, 2010-08-16 11:55.

I'm experiencing the same problems on new Ubuntu 10.04.1 LTS:

save_execute: Insecure dependency in open while running with -T switch at /etc/spamassassin/FuzzyOcr/Misc.pm line 76.
save_execute: Insecure dependency in open while running with -T switch at /etc/spamassassin/FuzzyOcr/Misc.pm line 76.
Aug 16 12:36:21.481 [4259] dbg: FuzzyOcr: Elapsed [4263]: 0.035510 sec. (/usr/bin/giftext: exit 8)
Aug 16 12:36:21.482 [4259] warn: readline() on closed filehandle INFILE at /etc/spamassassin/FuzzyOcr/Misc.pm line 189.
Aug 16 12:36:21.482 [4259] info: FuzzyOcr: Image is single non-interlaced...
Aug 16 12:36:21.482 [4259] warn: rules: failed to run FUZZY_OCR test, skipping:
Aug 16 12:36:21.483 [4259] warn:  (Insecure dependency in printf while running with -T switch at /usr/share/perl5/FuzzyOcr.pm line 469.
Aug 16 12:36:21.483 [4259] warn: )

Submitted by Error (not registered) on Wed, 2010-07-07 11:54.

Yep, got them also on the latest release 10.04 Ubuntu server

save_execute: Insecure dependency in open while running with -T switch at /usr/share/perl5/FuzzyOcr/Misc.pm line 92.
save_execute: Insecure dependency in open while running with -T switch at /usr/share/perl5/FuzzyOcr/Misc.pm line 92.

Please help!

Submitted by Eddo Jansen (not registered) on Wed, 2010-07-07 07:10.

Got the same problem here, any help would be appreciated!

Regards,

Submitted by Brandon (not registered) on Sat, 2010-07-03 15:53.
facing the same issues.  anyone found a solution to this?
Submitted by Anonymous (not registered) on Tue, 2011-09-20 22:03.
Google fuzzyocr untaint
Submitted by Joolee (not registered) on Tue, 2010-11-30 16:47.

Install version 3.6 of FuzzyOcr (http://fuzzyocr.own-hero.net/wiki/Downloads) and do not alter/uncomment/add the following line

focr_path_bin /usr/local/netpbm/bin:/usr/local/bin:/usr/bin

Works for me this way.

Submitted by Rocky (registered user) on Sat, 2010-12-04 01:20.

Correct, comment out the focr_path line and everything should be fine:

#focr_path_bin

 

Submitted by Ruben Schepers (not registered) on Wed, 2010-06-02 17:00.

In ubuntu mysq.sock is located at

/var/run/mysqld/mysqld.sock

not

/tmp/mysql.sock

(Regarding FuzzyOCR configuration)

Submitted by Rocky (registered user) on Sat, 2010-06-05 04:55.
That is what the guide has..
Submitted by PieterJ (not registered) on Wed, 2010-05-19 19:43.

After installing SQLGREY you see following errors in /var/log/mail.log:

May 19 16:55:56 smtp sqlgrey: dbaccess: can't connect to DB: Access denied for user 'sqlgrey'@'localhost' (using password: YES)
May 19 16:55:56 smtp sqlgrey: dbaccess: error: couldn't get now() from DB: Access denied for user 'sqlgrey'@'localhost' (using password: YES)
May 19 16:55:56 smtp sqlgrey: fatal: Can't call method "do" on an undefined value at /usr/sbin/sqlgrey line 308.

The SQLGREY db does not exist.

use mysql -p

CREATE USER 'sqlgrey'@'localhost' IDENTIFIED BY 'sqlgreypassword';
CREATE DATABASE IF NOT EXISTS `sqlgrey` ;
GRANT ALL PRIVILEGES ON `sqlgrey` . * TO 'sqlgrey'@'localhost';
FLUSH PRIVILEGES;

edit /etc/sqlgrey/sqlgrey.conf to change db_pass accordenly.


 

Submitted by PieterJ (not registered) on Wed, 2010-05-19 08:04.
When running  clamav-unofficial-sigs.sh you see this errors

/usr/bin/clamav-unofficial-sigs.sh: line 798: rsync: command not found
/usr/bin/clamav-unofficial-sigs.sh: line 803: rsync: command not found

======================================================================
Sanesecurity Database & GPG Signature File Updates
======================================================================

Sanesecurity mirror site used: www01.masbytes.es 213.194.159.34
Connection to www01.masbytes.es 213.194.159.34 failed - Trying next mirror site.                                                                                                                               ..


Access to all Sanesecurity mirror sites failed - Check for connectivity issues
or signature database name(s) misspelled in the script's configuration file.

You can solve this by installing rsync (aptitude install rsync)

 

Submitted by Rocky (registered user) on Sat, 2010-06-05 04:56.
Both updated.  Thanks for the catch.