PDA

View Full Version : DSPAM tutorial


bernd
26th May 2005, 20:06
I'd like to see a tutorial about using DSPAM (http://www.nuclearelephant.com/projects/dspam/) for filtering emails... :)

Bernd

Bailx
29th December 2005, 05:29
I'm going to be working on getting dspam to work in place of spamassassin for this tutorial http://www.howtoforge.com/book/print/181

i'll let you know if i'm successful... i'd be open to any suggestions in the meantime... as it might be a while before i'm to that stage, i'm still working on being "satisfied" with ubuntu smp kernel... as I'm coming off of using RHEL v4 (64-bit)... also that's a debian howto... ;)

falko
29th December 2005, 12:06
I'm going to be working on getting dspam to work in place of spamassassin for this tutorial http://www.howtoforge.com/book/print/181

I worked on exactly the same a few weeks ago (got it working), but there were some issues to which I couldn't find a solution, that's why I postponed such a tutorial:
- In such a setup DSpam scans also outgoing emails and puts the DSpam ID into the mail body which is not good if there's a spam scanner at the other end which doesn't like unique IDs in the mail body...
- When I sent mails to DSpam for learning, but with a faked DSpam ID in the mail body, the DSpam daemon got a fatal error and broke down... :( :( (and I could reproduce that behaviour)
- In the web interface you can re-learn messages only on the first page of the message list...

Bailx
29th December 2005, 17:01
Did you try all 3 methods of integration? Just curious because if not... perhaps one of the other methods would do the trick.

As a delivery proxy:
The default approach integrates DSPAM directly with the mail server and
filters spam as mail comes in. Please see the appropriate instructions
in doc/ pertaining to your MTA.

As a POP3 proxy:
This alternative approach implements a POP3 proxy where users
connect to the proxy to check their email, and email is filtered when
being downloaded. The POP3 proxy is a much easier approach, as it
requires much less integration work with the mail server (and is ideal
for implementing DSPAM on Exchange, etcetera). Please see the file
doc/pop3filter.txt.

As an SMTP Relay:
DSPAM can be configured as an SMTP relay, a.k.a appliance. You
can set it up to sit in front of your real mail server and then point
your MX records at it. DSPAM will then pass along the good mail to
your real SMTP server. See doc/relay.txt for more information. The
example provided uses Postfix and MySQL.

falko
29th December 2005, 18:52
I integrated DSpam into Postfix as a content filter:

[Postfix] (LMTP) -> [DSPAM] [Postfix] -> { Delivery }
|___ (SMTP Reinjection) ____|

because then I don't have to care about any setuid permissions.

his_royal_evilness
26th January 2006, 20:44
Could you provide a howto Falko? :)

falko
26th January 2006, 21:37
Sure - if I get it to work the way I want to... :D I don't write tutorials about half-cooked things...

his_royal_evilness
27th January 2006, 02:31
Thnx in advance , how far are you with the setup? :)

falko
27th January 2006, 08:18
I have postponed this because I had difficulties for which I couldn't find a solution then... :(

his_royal_evilness
27th January 2006, 22:59
What kind of ?
Perhaps we could work together on this one? :)

falko
28th January 2006, 08:28
What kind of ?
I mentioned them earlier in this thread.

his_royal_evilness
28th January 2006, 13:11
let work together on this one :)
If you could provide me with ur setup (dspam configure options / postfix configuration) it would be very nice to test it at my server :)

falko
29th January 2006, 09:51
These are my notes about what I've done so far (this setup is based on a basic Debian Sarge installation):

groupadd dspam
useradd -s /bin/false -m -c "DSpam" -g dspam dspam
usermod -G mail dspam

cd /tmp
wget http://dspam.nuclearelephant.com/sources/dspam-3.6.2.tar.gz
tar xvfz dspam-3.6.2.tar.gz
cd dspam-3.6.2
./configure --sysconfdir=/etc --with-dspam-home=/var/dspam --enable-domain-scale --with-storage-driver=mysql_drv,hash_drv --enable-daemon --with-mysql-includes=/usr/include/mysql --with-mysql-libraries=/usr/lib --enable-virtual-users --enable-preferences-extension --enable-clamav
make
make install

cp -r txt/ /var/dspam

mysqladmin -u root -p create dspam
mysql -u root -p
GRANT SELECT, INSERT, UPDATE, DELETE ON dspam.* TO 'dspam_admin'@'localhost' IDENTIFIED BY 'dspam_admin_password';
GRANT SELECT, INSERT, UPDATE, DELETE ON dspam.* TO 'dspam_admin'@'localhost.localdomain' IDENTIFIED BY 'dspam_admin_password';
FLUSH PRIVILEGES;
quit;
mysql -u root -p<password> dspam < src/tools.mysql_drv/mysql_objects-speed.sql
mysql -u root -p<password> dspam < src/tools.mysql_drv/virtual_users.sql

mkdir /var/dspam/sql
cp src/tools.mysql_drv/purge.sql /var/dspam/sql/

/var/dspam/sql/purge.sgl:
=========================
delete from dspam_token_data
where @a-to_days(last_hit) > 90; entfernen ---->

# $Id: purge.sql,v 1.4 2005/05/04 20:14:37 jonz Exp $
set @a=to_days(current_date());
delete from dspam_token_data
where (innocent_hits*2) + spam_hits < 5
and @a-to_days(last_hit) > 60;
delete from dspam_token_data
where innocent_hits = 1 and spam_hits = 0
and @a-to_days(last_hit) > 15;
delete from dspam_token_data
where innocent_hits = 0 and spam_hits = 1
and @a-to_days(last_hit) > 15;
delete from dspam_signature_data
where @a-14 > to_days(created_on);



rm -f /etc/alternatives/editor
ln -s /usr/bin/vi /etc/alternatives/editor

crontab -e

0 0 * * * mysql -u root -p<password> dspam < /var/dspam/sql/purge.sgl
0 0 * * * dspam_logrotate -a 30 /var/dspam/system.log `find /var/dspam/data -name "*.log"`

/etc/postfix/master.cf:
=======================
smtp inet n - - - - smtpd
-o content_filter=lmtp:unix:/tmp/dspam.sock

localhost:10026 inet n - - - - smtpd
-o content_filter=
-o receive_override_options=no_unknown_recipient_chec ks,no_header_body_checks
-o smtpd_helo_restrictions=
-o smtpd_client_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,rej ect
-o mynetworks=127.0.0.0/8
-o smtpd_authorized_xforward_hosts=127.0.0.0/8



/etc/dspam.conf:
================
#TrustedDeliveryAgent
DeliveryHost 127.0.0.1
DeliveryPort 10026
DeliveryIdent localhost
DeliveryProto SMTP

Trust root
Trust mail
Trust dspam

MySQLServer 127.0.0.1
MySQLPort 3306
MySQLUser dspam_admin
MySQLPass dspam_admin_password
MySQLDb dspam
MySQLCompress true

MySQLConnectionCache 10

MySQLVirtualTable dspam_virtual_uids
MySQLVirtualUIDField uid
MySQLVirtualUsernameField username

MySQLUIDInSignature on

Notifications on

ParseToHeaders on
ChangeModeOnParse on
ChangeUserOnParse full

ServerMode auto
ServerParameters "--deliver=innocent"
ServerIdent "localhost.localdomain"
ServerPID /var/run/dspam.pid
ServerDomainSocketPath "/tmp/dspam.sock"


/etc/init.d/postfix restart

dspam_admin ch pref default trainingMode TEFT
dspam_admin ch pref default spamAction quarantine
dspam_admin ch pref default spamSubject "[SPAM]"
dspam_admin ch pref default enableWhitelist on
dspam_admin ch pref default showFactors off

#dspam --daemon &
/etc/init.d/dspam anlegen
chmod 755 /etc/init.d/dspam
update-rc.d dspam defaults
/etc/init.d/dspam start

apt-get install apache-dev
apt-get install libgd-gd2-perl libgd2 libgd2-dev libgd2-xpm libgd2-xpm-dev
apt-get install libgd-graph3d-perl libgd-graph-perl libgd-text-perl
mkdir -p /var/www/dspam/web
mkdir /var/www/dspam/cgi-bin
cd /tmp
wget http://mesh.dl.sourceforge.net/sourceforge/modauthmysql/mod_auth_mysql-3.0.0.tar.gz
tar xvfz mod_auth_mysql-3.0.0.tar.gz
cd mod_auth_mysql-3.0.0
apxs -c -I/usr/include/mysql -lmysqlclient -lm -lz mod_auth_mysql.c
apxs -i mod_auth_mysql.so

/etc/apache/modules.conf:
=======================
LoadModule mysql_auth_module /usr/lib/apache/1.3/mod_auth_mysql.so

/etc/init.d/apache restart

cd /tmp/dspam-3.6.2/webui
cp -pfr cgi-bin/* /var/www/dspam/cgi-bin
cp -pfr htdocs/* /var/www/dspam/web/
cd /var/www/dspam
chown -R dspam:dspam cgi-bin/
chown -R dspam:dspam web/

/var/www/dspam/cgi-bin/configure.pl:
====================================
$CONFIG{'WEB_ROOT'} = ""; # URL location of included htdocs/ files
$CONFIG{'LOCAL_DOMAIN'} = "test.de";
$CONFIG{'HISTORY_SIZE'} = 799; # Number of items in history
$CONFIG{'HISTORY_PER_PAGE'} = 800;


/etc/apache/httpd.conf:
=======================
NameVirtualHost 192.168.0.101

<VirtualHost 192.168.0.101>
User dspam
Group dspam
DocumentRoot /var/www/dspam/web
DirectoryIndex index.html index.htm index.php index.php4 index.php3 index.shtml index.cgi index.pl index.jsp Default.htm default.htm
ScriptAlias /cgi-bin/ /var/www/dspam/cgi-bin/
AddHandler cgi-script .cgi
AddHandler cgi-script .pl
<Directory /var/www/dspam/web>
Options +Includes -Indexes
AllowOverride None
AllowOverride Indexes AuthConfig Limit FileInfo
Order allow,deny
Allow from all
<Files ~ "^\.ht">
Deny from all
</Files>
</Directory>
<Directory "/var/www/dspam/cgi-bin">
Options +ExecCGI
AuthName "DSPAM Quarantine Area"
AuthType Basic
AuthMySQLEnable on
AuthMySQLHost 127.0.0.1
AuthMySQLPort 3306
AuthMySQLUser mail_admin
AuthMySQLPassword mail_admin_password
AuthMySQLDB mail
AuthMySQLUserTable users
AuthMySQlNameField email
AuthMySQLPasswordField password
AuthMySQLPwEncryption crypt
Require valid-user
Order Deny,allow
Allow from all
</Directory>
</VirtualHost>

mv /usr/lib/apache/suexec.disabled /usr/lib/apache/suexec
/etc/init.d/apache restart


Retrain:
--------

/usr/local/bin/dspam-retrain:
=============================
#! /usr/bin/perl

# Get arguments
$class = $ARGV[0] || die; shift;
$sender = $ARGV[0] || die; shift;
$recip = $ARGV[0] || die; shift;

if ($recip =~ /^(spam|ham)-(\w+)@/) {
# username is part of the recipient
$user = $2;
} elsif ($sender =~ /^(\w+)@/) {
# username is in the sender
$user = $1;
} else {
print "Can't determine user\n";
exit 75; # EX_TEMPFAIL
}

# Pull out DSPAM signatures and send them to the dspam program
while (<>) {
if ((! $subj) && (/^Subject: /)) {
$subj = $_;
} elsif (/(!DSPAM:[a-f0-9]+!)/) {
open(F, "|/usr/local/bin/dspam --source=error --class=$class --user $user");
print F "$subj\n$1\n";
close(F);
} elsif (/(X-DSPAM-Signature: [a-f0-9]+)/) {
open(F, "|/usr/local/bin/dspam --source=error --class=$class --user $user");
print F "$subj\n$1\n";
close(F);
}
}




chmod 755 /usr/local/bin/dspam-retrain

/etc/postfix/master.cf:
=======================
dspam-retrain unix - n n - 10 pipe
flags=Ru user=dspam argv=/usr/local/bin/dspam-retrain $nexthop $sender $recipient


transport-Tabelle:
===================
spam@test.de dspam-retrain:spam
ham@test.de dspam-retrain:innocent

INSERT INTO transport(domain, transport) VALUES ('spam@test.de', 'dspam-retrain:spam');
INSERT INTO transport(domain, transport) VALUES ('ham@test.de', 'dspam-retrain:innocent');

INSERT INTO `users` (`email`, `password`, `quota`) VALUES ('spam@test.de', ENCRYPT('secret'), 10485760);
INSERT INTO `users` (`email`, `password`, `quota`) VALUES ('ham@test.de', ENCRYPT('secret'), 10485760);

/etc/postfix/main.cf:
=====================
postconf -e 'local_recipient_maps = proxy:unix:passwd.byname $alias_maps $transport_maps'
#recipient_delimiter = -

/etc/init.d/postfix restart


ClamAV:
=======
apt-get install libgmp3 libgmp3-dev
cd /tmp
groupadd clamav
useradd -g clamav -s /bin/false -c "Clam AntiVirus" clamav
wget http://mesh.dl.sourceforge.net/sourceforge/clamav/clamav-0.87.1.tar.gz
tar xvfz clamav-0.87.1.tar.gz
cd clamav-0.87.1
./configure --sysconfdir=/etc
make
make install

/etc/clamd.conf bearbeiten
/etc/freshclam.conf bearbeiten
/etc/init.d/clamd anlegen
chmod 755 /etc/init.d/clamd
update-rc.d clamd defaults
/etc/init.d/clamd start

/etc/dspam.conf:
================
#
# Virus Checking: If you are running clamd, DSPAM can perform stream-based
# virus checking using TCP. Uncomment the values below to enable virus
# checking.
#
# ClamAVResponse: reject (reject or drop the message with a permanent failure)
# accept (accept the message and quietly drop the message)
# spam (treat as spam and quarantine/tag/whatever)
#
ClamAVPort 3310
ClamAVHost 127.0.0.1
ClamAVResponse accept


/etc/init.d/dspam restart


/usr/local/bin/dspam --user ft@test.de --class=spam --source=error --signature=1,4399a03667787982714935

his_royal_evilness
29th January 2006, 10:09
Thanks , i'll have a look at this and will see if i can get it to work/enhance it
( Think of automatic spam-user@domain parsing with retraining spam )

Also : Could you translate the german words in English for me as German isn't my best language *looks nicely*

his_royal_evilness
31st January 2006, 04:54
AuthMySQLUser mail_admin
AuthMySQLPassword mail_admin_password

What does that mean? on what database?

till
31st January 2006, 12:09
I guess thats the login to the DSPAM mySQL database. Most DSPAM installations run on mySQL because it has the best performance of the DSPAM storage backends.

his_royal_evilness
31st January 2006, 16:06
Mkay , i want users to authenticate with their user@domain adress at the Webinterface of dspam , any idea on how to implement that?

brent.stephens
31st July 2006, 17:41
I am what I consider to be successfully running DSPAM "in front" of ISPConfig at this point. I am using the latest DSPAM CVS, which is the basis for what will be the 3.8 release (current release is 3.6.8).

I will create a howto with my internal wiki and once its finished will share it with you guys for editing and suggestions.

falko
1st August 2006, 14:05
I will create a howto with my internal wiki and once its finished will share it with you guys for editing and suggestions.
That would be great! :)

brent.stephens
7th August 2006, 16:11
Well its not on my internal wiki, since I love to share the DSPAM love ;)

This is my first real draft of the process I used to set up DSPAM. After a couple weeks now I am still very happy with it. Being that it is a first draft, it is basically a regurgitation of thought and config files. Think of it as a barebones kit for those who are comfortable with moderately technical Linux stuff :)

Please forgive me in advance for not being as unbelievably thorough as falko and till.

http://dspamwiki.expass.de/Installation/Postfix/DspamWithIspConfig

brent.stephens
8th August 2006, 15:49
Please reply with feedback in this thread or at least feel free to edit the wiki page. I'm quite sure that there are probably "wrong" things that I have done or maybe "better" ways to integrate than what I have chosen. I'm interested to see different approaches and to see people implement this awesome software.

falko
8th August 2006, 19:11
http://dspamwiki.expass.de/Installation/Postfix/DspamWithIspConfig
Looks good. :) I'm going to mention this in the ISPConfig Tips/Tricks forum.

standk
9th August 2006, 08:55
Please reply with feedback in this thread or at least feel free to edit the wiki page. I'm quite sure that there are probably "wrong" things that I have done or maybe "better" ways to integrate than what I have chosen. I'm interested to see different approaches and to see people implement this awesome software.

A small question, I've managed to get dspam working rather well, but when I receive an email, it seems to only parse the user@ part, and not the complete virtual user of user@domain.tld. I've searched dspam.conf a bit but I can't seem to figure out what's causing that to be set.

Kent

dklima
22nd August 2006, 20:51
- In such a setup DSpam scans also outgoing emails and puts the DSpam ID into the mail body which is not good if there's a spam scanner at the other end which doesn't like unique IDs in the mail body...
It's possible to put the ID into body's message...
dspam.conf
Preference "signatureLocation=headers"
But have a side effect: when you'll send the message to re-learn, it must go attached...


- When I sent mails to DSpam for learning, but with a faked DSpam ID in the mail body, the DSpam daemon got a fatal error and broke down... :( :( (and I could reproduce that behaviour)
It's not occour in 3.6.8 version that I'm using.


- In the web interface you can re-learn messages only on the first page of the message list...
I can't reproduce that... I've setup my message history to display 10000 messages :rolleyes:

Cya.

jeffrey404
8th January 2007, 07:22
Hi,


First post here. My question is, has anyone got dspam integrated with falko's postfix howto?

Thanks in advance!

Jeff

falko
9th January 2007, 00:42
I didn't work on it any further...

jeffrey404
9th January 2007, 03:58
Falko,

Thanks for the response. Do you have any plans in the future to work on it?

I am looking for a solution for postfix with mysql and dspam. dspam would get the user information from the postfix mysql database and go from there.

basically looking to setup a smtp gateway. any help would be great.

Jeff

falko
10th January 2007, 17:33
Do you have any plans in the future to work on it?

No, at least not now. But I might change my mind... :)