HOWTO: Spam control for POSTFIX

[crypted]

Spam is a major problem for anyone with a mail server. Many times, spam goes to email addresses that don't exist. But, it still is hitting your server even if it isn't delivered. Other times, a users inbox will be overflowing with annoying messages about Viagra, hookers, free software, and whatever else.

Below is a solution. It's an ongoing accumulation of my efforts to stop spam to the best of my abilities. So far, it has a 97% success rate with over 20,000 emails (spam and ham, alike) processed.

Follow the instructions. I will update/modify as necessary if things seem unclear. Feel free to ask.


1) Install Postgrey, RRD, a log parser, and Graphing tools.

apt-get install postgrey rrdtool mailgraph pflogsumm

Postgrey will have a delay of 5 minutes by default on email going to your mailbox. If this is too long, edit the /etc/default/postgrey file by adding "--delay=120" where 120 is seconds.

2) Restart the Postgrey server.

/etc/init.d/postgrey restart

3) edit the Postfix main.cf. We will be adding several things including the Postgrey configuration.

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = /usr/share/doc/postfix

# TLS parameters
smtpd_tls_cert_file = /etc/postfix/smtpd.cert
smtpd_tls_key_file = /etc/postfix/smtpd.key
smtpd_use_tls = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = my.derekgordon.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = my.derekgordon.com, localhost, localhost.localdomain
relayhost = 
mynetworks = 127.0.0.0/8 [::1]/128 66.118.142.78
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
html_directory = /usr/share/doc/postfix/html
virtual_alias_domains = 
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, mysql:/etc/postfix/mysql-virtual_email2email.cf
virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql-virtual_domains.cf
virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailboxes.cf
virtual_mailbox_base = /var/vmail
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes
smtpd_sasl_authenticated_header = yes
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, check_recipient_access mysql:/etc/postfix/mysql-virtual_recipient.cf, reject_unauth_destination, check_policy_service inet:127.0.0.1:60000, reject_rbl_client zen.spamhaus.org, reject_rbl_client smtp.dnsbl.sorbs.net, reject_rbl_client bl.spamcop.net, reject_rbl_client combined.rbl.msrbl.net, check_recipient_access regexp:/etc/postfix/spamtrap, permit
smtpd_tls_security_level = may
transport_maps = proxy:mysql:/etc/postfix/mysql-virtual_transports.cf
relay_domains = mysql:/etc/postfix/mysql-virtual_relaydomains.cf
relay_recipient_maps = mysql:/etc/postfix/mysql-virtual_relayrecipientmaps.cf
virtual_create_maildirsize = yes
virtual_maildir_extended = yes
virtual_mailbox_limit_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailbox_limit_maps.cf
virtual_mailbox_limit_override = yes
virtual_maildir_limit_message = "The user you are trying to reach is over quota."
virtual_overquota_bounce = yes
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $virtual_mailbox_limit_maps
smtpd_sender_restrictions = check_sender_access mysql:/etc/postfix/mysql-virtual_sender.cf
smtpd_client_restrictions = check_client_access mysql:/etc/postfix/mysql-virtual_client.cf
maildrop_destination_concurrency_limit = 1
maildrop_destination_recipient_limit = 1
virtual_transport = maildrop
header_checks = regexp:/etc/postfix/header_checks
mime_header_checks = regexp:/etc/postfix/mime_header_checks
nested_header_checks = regexp:/etc/postfix/nested_header_checks
body_checks = regexp:/etc/postfix/body_checks
content_filter = amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings
message_size_limit = 0

disable_vrfy_command = yes
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks, reject_invalid_hostname, permit

From a generic main.cf found in the Debian Lenny installation, I added/modified the BOLD areas.

4) Create a file named "spamtrap" in the /etc/postfix/ directory. This serves as a filter. If spam is emailed to this address and other addresses on your machine, it will drop that email so that it doesn't get to any other mailboxes.

The Spamtrap file looks like this:

/emailcontrol.*@derekgordon\.com/   DISCARD

This is regexp so the slashes have to be used. My filter email is emailcontrol@derekgordon.com so edit accordingly and place in the spamtrap file!!!

Side note: Do not create this mailbox using ISPConfig. There is absolutely no reason for it to exist on your mailserver. It's a fake address meant to catch annoying spam.

5) Open up local.cf for SpamAssassin and add the following bit. It will be an extra filter designed to work with SA more than it is with general Postfix.

nano /etc/spamassassin/local.cf

Add the following to the bottom:

urirhssub       URIBL_BLACK  multi.uribl.com.        A   2
body            URIBL_BLACK  eval:check_uridnsbl('URIBL_BLACK')
describe        URIBL_BLACK  Contains an URL listed in the URIBL blacklist
tflags          URIBL_BLACK  net       
score           URIBL_BLACK  3.0

6) Restart Postfix and Spamassassin

/etc/init.d/postfix restart

/etc/init.d/spamassassin restart

7) Copy the mailgraph CGI script to your websites CGI-BIN:

cp -p /usr/lib/cgi-bin/mailgraph.cgi /var/www/www.example.com/cgi-bin

8) Create and CHMOD the postfix_report.sh script:

nano /usr/local/sbin/postfix_report.sh

Paste the following into the script:

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

pflogsumm /var/log/mail.log | formail -c -I"Subject: Mail Statistics" -I"From: maillog@example.com" -I"To: youremail@yourdomain.com" -I"Received: from www.example.com ([ 127.0.0.1])" | sendmail youremail@yourdomain.com

##gzip /var/log/mail.log.0
exit 0

chmod 755 /usr/local/sbin/postfix_report.sh

9) Edit the RSYSLOG file so that your mail.log rotates daily and to setup an automatic email with postfix statistics:

nano /etc/logrotate.d/rsyslog

Delete the line that says "/var/log/mail.log" and add this at the VERY bottom of the file:

/var/log/mail.log
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        prerotate
              /usr/local/sbin/postfix_report.sh > /dev/null
        endscript
        postrotate
                invoke-rc.d rsyslog reload > /dev/null
        endscript
}

With this, every time the mail.log rotates (usually around 6am by default) you will get a detailed email about what Postfix has delivered, not delivered, greylisted, and so on.

So now you're all done! What did you do? You installed blacklist filters, greylisting, graphing for on-the-fly information about Postfix, daily emails with detailed Postfix stats, created a spam trap, and other minor things to make your mailserver a lot more secure and less susceptible to spam.

IMPORTANT: Let me know what you all do. Please respond with your choice, if you use it, and how well it worked. If there's much of a use, I will keep building upon the instructions and make it even better (hopefully). Responses are in the form of thread messages.

----------------------

BONUS INSTRUCTIONS: If you use the script I posted below, that gives you GREYLISTING SPECIFIC STATS, do the following:

1) DELETE it from having a CRONJOB if you added one originaly. Mostlikely, you did.

2) Open the RSYSLOG file again.

3) Modify the above entry so that it looks like this:

/var/log/mail.log
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        prerotate
	      /path/to/the/greylist_script.sh > /dev/null
              /usr/local/sbin/postfix_report.sh > /dev/null
        endscript
        postrotate
                invoke-rc.d rsyslog reload > /dev/null
        endscript
}

Make sure that the "/path/to/the/greylist_script.sh > /dev/null" matches the exact path to the script you were using.

Again, That script is found a few posts below.

---------------------------------------
TIPS AND FIXES:

1) If you have people sending you emails and they get rejected because SORBS says their dynamic IP is a bad one, change the following in /etc/postfix/main.cf: "reject_rbl_client dnsbl.sorbs.net" becomes "reject_rbl_client smtp.dnsbl.sorbs.net"
SORBS has a huge variation in repositories. Some AT&T DSL IP's will be found in the DNSBL it appears. So, that could cause a potential no deliver.

2) Similarly to #1, if you find out that there are some SORBS errors in delivering legit emails, you may want to experiement with the various Zones. https://www.au.sorbs.net/using.shtml lists all of them. I recommend SMTP, NOMAIL, SPAM, and ZOMBIE.

3) When upgrading ISPC3 to later versions, the main.cf for Postfix will be erased. This means you can either backup the one you created and replace the changes made by the update script for ISP3, or you can just copy the main.cf information above and use it again. This is the only step that has to be repeated. So, again if you upgrade ISPC3, you need to reconfigure the main.cf for Postfix as it will be replaced by the ISPC3 default main.cf again.

4) If you used this guide prior to 13 October 2010, I recommend removing "reject_rbl_client multihop.dsbl.org" from the main.cf for Postfix. That database has been taken offline. This was "effective" in 2009, but still worked until recently.

[crypted]

The next big step for fixing spam issues is to build a robust header_check and body_check file for Postfix.

As you see above, I have started on the header_check.

I'd like to determine how to setup a SUBJECT filter that would delete any email with more than 7 words. I have had NO legitimate email with more than 6 words in the subject. However, a lot of spam has between 7 and 12 words in the SUBJECT line.

I'm not finding great details on how it works, so it's going to take a lot of time. If you have any of your own mod's please post them so I can compile a better post above.

Also, filtering out all languages you DO NOT use would be sufficient. I will be filtering out all languages except ENGLISH, SPANISH, FRENCH, GERMAN, CZECH, POLISH, and PORTUGUESE when I have time.

[till]

The next big step for fixing spam issues is to build a robust header_check and body_check file for Postfix.

Thats implemented in ispconfig, so do not add such a file manually to ensure that you will not get conflicts. Use the header and body filter function from ispconfig instead.

By the way, I wont use such filter files for spam filtering. The normal spamassassin and blacklists filter already about 99.x % of all spam and adding such manual filters will just rise the risk of false positives. You can not win the battle against sapm with manual filter rules.

[crypted]

Yes, but if you have NO header blocks added in ISPCONFIG, you can use the header file directly.

In cases that it's not a production server, just for personal uses, it might be easier to copy/paste the above header_check.

However, I understand your point and if entries were being made it would be a conflict.

So, I would recommend doing what Till says if you're already using that feature or may use that feature.

[crypted]

I removed the HEADER_CHECKS I created from the HOWTO as I don't want to confuse.

Here they are for you expert experimenters.

Last piece, and this will be updated as I get more in depth, header_check in the postfix directory should contain these:

/^Subject: .*       / REJECT Spam Header Many Spaces 1

# Chinese, Japanese and Korean

/^Content-Type:.*?charset\s*=\s*�?(Big5|gb2312|euc-cn)�?/ REJECT HDR2100: Unaccepted character set: �$1?
/^Content-Type:.*?charset\s*=\s*�?(euc-kr|iso-2022-kr)�?/ REJECT HDR2110: Unaccepted character set: �$1?
/^Content-Type:.*?charset\s*=\s*�?(iso-2022-\w+|euc-jp|shift_jis)�?/ REJECT HDR2120: Unaccepted character set: �$1?


# Cyrrilic character sets: Russian/Ukrainian

/^Content-Type:.*?charset\s*=\s*�?(koi8-(?:r|u))�?/ REJECT HDR2200: Unaccepted character set: �$1?
/^Content-Type:.*?charset\s*=\s*�?(windows-(?:1250|1251))�?/ REJECT HDR2210: Unaccepted character set: �$1?

I will continue to build the header_check file as time goes on. So stay tuned.

[Turbanator]

Question/Recap:
- Till is saying not to perform header/body checks within main.cf and use the tools within ispc3, yes? If so, that makes perfect sense to me. I wish there was a service/blocklist that already did it on a realtime basis (somehow).

- Is Postgrey good to load or no? I still need to research postgrey to figure out what it is.

Sorry for my confusion and thank you for including me. By you posting your main.cf, I noticed I had limited my sorbs bl. By changing to yours I blocked a lot more immedietly.

[crypted]

Yes, I would do header_checks in ISPC3 if you're comfortable with that. It's pretty simple in the end. I haven't tested my new ones I wrote yet, so won't post them. But, I'm thinking that postgrey will be the best solution.

POSTGREY is an easy implementation of Greylisting:

Greylisting is a new method of blocking significant amounts of spam at the mailserver level, but without resorting to heavyweight statistical analysis or other heuristical (and error-prone) approaches. Consequently, implementations are fairly lightweight, and may even decrease network traffic and processor load on your mailserver.

Greylisting relies on the fact that most spam sources do not behave in the same way as "normal" mail systems. Although it is currently very effective by itself, it will perform best when it is used in conjunction with other forms of spam prevention. For a detailed description of the method, see the Whitepaper.

The term Greylisting is meant to describe a general method of blocking spam based on the behavior of the sending server, rather than the content of the messages. Greylisting does not refer to any particular implementation of these methods. Consequently, there is no single Greylisting product. Instead, there are many products that incorporate some or all of the methods described here.

Postgrey specific:

Postgrey is a Postfix policy server implementing greylisting.  Development of Postgrey started at the ISG.EE and is now sponsored by Open Systems AG.

When a request for delivery of a mail is received by Postfix via SMTP, the triplet CLIENT_IP / SENDER / RECIPIENT is built. If it is the first time that this triplet is seen, or if the triplet was first seen, less than 5 minutes ago, then the mail gets rejected with a temporary error. Hopefully spammers or viruses will not try again later, as it is however required per RFC.

I have POSTGREY running. It's blocked a significant amount of unwanted email. It's only downside, if there is any, that it delays mail delivery by 5 minutes default.

[crypted]

Once you're using Postgrey and what to see how it's working, install the following script somewhere on your server:


"postgrey_stats.sh"

#!/bin/sh

LOGFILE=/tmp/greylist-statistics
YOURMAIL=youremail@yourdomain.com
 
echo "Total amount of GreyListed messages
" > $LOGFILE
cat /var/log/mail.log | /usr/bin/postgreyreport --delay=300 >> $LOGFILE
echo -ne  "-------------------------------------\n" >> $LOGFILE
echo -ne  "-------------------------------------\n" >> $LOGFILE
echo "Get only the top 20 sources getting greylisted out
" >> $LOGFILE
cat /var/log/mail.log | postgreyreport | awk '{print $1}' | sort | uniq -c | sort -nr | head -n20 >> $LOGFILE
echo -ne  "-------------------------------------\n" >> $LOGFILE
echo -ne  "-------------------------------------\n" >> $LOGFILE
echo "Get a list of the top 20 email address that the greylisted sources are sending email to
"  >> $LOGFILE
cat /var/log/mail.log | postgreyreport | awk '{print $4}'  | sort  | uniq -c | sort -nr | head -n20 >> $LOGFILE
echo -ne  "-------------------------------------\n" >> $LOGFILE
echo -ne  "-------------------------------------\n" >> $LOGFILE
cat $LOGFILE | mail -s "Greylisting Statistics of `hostname` for `date +%Y-%m-%d`" $YOURMAIL

Edit the following parts of the above script:

1) Change the "YOUREMAIL = " line so that it goes to your personal mail box. This will give you details on how its working.
2) Make sure that /var/log/mail.log is the correct path to your current mail.log file. Distros are different.

Set the script to chmod +700 so that it is executable.

Finally, create a cronjob so that it will give you an automatic output to your email daily (or whenever you choose).

[Turbanator]

Have you seen sqlgrey? Not sure if it's better or not but it has an interface that's interesting. Any thoughts?

[crypted]

I have seen it. However, I will stick with Postgrey for the time being.

I recommended in a REQUEST that Till research including SqlGrey and its interface in a future release. At that point, it wouldn't be necessary to use Postgrey.

But, Postgrey is extremely simple (see above instructions) so it isn't worth the extra hassle in my opinion for roughly the same functionality.

I'm researching an addon to Postgrey that will make it even better.

Give Postgrey a chance, use that script above to see how well it works (you will be shocked after the first full day), and see what happens.

The only false positive I have heard of was Paypal. But, that's unconfirmed and worked without a hitch for me.

You can add whitelists for domains, ips, specific emails, clients, etc... with easy in the event there were problems. Whitelists also will skip the delay in delivery.

Let me know what you all do. Please respond with your choice, if you use it, and how well it worked. If there's much of a use, I will keep building upon the instructions and make it even better (hopefully).