The Perfect SpamSnake - Ubuntu Jeos 10.10 - Managing transport relay_domains etc

Discussion in 'HOWTO-Related Questions' started by mintydave, Dec 13, 2010.

  1. mintydave

    mintydave New Member

    First, thank you for the SpamSnake. I've been running two for almost 3 years now and have filtered MILLIONS of emails.

    On my Hardy (Ubuntu 8.04) SpamSnake, I got used to using Webmin's MySQL to manage the greylisting "whitelist" table, and the file in /etc/postfix for "transport", "relay_domains" and "relay_recipients" and others.

    I see now that MySQL is playing a bigger roll in the spamsnake.

    I copied my "transport" "relay_domains" etc.. from my old hardy spamsnake to my new spamsnake. Did a "postmap transport" and fed the snake a simple email from the command line using telnet.

    I tailed the "/var/log/mail.log" file and found that postfix used DNS to deliver the email instead of using the designated IP addresss of the host for deliver from the "transport" list.

    Is there a rerference for managing the postfix features now? Do we use MySQL or continue using the local files in /etc/postfix.

    Any assistance is GREATLY apreciated.


    Regards,
    David,
    Deconn Technical Services
     
  2. Rocky

    Rocky New Member

    Hey,

    3yrs is indeed a long time. I'm happy that it's working out for you.

    If you want to use the relay_domain and transport files from your older boxes, you'd have to edit /etc/postfix/main.cf and make it look like:

    relay_domains = hash:/etc/postfix/relay_domains
    transport_maps = hash:/etc/postfix/transports

    Make sure to edit those files accordingly and postmap them.

    I've gone with a much easier approach to setting up the Snake,
    almost everything can be done through Baruwa. The mysql setup is
    the best way to do it now, since all the records can be created through
    the frontend with the exception of relay_recipients. Of course you can
    always use the system the old school way, the way you've been doing it
    without any issues. You just have to make sure main.cf is set up
    appropriately and also the associated files.

    Rocky
     
  3. mintydave

    mintydave New Member

    I greatly appreciate you response and your work on the snake.

    Where in Baruwa do I manage the greylist feature's whitelist? I have hundreds of IP addresses that I would like the greylist to ignore.

    The way I use the snake is all the mx's for my clients point inbound to the snake, then the snake, sitting behind the firewall, delivers the email to my client's email server via IPSEC tunnels. I was not able to find out where to configure the transport host for a domain. i.e. where is the equivalent of this entry from the "transport" file in /etc/postfix:

    Code:
    deconn.com   smtp:[192.168.100.10]

    Regards,
    Dave.
     
  4. mintydave

    mintydave New Member

    So i'm digging around looking for the answers anyway.

    Found this:
    http://www.baruwa.org/users.html

    The document says:

    Looks like "SMTP delivery host management may be what I'm looking for. But on my Baruwa user interface, I only have "Accounts" and "Scanners" options. I don't have a "Domain Management" option, however I do see there is are "Domain Name", "Owner", "Status" and "Actions" columns when I click on the "Settings" tab.

    From that screen, I do not see any option buttons or text boxes to make any SMTP delivery host changes.

    Making the SMTP delivery settings is a critical feature for me. Thanks for your help.


    Regards,
    Dave.
     
  5. Rocky

    Rocky New Member

    Hey,

    This should walk you through the process.

    Baruwa Setup Instructions:

    Log into Baruwa as admin --> Settings --> Accounts --> Create Account
    Once you've created the user, you'll get 2 new tabs on that page, Profile Settings and Associated Addresses

    Fill out Profile Settings choosing Domain Admin and set a low score of 6 and a high score of 9 and check scan email.

    Fill out the Associated Addresses or domains for which the user is the admin of eg. domain.com. domain.com will show up under the Associated Addresses header.

    Click on domain.com and you'll be taken into Domain Information, where you'll be able to setup SMTP delivery information. Go ahead and add the smtp server, or the ip of the receiving smtp server. Select enable and if you use a non-standard port, set it, otherwise use 25. Once you've done that, you can click on the test button next to the pencil, to see if your receiving server will accept the connection.

    Now, you can log out as admin, and log in as the user you just setup and mails should start flowing.
    *Note: The user will be the Domain Admin for whatever domains you specify in Associated Addresses.

    The relay_domains and transports settings in main.cf will use the entries you've provided in Baruwa. Therefore, no hash file is required. The queries will pull the result in the proper format and feed it to postfix for use.

    However, if you'd like to use a hash for any of your config files, just set your /etc/postfix/main.cf to look like this example:
    relay_recipient_maps = hash:/etc/postfix/relay_recipients
    Of course, you would have to create the hash file, populate it and postmap it for postfix to use. You can also do this with relay_domains and transports if you'd like.
    *Note: If you do end up using a hash for relay_recipients, you'll have to remove the look_ahead feature.

    Also, since you need to do mx lookups, you'll have to edit /etc/postfix/mysql-transports.cf to look like:
    concat('smtp:', mail_hosts.address, ':', port) 'transport'I removed the [ and ] to allow MX lookups.

    Rocky
     
  6. mintydave

    mintydave New Member

    Once again, thank you for your help. I am super impressed with the new architecture of the snake.

    I actually did this howto on top of Ubuntu 10.04 Lucid because I wanted to run this box for quite a while.

    I got Baruwa configured and I like it mutch better as a front end for MailScanner. I actually had no idea that I could set up accounts for each client to manage their own spam trails, so I will continue to monitor my now 'in-production' snake and then slowly disclose to my clients how to log into the Baruwa and manage and track their own spam and quarantines.

    As I continue to monitor my snake, I see that the greyfix is a busy process. I use Splunk to monitor the box and there are a lot of greylist entries.

    I know the Hardy-based snake used PostGrey and I managed the greylist in MySQL (where I have almost 1000 entries.) As effective as the greylist feature is, it did create some confusion for my client's senders and for my help desk. Should I just try out the new greylist system and live with it's out-of-the box functionality? -- or -- should I use the suggestions from the website (http://www.kim-minh.com/pub/greyfix) to implement the whitelist?

    One of my next projects related to the snake is to setup either a central MySQL server so I can run two postfix gateways and one MySQL server. Alternatively, I will learn how and implement a MySQL synchronization so the databases will use the same features and I can have a truly centralized way to manage the snake "heads".

    I wonder how the snake compares in features and performance to the Baracuda appliance.

    Thanks again.

    Dave.
    Deconn Technical Services
     
  7. Rocky

    Rocky New Member

    Hey Dave,

    Greyfix is much like postgrey or sqlgrey in that it does the same thing when it sees a mail for the first time. The actions are identical, but greyfix is much lighter than any of those. Yes, you'll see those processes running because greyfix is setup to kill old processes and start new ones. This keeps the processes running and stops any from locking up.

    As for the whitelist entries, what I've done with this Snake was setup a global whitelist. If you log in to Baruwa --> List, you can add whitelist/blacklist entries. These entries are queried by postfix using the mysql-global_whitelist.cf file. If an entry is found in the whitelist, it bypasses grey, rbl, spf and mailscanner checks. This was something I thought long and hard about because I wanted to have a common whitelist.

    As for running multiple instances, the developer of Baruwa has that on his roadmap. He's going to develop Baruwa so that we can use a common database. Of course this is a way down the line.

    Give it a go, test it out. I think hooking Baruwa into the common db will be the trick, but the mysql replication is not hard at all.
     
  8. topdog

    topdog HowtoForge Supporter

    Actually you already can do that. You can run two nodes with multi master replication with both nodes logging to their own DB and the DB's replicating the changes between each other, you can login in to either box and you will see the same thing. You will not even notice that you are using a two node cluster.
     
  9. Rocky

    Rocky New Member

    There you go Dave, that's coming from the man who's making all this possible.:) I totally forgot that it's already capable, since I don't really use that feature.
     
  10. mintydave

    mintydave New Member

    Not being a DB Admin I'll have to dig in and research, but I'm sure I can figure it out. Thanks for pointing my sail on this.


    Dave.
    Deconn Technical Services
     
  11. mintydave

    mintydave New Member

    I now have my second new snake in production after getting the whitelisting working.

    When I set up the original user accounts in barawa in order to manage the transports for each domain, I set about half of the admin account to receive the quarantine report and the other half to not receive them.

    So this morning, I woke up to 400 quarantine reports for each of the domain admins' respective domains. So I went into baruwa->settings->accounts and edited each account and unchecked "send report"

    Those admins set to not receive are still receiving the quarantine reports. Am I missing where to actually disable the quarantine report?


    Thank you for your assistance,

    Dave
    Deconn Technical Services
     
  12. Rocky

    Rocky New Member

    Ok, as a temp, open cron and comment out the send pdf reports job. That should prevent the system from sending anymore reports.

    I'll have Andrew follow up with this.
     
  13. topdog

    topdog HowtoForge Supporter

    If the send quarantine report option is off, a report will definitely not be sent out.
     
  14. topdog

    topdog HowtoForge Supporter

    You may have this problem here http://www.howtoforge.com/forums/showthread.php?p=247474#post247474
     
  15. mintydave

    mintydave New Member

    Rocky,

    I have a conern that on my spamsnake the whitelist managed in Barawa is not behaving as you state above.

    I have the following IP in my whitelist:
    Code:
    46     Any address     216.241.219.0
    Then I get this in my syslog:
    Code:
    Jan 14 14:36:09 curve postfix/smtpd[17734]: NOQUEUE: reject: RCPT from mailout-01.cobaltgroup.com[216.241.219.148]: 450 4.7.1 <fleetguy@carlburger.com>: Recipient address rejected: SPF-Result=mailout-01-n2.tuk.cobaltgroup.com: 'SERVFAIL' error on DNS 'SPF' lookup of 'mailout-01-n2.tuk.cobaltgroup.com'; from=<214425@crm.cobaltgroup.com> to=<nickd@carlburger.com> proto=ESMTP helo=<mailout-01-n2.tuk.cobaltgroup.com>
    Just to cover myself, I researched how the SPF actually works and it appears that there should be an SPF record for "mailout-01-n2.tuk.cobaltgroup.com", but there is not, so that is why the server is failing on the DNS lookup.

    However, I would prefer that the whitelist just bypasses all the checks and let's the messages through the snake.

    Here is my main.cf:

    Code:
    # See /usr/share/postfix/main.cf.dist for a commented, more complete version
    
    
    # Debian specific:  Specifying a file name will cause the first
    # line of that file to be used as the name.  The Debian default
    # is /etc/mailname.
    #myorigin = /etc/mailname
    
    smtpd_banner = ESMTP SpamSnake
    biff = no
    
    # appending .domain is the MUA's job.
    append_dot_mydomain = no
    
    # Uncomment the next line to generate "delayed mail" warnings
    #delay_warning_time = 4h
    
    readme_directory = /usr/share/doc/postfix
    
    # TLS parameters
    smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
    smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
    smtpd_use_tls=yes
    smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
    smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
    
    # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
    # information on enabling SSL in the smtp client.
    
    myhostname = curve.dsh.com
    alias_maps = hash:/etc/aliases
    alias_database = hash:/etc/aliases
    myorigin = dsh.com
    mydestination =
    relayhost =
    mynetworks = 127.0.0.0/8, 192.168.0.0/16, 10.10.0.0/24
    mailbox_command = procmail -a "$EXTENSION"
    mailbox_size_limit = 0
    recipient_delimiter = +
    inet_interfaces = all
    html_directory = /usr/share/doc/postfix/html
    message_size_limit = 20485760
    local_transport = error:No local mail delivery
    local_recipient_maps =
    relay_domains = mysql:/etc/postfix/mysql-relay_domains.cf
    relay_recipient_maps = mysql:/etc/postfix/mysql-relay_recipients.cf
    transport_maps = mysql:/etc/postfix/mysql-transports.cf
    virtual_alias_maps = hash:/etc/postfix/virtual
    disable_vrfy_command = yes
    strict_rfc821_envelopes = no
    smtpd_delay_reject = yes
    smtpd_recipient_limit = 100
    smtpd_helo_required = yes
    smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, permit
    smtpd_helo_restrictions = permit_sasl_authenticated, permit_mynetworks, permit
    smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_non_fqdn_sender, permit
    smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unknown_recipient_domain, reject_unauth_destination, whitelist_policy, grey_policy, rbl_policy, spf_policy, permit
    smtpd_data_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_pipelining
    smtpd_restriction_classes = spf_policy, rbl_policy, grey_policy, whitelist_policy
    spf_policy = check_policy_service unix:private/policy
    rbl_policy = reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net
    grey_policy = check_policy_service unix:private/greyfix
    whitelist_policy = check_sender_access mysql:/etc/postfix/mysql-global_whitelist.cf
    header_checks = regexp:/etc/postfix/header_checks
    
    As always, your input and assistance are greatly appreciated.


    Dave.
    Deconn Technical Services
     
  16. Rocky

    Rocky New Member

    What happens if you whitelist this ip 216.241.219.148?
     
  17. mintydave

    mintydave New Member

    at 21:24 added:
    Code:
    52   any     216.241.219.148
    
    I also deleted the 216.241.219.0 entry as noted in the previous post.

    I get the same result.

    Code:
    Jan 15 21:25:24 curve postfix/smtpd[3078]: NOQUEUE: reject: RCPT from mailout-01.cobaltgroup.com[216.241.219.148]: 450 4.7.1 <fleetguy@carlburger.com>: Recipient address rejected: SPF-Result=mailout-01-n2.tuk.cobaltgroup.com: 'SERVFAIL' error on DNS 'SPF' lookup of 'mailout-01-n2.tuk.cobaltgroup.com'; from=<214425@crm.cobaltgroup.com> to=<fleetguy@carlburger.com> proto=ESMTP helo=<mailout-01-n2.tuk.cobaltgroup.com>

    Just as a sanity check, I added a /32 whitelist entry for another host and sent mail into the snake and it got hit by the greylist instantly. I used Outlook Express from a Windows host and simply used the spamsnake as the SMTP server to a known good/deliverable address.

    After the greylist activity completed, Baruwa shows the message whitelisted:
    Code:
    2011-01-15 21:22:03  ___@carlburger.com  _____@cwssandiego.com  test 301  1.8 KB 0.0  WL


    Dave.
    Deconn Technical Services
     
    Last edited: Jan 16, 2011
  18. Rocky

    Rocky New Member

    Hmm, I'll have to look into that. Have you tried using the dns name?
     
  19. mintydave

    mintydave New Member

    I assume you mean the hostname of the sending email server. "mailout-01.cobaltgroup.com" in this case.

    Done.

    No change.

    In the mean time, is there a way I can disable SPF just so I can get these emails flowing?


    Dave.
    Deconn Technical Services.
     
  20. Rocky

    Rocky New Member

    The guide has been updated with the fix for this problem.
     

Share This Page