this problem was present on my up to date system after following the ispconfig3 guide for ubuntu 9.10 and google says some debian users had a similar problem too.(bug 573314)
If you want to block smtp brute force attempts you have to enable the sasl filter in jail.conf and change failregex in /etc/fail2ban/filter.d/sasl.conf to
failregex = (?i): warning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed
To test it:
fail2ban-regex /var/log/mail.log /etc/fail2ban/filter.d/sasl.conf
This is a "works for me solution"
Thanks for the great guide, Ispconfig makes things so easy....