you have in your jail.local
enabled = true
port = ftp
filter = proftpd
logpath = /var/log/auth.log
failregex = proftpd: \(pam_unix\) authentication failure; .* rhost=<HOST>
maxretry = 5
this means that fail2ban looks into
for something that begins with
proftpd: pam_unix *something* authentication failure; *something* rhost=<*IP*>
if it found this 5 times it will block *IP* from that line in log file.
if proftpd creates failed authentication logs in /var/log/auth.log that looks like failregex line then it will block that IP.
here is example from my log:
Sep 6 19:57:15 my_domain proftpd: pam_unix(proftpd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser=backup rhost=::ffff:220.127.116.11 user=backup
and fail2ban response in /var/log/fail2ban.log
2009-09-06 19:58:03,248 fail2ban.actions: WARNING [proftpd] Ban 18.104.22.168