Fail2ban - process own logs

Discussion in 'Installation/Configuration' started by wojtekgoral, Apr 15, 2019.

  1. wojtekgoral

    wojtekgoral New Member

    Hi
    Can anyone help in creating a proper regular expression for such entries for fail2ban please ?

    I created this regex, but it does not work :(

    failregex = .* fail2ban.filter \[(.*)\]: INFO \[(.*)\] Found <HOST>
    I need to match this string:

    2019-04-15 11:59:07,255 fail2ban.filter [13677]: INFO [sasl] Found 185.234.216.104
    2019-04-15 12:01:46,062 fail2ban.filter [13677]: INFO [sasl] Found 45.227.253.98
    2019-04-15 12:01:53,479 fail2ban.filter [13677]: INFO [sasl] Found 45.227.253.98
    2019-04-15 12:06:16,971 fail2ban.filter [13677]: INFO [pure-ftpd] Found 123.133.161.68
    2019-04-15 12:06:24,690 fail2ban.filter [13677]: INFO [pure-ftpd] Found 123.133.161.68

    Also please kindly advise how to convert NOTICE to WARNING according such log entries. Do I have to change loglevel and to which option ?
    2019-04-15 12:22:10,168 fail2ban.actions [13677]: NOTICE [sasl] Ban 45.227.253.98
    2019-04-15 12:06:25,348 fail2ban.actions [13677]: NOTICE [pure-ftpd] Ban 123.133.161.68
     
  2. Jesse Norell

    Jesse Norell Well-Known Member

    You are trying to match fail2ban's own log entries, which is not normally what you want to do - those items have already been matched by other fail2ban jails. If you're sure that's what you want to do, you are probably trying to recreate the functionality of the recidive jail, and I would recommend either using it as your starting point, or simply use that jail directly if it suits you. Read the warnings preceding the [recidive] jail in /etc/fail2ban/jail.conf and see /etc/fail2ban/filter.d/recidive.conf:
    Code:
    # Fail2Ban filter for repeat bans
    #
    # This filter monitors the fail2ban log file, and enables you to add long 
    # time bans for ip addresses that get banned by fail2ban multiple times.
    #
    # Reasons to use this: block very persistent attackers for a longer time, 
    # stop receiving email notifications about the same attacker over and 
    # over again.
    #
    # This jail is only useful if you set the 'findtime' and 'bantime' parameters 
    # in jail.conf to a higher value than the other jails. Also, this jail has its
    # drawbacks, namely in that it works only with iptables, or if you use a 
    # different blocking mechanism for this jail versus others (e.g. hostsdeny 
    # for most jails, and shorewall for this one).
    
    [INCLUDES]
    
    # Read common prefixes. If any customizations available -- read them from
    # common.local
    before = common.conf
    
    [Definition]
    
    _daemon = fail2ban\.actions\s*
    
    # The name of the jail that this filter is used for. In jail.conf, name the 
    # jail using this filter 'recidive', or change this line!
    _jailname = recidive
    
    failregex = ^(%(__prefix_line)s| %(_daemon)s%(__pid_re)s?:\s+)NOTICE\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+<HOST>\s*$
    
    ignoreregex = 
    
    [Init]
    
    journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=5
    
    # Author: Tom Hendrikx, modifications by Amir Caspi 
     
  3. wojtekgoral

    wojtekgoral New Member

    Hi
    Thanks for an advice. I've set up fail2ban , but I get 'already banned' in var/logs/fail2ban.logs Could You please help and tell me how to avoid it ?
    I've set up 10 mins ban for postfix-sasl , so 'repeat' jail can see the same IP again in logs, but it comes from different jail. I have a jail rule for actionban to avoid double bans etc, but how to preven postfix jail before checking the IP already banned by 'repeat' jail ?
    Code:
    2019-04-17 16:22:07,013 fail2ban.filter         [4036]: INFO    [repeat] Found 45.227.253.98
    2019-04-17 16:22:07,441 fail2ban.actions        [4036]: NOTICE  [repeat] 45.227.253.98 already banned
    2019-04-17 16:22:42,415 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.234.218.176
    2019-04-17 16:22:43,267 fail2ban.actions        [4036]: NOTICE  [postfix-sasl] Ban 185.234.218.176
    2019-04-17 16:22:43,269 fail2ban.filter         [4036]: INFO    [repeat] Found 185.234.218.176
    2019-04-17 16:22:43,482 fail2ban.actions        [4036]: NOTICE  [repeat] 185.234.218.176 already banned
    2019-04-17 16:23:05,506 fail2ban.actions        [4036]: NOTICE  [postfix-sasl] Unban 185.222.209.66
    2019-04-17 16:25:52,224 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.211.245.170
    2019-04-17 16:26:00,269 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.211.245.170
    2019-04-17 16:29:04,004 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.222.209.66
    2019-04-17 16:29:13,483 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.222.209.66
    2019-04-17 16:32:07,325 fail2ban.actions        [4036]: NOTICE  [postfix-sasl] Unban 45.227.253.98
    2019-04-17 16:32:43,576 fail2ban.actions        [4036]: NOTICE  [postfix-sasl] Unban 185.234.218.176
    2019-04-17 16:33:24,116 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.234.218.239
    2019-04-17 16:33:30,396 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.234.218.239
    2019-04-17 16:33:30,840 fail2ban.actions        [4036]: NOTICE  [postfix-sasl] Ban 185.234.218.239
    2019-04-17 16:33:30,842 fail2ban.filter         [4036]: INFO    [repeat] Found 185.234.218.239
    2019-04-17 16:33:31,216 fail2ban.actions        [4036]: NOTICE  [repeat] 185.234.218.239 already banned
    2019-04-17 16:33:37,314 fail2ban.filter         [4036]: INFO    [postfix-sasl] Found 185.234.218.176
    
    If it helps I will publish my whole config.
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

  5. wojtekgoral

    wojtekgoral New Member

    I cannot find the other answer to this problem than changing short term jails into long term and it perhaps will decrease qty of 'already banned' errors. Do You see any different solution ?

    As I see in iptables there is the following entry with my ip and I still can access ssh, smtp/imap, www, ftp.
    -A f2b-repeat -s 31.179.81.152/32 -j DROP

    I am banned by fail2ban and I see the following in logs:
    2019-04-18 02:12:27,521 fail2ban.actions [23606]: NOTICE [pureftpd] Ban 31.179.81.152
    2019-04-18 02:12:28,501 fail2ban.actions [23606]: NOTICE [repeat] Ban 31.179.81.152
    2019-04-18 03:34:50,369 fail2ban.actions [3496]: NOTICE [repeat] Ban 31.179.81.152

    Below is my configuration:
    /etc/fail2ban/jail.local :
    Code:
    [DEFAULT]
    bantime=3600
    port=0:65535
    
    [pureftpd]
    enabled=true
    port=ftp
    filter=pure-ftpd
    logpath=/var/log/syslog
    findtime=7200
    bantime = 600
    maxretry=3
    
    [ispconfig]
    enabled  = true
    port     = 8080
    filter   = ispconfig
    logpath  = /var/log/ispconfig/auth.log
    bantime = 600
    findtime = 7200
    maxretry = 3
    
    [postfix-sasl]
    enabled  = true
    port     = smtp
    filter   = postfix-sasl
    logpath  = /var/log/mail.log
    maxretry = 4
    findtime = 7200
    bantime = 600
    ignoreip = 31.179.81.152
    
    [repeat]
    enabled  = true
    filter   = repeat
    logpath  = /var/log/fail2ban.log
    action = repeat[name=repeat]
    bantime  = 31536000   ; 1 year
    findtime = 31536000   ; 1 year
    maxretry=3
    
    /etc/fail2ban/filter.d/repeat.conf :
    Code:
    # Fail2Ban filter for repeat bans
    #
    # This filter monitors the fail2ban log file, and enables you to add long
    # time bans for ip addresses that get banned by fail2ban multiple times.
    #
    # Reasons to use this: block very persistent attackers for a longer time,
    # stop receiving email notifications about the same attacker over and
    # over again.
    #
    # This jail is only useful if you set the 'findtime' and 'bantime' parameters
    # in jail.conf to a higher value than the other jails. Also, this jail has its
    # drawbacks, namely in that it works only with iptables, or if you use a
    # different blocking mechanism for this jail versus others (e.g. hostsdeny
    # for most jails, and shorewall for this one).
    
    [INCLUDES]
    
    # Read common prefixes. If any customizations available -- read them from
    # common.local
    before = common.conf
    
    [Definition]
    
    _daemon = fail2ban\.actions\s*
    
    # The name of the jail that this filter is used for. In jail.conf, name the
    # jail using this filter 'recidive', or change this line!
    _jailname = repeat
    failregex = ^(%(__prefix_line)s| %(_daemon)s%(__pid_re)s?:\s+)NOTICE\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+<HOST>\s*$
    # failregex = ^%(__prefix_line)s: NOTICE /[w+/] Ban <HOST>
    ignoreregex =
    
    [Init]
    
    journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=5
    
    # Author: Tom Hendrikx, modifications by Amir Caspi
    
    /etc/fail2ban/action.d/repeat.conf :
    Code:
    # Fail2Ban configuration file
    #
    # Author: Cyril Jaquier
    # Modified by Yaroslav Halchenko for multiport banning
    #
    
    [INCLUDES]
    
    before = iptables-common.conf
    
    [Definition]
    
    # Option:  actionstart
    # Notes.:  command executed once at the start of Fail2Ban.
    # Values:  CMD
    #
    actionstart = <iptables> -N f2b-<name>
                  <iptables> -A f2b-<name> -j <returntype>
                  <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
                  # Load local list of offenders
                  if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do <iptables> -I f2b-<name> 1 -s $IP -j DROP; done; fi
                  #cat /etc/fail2ban/ip.blacklist | grep -v ^\s*#|awk '{print $1}' | while read IP; do <iptables> -I f2b-<name> 1 -s $IP -j DROP; done
    
    # Option:  actionstop
    # Notes.:  command executed once at the end of Fail2Ban
    # Values:  CMD
    #
    actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
                 <iptables> -F f2b-<name>
                 <iptables> -X f2b-<name>
    
    # Option:  actioncheck
    # Notes.:  command executed once before each actionban command
    # Values:  CMD
    #
    actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
    
    # Option:  actionban
    # Notes.:  command executed when banning an IP. Take care that the
    #          command is executed with Fail2Ban user rights.
    # Tags:    See jail.conf(5) man page
    # Values:  CMD
    #
    # actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
    
    actionban = if ! <iptables> -C f2b-<name> -s <ip> -j DROP; then <iptables> -I f2b-<name> 1 -s <ip> -j DROP; fi
                # first command line: Check existence of an 'ip' in chain and if not - inserts it into this chain
                # second command line: Check if there is certain 'ip, repeat' string in /etc/fail2ban/ip.blacklist and if not - append it to /etc/fail2ban/ip.blacklist
                if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi
    
    
    # Option:  actionunban
    # Notes.:  command executed when unbanning an IP. Take care that the
    #          command is executed with Fail2Ban user rights.
    # Tags:    See jail.conf(5) man page
    # Values:  CMD
    #
    
    #actionunban = <iptables> -D f2b-<name> -s <ip> -j <blocktype>
    
    actionunban = # Do nothing becasuse their IP is in the blocklist file
    
    # To manually unban from the ip blocklist file run this command:
    # Be warned that if the ip is in log rotated files it must be whitelisted
    #
    # sed -i ā€š/^/dā€™ /etc/fail2ban/ip.blocklist.repeatoffender
    #
    
    [Init]
    
    
    Please help ! :(
     
    Last edited: Apr 18, 2019
  6. Jesse Norell

    Jesse Norell Well-Known Member

    I don't believe you can, short of changing fail2ban source not to log the message, as it's completely normal. Is this the only issue you have, or does one/some of your jails not work?
     
  7. wojtekgoral

    wojtekgoral New Member

    Seems my filter.d\repeat.conf is still wrong. I think failregex should recognize lines similar to this one:
    2019-04-17 09:19:03,941 fail2ban.actions [4036]: NOTICE [postfix-sasl] Ban 185.222.209.66
    (I use jails for postfix-sasl, ssh etc)
    and ignore:
    2019-04-22 08:21:38,230 fail2ban.actions [3496]: NOTICE [repeat] Ban 185.211.245.198
    Could You help in creating regular expressions please ?
     
  8. Jesse Norell

    Jesse Norell Well-Known Member

    Other than the jail name (postfix-sasl vs. repeat), those are formatted exactly the same, are they not? You don't need to be concerned with matching the 'repeat' jail there - by the time an IP shows up for the 'repeat' jail, they have already been firewalled off for an entire year per your jail config, so just ignore that one extra match. Am I missing something?
     
  9. wojtekgoral

    wojtekgoral New Member

    Thanks. I was able to set it up completely and it works perfect. If anyone needs working config - let me know and I will publish it.
     

Share This Page