Thanks Falko for your suggestion,
No it's not on the whitelist.
But I started thinking of another route as the attack is a very slow one + the fact that a properly written daemon ( fail2ban ) wouldn't parse the complete logs ( as to resource intensive ).
So I asked an their mailing list if there would be a time setting the daemon uses to parse logs back in time counting offending IP's.
Lo and behold there is....
As reference for other users ...
The default is at 10 min.
the parameter is called "findtime = 600" ( time in seconds ).
and should go in jail.local under [DEFAULT]
I have set it now at 4 hours. > 14400 sec
ignoreip = 127.0.0.1
destemail = *****@*****
maxretry = 3
bantime = 86400
findtime = 14400
backend = polling
banaction = iptables-multiport
mta = sendmail
protocol = tcp
my 5 cents