Got the problem nailed !
In case anyone else suffers the same problem, here are the details.
In Jail.conf there is a "backend" option, this can be set to "gamin", "polling" or "auto" which chooses whatever method is available.
If I set this to "polling" Fail2Ban started working but "auto" or "gamin"! it didn't.
I could not find why gamin was not working and could find little information on testing/debugging it.
In the end, out of desperation I rebooted the server.
And the problem went away.