ISPConfig contains already a mechanism that blocks users automatically after a few logins, so what fail2ban does is already implemented in ISPConfig.
If you want to add a log file for denied logins to block on network level as well, then add code to the file /usr/local/ispconfig/interface/web/login/index.php in the same place where the internal ispconfig lock mechanism is implemented, you have to add just a simple fwrite to your log file in that place. Use /var/log/ispconfig/auth.log as log file name, ensure that this file is created in the installer with touch() and chowned to user and group ispconfig, otherwise you cant write to that file. You might want to log the successfull logins as well to that log. The third thing that would have to be implemented is a log rotation similar to the one of the cron.log in the cron_daily.php file in ispconfig.
Please dont use any exec, passthrus etc. commands in the interface.
PLease dont add a separate cronjob or server plugin or similar solution.