fail2ban apache-badbots doesn't work on ubuntu 20.04

Discussion in 'Tips/Tricks/Mods' started by gb78, Oct 26, 2020.

  1. gb78

    gb78 New Member

    I activated apache badbots on an Ubuntu 20.04 but it didn't block anything.
    It works with the following regex:
    failregex = (?i)<HOST> -.*"(GET|POST|HEAD|OPTIONS).*HTTP.*(?:%(badbotscustom)s|%(badbots)s).*"$
    Edit: you have to add the client log dir to error and access lines in paths-common.conf:
    apache_error_log = /var/log/apache2/*error.log
    apache_access_log = /var/log/apache2/*access.log
    Jesse Norell likes this.
  2. TonyG

    TonyG Active Member

    There is more to this...
    This mod should really be in paths-overides.local rather than in the default paths-common.conf.
    /var/log/apache2/access.log is loaded, as is other_vhosts_access.log. And jail apache-badbots is started. But none of the error logs are getting loaded. The apache-badbots jail doesn't use the apache_error_log logpath. But the idea is still valid that there are many more error.log files, and other jails do use that logpath.
    For anyone not using the default document root web path, that specific path "/var/www/clients/client*/web*/log/access.log" needs to be modified to point to your own /whatever/html/all_clients/client*/web*/log folder. (Mine is clouds/cloud*/site*/log)
    This might be even more config-specific: I don't have logs all lumped together, and I don't remember at the moment where I set it like this. Vhosts can/do have their own folder under .../log. I'm just finding this out and now I'm wondering what log info we're missing by having log files in so many places. To be clear, here is the path to the access.log for a vhost subdomain:
    ... and that's a symbolic link to a date-stamped file. The error.log file is Not symlinked.

    There are a couple ways to get a more complete data source of logs.

    First, try something like this to find all error.log and access.log files:
    (find $logpath -maxdepth 6 -type d | grep \/log | awk '{print $1"/error.log"}') | more
    Rather than piping to more, build a file, then get paths-overides.local to Include that file.
    But that quickly results in a huge list of files and will certainly be a performance issue.

    So the second approach, and I think best, is to get apache to log ALL errors for ALL sites into a single file, and ALL access for ALL sites into a single file that is intended only for fail2ban. This would be in addition to all of the individual files … I don't know how to setup multiple log files for a single site. That single file should be limited to a small size, just big enough to ensure fail2ban can process through it before it gets pruned. Again, I don't know the ramifications of having a dynamic file under fail2ban, but you get the idea.

    So, if you're concerned about getting more data into fail2ban, the OP suggestion is good, but to be effective the idea needs to be expanded. I have to believe that somewhere in the fail2ban world this has been discussed/done already.
  3. TonyG

    TonyG Active Member

    Additional note: I just re-stumbled upon /var/log/ispconfig/httpd which has a duplicate of the logs under the docroot tree. It looks like it's just another option - and again is probably only there because of some setting I checked. I'll continue to look for an elegant solution here unless/until someone else speaks up.
  4. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    the logs in /var/log/ispconfig/httpd are not duplicates, if you double check your sites /log folders, you'll see that the entries in /var/www/clients/client##/web##/log/ are symlinks to the actual files in /var/log/ispconfig/httpd/<domain.tld>/

    this is so the log files are all in one location, which can be a mounted partition to allow for expanding the storage space if necessary, whilst allowing site owners to access their own sites log files without having to call the hosting supplier and requesting the log files are sent to them.
  5. TonyG

    TonyG Active Member

    Ugh! They're not Symlinks/softlinks, they're hardlinks! The only way to see this is to look at the inode of the log folder. OK, that makes sense. So my last note is invalid but the general premise is still valid. Thanks for that tip.
  6. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    yep, sorry, you're right, they not symlinks. was answering from memory, and with the number of symlinks for various things on the servers, it's hard to keep track of what's what. bit easier now, when i can actually see the bash shell... ;)
    the log folders are actually bind mounts. and you can see what they all are, they're all listed in /etc/fstab on each server.

    and perhaps the /var/log/ispconfig/httpd/*/error.log or /var/log/ispconfig/httpd/*/access.log would be the better path to use. that path will always exist regardless of any docroot customization an admin uses.

    And on very rare occasions, after a reboot, the bind mount isn't always created/mounted and the log files aren't visible in the /var/www/clients/client*/web*/log path, but will still be accessible in /var/log/ispconfig/httpd/*/ only seen this happen when doing some maintenance/reconfiguration in testing though, so don't expect it to be an issue on a stable production server.
  7. TonyG

    TonyG Active Member

    OK, so let's say /var/log/ispconfig/httpd/*/error.log or /var/log/ispconfig/httpd/*/access.log are canonical paths. I'm concerned that Fail2Ban would be overloaded with the volume of files. I'm thinking it would be ideal for each site to in their own mysite/access.log AND for the same detail to get appended to a allsites_access.log. This can be accomplished with more than one CustomLog entry - or it might already be in place with a high level CustomLog and then individual CustomLog directives in each VHost. I don't know yet but will look as time permits. After all of this it would be funny if /var/log/apache2/access|error.log already has all of the data that's being discussed here.

Share This Page