I activated apache badbots on an Ubuntu 20.04 but it didn't block anything. It works with the following regex: Code: failregex = (?i)<HOST> -.*"(GET|POST|HEAD|OPTIONS).*HTTP.*(?:%(badbotscustom)s|%(badbots)s).*"$ Edit: you have to add the client log dir to error and access lines in paths-common.conf: Code: apache_error_log = /var/log/apache2/*error.log /var/www/clients/client*/web*/log/error.log apache_access_log = /var/log/apache2/*access.log /var/www/clients/client*/web*/log/access.log
There is more to this... 1) This mod should really be in paths-overides.local rather than in the default paths-common.conf. 2) /var/log/apache2/access.log is loaded, as is other_vhosts_access.log. And jail apache-badbots is started. But none of the error logs are getting loaded. The apache-badbots jail doesn't use the apache_error_log logpath. But the idea is still valid that there are many more error.log files, and other jails do use that logpath. 3) For anyone not using the default document root web path, that specific path "/var/www/clients/client*/web*/log/access.log" needs to be modified to point to your own /whatever/html/all_clients/client*/web*/log folder. (Mine is clouds/cloud*/site*/log) 4) This might be even more config-specific: I don't have logs all lumped together, and I don't remember at the moment where I set it like this. Vhosts can/do have their own folder under .../log. I'm just finding this out and now I'm wondering what log info we're missing by having log files in so many places. To be clear, here is the path to the access.log for a vhost subdomain: /var/www/clients/client_1/web_1/log/vhostname/access.log ... and that's a symbolic link to a date-stamped file. The error.log file is Not symlinked. There are a couple ways to get a more complete data source of logs. First, try something like this to find all error.log and access.log files: Code: $logpath=/var/www/clients/ (find $logpath -maxdepth 6 -type d | grep \/log | awk '{print $1"/error.log"}') | more Rather than piping to more, build a file, then get paths-overides.local to Include that file. But that quickly results in a huge list of files and will certainly be a performance issue. So the second approach, and I think best, is to get apache to log ALL errors for ALL sites into a single file, and ALL access for ALL sites into a single file that is intended only for fail2ban. This would be in addition to all of the individual files … I don't know how to setup multiple log files for a single site. That single file should be limited to a small size, just big enough to ensure fail2ban can process through it before it gets pruned. Again, I don't know the ramifications of having a dynamic file under fail2ban, but you get the idea. So, if you're concerned about getting more data into fail2ban, the OP suggestion is good, but to be effective the idea needs to be expanded. I have to believe that somewhere in the fail2ban world this has been discussed/done already.
Additional note: I just re-stumbled upon /var/log/ispconfig/httpd which has a duplicate of the logs under the docroot tree. It looks like it's just another option - and again is probably only there because of some setting I checked. I'll continue to look for an elegant solution here unless/until someone else speaks up.
the logs in /var/log/ispconfig/httpd are not duplicates, if you double check your sites /log folders, you'll see that the entries in /var/www/clients/client##/web##/log/ are symlinks to the actual files in /var/log/ispconfig/httpd/<domain.tld>/ this is so the log files are all in one location, which can be a mounted partition to allow for expanding the storage space if necessary, whilst allowing site owners to access their own sites log files without having to call the hosting supplier and requesting the log files are sent to them.
Ugh! They're not Symlinks/softlinks, they're hardlinks! The only way to see this is to look at the inode of the log folder. OK, that makes sense. So my last note is invalid but the general premise is still valid. Thanks for that tip.
yep, sorry, you're right, they not symlinks. was answering from memory, and with the number of symlinks for various things on the servers, it's hard to keep track of what's what. bit easier now, when i can actually see the bash shell... the log folders are actually bind mounts. and you can see what they all are, they're all listed in /etc/fstab on each server. and perhaps the /var/log/ispconfig/httpd/*/error.log or /var/log/ispconfig/httpd/*/access.log would be the better path to use. that path will always exist regardless of any docroot customization an admin uses. And on very rare occasions, after a reboot, the bind mount isn't always created/mounted and the log files aren't visible in the /var/www/clients/client*/web*/log path, but will still be accessible in /var/log/ispconfig/httpd/*/ only seen this happen when doing some maintenance/reconfiguration in testing though, so don't expect it to be an issue on a stable production server.
OK, so let's say /var/log/ispconfig/httpd/*/error.log or /var/log/ispconfig/httpd/*/access.log are canonical paths. I'm concerned that Fail2Ban would be overloaded with the volume of files. I'm thinking it would be ideal for each site to in their own mysite/access.log AND for the same detail to get appended to a allsites_access.log. This can be accomplished with more than one CustomLog entry - or it might already be in place with a high level CustomLog and then individual CustomLog directives in each VHost. I don't know yet but will look as time permits. After all of this it would be funny if /var/log/apache2/access|error.log already has all of the data that's being discussed here.
