Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > ISPConfig 3 > General

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 29th August 2012, 03:32
scottrill2 scottrill2 is offline
HowtoForge Supporter
 
Join Date: Dec 2009
Posts: 93
Thanks: 21
Thanked 2 Times in 1 Post
Default Ban bots (ISPConfig 3)

Hello all,

I am letting my brother in law run a non-profit club website for US military folk. His site has been slowing. From the logs one of the main culprits is a bot/spider from China called Baidu. I have blocked it using robots.txt but it doesn't heed the disallow.

Anyway, he has several gigabytes usage per month just from Baidu and some Russian bot. Oddly enough, all these bots from overseas combined don't take up the same bandwidth as this Baidu and russian bot are using lol

His military folks could very well be overseas but they do NOT need to find the site via search engine. And since my server is mainly family and friends site I want to block out these bots server wide, rather than altering .htaccess files on each site.

So I read this page:

http://searchenginewatch.com/article...-the-Parasites

The article mentions adding the following info to my httpd.conf file which I can find simple enough.

Code:
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bots
SetEnvIfNoCase User-Agent "^Sogou" bad_bots
SetEnvIf Remote_Addr "212\.100\.254\.105" bad_bot

Order allow,deny
Allow from all

Deny from env=bad_bots
But httpd.conf is currently empty. Is there a different file I should put the info into for ISPConfig 3?

Are there any better suggestions for banning these bots?

Thanks as always folks,

Scott
Reply With Quote
Sponsored Links
  #2  
Old 29th August 2012, 08:09
fbartels fbartels is offline
Junior Member
 
Join Date: Feb 2012
Location: Hanover, Germany
Posts: 21
Thanks: 0
Thanked 4 Times in 4 Posts
Default

The httpd.conf is only used in a stock apache installation, in most cases this default configuration file is even overridden by distribution dependent config files (for example in Debian). If you want your code to be executed in every site you also have to add it to every vhost.

The best way to do this is to create a customer vhost.conf.master (copy it from /usr/local/ispconfig/server/conf to /usr/local/ispconfig/server/conf-custom and do your changes).

After that you have to manually "resave" every vhost for the file to be recreated. Else you could use the remote api to recreate the file for every vhost. The following post suggests the needed steps:
http://www.howtoforge.com/forums/showthread.php?t=56195

Regards Felix

Overthinking this, there is a way simpler way to achieve this. As the new information is static (doenst need to be changed for every host) you could also create a file inside /etc/apache/conf.d with the needed input. This also affects all vhosts.

Here is the needed layout of the file:
Quote:
<Directory /var/www/>
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bots
SetEnvIfNoCase User-Agent "^Sogou" bad_bots
SetEnvIf Remote_Addr "212\.100\.254\.105" bad_bots

<Files *>
Order Allow,Deny
Allow from all
Deny from env=bad_bots
</Files>
</Directory>

Last edited by fbartels; 29th August 2012 at 08:30. Reason: Easier solution
Reply With Quote
  #3  
Old 31st August 2012, 02:13
scottrill2 scottrill2 is offline
HowtoForge Supporter
 
Join Date: Dec 2009
Posts: 93
Thanks: 21
Thanked 2 Times in 1 Post
Default Thanks

Hello fbartels,

Thanks for the reply. Two quick questions:

1. Can I call this file anything I want? i.e. (banned-bots.conf)

2. Do I have to tell any other apache file to include this file or will it be automatic?


Thasnks again for the reply.

Scott
Reply With Quote
  #4  
Old 31st August 2012, 07:45
fbartels fbartels is offline
Junior Member
 
Join Date: Feb 2012
Location: Hanover, Germany
Posts: 21
Thanks: 0
Thanked 4 Times in 4 Posts
 
Default

When you have a look into /etc/apache2/conf.d you will already find other files like "charset" and "phpmyadmin.conf", so you can basically name it how you want it.

There is also no extra include necessary as this is already done in apache.conf.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Relay access denied when using SMTP to external recipients Kamran Shah Installation/Configuration 111 14th May 2014 19:10
Control Panel - E-mail Tab is empty domi-nik General 15 14th April 2011 19:17
chroot SSH IspConfig2 kextra1 General 4 20th January 2011 19:41
Fail2ban attacker Toucan General 2 6th October 2010 00:00
SP-Server Setup - Ubuntu 5.10 "Breezy Badger" - Page 6 (changes) LuisC-SM HOWTO-Related Questions 0 21st April 2006 16:16


All times are GMT +2. The time now is 06:02.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.