#1  
Old 10th February 2012, 18:47
midcarolina midcarolina is offline
Senior Member
 
Join Date: Jul 2011
Posts: 254
Thanks: 1
Thanked 5 Times in 5 Posts
Default Slowing Down Bots

Hi,

Really wasn't sure if I should post this topic in general Linux or ISPConfig 3 since it is the CPanel I use. Fundamentally it will probably be general Linux as I would like to implement this server-wide...not site by site.

Fedora 15 The Perfect Server x86_64 500GB HD Raid 5 3.2 Ghz I5 duo-core Intel and 16 GB Ram

My Access logs (httpd / Apache) are absolutely overwhelmed with GET requests from robots. It is actually quite unbelievable how often they visit (Numerous major search engines, but mostly Google, Bing, Baidu, and Yandex). It bogs down the server tremendously.

Since 80% of the sites are content driven (mostly WordPress) these Bots are pinged constantly.

Is there a way to force the bots to come only every say 8 hours? maybe 12?

Thanks for the direction.
Reply With Quote
Sponsored Links
  #2  
Old 11th February 2012, 12:42
falko falko is offline
Super Moderator
 
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,741 Times in 2,575 Posts
Default

http://www.htaccess-guide.com/blocki...-and-bad-bots/
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote
  #3  
Old 11th February 2012, 17:38
midcarolina midcarolina is offline
Senior Member
 
Join Date: Jul 2011
Posts: 254
Thanks: 1
Thanked 5 Times in 5 Posts
Default

Hi Falko,

Thanks for this link. I couldn't really find one I was comfortable with and one thing I didn't think to address are "bad bots". I created a robots.txt file for time delay, but found that Google ignores this rule. They do follow the Disallow for a file path or out-right noindex, nofollow, but not time. This can only be adjusted in their webmaster tools (which I believe is crazy). Anyhow, I followed it anyway and the server load is dramatically different. Thanks again,

Mid Carolina
Reply With Quote
  #4  
Old 11th February 2012, 18:21
edge edge is offline
Moderator
 
Join Date: Dec 2005
Location: The Netherlands
Posts: 2,034
Thanks: 265
Thanked 152 Times in 132 Posts
 
Default

I think that for Google you can set this in the Google Webmasters.
http://www.google.com/webmasters/
__________________
Never execute code written on a Friday or a Monday.
Reply With Quote
Reply

Bookmarks

Tags
apache, httpd, robots

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Spam Attack slowing postfix azteca2137 Installation/Configuration 4 29th July 2010 06:07
server slowing down / apache2 / error 500 jeeva General 9 20th October 2009 20:11
Training SA & libwww-perl bots trashed Server Operation 1 29th May 2008 17:47
IPtables slowing down my proftpd server. Tohubohu Installation/Configuration 3 19th March 2007 12:47
IPtables slowing down my SSH login!? edge Server Operation 7 4th July 2006 13:56


All times are GMT +2. The time now is 12:30.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.