HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   General (http://www.howtoforge.com/forums/forumdisplay.php?f=25)
-   -   Slowing Down Bots (http://www.howtoforge.com/forums/showthread.php?t=56130)

midcarolina 10th February 2012 19:47

Slowing Down Bots
 
Hi,

Really wasn't sure if I should post this topic in general Linux or ISPConfig 3 since it is the CPanel I use. Fundamentally it will probably be general Linux as I would like to implement this server-wide...not site by site.

Fedora 15 The Perfect Server x86_64 500GB HD Raid 5 3.2 Ghz I5 duo-core Intel and 16 GB Ram

My Access logs (httpd / Apache) are absolutely overwhelmed with GET requests from robots. It is actually quite unbelievable how often they visit (Numerous major search engines, but mostly Google, Bing, Baidu, and Yandex). It bogs down the server tremendously.

Since 80% of the sites are content driven (mostly WordPress) these Bots are pinged constantly.

Is there a way to force the bots to come only every say 8 hours? maybe 12?

Thanks for the direction.

falko 11th February 2012 13:42

http://www.htaccess-guide.com/blocki...-and-bad-bots/

midcarolina 11th February 2012 18:38

Hi Falko,

Thanks for this link. I couldn't really find one I was comfortable with and one thing I didn't think to address are "bad bots". I created a robots.txt file for time delay, but found that Google ignores this rule. They do follow the Disallow for a file path or out-right noindex, nofollow, but not time. This can only be adjusted in their webmaster tools (which I believe is crazy). Anyhow, I followed it anyway and the server load is dramatically different. Thanks again,

Mid Carolina

edge 11th February 2012 19:21

I think that for Google you can set this in the Google Webmasters.
http://www.google.com/webmasters/


All times are GMT +2. The time now is 20:26.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.