Prev Previous Post   Next Post Next
Old 10th November 2012, 15:31
blinky blinky is offline
Join Date: Sep 2012
Posts: 34
Thanks: 0
Thanked 0 Times in 0 Posts
Default blocking bots

What's the best way to block bots from searching your website?

I have created a robots.txt file which looks like this:
User-agent: *
Disallow: /
Disallow: /cgi-bin/
I have included the following in my index.html file:
<meta name="robots" content="NOINDEX, NOFOLLOW">
And I have also included an .htaccess file in my root which looks like this:
SetEnvIfNoCase User-Agent "^Yandex*" bad_bot
Order Deny,Allow
Deny from env=bad_bot
Yet I'm still seeing entries in Apache's access.log:
Code: - - [10/Nov/2012:04:33:14 -0500] "GET /robots.txt HTTP/1.1" 200 324 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +" - - [10/Nov/2012:04:33:14 -0500] "GET /phpbb/search.php?search_id=active_topics&sid=3a033d745efebc4ace615dd64e8f63f7 HTTP/1.1" 200 3735 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +" - - [10/Nov/2012:04:33:17 -0500] "GET /phpbb/ucp.php?mode=login&sid=3a033d745efebc4ace615dd64e8f63f7 HTTP/1.1" 200 3513 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +" - - [10/Nov/2012:06:05:11 -0500] "GET /robots.txt HTTP/1.1" 200 368 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +" - - [10/Nov/2012:06:32:14 -0500] "GET /phpbb/index.php?sid=3a033d745efebc4ace615dd64e8f63f7 HTTP/1.1" 200 3908 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +" - - [10/Nov/2012:06:35:02 -0500] "GET /robots.txt HTTP/1.1" 200 331 "-" "Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2"
I have even included the IP address in my iinbound filter list on my router. (The fact that I see that address still listed in my Apache logs suggests (at least to me) that Yandex isn't coming from that address.

Thoughts anyone?

Last edited by blinky; 10th November 2012 at 23:43.
Reply With Quote
Sponsored Links


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Blocking Based on Country with GeoIP, xtables-addons, and iptables linus3x Installation/Configuration 2 5th October 2013 00:56
Ban bots (ISPConfig 3) scottrill2 General 3 31st August 2012 07:45
Slowing Down Bots midcarolina General 3 11th February 2012 19:21
blocking spam emails in ispconfig3 Mitz General 4 3rd February 2010 13:26
blocking top-level-domains? Libor Server Operation 1 13th October 2008 12:45

All times are GMT +2. The time now is 11:48.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.