Thread: blocking bots
View Single Post
  #5  
Old 12th November 2012, 16:29
blinky blinky is offline
Member
 
Join Date: Sep 2012
Posts: 34
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by falko View Post
You should be aware that this will also block the Google and BING bots...
Yes, I'm aware that is should stop ALL bots. And it does if they observe the rules in robots.txt. But if they don't, they'll keep knocking away with the zeal of a vaccuum cleaner salesman pounding on my front door.

Quick question you might know the answer to...

When I see an entry in my Apache logfile that says : GET /robots.txt" does that mean the robot has tried to do a search and then has recieved my robots.txt file? I guess what I'm really asking here is must a robot search at least once from 999.999.999.999 to recieve the robots.txt file after which searches from 999.999.999.999 will stop?
Reply With Quote