Thread: blocking bots
View Single Post
  #7  
Old 13th November 2012, 16:49
falko falko is offline
Super Moderator
  
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,665
Thanks: 1,896
Thanked 2,594 Times in 2,445 Posts
Default
Quote:
Originally Posted by blinky View Post
When I see an entry in my Apache logfile that says : GET /robots.txt" does that mean the robot has tried to do a search and then has recieved my robots.txt file?
Bots explicitly request that file to learn what URLs they are allowed to index.
__________________
Falko
--
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!

FB: http://www.facebook.com/howtoforge

nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote