Thread: blocking bots
View Single Post
Old 13th November 2012, 17:49
falko falko is offline
Super Moderator
Join Date: Apr 2005
Location: Lüneburg, Germany
Posts: 41,701
Thanks: 1,900
Thanked 2,749 Times in 2,579 Posts

Originally Posted by blinky View Post
When I see an entry in my Apache logfile that says : GET /robots.txt" does that mean the robot has tried to do a search and then has recieved my robots.txt file?
Bots explicitly request that file to learn what URLs they are allowed to index.
Download the ISPConfig 3 Manual! | Check out the ISPConfig 3 Billing Module!


nginx-Webhosting: Timme Hosting | Follow me on:
Reply With Quote