Go Back   HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials > ISPConfig 3 > Tips/Tricks/Mods

Do you like HowtoForge? Please consider supporting us by becoming a subscriber.
Reply
 
Thread Tools Display Modes
  #1  
Old 10th May 2013, 15:08
SupuS SupuS is offline
HowtoForge Supporter
 
Join Date: May 2006
Posts: 201
Thanks: 68
Thanked 13 Times in 11 Posts
Default How to prevent search engine from indexing of Website auto alias domains

Hi,

I use Website auto alias in ISPConfig and I've found that search engines are indexing domain aliases. This is not desired status. So I decided to use X-Robots-Tag HTTP header (more info on developers.google.com for example). When you have in Website auto alias for example:

[client_id].test.domain.ltd

so you can put this directive to apache directive settings:

Code:
# Prevent search engine indexing on temporary domain sites:
<IfModule setenvif_module>
  <IfModule headers_module>
    SetEnvIf Host test\.domain\.ltd$ temporary_domain 
    Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
  </IfModule>
</IfModule>
Apache mod headers and mod setenvif has to be activated for properly work.

It would be nice if this functioin was included in ISPConfig directly as optional

SupuS
Reply With Quote
The Following 2 Users Say Thank You to SupuS For This Useful Post:
mattltm (24th June 2013), till (11th May 2013)
Sponsored Links
  #2  
Old 24th June 2013, 14:02
mattltm mattltm is offline
Member
 
Join Date: Jun 2011
Posts: 59
Thanks: 11
Thanked 2 Times in 2 Posts
Default

I've added this to System >> Directive Snippets

Is that the right place?
Reply With Quote
  #3  
Old 24th June 2013, 14:08
SupuS SupuS is offline
HowtoForge Supporter
 
Join Date: May 2006
Posts: 201
Thanks: 68
Thanked 13 Times in 11 Posts
Default

Quote:
Originally Posted by mattltm View Post
I've added this to System >> Directive Snippets

Is that the right place?
Yes. You have to create directive snippet for Apache and than choose this directive for each web on which you want prevent indexing by crawlers.
Reply With Quote
  #4  
Old 24th June 2013, 14:19
mattltm mattltm is offline
Member
 
Join Date: Jun 2011
Posts: 59
Thanks: 11
Thanked 2 Times in 2 Posts
Default

Thanks SupuS

Thats the part I was missing. Enabling it in the site config.

Looks like it's working but on checking the headers, I don't get the X-Robots-Tag.

I added a new line to the directive like this to test it:

Code:
Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
Header set MyHeader "Hello. It took %D microseconds for Apache to serve this request."
The headers show:
Code:
MyHeader:	 Hello. It took D=4463 microseconds for Apache to serve this request.
So the directive is loaded. Should I be able to see the X-Robots-Tag in the headers?
Reply With Quote
  #5  
Old 24th June 2013, 14:52
SupuS SupuS is offline
HowtoForge Supporter
 
Join Date: May 2006
Posts: 201
Thanks: 68
Thanked 13 Times in 11 Posts
Default

Please check your setting in Server configuration => Website auto alias. You have to specify on which URL this directive will be launched. If you use for example:

[website_domain].test.testingdomain.tld

you have to set:

SetEnvIf Host test\.testingdomain\.tld$ temporary_domain
Reply With Quote
The Following User Says Thank You to SupuS For This Useful Post:
mattltm (24th June 2013)
  #6  
Old 24th June 2013, 14:56
mattltm mattltm is offline
Member
 
Join Date: Jun 2011
Posts: 59
Thanks: 11
Thanked 2 Times in 2 Posts
 
Default

Perfect.

100% working.

Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Alias Domains creating Trailing Slash Vamiris Installation/Configuration 1 3rd December 2012 09:55
howto Domain module, mail domains limit & mail alias unlimited ? radim_h Tips/Tricks/Mods 1 20th April 2011 11:24
alias domains vwpete Installation/Configuration 2 4th April 2011 12:23
Exim Gateway with mailwatch (Unable to receive emails) siul0_0 HOWTO-Related Questions 10 8th May 2009 23:00
postfix domains alias alxgomz Installation/Configuration 0 5th March 2007 12:53


All times are GMT +2. The time now is 06:31.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.