HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials

HowtoForge Forums | HowtoForge - Linux Howtos and Tutorials (http://www.howtoforge.com/forums/index.php)
-   Tips/Tricks/Mods (http://www.howtoforge.com/forums/forumdisplay.php?f=29)
-   -   How to prevent search engine from indexing of Website auto alias domains (http://www.howtoforge.com/forums/showthread.php?t=61790)

SupuS 10th May 2013 15:08

How to prevent search engine from indexing of Website auto alias domains
 
Hi,

I use Website auto alias in ISPConfig and I've found that search engines are indexing domain aliases. This is not desired status. So I decided to use X-Robots-Tag HTTP header (more info on developers.google.com for example). When you have in Website auto alias for example:

[client_id].test.domain.ltd

so you can put this directive to apache directive settings:

Code:

# Prevent search engine indexing on temporary domain sites:
<IfModule setenvif_module>
  <IfModule headers_module>
    SetEnvIf Host test\.domain\.ltd$ temporary_domain
    Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
  </IfModule>
</IfModule>

Apache mod headers and mod setenvif has to be activated for properly work.

It would be nice if this functioin was included in ISPConfig directly as optional ;)

SupuS

mattltm 24th June 2013 14:02

I've added this to System >> Directive Snippets

Is that the right place?

SupuS 24th June 2013 14:08

Quote:

Originally Posted by mattltm (Post 299022)
I've added this to System >> Directive Snippets

Is that the right place?

Yes. You have to create directive snippet for Apache and than choose this directive for each web on which you want prevent indexing by crawlers.

mattltm 24th June 2013 14:19

Thanks SupuS :)

Thats the part I was missing. Enabling it in the site config.

Looks like it's working but on checking the headers, I don't get the X-Robots-Tag.

I added a new line to the directive like this to test it:

Code:

Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
Header set MyHeader "Hello. It took %D microseconds for Apache to serve this request."

The headers show:
Code:

MyHeader:        Hello. It took D=4463 microseconds for Apache to serve this request.
So the directive is loaded. Should I be able to see the X-Robots-Tag in the headers?

SupuS 24th June 2013 14:52

Please check your setting in Server configuration => Website auto alias. You have to specify on which URL this directive will be launched. If you use for example:

[website_domain].test.testingdomain.tld

you have to set:

SetEnvIf Host test\.testingdomain\.tld$ temporary_domain

mattltm 24th June 2013 14:56

Perfect.

100% working.

:D


All times are GMT +2. The time now is 02:00.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.