How to prevent search engine from indexing of Website auto alias domains

Discussion in 'Tips/Tricks/Mods' started by SupuS, May 10, 2013.

  1. SupuS

    SupuS HowtoForge Supporter

    Hi,

    I use Website auto alias in ISPConfig and I've found that search engines are indexing domain aliases. This is not desired status. So I decided to use X-Robots-Tag HTTP header (more info on developers.google.com for example). When you have in Website auto alias for example:

    [client_id].test.domain.ltd

    so you can put this directive to apache directive settings:

    Code:
    # Prevent search engine indexing on temporary domain sites:
    <IfModule setenvif_module>
      <IfModule headers_module>
        SetEnvIf Host test\.domain\.ltd$ temporary_domain 
        Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
      </IfModule>
    </IfModule>
    
    Apache mod headers and mod setenvif has to be activated for properly work.

    It would be nice if this functioin was included in ISPConfig directly as optional ;)

    SupuS
     
  2. mattltm

    mattltm Member

    I've added this to System >> Directive Snippets

    Is that the right place?
     
  3. SupuS

    SupuS HowtoForge Supporter

    Yes. You have to create directive snippet for Apache and than choose this directive for each web on which you want prevent indexing by crawlers.
     
  4. mattltm

    mattltm Member

    Thanks SupuS :)

    Thats the part I was missing. Enabling it in the site config.

    Looks like it's working but on checking the headers, I don't get the X-Robots-Tag.

    I added a new line to the directive like this to test it:

    Code:
    Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
    Header set MyHeader "Hello. It took %D microseconds for Apache to serve this request."
    
    The headers show:
    Code:
    MyHeader:	 Hello. It took D=4463 microseconds for Apache to serve this request.
    So the directive is loaded. Should I be able to see the X-Robots-Tag in the headers?
     
  5. SupuS

    SupuS HowtoForge Supporter

    Please check your setting in Server configuration => Website auto alias. You have to specify on which URL this directive will be launched. If you use for example:

    [website_domain].test.testingdomain.tld

    you have to set:

    SetEnvIf Host test\.testingdomain\.tld$ temporary_domain
     
  6. mattltm

    mattltm Member

    Perfect.

    100% working.

    :D
     

Share This Page