How to prevent search engine from indexing of Website auto alias domains

Discussion in 'Tips/Tricks/Mods' started by SupuS, May 10, 2013.

  1. SupuS

    SupuS Member HowtoForge Supporter

    Hi,

    I use Website auto alias in ISPConfig and I've found that search engines are indexing domain aliases. This is not desired status. So I decided to use X-Robots-Tag HTTP header (more info on developers.google.com for example). When you have in Website auto alias for example:

    [client_id].test.domain.ltd

    so you can put this directive to apache directive settings:

    Code:
    # Prevent search engine indexing on temporary domain sites:
    <IfModule setenvif_module>
      <IfModule headers_module>
        SetEnvIf Host test\.domain\.ltd$ temporary_domain 
        Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
      </IfModule>
    </IfModule>
    
    Apache mod headers and mod setenvif has to be activated for properly work.

    It would be nice if this functioin was included in ISPConfig directly as optional ;)

    SupuS
     
  2. mattltm

    mattltm Member

    I've added this to System >> Directive Snippets

    Is that the right place?
     
  3. SupuS

    SupuS Member HowtoForge Supporter

    Yes. You have to create directive snippet for Apache and than choose this directive for each web on which you want prevent indexing by crawlers.
     
  4. mattltm

    mattltm Member

    Thanks SupuS :)

    Thats the part I was missing. Enabling it in the site config.

    Looks like it's working but on checking the headers, I don't get the X-Robots-Tag.

    I added a new line to the directive like this to test it:

    Code:
    Header set X-Robots-Tag "noindex, nofollow" env=temporary_domain
    Header set MyHeader "Hello. It took %D microseconds for Apache to serve this request."
    
    The headers show:
    Code:
    MyHeader:	 Hello. It took D=4463 microseconds for Apache to serve this request.
    So the directive is loaded. Should I be able to see the X-Robots-Tag in the headers?
     
  5. SupuS

    SupuS Member HowtoForge Supporter

    Please check your setting in Server configuration => Website auto alias. You have to specify on which URL this directive will be launched. If you use for example:

    [website_domain].test.testingdomain.tld

    you have to set:

    SetEnvIf Host test\.testingdomain\.tld$ temporary_domain
     
  6. mattltm

    mattltm Member

    Perfect.

    100% working.

    :D
     
  7. Antti Mäkelä

    Antti Mäkelä New Member

    I added this snippet and my sites and admin panel urls won't respond anymore. Any ideas how to recover? I didin't find a file where this is located in debian.
     
  8. SupuS

    SupuS Member HowtoForge Supporter

    Directive snippets for Apache are saved in vhost files in:
    /etc/apache2/sites-available/yourdomain.tld.vhost
    Try to remove your snippet from vhost file of domain you edited and restart apache manualy.
     
  9. Antti Mäkelä

    Antti Mäkelä New Member

    I'm quite sure that snippet was not in my vhost file, but I found it in *.vhost.error file. I removed my corrupted vhost file from both sites-available and sites-enabled folders and tried to restart apache, not working:
    Code:
    root@dev-anttimakela:/etc/apache2/sites-available# systemctl status apache2.service
    ● apache2.service - LSB: Apache2 web server
       Loaded: loaded (/etc/init.d/apache2)
      Drop-In: /lib/systemd/system/apache2.service.d
               └─forking.conf
       Active: active (running) (Result: exit-code) since Wed 2017-09-27 00:07:36 CEST; 9min ago
      Process: 11025 ExecStop=/etc/init.d/apache2 stop (code=exited, status=1/FAILURE)
      Process: 11329 ExecReload=/etc/init.d/apache2 reload (code=exited, status=1/FAILURE)
      Process: 11038 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
       CGroup: /system.slice/apache2.service
               └─31440 /usr/sbin/apache2 -k start
    
    Sep 27 00:09:05 dev-anttimakela systemd[1]: Reloading LSB: Apache2 web server.
    Sep 27 00:09:05 dev-anttimakela apache2[11149]: Reloading web server: apache2 failed!
    Sep 27 00:09:05 dev-anttimakela apache2[11149]: Apache2 is not running ... (warning).
    Sep 27 00:09:05 dev-anttimakela systemd[1]: apache2.service: control process exited, code=exited status=1
    Sep 27 00:09:05 dev-anttimakela systemd[1]: Reload failed for LSB: Apache2 web server.
    Sep 27 00:14:00 dev-anttimakela systemd[1]: Reloading LSB: Apache2 web server.
    Sep 27 00:14:00 dev-anttimakela apache2[11329]: Reloading web server: apache2 failed!
    Sep 27 00:14:00 dev-anttimakela apache2[11329]: Apache2 is not running ... (warning).
    Sep 27 00:14:00 dev-anttimakela systemd[1]: apache2.service: control process exited, code=exited status=1
    Sep 27 00:14:00 dev-anttimakela systemd[1]: Reload failed for LSB: Apache2 web server.
    root@dev-anttimakela:/etc/apache2/sites-available# journalctl -xn
    -- Logs begin at Tue 2017-09-19 21:05:02 CEST, end at Wed 2017-09-27 00:17:01 CEST. --
    Sep 27 00:16:49 dev-anttimakela postfix/smtpd[11315]: disconnect from unknown[14.177.236.10]
    Sep 27 00:17:01 dev-anttimakela CRON[11472]: pam_unix(cron:session): session opened for user root by (uid=0)
    Sep 27 00:17:01 dev-anttimakela CRON[11470]: pam_unix(cron:session): session opened for user root by (uid=0)
    Sep 27 00:17:01 dev-anttimakela CRON[11474]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
    Sep 27 00:17:01 dev-anttimakela CRON[11471]: pam_unix(cron:session): session opened for user root by (uid=0)
    Sep 27 00:17:01 dev-anttimakela CRON[11473]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/i
    Sep 27 00:17:01 dev-anttimakela CRON[11470]: pam_unix(cron:session): session closed for user root
    Sep 27 00:17:01 dev-anttimakela CRON[11475]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/isp
    Sep 27 00:17:01 dev-anttimakela CRON[11471]: pam_unix(cron:session): session closed for user root
    Sep 27 00:17:01 dev-anttimakela CRON[11472]: pam_unix(cron:session): session closed for user root
     
  10. HSorgYves

    HSorgYves Active Member

    What's the output of: apachectl configtest
     
  11. Antti Mäkelä

    Antti Mäkelä New Member

    Rebooted my server, apache is working again and got back in ispconfig admin area. Solved, thanks.
     

Share This Page