Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
By gtaylor on
I have a 5.1 multi-site, single code line install. One site I want indexed for search. The other is private. I clearly need different robots.txt settings.
I see that the robots.txt module is not available for 5.1.
Also while digging, I found a thread that lead me to believe it's now in 5.1 drupal core. But I don't see any instructions on how to configure/use it.
Can somebody point me in the right direction?
Comments
Same problem :) www :
Same problem :)
www : production site and
beta : to not indexed
but I don't find solution.:(
ZebX
.htaccess could be used
if you have access and are able to use to override the system wide file then you can do it in this file using
RewriteCond %{HTTP_HOST} ^www1\.example\.com$ [NC]
RewriteRule ^robots\.txt$ robots_www1.txt [L]
RewriteCond %{HTTP_HOST} ^www2\.example\.com$ [NC]
RewriteRule ^robots\.txt$ robots_www2.txt [L]
and then put what you like in each robots_xxx.txt
also, be warned that errors/typos in .htaccess can make 500 erors and make your site not work at all.
Alternative for all sites...
Something like:
RewriteCond %{HTTP_HOST} \.([a-z-]+\.[a-z]{2,6})$ [NC]
RewriteCond %{REQUEST_URI} !^(.*)/sites/(.*)$ [NC]
RewriteRule ^(.*)robots.txt$ /sites/%1/files/robots.txt[L]
You will of course need a robots.txt in all /sites/files (I suggest symlinking to /drupal/robots.txt when unique is not needed) and to mod .htaccess to allow handling of the robots.txt file.
~kwt
Ahem. Above code only works
Ahem. Above code only works on subdomains. Please stand by.
I have made one change to
I have made one change to support particular domain name containing some numbers.
This snippet still works well under Drupal 6 only for site with sub domain, if you don't install module RobotsTxt.
If you want to apply this rule to all domains not only to sub domain, just remove the first line. However, be sure prepare all roborts.txt for each sites, else even a small error may raise a big problem:)
work for me. thanks! and
work for me. thanks!
and subscribe ...
not working using drupal 7
not working using drupal 7 multisite
http://drupal.org/project/rob
http://drupal.org/project/robotstxt
Composer is a tool for dependency management in PHP (see https://getcomposer.org). It is not a website development tool.
Allow webbuilders using to choose their own building strategy. Do not require composer for webdevelopment or production websites.
Haven't tried it but this is
Haven't tried it but this is available..
http://drupal.org/project/robotstxt