Posted by gtaylor on March 9, 2007 at 2:26pm
I have a 5.1 multi-site, single code line install. One site I want indexed for search. The other is private. I clearly need different robots.txt settings.
I see that the robots.txt module is not available for 5.1.
Also while digging, I found a thread that lead me to believe it's now in 5.1 drupal core. But I don't see any instructions on how to configure/use it.
Can somebody point me in the right direction?
Comments
Same problem :) www :
Same problem :)
www : production site and
beta : to not indexed
but I don't find solution.:(
ZebX
Sylvain Lasnier
.htaccess could be used
if you have access and are able to use to override the system wide file then you can do it in this file using
RewriteCond %{HTTP_HOST} ^www1\.example\.com$ [NC]
RewriteRule ^robots\.txt$ robots_www1.txt [L]
RewriteCond %{HTTP_HOST} ^www2\.example\.com$ [NC]
RewriteRule ^robots\.txt$ robots_www2.txt [L]
and then put what you like in each robots_xxx.txt
also, be warned that errors/typos in .htaccess can make 500 erors and make your site not work at all.
--
http://eventzs.com
Alternative for all sites...
Something like:
RewriteCond %{HTTP_HOST} \.([a-z-]+\.[a-z]{2,6})$ [NC]
RewriteCond %{REQUEST_URI} !^(.*)/sites/(.*)$ [NC]
RewriteRule ^(.*)robots.txt$ /sites/%1/files/robots.txt[L]
You will of course need a robots.txt in all /sites/files (I suggest symlinking to /drupal/robots.txt when unique is not needed) and to mod .htaccess to allow handling of the robots.txt file.
~kwt
Ahem. Above code only works
Ahem. Above code only works on subdomains. Please stand by.
I have made one change to
I have made one change to support particular domain name containing some numbers.
RewriteCond %{HTTP_HOST} \.([a-z0-9-]+\.[a-z0-9]{2,6})$ [NC]RewriteCond %{REQUEST_URI} !^(.*)/sites/(.*)$ [NC]
RewriteRule ^(.*)robots.txt$ /sites/%{HTTP_HOST}/files/robots.txt [L]
This snippet still works well under Drupal 6 only for site with sub domain, if you don't install module RobotsTxt.
If you want to apply this rule to all domains not only to sub domain, just remove the first line. However, be sure prepare all roborts.txt for each sites, else even a small error may raise a big problem:)
work for me. thanks! and
work for me. thanks!
and subscribe ...
not working using drupal 7
not working using drupal 7 multisite
http://drupal.org/project/rob
http://drupal.org/project/robotstxt
Haven't tried it but this is
Haven't tried it but this is available..
http://drupal.org/project/robotstxt
⎋ joon park