I have a 5.1 multi-site, single code line install. One site I want indexed for search. The other is private. I clearly need different robots.txt settings.

I see that the robots.txt module is not available for 5.1.

Also while digging, I found a thread that lead me to believe it's now in 5.1 drupal core. But I don't see any instructions on how to configure/use it.

Can somebody point me in the right direction?

Comments

Sylvain Lasnier’s picture

Same problem :)
www : production site and
beta : to not indexed
but I don't find solution.:(

ZebX

eventzsguy’s picture

if you have access and are able to use to override the system wide file then you can do it in this file using

RewriteCond %{HTTP_HOST} ^www1\.example\.com$ [NC]
RewriteRule ^robots\.txt$ robots_www1.txt [L]

RewriteCond %{HTTP_HOST} ^www2\.example\.com$ [NC]
RewriteRule ^robots\.txt$ robots_www2.txt [L]

and then put what you like in each robots_xxx.txt

also, be warned that errors/typos in .htaccess can make 500 erors and make your site not work at all.

1kenthomas’s picture

Something like:


RewriteCond %{HTTP_HOST} \.([a-z-]+\.[a-z]{2,6})$ [NC]
RewriteCond %{REQUEST_URI} !^(.*)/sites/(.*)$ [NC]
RewriteRule ^(.*)robots.txt$ /sites/%1/files/robots.txt[L]

You will of course need a robots.txt in all /sites/files (I suggest symlinking to /drupal/robots.txt when unique is not needed) and to mod .htaccess to allow handling of the robots.txt file.

~kwt

1kenthomas’s picture

Ahem. Above code only works on subdomains. Please stand by.

eric.chenchao’s picture

I have made one change to support particular domain name containing some numbers.

  RewriteCond %{HTTP_HOST} \.([a-z0-9-]+\.[a-z0-9]{2,6})$ [NC]
  RewriteCond %{REQUEST_URI} !^(.*)/sites/(.*)$ [NC]
  RewriteRule ^(.*)robots.txt$ /sites/%{HTTP_HOST}/files/robots.txt [L]

This snippet still works well under Drupal 6 only for site with sub domain, if you don't install module RobotsTxt.

If you want to apply this rule to all domains not only to sub domain, just remove the first line. However, be sure prepare all roborts.txt for each sites, else even a small error may raise a big problem:)

Guo’s picture

work for me. thanks!
and subscribe ...

marcoka’s picture

not working using drupal 7 multisite

cmseasy’s picture

Composer is a tool for dependency management in PHP (see https://getcomposer.org). It is not a website development tool.
Allow webbuilders using to choose their own building strategy. Do not require composer for webdevelopment or production websites.

dvessel’s picture

Haven't tried it but this is available..

http://drupal.org/project/robotstxt