Hi everyone,

I followed the instructions,:

"Note: You must delete or rename the robots.txt file in the root of your Drupal installation for this module to display its own robots.txt file(s)."

...deleted the robots.txt file in the root of my installation, went through my 3 sites and enabled the module and saved configuration.

So where are the robots.txt files located for my 3 sites? I thought they might be in:

/public_html/sites/site1/files
/public_html/sites/site2/files
/public_html/sites/site3/files

..but I can't see them via my ftp interface. Maybe I'm missing something. I wanted site1 and site2 to be spidered as normal, but site3 (dev) needs the spiders to be excluded.

I noticed this line at /admin/config/search/robotstxt:

"# This file will be ignored unless it is at the root of your host:
# Used: http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt"

But the nature of multi site means there *is* only one root directory, public_html. A single robots.txt in there would affect every site on the multi site installation using one codebase.

The info seems rather conflicting.

If anyone can help out or clarify, that'd be great. Thanks.

Sam.

Comments

samwillc’s picture

samwillc commented

Is anyone able to help with this one?

Thanks.

Sam.

wayne2011’s picture

wayne2011 commented

I as well just installed the module on a multi-site and was looking around for the robots.txt file when I recalled the wording on the project page that it is generated dynamically. It is a little scary to not be able to look, see and check that the file is not located such that it will exclude googlebot from one site as desired and not another highly ranked site that welcomes googlebot. I think I'm going to look for another solution.

wayne2011’s picture

wayne2011 commented

Ok, I did a bit of kicking around and realized that though there is no static file "robots.txt" for any of the sites within the multi site, you can of course look at the dyanmically generated file by simply pointing your browser to examplesite.com/robots.txt. By simply enabling the module on each site, the default drupal robots.txt is generated, I checked. As would be expected, if you delete the robots.txt from the root, you will get a 404 by going to thatsite.com/robots.txt for any sites where you have not enabled the module. Hope this helps.

samwillc’s picture

samwillc commented

Thanks wayne2011, that puts my mind at rest!

Sam.

hass’s picture

hass commented

robots.txt is inside /public_html/

hass’s picture

hass commented
Status: Active » Fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.