Jump to:
| Project: | RobotsTxt |
| Version: | 4.7.x-1.x-dev |
| Component: | Code |
| Category: | bug report |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | closed (works as designed) |
Issue Summary
Thank you for the great module.
After several different configurations, I have not be able to get a robotstxt to function on a site by site basis. Part of the confusion is that multisite, single db, single codebase, with sites/mysite.com/modules/robotstxt installation instructions are a bit varied and I may just not have gotten the correct combination of tricks (serveralias, synlinks,staged installation of modules...). My primary difficulty is understanding how drupal manages the 'variable' table. Whenever I update the robots.txt file via the robotstxt.module UI a single variable 'robotstxt' is updated in the 'variable' table. For those using a prefix stategy 'ex: mysite_com_' which prefixes the table ( mysite_com_viariable), this would not be a problem. [For a long term fix, druapl may need to updates its multisite code, such that a new variable is created and used (ex: mysite_com_robotstxt) for persisting a unique variable for that module.]
My workaround:
Until I can figure out the multisite magic, here is my work around. I installed the robotstxt module inside the main /module area. Then I used the settings.php, located inside the site specific installation (sites/mysite.com/settings.php) file for customizing the sites in which I wish to have a unique robots.txt file.
$conf = array(
'robotstxt' => "User-agent: *
Disallow: /"
);
This is a bit simplified, since it blocks all robots from that site. This works great if you wish to only have bots hit you primary sites and avoid a specific set of secondary sites. It also allows you a single robotstxt UI to update you primary sites. The downside is that this misses the goal of multisite robotstxt UI. I look forward to your install.txt file to clarify how to properly install this module for a multisite structure with a shared db.
JJ
Comments
#1
Thanks for reporting. I believe this is just a bug in the way I wrote the code. Drupal is equipped to deal with aliased tables. I'll look into it when I get a chance.
#2
Because robotstxt uses variable_get and variable_set, this should be handled at the Drupal core level. Are you sure you're using different databases or different table prefixes for your multiple sites?
#3
For a multisite configuration using Domain Access ( http://drupal.org/project/domain ), variable_get / variable_set is a poor choice, because the variable table is likely to be shared between the sites.
It would be better if robotstxt would use its own table, so that it could be prefixed to each sub-site. Or perhaps it could handle all of the sites under the same table, determining the sites via $base_url and then providing the correct robots.txt.
Honestly, multisite configurations are the best purpose of this module, so it would be nice if it could handle all of the configurations that may get used. It needs a bit more flexibility. Seems like more of a feature-request than a bug, but a strong need regardless.
#4
Seems to be fixed by the "domain access" module themself:
#5
$conf = array('robotstxt' => "User-agent: *
Disallow: /"
);
will this work with drupal 6 ? For blocking a subdomain used as a test site