Int the online documentation, i don't have founded nothing speaking about robots.txt , how I can manage it using domain access module?

Comments

matteoraggi’s picture

matteoraggi commented

I have done some testa about it (domain access was configured from a friend and I checked what I can only) and this is my test result:
1) If placed on root of main domain, using domain access, then robot.txt is the same on all other subdomains
2) Putting robots.txt into maindomain/sites/all/themes/themename or maindomain/sites/default/domain_name/ or sites/default/files/domain-N (N is idnumber of subdomain) it don't work.
Conclusion:It look that with domain access module actually is impossible to put a different robots.txt per web site, but I think that it could be useful in soem cases, and more are domains and more probably it will be useful.

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

There is a robots.txt module that is designed for multisite setups and may be helpful. You need some manner of redirect to point the bots to the proper version of the file.

matteoraggi’s picture

matteoraggi commented

I founded the module, reading multisite, I was thinking that it can't work with domain access module
http://drupal.org/project/robotstxt I don't feel so much able to configure it on domain access module configuration, could you please be more detailed with instructions please?

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

I cannot. I have not used that module.

jeremyr’s picture

jeremyr commented

I was recently checking into this module and have found that it only works in the sense of mutli-site installs... as in having multiple instances of drupal installed from the same code base. You would have to activate that module for each instance to get it to work properly.

What I have done to prevent search engines from seeing the same content on my other domains is I enabled the search engine rewrite in the domain settings. I assume this works fine but if you need to specify different items in the robot.txt for each domain I think we'll have to create a new module.

And if you are into "Hacking" and doing non-recommended practices you could do what I do on my development site. In my index.php file I added some php logic to identify robots and then redirect them back to the live site. You could add something like that to redirect them back to the primary domain if they try to scrub the other domains. I'm sure there are all sorts of things wrong with doing that but it would work.

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

You might also be able to do a rewrite rule in .htaccess that points to different versions of robots.txt based on the inbound HTTP_HOST.

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented
Status: Active » Closed (fixed)

With the robotstxt module, you just need hook_domainbatch() or hook_domainconf() to support changing the 'robotstxt' variable.

abaddon’s picture

abaddon commented

there is a request on http://drupal.org/node/624372 for the robotstxt module to support domain access, i am going to work on it on behalf of a sponsor and ill post a patch there soon

kfrank’s picture

kfrank commented

I'm so excited to see that someone is going to create a patch!

gideon_ph’s picture

gideon_ph commented

An alternative solution to this, assuming you are using different themes for each domain, is to add a robots meta tag having noindex,nofollow to the theme's page.tpl.php to avoid all the pages on that domain from getting indexed by search engines.