By venkat-rk on

A few questions about robots.txt that I hope someone will be kind enough to answer:

- Does drupal ship with a default robots.txt in core? Or does one have to add it manually?
- If I am running multiple sites off one code base, I need to use the robots.txt module to give each site its own robots.txt, correct?
- The book page at http://drupal.org/node/22265 says that the file should be uploaded to the root of the web server. If I have a shared hosting account with access to just my domain, does this actually mean I should upload the file to my web root? That is, public_html in most cases?
- If I am using the global redirect module to avoid duplicate content issues (node/nid and its path alias), can I can avoid putting /node in the disallow directives in the robots.txt file?

Thanks in advance.

⋅ Categories: Drupal 5.x

Comments

jonfhancock’s picture

jonfhancock commented
  1. Drupal 5.x and 6.x ship with a default robots.txt. I think there might be a module that helps you write custom robots.txt, and Google webmaster tools can definitely help.
  2. Not sure about that one. In my setup, since all the files in my drupal code base are symlinked to each domain's root, they all share a robots.txt. Not sure if they should be unique or not.
  3. Yes, put it in your web root (public_html, or www, or whatever). Wherever index.php is, that's where robots.txt goes. The idea is that example.com/robots.txt brings up the robots.txt file
  4. Don't know about that one
venkat-rk’s picture

venkat-rk commented

Thank you for taking the time to answer. For some reason, I couldn't find a robots.txt in my default install, that's what really prompted my questions. It looks like the robots.txt module is meant to provide a unique robots.txt file for a multisite install.

I hope someone can clarify that question about global redirect and robots.txt file.

----
Previously user venkat-rk.