Controlling what gets indexed -- the robots.txt file

The robots.txt file is the mechanism almost all search engines use to allow website administrators to tell the bots what they would like indexed.

A robots.txt is included with Drupal 5.x. Drupal 4.x does not come with a robots.txt file.

By adding this file to the root of your (virtual) webserver, you can guide the search engines' "bots" through your site or forbid indexing parts of your site. See for an example the file for drupal.org itself at http://drupal.org/robots.txt.

If you want to have a robots.txt file, please follow the instructions below. For more details check http://www.robotstxt.org

Create a file with the content as shown below and call it "robots.txt"

This file tells indexing robots that they should avoid pages that contain content for users only, for example the search page, or the 'Add a comment' forms for nodes.

Many robots obey the "Crawl-delay:" parameter. Since Drupal sites seem to be popular with search engines and lots of people have more aggressive bots than visitors at their site, it might be wise to slow down the robots by adding a robots.txt line like this:

Here 10 is the delay in seconds between page requests.

Both "Slurp" (The robot that is indexing for yahoo and altaVista) and the Microsoft robots for the MSN sites obey this parameter. Googlebot does not use the "crawl-delay" parameter yet but will likely do so in an upcoming version.

Change the file as you wish and save it. Now upload it to your webserver and make sure it is in the root of the (virtual) webserver. If you have installed Drupal in a subdirectory (for example /drupal), then change the URL's in the robots.txt file but place the file in the root of the webserver, not in the root of your drupal installation.

Now watch the robots visit your site and after some time, monitor your log files ("referrer log") to see how many visitors came from a search engine.

For multi-site installations if the sites need different robots.txt files for each site it is necessary to remove the robots.txt file and use the RobotsTxt module.