I did a site:www.example.com search on Google and I noticed that a lot of the URLs that I don't want indexed are still being indexed, even ones that are explicitly disallowed in the robots'txt file.

I looked into the robots.txt file and noticed that all the entries have a trailing backslash on them and thought that maybe Google isn't respecting the robots.txt file because of this.

I've duplicated most of the entries and removed the backslash off the end of the line.

Any thoughts?

Comments

devin carlson’s picture

devin carlson commented
Category: bug » support
Status: Active » Closed (fixed)

In robots.txt file in Drupal 6.x correctly follows the robots.txt standard outlined on the official robots.txt website http://www.robotstxt.org/robotstxt.html. Any problems you're having probably relate to configuration.