Posted by Cactii1 on November 7, 2009 at 6:12am
Jump to:
| Project: | Drupal core |
| Version: | 6.10 |
| Component: | base system |
| Category: | support request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | closed (fixed) |
Issue Summary
I did a site:www.example.com search on Google and I noticed that a lot of the URLs that I don't want indexed are still being indexed, even ones that are explicitly disallowed in the robots'txt file.
I looked into the robots.txt file and noticed that all the entries have a trailing backslash on them and thought that maybe Google isn't respecting the robots.txt file because of this.
I've duplicated most of the entries and removed the backslash off the end of the line.
Any thoughts?
Comments
#1
In robots.txt file in Drupal 6.x correctly follows the robots.txt standard outlined on the official robots.txt website http://www.robotstxt.org/robotstxt.html. Any problems you're having probably relate to configuration.