robots.txt entries all with trailing backslashes
Cactii1 - November 7, 2009 - 06:12
| Project: | Drupal |
| Version: | 6.10 |
| Component: | base system |
| Category: | bug report |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Description
I did a site:www.example.com search on Google and I noticed that a lot of the URLs that I don't want indexed are still being indexed, even ones that are explicitly disallowed in the robots'txt file.
I looked into the robots.txt file and noticed that all the entries have a trailing backslash on them and thought that maybe Google isn't respecting the robots.txt file because of this.
I've duplicated most of the entries and removed the backslash off the end of the line.
Any thoughts?
