Filter /taxonomy/ after refine by taxonomy installation
FlemmingLeer - September 9, 2008 - 10:10
| Project: | Refine by taxonomy |
| Version: | 5.x-0.1 |
| Component: | Miscellaneous |
| Category: | task |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Description
This is not quite an issue as much as a small warning.
I would advise users of refine by taxonomy to add a blocking in robots.txt or else thousands of additional pages will be indexed by search engines.
If used together with pathauto it'll cause multiple pages for the same content.
Therefore a blocking of the taxonomy pages in robots.txt is advisable:
User-agent: Googlebot
# don't allow taxonomy directory to be indexed
Disallow: /taxonomy/
#Other search engine robots
User-agent: *
Disallow: /taxonomy/Please include this in the readme.txt file somehow.
I also posted about this in the search engine group: http://groups.drupal.org/node/14690