By adlitinc on

I'm currently looking at my logs which shows under recent hits:

2005-07-04 16:08 archives
archive/1948/06/21 Anonymous details
2005-07-04 16:08 archives
archive/1949/09/25 Anonymous details
2005-07-04 16:08 archives
archive/1949/10/25 Anonymous details
2005-07-04 16:08 archives
archive/1948/07/21 Anonymous details
2005-07-04 16:08 archives
archive/1951/07/03 Anonymous details

This looks like a crawler going crazy. Is there a way to control the archive module?

Comments

oNyx’s picture

oNyx commented

You can exclude directorys by specifying em in your robots.txt file.

See... here (or google):
http://www.searchengineworld.com/robots/robots_tutorial.htm