I noticed in my log that I had spiders wandering into my glossary terms. Since this generated quite a bit of unnecessary server load, I added the following line to robots.txt:

User-agent: *
Disallow: /glossary

It seems to have helped quite a bit. Might want to add it to the install docs.

Comments

moshe weitzman’s picture

moshe weitzman commented
Status: Active » Closed (won't fix)

this could be said for any group of pages on any web site. not glossary specific.