The urlfilter only lists items in the node table. But what about some of the other important nodes ...

e.g. user/###
e.g. blog/###
e.g. topics/x/y/z
e.g. taxonomies

And what about mapping them through a path alias? (oh, I see that's handled in the code)

Comments

deekayen’s picture

deekayen commented
Category: bug » feature

You wouldn't happen to know any modules off hand that might have similar base functions for listing stuff like blog, user, topics, etc so I don't have to reinvent the wheel, would you?

deekayen’s picture

deekayen commented
Status: Active » Closed (works as designed)

I added taxonomy/term/# and /blog to HEAD, DRUPAL-4-7, and DRUPAL4-6. I'm hesitant to go beyond that since spiders should be expected to do a bit of crawling beyond just the URLs in the URL list. For example, I can easily see admins adding /user to robots.txt (because I do).