By kiwibird on
Hi, I couldn't find any good answers about this in the forums. I want to allow robots to crawl the file uploads on a site (mostly PDFs with public content, very search-engine relevant...). What is the "standard" or best way to do this in Drupal 6?
Should I just have the following?
Allow: /sites/default/files/
Disallow: /sites/
(Wikipedia calls "Allow" nonstandard...hmm..)
