We want to improve the Drupal core file robots.txt.

See also:

CommentFileSizeAuthor
#10 robots.txt.drupal-7.50.patch1.29 KBkienan
#6 robots.txt.drupal-6.25.patch3.45 KBj0nathan
#2 robots.txt.drupal-7.4.patch3.47 KBj0nathan
#1 robots.txt.drupal-6.22.patch3.43 KBj0nathan

Comments

j0nathan’s picture

j0nathan commented
StatusFileSize
new robots.txt.drupal-6.22.patch3.43 KB

Here is a patch to apply to Drupal 6.22 installations.

j0nathan’s picture

j0nathan commented
StatusFileSize
new robots.txt.drupal-7.4.patch3.47 KB

Here is a patch to apply to Drupal 7.4 installations.

j0nathan’s picture

j0nathan commented
Status: Active » Fixed

These patches will be called in their respective branch.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

j0nathan’s picture

j0nathan commented
Version: » 6.x-1.x-dev
Status: Closed (fixed) » Needs work
j0nathan’s picture

j0nathan commented
Status: Needs work » Fixed
StatusFileSize
new robots.txt.drupal-6.25.patch3.45 KB

Here is a patch to apply to Drupal 6.25 installations.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

j0rd’s picture

j0rd commented

Thanks a bunch J0nathan. This appears to be the best robots.txt I've found for D6.

Patch #6 applies cleanly to D6.26 as well.

j0nathan’s picture

j0nathan commented
Component: Code » Patch request
kienan’s picture

kienan commented
Issue summary: View changes
StatusFileSize
new robots.txt.drupal-7.50.patch1.29 KB

Slightly updated robots.txt for drupal 7.50. Removed boost module disallows.