I just read http://drupalzilla.com/module/image and it is quite compelling advice about the image module, SEO, and how to fix it with a simple change to robots.txt

I suggest adding the following documentation to the README.txt and/or to the Handbook.

To reduce load on your site from robots and to reduce duplicate content from getting into the search engine, you can add the following line to your robots.txt file in the root of your Drupal installation:

Disallow: /*size=
CommentFileSizeAuthor
#6 image-install-note_200029_robots-txt.patch761 bytesHetta
#2 image_install-txt_200029.patch628 bytesHetta

Comments

drewish’s picture

drewish commented

sounds good to me. a patch would be great.

Hetta’s picture

Hetta commented
StatusFileSize
new image_install-txt_200029.patch628 bytes

OK, a patch against INSTALL.txt appended.

greggles’s picture

greggles
he/him
Primary language English
Location Denver, Colorado, USA
commented
Status: Active » Needs review

I just realized that "into the search engine" should probably be "into search engines".

I can't easily create patches here or I would do this. It's just a copy/paste anyway...

nschindler’s picture

nschindler commented

Just my $0.02...
The standard for robots.txt doesn't support wildcards in Disallow lines. This may invalidate your robots.txt and cause some spiders to error out.
Googlebot and Yahoo! Slurp DO support wildcards in Disallow lines as an extension to the standard. So, you may want to have robots.txt speak to them specifically, like so:

User-Agent: Yahoo! Slurp
Disallow: /*size=

User-Agent: Googlebot
Disallow: /*size=

References:

Yahoo! Slurp wildcard support: http://www.ysearchblog.com/archives/000372.html
Googlebot wildcard support: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40367
Robots.txt validator (try running http://www.redhat.com/robots.txt through it): http://tool.motoricerca.info/robots-checker.phtml
Robots.txt standard: http://www.robotstxt.org/robotstxt.html

greggles’s picture

greggles
he/him
Primary language English
Location Denver, Colorado, USA
commented

While not part of the standard, I believe that google, yahoo, msn all support them for sure. So, given that over 85% of search engine traffic will respect them and that we don't want to suggest 20 "User-Agent: " lines, I think we could add:

Wildcards may not be supported by all search engines. Confirm whether it will work for search engines important to you.

Hetta’s picture

Hetta commented
Status: Needs review » Reviewed & tested by the community
StatusFileSize
new image-install-note_200029_robots-txt.patch761 bytes

Let's get this one done (patch for 5.x-2.x-dev, though).

greggles’s picture

greggles
he/him
Primary language English
Location Denver, Colorado, USA
commented

Looks good to me.

drewish’s picture

drewish commented
Status: Reviewed & tested by the community » Fixed

thanks, committed to DRUPAL-5, DRUPAL-5--2, and HEAD.

greggles’s picture

greggles
he/him
Primary language English
Location Denver, Colorado, USA
commented

Thanks to you, drewish!

Anonymous’s picture

Anonymous (not verified) commented
Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.