I just ran a search for a unique site item in Google and the sitemap was listed in the search results. It seems that the sitemap is being indexed as if it were a normal page. Perhaps a NoIndex Meta tag should be added to the sitemap.xml file to prevent this. Not sure if this would be the proper technique, however the sitemap should be allowed for crawling but prevented from inclusion in search results .

Comments

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Category: bug » support

I'm curious which search engine you used? You can't add meta tags like that to raw XML, and any decent search engine should not show XML sitemaps in results...

Coupon Code Swap’s picture

Coupon Code Swap commented

I ran a Google search that is setup to only search the site. There were two results, one linked to the node, the other linked to the sitemap.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Wierd. I have no idea why Google would show the sitemap in results. Maybe it's a setting for your site-specific search that needs to be adjusted? There's nothing that the module could or should add to the XML. :/

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

I confirm what Dave said.
Meta tags are not used, when inserted in a XML file; furthermore, the tag meta is not included in the XML schema for a XML sitemap.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Active » Fixed

So I did a little more digging on this today:
http://www.google.com/support/forum/p/Webmasters/thread?tid=5b2bf3c1ce7b...
http://www.webmasterworld.com/google/3971249.htm
http://googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-wit...

The common theme seems to be that you probably have a link on your site's content somewhere to the sitemap.xml, which causes google to include it in the index. I'd double check that you don't have any links like that.

Fortunately Google (and Yahoo I think) has added support for an X-Robots-Tag header that can be used instead of meta tags, so I've included 'X-Robots-Tag: noindex, follow' in any of the module's XML or XSL output. It will probably take a while to affect Google's index, but it should help if this problem continues.

http://drupal.org/cvs?commit=255502

Coupon Code Swap’s picture

Coupon Code Swap commented

That should do the trick. I don't have any links to the sitemap, so, not sure why Google is including it in the results. Obviously, this is a usability issue for visitors who run a search.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.