Hello,
I have a new ecommerce site with drupal + ubercart. I am testing the website with a close circle of my friends before going to advertising it. I have google webmasters account for the website and shockingly I found that Google has the following keywords at the top of the list for my website:

div
product
class
form
span
cart
edit
type
imagecache
name
display
label
files
hidden
image

After researching a couple of drupal posts and other post on internet, I found that google is indexing xml files of taxonomy feed in the url format http://www.example.com/taxonomy/term/15/all/feed and parses the html entities as content of the website. Since I had large number of taxonomy terms, these pages got indexed and the html tags got into my website's keyword list.

I guess with a simple 'Disallow: /taxonomy/term/*/*/feed' this serious seo blunder can be avoided.

For my case, I added 'Disallow: /taxonomy/*' and 'Disallow: /taxonomy/' as I had path alias urls for accessing taxonomy terms.

this issues is related all drupal releases but not sure how it tag it like that.

Comments

snorkers’s picture

snorkers commented
Version: 6.9 » 6.16

Thanks for your suggestion - I had wondered why DIV, FIELD and CLASS were in my top 10 keywords.

Agree this is a bit of a crucial bug. Not sure if being able to change the URL to end /feed.xml might steer Google in the right direction (odd, as all the right tags are at the beginning of the XML document:

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xml:base="http://www.example.com"  xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/" xmlns:georss="http://www.georss.org/georss">

... is there a Google name space (xmlns:) that might help?)

almalino’s picture

almalino commented

Same problem I have reported here:
http://drupal.org/node/666930#comment-3040050

powery’s picture

powery commented

I have the same problem. Subscribing.

powery’s picture

powery commented
Version: 6.16 » 6.17
dpearcefl’s picture

dpearcefl commented
Priority: Critical » Normal
Status: Active » Postponed (maintainer needs more info)

Is this issue still occurring in the current version of Drupal v6?

neokrish’s picture

neokrish commented
Priority: Normal » Critical
Status: Postponed (maintainer needs more info) » Active

Yes, it's still the same problem.

To check the problem, go to http://drupal.org/taxonomy/term/15/all/feed and view the html source code in your browser. I am pasting a small snippet from the above url below. Notice the occurrence of 'div' 'href', 'class' etc... These are the words that are topping my Google Webmaster Keyword list.

 <description>&lt;div class=&quot;all-attached-images&quot;&gt;&lt;div class=&quot;image-attach-body image-attach-node-1185762&quot; style=&quot;width: 218px;&quot;&gt;&lt;a href=&quot;/node/1185762&quot;&gt;&lt;img src=&quot;http://drupal.org/files/images/The Coding Designer’s Survival Kit-1.thumbnail.png&quot; alt=&quot;The Coding Designer&amp;#039;s Survival Kit &quot; title=&quot;The Coding Designer&amp;#039;s Survival Kit &quot;  class=&quot;image image-thumbnail &quot; width=&quot;218&quot; height=&quot;56&quot; /&gt;&lt;/a&gt;&lt;/div&gt;

&lt;/div&gt;&lt;p&gt;When you design web sites in the browser you&#039;re left alone in the dark without any of your familiar design tools. By nature there&#039;s no application you can load up and start building your design. Well, this isn&#039;t that. However I think it&#039;s the next best thing.&lt;/p&gt;
&lt;p&gt;Instead of starting from square one every time you begin a project I think we need a collection of good starting values in markup and css to start designing, and a library of javascript tools to help us rapidly prototype as we think creatively. In the Kit you&#039;ll find HTML inspired by HTML5 Boilerplate, a system for writing highly flexible and powerful css with Sass, Compass, Susy, and other frameworks, and a bevy of javascript libraries at your disposal.&lt;/p&gt;
&lt;p&gt;The Coding Designer&#039;s Survival Kit is two related projects. First is the project on Github, which is not Drupal, but a generalized set of tools for doing this work in any CMS (or none). Second is the theme based on that project, which will have a full release here on drupal.org shortly.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/canarymason/The-Coding-Designers-Survival-Kit&quot; rel=&quot;nofollow&quot;&gt;Link to the project on Github&lt;/a&gt;&lt;/p&gt;
</description>

The latest robots.txt also does not disallow indexing of taxonomy term feed pages. Hence it's critical and not yet resolved.

dddave’s picture

dddave commented
Version: 6.17 » 6.22
Priority: Critical » Major

I am not sure I understand the problem here correctly so please help me out. Simply editing the robots.txt file helps, doesn't it?

What do you want Drupal to do here? Is this really a bug or a feature request? And this is definitely not critical per http://drupal.org/node/45111

neokrish’s picture

neokrish commented

@dddave.. agree that it's not critical.

Simply editing the robots.txt file helps, doesn't it?

Yes, editing robots.txt helps to avoid google indexing these urls and prevents it from mistaking divs and spans as content of the website.

However, how many people ever know (I am not talking of developers here) to make these changes and how many does know that this problem exists at first place?

The solution that I am proposing is that Drupal ships with an updated robots.txt that handles this issue so that the millions of drupal installations do not have to edit their robots.txt to do the same.

dddave’s picture

dddave commented
Version: 6.22 » 8.x-dev

You stated in you OP that this is a problem for any Drupal version so lets kick it some versions up to see what the elders say about this issue. ;)

damien tournoud’s picture

damien tournoud commented

I never heard of a recommendation to block RSS feeds from robots.txt (that just feels retarded). Anyone has a link for that?

neokrish’s picture

neokrish commented

@Damien
These links may not be what you are asking for, still these may be related.
http://drupal.org/node/660784
http://sebastians-pamphlets.com/google-to-remove-feeds-from-web-search-r...
http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-o...

damien tournoud’s picture

damien tournoud commented

If nothing, it seems to be a bug in Google Webmasters tool.

catch’s picture

catch
he/him
Primary language English
commented
Priority: Major » Normal

Not a major bug at least, I'm sure other search engines manage to deal with feeds just fine.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

lendude’s picture

lendude
Primary language Dutch
Location Amsterdam
commented
Issue summary: View changes
Status: Active » Postponed (maintainer needs more info)

I'm going to agree with #12 here, indexing a feed page as HTML seems like a bug in the tooling.

Several things have changed over the years. In D8 these feeds are Views and they are disabled by default, so at the very least this isn't an out of the box problem anymore.

But, is this still an issue, has anybody seen this happen lately or can we close this?

lendude’s picture

lendude
Primary language Dutch
Location Amsterdam
commented
Issue tags: +Bug Smash Initiative

Cleaning up old bugs, tagging for the initiative

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev
quietone’s picture

quietone commented
Status: Postponed (maintainer needs more info) » Closed (outdated)

There has been no activity here for 11 years, except to ask for more information over a year ago and there has been no reply.

There is agreement that this is a problem in tooling and given that changes to Drupal 8/9 it is time to close this. Therefore, closing as outdated.