Although I didn't have a link to the amazon_store page from within the site I must have had a link from a views page which was picked up by Google spidering the site, and my recent hits logs where full of bizarre Amazon product links... so now everyday Google is spidering endless product links from Amazon. As every page creates product links suggestions the links the spider just keeps going. I don't think this is useful for the site and unnecessary load on the server. I put an exclude in the robots.txt file, but its too late, google just keep revisiting those pages and continues to add links. Perhaps there should be a nofollow on the links created by the store suggestions?
| Comment | File | Size | Author |
|---|---|---|---|
| #2 | amazon_store.nofollow_735794.patch | 7.95 KB | rfay |
Comments
Comment #1
rfayAmazon module already has nofollow: #673500: Use rel="nofollow" on detailpageurl links (links to amazon.com), so if it was a view, that's already dealt with.
I definitely think something should be added to both the Amazon Store and Amazon docs pages about adding to the robots.txt. Since you're recently back from this journey, could you add something there?
And we should get nofollow onto Amazon Store links as well.
Patches and community contributions are welcome!
Comment #2
rfayI think I got all of them. Would appreciate your review.
Committed to HEAD: http://drupal.org/cvs?commit=350264