Download & Extend
Views » Issues

Use Robots.txt to Prevent Duplicates

Posted by mikeytown2 on December 9, 2008 at 10:05am
Project:Views
Version:6.x-2.1
Component:exposed filters
Category:support request
Priority:normal
Assigned:Unassigned
Status:closed (fixed)

Issue Summary

I'm fairly certain that Google will reindex the same page, trying every single value from a drop down list if you have one as an exposed filter. I'm suggesting that

# URL Variables
disallow: /*list
disallow: /*of
disallow: /*drop
disallow: /*down
disallow: /*url
disallow: /*variables

be added to robots.txt, or at least a warning somewhere so web admins know how to handle the issue. Spiders might try other values in other url variables as well? Guessing on the robots.txt syntax, it's correct right?

Similar yet different issue
#280281: Views pager duplicates content

Login or register to post comments

Comments

#1

Posted by mikeytown2 on December 9, 2008 at 6:54pm

or would you do something like this

Disallow: /path/of/view?
Allow: /path/of/view
Allow: /path/of/view?page=*

Or this

Disallow: /*?
Allow: /*?page=

using clean urls.

The option of having views insert a "noindex" meta tag might be the answer.

EDIT:
http://www.google.com/support/webmasters/bin/answer.py?answer=76329&hl=en

#2

Posted by mikeytown2 on December 10, 2008 at 7:18am

Actually now that I think about it, I'm disabling anything that makes a url variable until we get robots.txt figured out. Duplicate content is a killer. ?page= is fine, everything else isn't.

#3

Posted by mikeytown2 on December 11, 2008 at 5:42am

https://www.google.com/webmasters/tools/dashboard

  1. Click on your URL, or add it in.
  2. Go to tools
  3. Analyze robots.txt
  4. Find the Test URLs against this robots.txt file box
  5. Type some test URLs

Once I get some feedback on this, I'll add it to the handbook page

This is what I came up with

# URL Variables
Disallow: /*?
Allow: /*?page=
Disallow: /*?page=*&*

#4

Posted by mikeytown2 on December 11, 2008 at 8:17am

Here's the handbook page...
SEO (Search Engine Optimization) Guidelines When Using Views

#5

Posted by halisemre on May 10, 2009 at 8:14pm

I have a question?
If i use
Disallow: /*?
Allow: /*?page=
Disallow: /*?page=*&*

http://www.mysite.com/?page=1 is ok
http://www.mysite.com/?page=2 is ok

but what about

http://www.mysite.com/?page=0
it is the same as http://www.mysite.com/ so it is kind of you are duplicating the frontpage.

Is there a way to eliminate this problem

#6

Posted by mikeytown2 on May 12, 2009 at 11:27pm

I updated the handbook page
http://drupal.org/node/345620

Another way is to not link to page=0; or 301 it.

#7

Posted by merlinofchaos on May 30, 2009 at 6:29pm
Category:bug report» support request
Status:active» fixed

How is this a bug? Views has no control over the robots.txt. Looks like you guys have it sorted, anyhow.

#8

Posted by System Message on June 13, 2009 at 6:30pm
Status:fixed» closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.