Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
By Helen-kh on
Hi,
I have a question on Google indexing pages. I have noticed that google has indexed the user/register page and password page as example below:
mysite.com/user/
mysite.com/user/password?page=1
mysite.com/user/register
Is this normal or is there anything to prevent it?
Please advise in as detailed as possible so I can understand what to do to prevent this if possible.
Thanks a lot :)
Comments
robots.txt
I just wanted to add that I checked the robots.txt and I found this scripts:
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/
Does this mean that these pages are disallowed to show on search engines? And if so, why it is not effective in my case?
(The robots.txt is located on the same directory as other pages and folders of drupal)
I'm having the same problem.
I'm having the same problem. I don't have any need for that login page to be accessed by anyone but me. But Google keeps indexing it. My robots.txt disallows it. Don't understand why.
Another way.
The robots.txt is the best method, you could also use rel="noindex" on your login (& register if you have one) link(s).
Run into the same problem.
It seems the problem is caused by the trailing slashes in Drupal's robots.txt. Now I'm playing with my robots.txt in webmaster robots testing tool and when I add the URLs like this:
Disallow: /user/register
Disallow: /user/password
Disallow: /user/login
Disallow: /user/logout
It seems to work. Now I'm waiting for the real googlebot to come, I hope it will help.