By Burak on

I have been using a Drupal implementation since May, and everything has been going fine.

Now let's say I write an article about flowers. I use clean URL's and the path module so that I can change path from something like:

example.com/node/23

into

example.com/flowers.html

--

Now I have a problem since the search engines sometimes (for whatever reason) pick up the URL of:

example.com/node/23

...and it will show the article about flowers.

My question is: how can I make /node/23 a 404? How can I make sure that the only way to access the article is through that single URL?

I would like to do this with about 150 url's. I thought of using mod rewrite but I also want to make sure that I can access /node/add when I wish to create a new page.

This issue is becoming a real problem as the search engines index me and seemingly duplicate content is appearing under different URL's. Please let me know of anything that can fix this problem. Thanks :)

⋅ Categories: Drupal 4.7.x

Comments

firstov’s picture

firstov commented

create robots.txt file in your web site main directory and add these two lines:
User-agent: *
Disallow: /node

Make sure you can view your robots.txt file by accessing it via your browser using http://your_site.com/robots.txt link

Burak’s picture

Burak commented

I really appreciate your reply, and I agree that is a good solution to keep the well behaved spiders out. However, the /node/23 url still exists and shows the content. I'm trying to find a way to avoid that. Even if the spiders don't look at it, it's still there under both URL's.

Is it not possible to make the content accessible through only a single URL?

firstov’s picture

firstov commented

search engines, from where the majority of the traffic comes from will eventually (after about two three months) drop /node/xx format from their indexes after you modify your robots.txt file.

I think this is probably the best thing to try. Another solution would be modifying your .htaccess file to either restrict /node/xx URLs to certain IPs either deny them altogether, however this will impact an ability to modify your site content as Drupal uses /node/xx/edit format when editing nodes.
try these in your .htaccess:

rewritecond %{REQUEST_URI} ^/node
RewriteRule ^(.*) - [F]

this should deny acess to ANY URL that starts with /node
Make sure you test this first on your dev server...

Burak’s picture

Burak commented

Thanks for the reply. I had thought mod rewrite might be the best/only solution but I wanted to ask here just in case.

eagereyes’s picture

eagereyes commented

This is an interesting topic, I think. Would it be possible to write a module that takes a /node/xy URL, looks it up in the path table, and then redirects to that node's clean URL (if one exists)? That way, even if there is a link somewhere, it would be clear that the /node URL is not supposed to be used, and it's cleaner and perhaps less problematic than doing a URL rewrite.

Burak’s picture

Burak commented

This is the kind of solution I was seeking. Or just in general how everyone else deals with it. Or if it has become problematic for them. Knowing that others may link to a /node/XX is part of the problem.

Example:

Someone sees the link as /node/23 and they try node/24 and node/25 etc. They find some content like in this way then they link to it (they wouldn't know what the proper url alias is to link to anyway). robots.txt is not going to help with that, kind of situation. They should not exist outside of the CMS guts. You can't just ignore them.

Before I posted my question I searched the forum for others asking similar questions, but usually the response is to use robots.txt to hide it, or never link to those pages. It's not really a good solution. It's like saying Problem: My watch is broken, Solution: Don't look at it.

I'm not a fan of using robots.txt to make a list of places that you want people to avoid. It's just asking for it. Hell they are MORE likely to find the node/xx urls if I give them a list than if not.

These same issues also apply to taxonomy, vocabulary and feeds.

I guess for now I'll just nix the urls that start with /node/ and are followed by numbers using mod rewrite, but that's a very ungraceful solution :(

Thanks for the reply and the interest

planetholt’s picture

planetholt commented

I know it's been a while since this was asked, but in case anyone stumbles upon it, I wanted to point out that you should use a HTTP 301 instead of a 404. 301 is a permanent redirect and it tells the spiders from where the content is supposed to be accessed. Apache rewrite rules can do this if you set the right rule flags. (I think it's [R=permanent,L] but you'd need to check the documentation.)