redirect from www2.drupal.org to drupal.org

@emsearcy - your explanation makes sense.

@killes - I have no idea ;) I didn't think it was possible.

There is a "select canonical url" option...perhaps that's what emsearcy is referring to:

From the site overview page
Tools > Set preferred domain > { radio buttons which may include the www2, though may not }

I'm not sure that helps with the originally stated problem of duplicate content, though.

I marked http://drupal.org/node/193527 as a duplicate of this.

Also, I'm starting see weird search results e.g. this one which for me only shows a single result from www2.d.o

The two reasons for doing this were 1) monitoring 2) debugging.

Can't we use IP based monitoring? Or use a different url on those hosts that is not quite as important as the www.d.o ? E.g. association.d.o or infarstructure.d.o or groups.d.o ?

About debugging - couldn't people who need to debug add entries to their hosts file to achieve the same result?

I believe that dropping www1 and www2 instead of doing a 301 redirect is hurting us pretty badly in the short term. I know that in the long run we'll get pages re-indexed, but these changes have confused the poor little googlebot which makes it harder for people to find us and harder for regular community members to find content as well. I know we have the drupal.org search but the fact remains that (for various reasons) it is not the preferred way to get this information for many people.

Eric - I've re-read your question about keeping them active and I'm said to say I don't get it. If it's not possible/easy to set up a 301 redirect then I'm ok letting this drop and being patient. But if it's possible/reasonable then it would be *awesome* to get 301s.

One example: there are 109,000 pages that it thinks are on www2.drupal.org.

Any search that returns one of those pages is now going to send the user to a 404. That's a lot of incoming traffic for us to drop on the floor. we know that we can simply remove the www2 but not everybody does. Especially at the reduced crawl rate that we have instructed google to use via our cache patch on drupal.org, it could be a while before we see an improvement.

Pardon my premature submittal and followup to my own post, but I just saw that there are only 64,000 pages in the index for drupal.org so by not using a 301 to redirect this we are reducing the number of valuable pages in the index by 2/3. That feels like a big enough problem to me that we should look into it.

I know that the "result number 1-10 of about XYZ" numbers are not supremely accurate but even if they are off by 50% I think they show enough the problem that we have.

1) the reduced google bot crawl rate has expired two weeks ago. It is still much lower than it used to be, though.

We are still running your patch that sets the updated header to a more accurate (older) number though, right? Even if not, we were for months which has "trained" Google bot about how often it should visit our site. That will mean that for many of those 109,000 pages Google thinks it doesn't need to come back very often since they hadn't been updated in 3 months, 3 years, etc.

I agree with emsearcy about the webmaster tools idea not being feasible.

emsearcy - if we just tell it that those pages no longer exist then I imagine we lose the pagerank they had but, more importantly, now Google has to rediscover them all. Also, afaik there is no such thing as "the next crawl of d.o". It's an ongoing crawl of different parts of the site weighted to their ranks and frequency of update (based on header data and sitemap if available). So it would take some time (see above about how often it crawls pages that haven't been updated for months/years).

I take it from people posting alternate solutions that a 301 redirect isn't simple to do *for the long term*. What about Gerhard's proposal that we go back and start accepting incoming www2.drupal.org requests but then instead of using webmaster tools we use a 301 redirect to forward them to drupal.org. We could then monitor the number of pages that are listed for the www2 subdomain and when it gets reasonably small we could stop serving them up again. That mechanism has worked quite effectively for www.d.o

I agree with you, emsearcy, that it probably does have data about drupal.org pages somewhere in the bowels of its cloud, but the trick is how to get those back to being *the* authoritative version and I believe that a 301 is the best way. The information for webmasters page and dealing with duplicate content pages suggest a couple alternatives. We've done most of the things on the duplicate content ticklist, but haven't done the 301s.

At the risk of opening a new can of worms - another potential solution is to combine emsearcy's idea of removing all wwwN.d.o content with the sitemap module. Of course that requires a security review and performance testing which are beasts of their own.

I decided to check Yahoo! and Live to see what they think - both of them are actually doing relatively well on at least this problem.

Just an update -

109,000 pages that it thinks are on www2.drupal.org

Now down to 58,000.

there are only 64,000 pages in the index for drupal.org

Now up to 424,000

So, the process is moving although not super-fast.

I had also noticed in the past that new pages on d.o were slow to get indexed/cached. This seems to have gone away since pages are now being crawled and included with the index within hours/minutes of being created. Not sure if that's related to the crawl-speed change being lifted or the duplicate page situation or what.

It appears that someone has fixed this since my update on the 22nd.

Many thanks to whomever did that.

I just tested again and the numbers continue to improve. This change to a 301 should help as well so I think it's time to mark it fixed.