Add XML Sitemap to Core

+1

+1

+1

+1 for core - a must module for SEO

OK a few things with this:

1. In my opinion, core should only handle things which are difficult to do in contrib, or make things in contrib much easier, not simply because they're popular (and there's more popular modules than xmlsitemap which aren't in core).
2. I think I used the google sitemap module for a week or so back in 4.7, and with 10,000 nodes or so on a small VPS it used to almost bring my site down every time it got hit. Has this been dealt with in the new version at all?
3. The specification has changed within the past couple of years, once it's in core, it's frozen for a year at a time (like jQuery) so if there's an update it'll be much harder to modify than if it stays in contrib

So I'm finding it hard to see the benefit of this moving into core.

@catch: Yep this overloading of a site still exists... in the latest D6 patch it is not fixed yet, but we hopefully can solve this with D6 batch api... The main issue is performance of your site that this install step runs through. The sitemap installer does one or more inserts into a xmlsitemap table for every node that exists. This will overwhelm you server or better - PHP will run out of time (main issue). I tested it with 18.000 nodes once and darren added many performance enhancements in D5 times... it behaves much better now, but the issue is not fully solved.

However we must move to batch api about this to get it fully fixed...

Hmm. I believe that there's a redesign in the works too. Because of the code freeze, I think we should postpone it until after that, at the very least.

D7 is in code freeze??? Maybe in a half year, but i haven't read anything about a date yet...

May be it doesn't have to be as fancy as the current module. But at least something in core that will create a sitemap!

it's basically essential to any website. Anyone who doesn't have a sitemap only doesn't because they don't know to do so or their site is for private use only.

My main drupal site site only ever had a sitemap for one week (which made it crash), it's got over 40,000 pages indexed by google and gets over 200,000 referrals from them per month. Please explain why it's essential.

+1

As i know there is no issue if you run gsitemap/xmlsitemap from the first day... the issue with overloading is temporarily... after all nodes are inserted the issue goes away. There are a few ways... wait some time for cron to complete this or do it by hand and ignore the PHP timeouts. For D6 we can solve this in a much better way.

I'm not saying that there's a code freeze right now, but that inclusion in core makes the code "more frozen", so we should wait until after the big rewrite (2.x?) before we try to get it into core. My estimate is that, even if the 2.x rewrite is done by the end of the year, it'll probably be too late for D7, but I'd still like to see it in D8.

@IceCreamYou: There is that, although the multisite one I've been working on ( http://drupal.org/node/202294 ) hasn't been turned down, simply ignored. But it's been getting attention this week from non-Darren people. We'll have to see what happens.

I mentioned 2.x because Darren apparently has some big plan for 2.x. I have some ideas too, which I'm planning to put into a 2.x issue soon. But if we don't get some co-operation within the next month, I wouldn't be surprised if there's enough people interested that we could fork the module. But I'd prefer it if we didn't have to.

To see the ideas for the 2.x version: http://drupal.org/node/253762

My main drupal site site only ever had a sitemap for one week (which made it crash), it's got over 40,000 pages indexed by google and gets over 200,000 referrals from them per month. Please explain why it's essential.

With that many pages indexed by webcrawlers, I'm surprised you don't agree that it's important (I won't go as far to say essential). A while back I had another CMS package with Gallery on a local box with limited bandwidth to the internet. Over time, I built up my Gallery (and eventually Gallery2) to thousands of images. After a while, the web crawlers began discovering my site and crawling it regularly. Since it was loaded with images, they (especially the Googlebots) practically brought my internet connection to its knees every time they crawled, and I had crawlers from all over hitting it many times a day. And they'd keep coming back and downloading the same content over and over again. That's when I discovered the benefit of sitemaps as it was able to define which content has changed without the need for the Googlebots to download the entire site just to see if anything changed. Soon after I fed the sitemap information to Google, I noticed a substantial drop in Googlebot downloading activity, as Google was only needing to periodically check the sitemap to see if any content had changed. Since then, I enjoyed improved performance of the site since they're not pounding on it for several hours per day, which far outweighs the few seconds of processing time for xml generation at the moment that Google comes to grab the sitemap. Sure, you can control many of the bots with robots.txt, but rather than just blocking their ability to index the content, why not just allow them to do their work more efficiently? I've left the other CMS behind now, but the sitemap is probably the thing I miss the most, so very anxious to see it supported in Drupal 6.x, whether it be core or contrib.

+1 from me

+1

@Designer:

Reading about problems with large sites in the posts above. If this is the case this is something which could damage the reputation of Drupal as having buggy parts.

This is not a "Drupal" issue and have nothing to do with reputation.

Despite what some marketing folk would sell you, SEO is an optional function for some websites.

If your intention of the website is to make money, then pagerank is probably a priority to you. However, there are also intranets, extranets, hobby sites, members-only sites, private blogs and resources that do not need to push it.
Plus, it's actually dangerous to have sitemap enabled prematurely, when a site is being built. It's probably a good thing that amature developers don't have it on by default.

It can easily continue to live as a contrib module, as it doesn't need to be tied any tighter to core code than is already possible.
Core is not for cool or even popular modules - it's for things that make the guts of the CMS possible. SEO tweaking is an optional choice.

I agree this feature belongs to a core Drupal more than some other modules (already being a part of core). Besides, if it would be optional, it could be switched off by default in any case...

Putting more features into core will eventually make core unmanageable. Contributed modules allow interested developers to collaborate much more effectively. Features should not go into core unless they change the way core works. CCK was such a feature. Views may be eventually. We have solved most of the problems in XML Sitemap without any changes to core. The remaining problem, of detecting when URL aliases are updated, is also not likely to involve changing core.

If specific changes to core can improve XML Sitemap, we can open issues for them. It is the responsibility of interested developers to become familiar enough with the code to know what those changes should be.

What would really help beginners would be an install profile that would include XML Sitemap. But that's not an issue for Drupal core.

IMHO, Drupal Core does not need a tool to generate sitemaps for Search Engines; some might need it, others definitely don't, it's simply not a feature for Core. It's a feature for end users, not a part of a Framework.

However, what Drupal Core *badly* needs is a mechanism to handle hierarchically (by something like the menu or the book mechanism), or dynamically (by taxonomy) structured content. Such a mechanism would need to be able

* to scale massively (potentially millions of nodes, as the amazing "Eureka!" showcase illustrates (http://drupal.org/node/261340), and
* to output data about the site's structure to contributed tools like "XML sitemap" or *other* sitemap generators not targeted for search engines but humans (like "Sitemap" or "Sitemenu" modules).

Drupal core needs a feature enabling the Framework to *know* about the content it's maintining, and an API to harvest this knowledge by 3rd-party tools. That would allow to solve scalability problems *once*, and offer solutions *multiple* times. That's what a Framework should do.

All tools like "XML sitemap", or "Site map", or "Site Menu" modules do have the same problems coping with mid-sized sites (a few thousand categories, some 10k nodes), not even thinking about really large sites (imagine someone wanting to import a dump of en Wikipedia into Drupal with 2.5 million articles plus additional millions of discussion and administration pages). This problem should be solved conceptually *once*, and that mechanism should be in Drupal Core. The hard work invested in making "XML sitemap" perform reasonably on mid-sized sites could possibly offer a starting point for such a conceptual approch in Core.

Regards, -asb

> will you open a separate issue?

Where would be the right place for this? Should this be an XML Sitemap issue, or a "feature request" for Drupal Core (e.g. http://drupal.org/node/add/project_issue/drupal/feature)?

-asb

I don't think XML sitemaps should be added to the Drupal core. There is still debate over whether XML sitemaps are useful for SEO. (I work in SEO and I don't use them.)

XML sitemaps don't increase your rankings -- they only help search engines discover pages. Search engines don't need XML sitemaps to discover pages though. You just need a good internal linking structure. If you want Google to discover new content immediately, you can ping Google Blogsearch when you add new content.

More thoughts here:
http://groups.drupal.org/node/11304

@#29: Drupal already has a system for building large hierarchies. It's called the menu system. It already has an alternate interface for constructing them. It's called the book module. Those are already in core.

There's also a "Node Hierarchy" module somewhere. I'm not sure the menu system solves the perceived problem.

http://drupal.org/project/nodehierarchy

It sounds to me like asb wants that module in core. That would be pretty cool, because then XML Sitemap would be able to depend on it.

[ Node Hierarchy, http://drupal.org/project/nodehierarchy ]

> It sounds to me like asb wants that module in core. That would be pretty cool, because then XML Sitemap would be able to depend on it.

Yes and no; I think, Drupal core needs to know about relations of it's nodes; those relations do not necessarily need to be hierarchical. That knowledge has to be somehow abstract, contributed modules can harvest this, that's my point.

Some of this functionality already is in core (menues, books), but can not harvested by contributed modules like "XML Sitemap" as far as I know. Simple example: Even a tightly hierarchical site that is completely structured into one "book" does not represent all information required for a Google or Yahoo Sitemap. Also, there is a bunch of contributed modules that try to add this functionality; some examples:

* Simple Sitemap, http://drupal.org/project/simplesitemap
* Node Relativity, http://drupal.org/project/relativity
* Related Links, http://drupal.org/project/relatedlinks
* Site map, http://drupal.org/project/site_map

Some of this modules help creating relations between nodes, but don't display them; some of them solely display nodes and its (herarchical) relations; some of them are targted to end users, others only to machines. All of them need the same information about a site's structure, and that's what IMHO needs to be in Core - nothing else.

I've created a feature request for this: Abstraction layer plus API for structural knowledge about a site's content in Core. Those liking the idea to integrate structural knowledge about a site's content into content should join in there; I'm hoping that there will be pointers to similar discussions.

Greetings, -asb

YES, I would LOVE to see this module in core! Sitemap belong in core as a feature that will be virtually required moving forward. Drupal is amazing but can improve on its SEO capabilities out of the box.

Also desirable would be adding a few existing modules to the core:

XML Sitemap - sitemaps are not optional for SEO
Meta tags - still needed by many engines, the configurable title tags here is also vital.
Path auto - clean urls don't cut it for SEO, they need to be more configurable. At this point they are not even humanly readable or usable either!

I hope XML sitemaps become part of core like RSS has because it's fundamentally important to the success of a website. Keep up the great work, you will always have my support. =)

I totally agree with a simple sitemap creation option in core. Doesn't have to be as complicated as the current module...

I think it would be a mistake to put the sitemaps module in core. It's not essential for SEO. I think sitemaps benefit Google more than webmasters.

Vanessa Fox from Google says, "It’s really not about the ranking; it’s more about crawling… Sitemaps doesn’t impact your ranking at all."

(source)

It's not essential, but it helps. A sitemap, then, it's not just used by Google, but also by Microsoft Live, Yahoo, and Ask.com (at least).

Rather than to add the module, I would like to see ported some of the code, which would be the base for other modules. Apart the normal sitemaps, there are other types of sitemaps. like the code search sitemap (that is actually just used from Google); therefore, there could be a module for the normal sitemap, a module for the code search sitemaps, etc...