XML sitemap 6.x-1.x-dev progress

I think the module can be ready in weeks.
Thanks to the help of andreiashu, I am changing the code to avoid the memory problem caused by the XML site map being cached in a Drupal database table, and then loaded completely in memory before to be shown. I am also changing the code so that the xmlsitemap database table is not emptied before to collect the XML site map links from the modules.

I am changing the code to use only the xmlsitemap table.
It doesn't make sense that a module saves the data about the links which would go in the site map in its own table, and then copy this data into the xmlsitemap table.

Like the 6.x-1.x-dev, and 5.x-1.x-dev versions are now working, the xmlsitemap table gets completely rewritten every time hook_xmlsitemap_links() is called; this means that in a web site with 20000 nodes there are 20001 table rows that are going to be rewritten. hook_xmlsitemap_links() can be called as few times as possible, but still there is the problem on writing 20001 table row when it is not necessary.

subscribing

subscribing

subscribing anxiously..

subscribing

subscribing....

I can hardly wait! One of the big missing modules in 6....

Subscribing. Thanks!

I'm also very keen to know when the Drupal 6 version will be stable. Thanks for all the hard work on this lads!

It is nice to see that this is in progress! Subscribing!

Subscribing. Thanks for this future release.

Subscribing

Subscribing

@earnie. There is no other version we can use for the issue, so 6.x-1.x-dev it is.

Just like what is currently on the main module page http://drupal.org/project/xmlsitemap

"ATTENTION
The 6.x-1.x-dev, and 6.x-2.x-dev versions are still work in progress to port the code from 5.x-2.x-dev; don't use them in production web sites.
6.x-2.x-dev has been created to resolve a problem I had on submitting the project files in CVS repository; the branch has been kept, and it now contains code that is not compatible with the older branches.
6.x-0.x-dev, and 6.x-1.x-dev will be soon declared unsupported."

Subscribing!

Subscribing

You don't see any 6.x-2.x-dev release because the code of that branch is still not associated to any public releases; that part on the project page has been written so people read it, and they know the procedure to install the 6.x-2 version is. I wrote it before to release that version so people will read it before to install the 6.x-2 branch of the code.

subscribing

@earnie - You seems to be a programmer, so maybe contacting Kiam (http://drupal.org/user/55077) directly to help with his tasks would be the best approach. He seems to assign to himself quite a lot http://drupal.org/project/issues/xmlsitemap

could not get sitemap xml module to work in 6.9 linux install - only seems to create the base url in the page and no other nodes (reinstalled and configured corectly) - here is a simple alternate mod i found
http://dinaiz-two-dot-zero.blogspot.com/2008/07/easily-add-sitemap-to-dr... - quick and dirty until xml sitemaps module is fully ported and tested.

subscribe, thx!

@earnie: Thanks for your report.
If the declaration of those indexes is causing the problem you described, then I will remove such SQL indexes; the database table has already the most important indexes, and frankly using like index a field that is arbitrarily used by third party modules doesn't make any sense.

Thanks again for pointing out this; your help is well appreciated.

subscribing

@ripcloud: did you enable any other modules apart xmlsitemap.module? If you just enabled that module, then it's normal that you don't see any other link in the site map.

For all the commenters here: this is a support request which is supposed to have the progress in the Drupal 6 port like topic; it's not an all-purpose container for all the reports one can think of.

Any comments which are not strictly related to the progress of porting the module for Drupal 6 should be avoided. If you have any comment about a different topic, please open a different report.

So far, the only comments which should be present here are the ones from who is submitting to this issue, or the comments of a maintainer who would report about the progress in the module developing.

Thank you for your cooperation.

subscribed

subscribing

subscribe

subscribe

subscribe

subscribing.

subscribing...

With the risk of being repetitive... subscribing.

Definitely subscribing (the site should have an option to do that without spamming)

Guess what? I'm also subscribing :p

me too :)

Subscribing. This is an important module, so getting a stable version would be very much appreciated, even if it doesn't do everything. There's always the next version. ;)

Looking forward to it!

subscribing

Subscribing

Guess what? ... Subscribing!

it's pretty retarded that we have to leave a "subscribing" comment to follow a node. There are module for this so why aren't they used on drupal.org???

btw..... subscribing

subscribing

To all subscribers:

- http://drupal.org/node/371663

subscribing

Yeah, me too

subscribing

Subscribing )
Respect for your work!

well
subscribing...
:)

subscribing

subscribing

subscribing

Guess what? Exactly! subscribing

Subscribing....great modules. Thanks for your effort.

Is there any version or other module wich i can use for D6 on a production serveR?

I am using views and views node feed to temporary build a xml sitemap like structure, while waiting for xml sitemmap's official release. Thumbs up for xml sitemap!

subscribing

subs...

subscribing

sorry... only subscribing to this must-have-module. :)

Subscribing......

sub

Subscribing...

subscribing!

subscribing!

subscribing

i'm waiting ...

Subscribing and hoping it's available soon. :)

subscribing

Subscribing

Subscribing.

Subscribing

subscribing

subscribing

The 6.x-1.x-dev, and 6.x-2.x-dev versions are still work in progress to port the code from 5.x-2.x-dev; don't use them in production web sites. 6.x-2.x-dev has been created to resolve a problem I had on submitting the project files in CVS repository; the branch has been kept, and it now contains code that is not compatible with the older branches.
6.x-0.x-dev, and 6.x-1.x-dev will be soon declared unsupported.

Why is the 6.x-1.x-dev branch still getting regular updates if it "will be soon declared unsupported"??

J

The dev version from the 15th seems to be working OK for me.

subscribing

subscribing

subscribing

subscribing

I have to say that 15th March version works fine to me too. You can start using it and report bugs instead of keep "subscribing"

Just a heads up for anyone that experiences the same issue. This module was the straw that broke the camel's back as far as MySQL went. We had to adjust our MySQL settings. You'll know if this happens to you because you'll get the "MySQL server has gone away" error.

I have been waiting for a drupal 6 compatible XML sitemap module to be developed since November 2008. Quite frankly I am growing very impatient.

What is so difficult? Do you need money to finish the module? I would gladly pay. I was disappointed by the things I read in the support forum for this module. There was far too many foolish restrictions.

Let's be honest. We (as drupal users) just need this module to function just as it did for Drupal 5.

With the advent of Drupal 7, it seems this module is well overdue. This module is the only module preventing me from jumping to Drupal 6. Everytime I check to see a working version of XML Sitemap for Drupal 6, I have to retreat to using Drupal 5.

Please let us know what the hold up is, and how much money it will take to fix it. I need the XML Sitemap for Drupal 6 to perform and function just as it did for Drupal 5. If you cannot complete this task, please just say so. I will be happy to outsource this module.

Thank you.

@puli6912 : This is not a discussion board. This is here so that all subscribers will know when an official version 6 will be online.

Until then you can...

1- If you want to discuss, use the forums http://drupal.org/forum

2- If you want to pay for some developpement, contact the author Kiam (http://drupal.org/user/55077) or use this http://drupal.org/services or/and this http://drupal.org/paid-services

3- You can always test the dev module and if it work for you, keep that version and upgrade to version 6. Of course, that's unsupported.

... or (4) - re-port the xmlsitemap 5.x branch yourself, which I just did (see attached tarball).

I have lost confidence that, despite Kiam's valiant efforts, we're going to get what we need any time soon. The 6.x port has deviated far from the functionality of 5.x, with additions the community does not want (like lack of admin access to sitemap.xml), and no workable governance or change control processes for this project.

What can be so difficult about upgrading this module to 6.x? As it turns out, nothing. It took me 4 hours from start to finish, and I've never built a Drupal module before. The cause of the delays with the 6.x port is that the 6.x-1.x-dev branch is not a simple port of 5.x to 6.x. Rather, it is a complete re-write and re-organization of the xmlsitemap code, with many feature changes.

The attached tarball is working fine on my production site. It is an automated conversion of 5.x-1.6 to 6.x, with a few small fixes detailed below. I propose that we create a new branch to house it, 6.x-3.x-dev. A larger group of developers should be given access to this branch, and its charter should be to exactly duplicate 5.x-1.6 functionality with NO changes. Once that is stable, then people can start adding new bells and whistles.

Here's how this tarball was built. I followed the recommendations of http://drupal.org/node/114774 for "Converting 5.x modules to 6.x":
(1) Uploaded the 5.x-1.6 xmlsitemap tarball to http://boombatower.com/tools/deadwood to run it through the Deadwood project
(2) Installed the ported xmlsitemap code (output of running Deadwood on it) on my 6.x site
(3) Installed and ran the Schema project (http://drupal.org/project/schema), and fixed the 4 schema errors in the .install files
(4) Installed and ran the Coder project (http://drupal.org/project/coder). This identified problems with the menus and themes.
(5) Upgraded xmlsitemap.module.xmlsitemap_menu (hook_menu) function, per Drupal recommendations (http://drupal.org/node/103114).
(6) Fixed an access problem for the Engines and Additional tabs by adding appropriate 'access callback' commands
(7) Didn't bother to fix the hook_theme alert from Coder, as it doesn't affect me.
(8) Rebundle everything
(9) Created a clean install of Drupal, cleanly installed 6.x-3.x-dev, and tested. It did load all old content, and is working perfectly for my needs (e.g. main modules, terms, and engines). I don't use the user module.

Some of the things Kiam is working on sound quite nice, and I'll be glad to see the finished product. In the interim, I think we can make this new branch stable much more quickly since it has no features changes.

Rather than being a step forward, the proposed 6.x-3.x-dev release is a step back.
Just to make an example, the xmlsitemap_settings_sitemap_validate() function checks if the xmlsitemap_chunk_size parameter is not greater than 50000, but it doesn't check if the parameter is positive.

Like reported by alexandreracine, this is not the right place to discuss of things like this that should find a better place in the Drupal.org forums.

Well said brother. Well said. I too am discouraged by the silly modifications that have been projected for the 6.x release. I too am looking for a 6.x compatible version that functions identically as XML Sitemap 5.x did.

I think everyone has been quite calm and respectful. There is nothing to "calm down" about. What's needed here is some clarification and project management. For example, on the xmlsitemap project page, there is this confusing statement that, despite my email to Kiam, has not been resolved.

It says: "6.x-2.x-dev has been created to resolve a problem I had on submitting the project files in CVS repository; the branch has been kept, and it now contains code that is not compatible with the older branches. 6.x-0.x-dev, and 6.x-1.x-dev will be soon declared unsupported."

Well, there is no "6.x-2" rollup listed for download. Are we supposed to go into CVS and get this manually? Please explain. And why is the "6.x-1.x-dev" version getting many recent updates when it says it will "be soon declared unsupported"?

In any case, I agree with several other postings that the community just needs the basic functionality without additions. Once that basic functionality is solid, enhancements are welcome.

J

Yes I second that same question:

Can anyone verify tw176's module?

Any feedback would help greatly. Thank you.

The project page contains a reference to a 6.x-2 branch so Everybody can read from now what they're supposed to do when the new branch will be ready. The code that is being committed in the 6.x-1 branch will be present also in the newer branch.

The proposed branch doesn't simply avoid any enhancements, but it also removes the code that is thought to fix some incorrect behaviors present in the previous commits. Removing a stricter checking on the settings passed to a module is not to remove a new feature but it's to remove code that should avoid bugs in the code; I can just imagine what would happen if the module would accept negative numbers for the site map chunks.

Hi Kiam,

If you mention version 6.x-2 without offering a rollup of the code, it's very confusing. I'd recommend two changes to the xmlsitemap project page:

1. "Upgrading to XML Sitemap 6.x-2...."

Remove this section until you have posted a 6.x-2 rollup

2. "The 6.x-1.x-dev, and 6.x-2.x-dev versions are still work in progress to port the code from 5.x-2.x-dev; don't use them in production web sites. 6.x-2.x-dev has been created to resolve a problem I had on submitting the project files in CVS repository; the branch has been kept, and it now contains code that is not compatible with the older branches. 6.x-0.x-dev, and 6.x-1.x-dev will be soon declared unsupported. "

Change this to:

"The 6.x-1.x-dev version is still a work in progress to port the code from 5.x-2.x-dev; don't use the 6.x-0 or 6.x-1 versions in production web sites. I am working on a version 6.x-2.x-dev to fix various problems with previous branches. 6.x-0.x-dev, and 6.x-1.x-dev will be soon declared unsupported. "

You might also add:

"There is currently NO Drupal 6 version of xmlsitemap ready for production web sites."

J

If anybody is looking for a way to create a site map, and cannot wait until the module will be bug-free, there is a different way to create a site map.

A site map can contains links to RSS feeds, which can be generated dynamically from Views.
That is surely a better solution than to use a module that does not check the input given for its settings (and that should replace another module because it's said to be buggy).

See http://www.sitemaps.org/protocol.php, under Other Sitemap formats.

Kiam: Thank you for your efforts on behalf of the drupal community.

In researching this issue I stumbled upon the following link from where one can download a very small 6.x module that generates a sitemap for one's site. It is not nearly as elaborate or sophisticated as the version you are developing, but it may nonetheless satisfy those here who are looking for a short term solution.

http://dinaiz-two-dot-zero.blogspot.com/2008/07/easily-add-sitemap-to-dr...

I am using it and it works flawlessly.

At the same time, I look forward to the masterpiece you are developing.

bwv, but your module does not generate map in xml-format, correct?

I think it does... you can find it here:

http://horsezilla.com/sitemap.xml

Google webmasters' tools also shows no errors.

It surely does (if the question was for me).

Kiam, what approximitly how much time do you need to finish 6.x stable?

I cannot reply to the question, as it's not a task I am actually doing on a daily schedule, lately. My intention is to develop the 6.x-2, which should resolve the timeout issue.

I installed the 6.x-1.x-dev release today on a small site (~45 urls) and it is handling basic nodes correctly. Not sure if this is what you are referring to as the 2.x branch, but it says it was updated today and its basic uses seem stable to me. Thanks for all the hard work you have put into this so far.

To patch tw176's 6.x-3.x code for the problem Kiam mentions, simply add these lines to linkchecker.module at line 199:

  if ($form_state['values']['xmlsitemap_chunk_size'] < 1) {
  	form_set_error('xmlsitemap_chunk_size', t('Cannot send less than 1 link at a time.'));
  }

If this code from tw176 works (still testing here) I think we should consider forking this code and potentially creating a new module: NoFrillsXMLSitemap. Some of us want a module that is:
(1) production-ready
(2) now
(3) without lots of new features

I am sure there are other challenges with the 3.x code -- there always are. But those with a strong desire for the three attributes mentioned above may be able to focus some development effort against the code and get something out quickly. Might not work, but it might.

When Kiam's version is production-ready, I'm sure many NoFrills... users would switch back, myself included. Kiam, please do not take this personally: no one can please everyone all of the time, and there are users here with very different needs. Your efforts are appreciated, and a forking of the code would *not* constitute any less faith in your ability to deliver a tremendous module.

Creating a new module that has the same purpose of an already existing module is contrary to Drupal.org, which suggests to co-operate with the maintainers of existing modules; that is the reason I asked to co-maintain this project.

To fork the existing code could serve a purpose, but if the fork doesn't include the fixes implemented in the last commits, then it's a waste of time; if then it does include the last bug fixes, it still is a waste of time because is a fork of code that already does the same things.
I think the better solution would be to propose patches for the currently committed code; the patch should take in consideration that there is already a plan to change the code so that is not anymore needed to copy the database content from a table to another one.

With Drupal 6 close to a year old and this module in a perpetual state of limbo.. is there any chance of opening up development to contributors? On behalf of the eager hordes, specifically those of us who *could* help the effort, I'd like to request the following:
* That the v6.x-2.x branch be made available as a dev release,
* That all planned changes & fixes be added to the issues list so that the community can help speed up the development process,
* A message be posted on the project main page indicating that the v6.x-2.x branch is only for development and that anyone wishing to help should start working on the tasks.

Kiam, would you be willing to do this? I think this would take some of the load off your shoulders *and* get the project closer to a solid release much more quickly, and speaking for myself I have both myself and another developer working with me on a project that could immediately start helping but we need the above in place first. Please please please please please :)

I would like to help as well. yes please open the -dev-2 branch to development.

I've been maintaining d5-dev2 fork at work for quite a while now and developed various optimization methods that don't kill our site when we try to rebuild xmlsitemap index from a node table of over 1.3mil records and 200k users and (a ton?) or taxonomies. I would like to work these optimizations into the core of this module so i can finally stop maintaining it on the side :-D.

So what Damien said, Please please please please please :)

@earnie: litwol was probably seconding my request to have v6.x-2.x made "open" by "be[ing] made available as a dev release", I (personally) know it's available via CVS but Kiam's comments suggests he doesn't want anyone to touch it until he's finished. Also, having a list of known issues & planned changes made public we know where to start, rather than each of us having to duplicate this initial groundwork. With such an important tool for modern websites, open communication and collaboration will help everyone.

@earnie, offense taken. I was only seconding Damien's post. However, you're absolutely right, i should have done my research regarding -dev2 branch rather than just seconding the post.

It's my intention to create a 6.x-2 branch, but I would prefer to resolve the problems present in the 6.x-1 branch first; the code of 6.x-1 will be ported to 6.x-2, with the exception of the code handling the database tables.

@kiam: Thanks for that quick response and for clearing up your current plans. I'll see what I can do to verify the output generated and work on patches to rectify any issues. Do you think you could update the project description to remove references to v6.x-2.x-dev until you are in a position to release a dev version, as you can see it is confusing people as to your intentions?

@litwol: Please try to port your improvements to v6.x-1.x and submit a patch for them.

I will change the description in the project page to make it less confusing; I will be do it as soon as possible.

I'm completely +1 on adding multiple maintainers for this project. It's too important not to, especially when there are a lot of major architectural problems that need solving. It seems like for every step forward, the module takes two steps back. We can avoid those types of problems with coordination and discussion.

Tried installing tw176's module...and instantly:

# warning: Missing argument 4 for xmlsitemap_node_views_query_alter(), called in /var/www/sites/all/modules/views/includes/view.inc on line 618 and defined in /var/www/sites/all/modules/xmlsitemap/xmlsitemap_node/xmlsitemap_node.module on line 520.

(error above showed about 6 times)

# The installed version of at least one of your modules or themes is no longer supported. Upgrading or disabling is strongly recommended! Please see the project homepage for more details. See the available updates page for more information.

missed that?

I think we should close out this topic as it has ran its course.

At this point I think we should:
* Continue to use the v6.x-1.x codebase,
* Anyone interested in helping should use the code from CVS,
* Submit bug reports and, more importantly, patches,
* Review patches that are submitted.

Subcribing...

Re: #139

@earnie: there were a few people talking back and forth about the port that tw176 posted in this thread. many people asked if it had been tested. What you linked to is irrelevant.

agrees with earnie http://drupal.org/node/359104#comment-1417638

everything works out better and moves faster as a working community

@#136: That happens because the implementation of the hooks required by Views is changed. The fork didn't change that part of the code; therefore, it's perfectly useless (and that is the reason I suggest to not use the alternative fork).

/me applauds Damian and crosses his fingers... ;)

There are still some issues that needs a work out, before I continue to develop the 6.x-2 branch. I would like to fix those issues, before to continue. It would really help, if somebody could check the reports I set as postponed (maintainer needs more info), and verify if there is really a bug in the code.

earnie: I think working through some of the v6.x-1.x issues would be a good start. I'm still waiting to hear back from someone on the output..

Anxiously awaiting this port. Thanks for working on this!

Subscribing!

I just pieced together what I believe to be a good temporary fix for 6.x users needing a valid sitemap.

I know this issue is supposed to be reserved for the progress of the actual 6.x port but for one, I didn't want to create a new issue because this really isn't a bug or support request, and two, because the thousand of people subscribing to this issue will most likely find this very useful.

Well like all of you, I desperately needed/wanted a sitemap module for 6.x - we all know how important it is. I'm not at all being rude or greedy, but we all need a solution now. I love the XMLSitemap module and thank every single person who made it possible. But I had to take this one into my own hands for the time being - until the 6.x release is completed and bug-free. I spent the last few hours writing a quick and simple module which will provide your site with a valid, complete sitemap.

*IMPORTANT* - To use this module, you must do some code configuring yourself to achieve what you want. I didn't have time to make an admin settings page (although I welcome someone to edit it so). I'll explain more below. Also, this module is only meant to be used until a full 6.x release of xmlsitemap is ready. There aren't any features in this module besides simply creating and returning a sitemap - it is very crude and straight-forward (please expand upon it if you'd like - don't forget to repost though).

OK. Now to explain how it works..

A sitemap is generated upon invoking cron - so don't install it, access /sitemap.xml, see no sitemap and freak out. After installing, run cron once. You can edit this module to determine how often to create a new sitemap. The sitemap is generated, saved at /sites/default/files/sitemap.xml. When you access www.site.com/sitemap.xml, the file is grabbed and returned, just like it existed at that location (so obviously, don't have xmlsitemap installed while using this). By default, this module will generate a new sitemap every 24 hours (see isitemap_cron() and edit the number in the first IF statement if you want it longer/shorter). There is no database - nodes aren't added to the sitemap right after they are created/deleted.

Sitemap content: Also in isitemap_cron() you can decide what content to add to the sitemap. By default the sitemap first adds the root address, then all of the primary links which are not nodes, then all of the taxonomy term pages (edit the SQL to specify which vocabID's to include), then every published node on the site (edit the SQL to specify which node types). For the node entries, the most recent update (content or comments) time is used. Feel free to change the priority and change frequency numbers as well. When I have time, I'll create an admin settings page for all of this. For now, just tear up the simple code.

Like xmlsitemap - the sitemaps are only accessible to anonymous users. I didn't want it this way, but since I'm used to 5.x modules, I didn't have the time to figure out how to allow all user access.

That's it...

If I find any bugs, I'll fix and reupload. Any comments, bugs, or feature requests/additions, post or PM me.

Sorry for hijacking this thread but I'm sure this will help a lot of people.

**EDIT**
About 30 minutes after submitting to Google Webmaster Tools, Google reports a successful download, and all 13,000+ URLs submitted without a problem. Also, this module doesn't create separate sitemaps so do not use if your site has +50,000 URLs

I think it's not possible. I would remember, anyway, that the issue queue is thought for the reports related with the project; patches to apply to the project code are welcome, but a complete rewritting of the project should be posted in a different place, like the Drupal general forum.

As one who previously introduced (http://drupal.org/node/359104#comment-1404928) a suggested alternative, I can report that I followed the instructions here (thank you earnie):

http://drupal.org/node/379854#comment-1421236

And am having great success with Kiam's code.

subscribing stressed....

subscribing

subscribe

++1

I read through this entire thread and am still a bit confused about which branch I should be testing/fixing bugs in. Would love a notice on http://drupal.org/project/xmlsitemap that says in bold, blinking, marquee font:

"If you are a developer wanting to help get a 6.x release of this module out as soon as possible, please hack on the XXXX branch."

It sounds like XXXX is DRUPAL-6--1 unless your site has a lot of nodes (which my client's does; 100,000+) in which case it should be DRUPAL-6--2, but DRUPAL-6--2 won't be done until DRUPAL-6--1 is done..? I'm so confused. :\

Basically, if #126 could be made a priority as soon as humanly possible, that would probably make everyone a heck of a lot happier. :)

In the meantime, I'm going to take a stab and work on DRUPAL-6--1, since that has a commit from 5 hours ago, and DRUPAL-6--2 has one from 8 weeks ago.

For others who might be wondering, that does indeed seem to be the case.

The CHANGELOG.txt of both branches actually look very nicely maintained, so it's easy to tell from a quick diff which fixes are in which branch. Thanks for that, Kiam.

I started to develop the 6.x-2 when I understood that the modules should use a unique database table to save the links which should populate the site map; as the new code is supposed to be not compatible with the old code, I created a new branch.
I have been then recalled about the necessity to back-port the code, and I started to develop again the 6.x-1.

To make a long story short, the branch currently developed is the 6.x-1 one; its code will be ported to the 6.x-2 branch that will be the branch where all links for the site map will be saved in a unique table.

I am sorry for the confusion I generated, which has been caused by my mistake to create a development snapshot from code that was ready to be tested (and from not knowing some details about how the Drupal CVS repository works).

- Duplicated comment -

Unpublished content is currently not being populated to the sitemap.xml

Do you mean that unpublished nodes do not appear in the site map? That is the behavior I would expect.
I am sure I am misunderstanding the sentence you wrote (it must be so; next, I will see mirages :-) ).

There is nothing to be sorry; we are all here to try to resolve issues, and you already helped many times with XML Sitemap issues.

I can report that the nodes in my web site are not promoted to front page, and they are present in the site map. This doesn't mean anything, as I don't have part of the issues the other people have with this project code.

The development plan is to resolve the most important issue with the project (#379854: The site map is not being populated), and then start to port the code to the 2.x branch, which will change the database tables used.
Before this happens, the code to populate the project database tables during cron tasks must be implemented.

The multilingual support will be completed in the 2.x branch (if it still needs some work).

Kiam: Please post a quick update on the project summary to list the planned releases and expected functionality, e.g. at what point do you expect it to list all nodes. Currently there's confusion on what is expected so there's confusion on what you anticipate fixing versus what you would like help to work on. Thanks.

subscribing

169 comments! Wow!

If everyone who responded here donated 5 bucks we probably could have paid a developer to make a new sitemap module from scratch in the last 3 months. :)

In 169 comments over 80% is people subscribing to track this issue and make the maintainers know that they are waiting...and waiting...

The code is going to be further changed, after the code review made from webchick.
Who wants to help in the development can confirm if the site map is being populated, and the multilingual sites are correctly handled.

Thank anyone for the help given with this project.

@#168: I am not sure I understand what you mean; may you be more explicative (English is not my first language; that can explain why I don't understand all I read :-))?

For the person who posted comment #170:

Your remarks are out of line. Nothing in life is free except the sun rising in the east. If you are impatient please feel free to contact a developer on your own. Meantime, please exhibit a little more respect, and show a little more support, for the individual(s) who, are not complaining, but rather doing all the work.

I am not a coder, but I would be happy to help with the documentation for the project. Also may be able to test on a Drupal 6.10 site. Let me know.
Tom

bwv,

How is my comment out of line again? And what exact purpose does your comment#174 add to this conversation?

My point was through 169+ comments - 90% is a lot of junk - people subscribing.

Get a grip man...you been here long enough to know better than to flame.

Thanks though!

We shall not start such arguments here; the issue queue is not the place to start a battle about what somebody said, and what he meant. It is already happened in the past, and it's not a good thing for the project developing.

thats better, now I'm feeling the love :-)

Happy Earth Day everyone

are you "recycling" your code ;-)

That is the very point I was making. If SOOO many people are keen on this, let's all pitch in and get this going. Kiam if some cash will speed up the development place a chip-in widget mate.

Thank you for the progress you have made thus far.

I agree; the more useful help is to help in the development of the code.

I understand your perspective, but keep in consideration that I am co-maintaining this module in my spare time, without to get paid for what I do here. I am not absolutely developing the code in slow motion, and waiting for somebody to offer me money (and then, magically starting to develop at the speed of light).

Drupal is developed like free software, and the code you can download from Drupal.org is available freely. You cannot pretend the modules are developed in few time, or that who develops the modules is able to predict all the needs of who will use his module.

I hope I will not misunderstood for what I right said.

@Kiam: I don't really read it like that; I think what people are saying is that they appreciate your work and really would like to help you achieve it, but since they're not developers they feel somewhat stranded. But something non-developers can do is send a few dollars your way. The idea is not to imply that you're being somehow sneaky with how you're spending your time, but more to say thanks. :)

If time is the main constraint and money won't sway things one way or the other, then you're probably right to not put a Paypal link up. But if there's a chance that if enough money was raised you could take a break from your day job / school / whatever for a couple days and just focus on the module, then it might be a good move, and would give those who can't help with code reviews and patches a constructive way to contribute to something they'd like to see happen.

I didn't mean somebody was saying that; still, I wanted to be clear on that.

In my case, money doesn't help me to have more time to develop the project, nor raises the quality of my code. :-)

Webschick, spot on. That is exactly how I have read this thread. I think it is easy to forget that it is not just coders in the community, but a lot of people who want to "chip in" with whatever they can offer. Unfortunately if you are not a coder it can sometimes be miss-interpretatied in the wrong way. We are all here to try and help out the best we can from our own needs, knowledge and experience...

What is the point of all of the complex tables on the backend, and extra features? I think the sitemap should just be generated on the fly dynamically based on which node types, tax terms, menu items, etc, the user specifies in the admin settings. XML file gets stored as sitemap.xml and returned when a SE hits it. You can update it on cron at whatever interval you specify. I know everyone yelled at me for the temporary sitemap module I posted here, but to be honest, it took me about 3 hours to write from scratch and works perfectly for me (for now).

The keyword here is from scratch. It's surely easier to make a module from scratch than to modify existing code.

Still, when someone spend 3 hours of their time and duct tape something together, still manage to satisfies my need better then a project that have been on going for months, say something.

None of us are saying the XML module is not on the right track, but problem being it try to race to the goal without stopping for gas, and gave us the feeling that we are all stalled on the side of the highway waiting for someone to help. An important module like this should, at the very least, have one usable and stable version for the most recent version, however feature lacking it might be.

It doesn't really matter how much bell and whistle there is if all we get to do is stares at it for months on end. Some kinda balance need to be made.

http://drupal.org/node/359104#comment-1473510

I'm with mike on this one. I don't need any fancy features for a sitemap module, I just need my node URLs printed in an XML format for Google. Updated on cron and cached otherwise.

Currently I can't believe Drupal 6 does not have a working sitemap module.

XML sitemaps for my site with about 500-750 nodes, is unable to produce a sitemap on my shared hosting (dreamhost) with a 120 second execution time, it appears to timeout and I receive a 404 error. I'm not really interested in any of the complex features this module provides. I feel waiting until 1, this module is complete and 2, this module is optimized enough, so that sitemaps get created in a reasonable amount of time....I'll be waiting for another couple months to come. My sites can't afford this.

I'll be going mikes road and either downloading his, or creating my own module to get things done. Like mike, I do not believe it will take more than a couple hours to create something for my (and probably most users) needs.

I do appreciate all the work for people who are working on this project (and all other drupal projects), but when a stalled project is holding other people back, those people should step up and create/share something to fill the gap.

Have to agree with the last post. Many of us need the basic xml site map so that we can migrate our Drupal 5 websites to Drupal 6. We also have clients who want to take advantage of some of the functionality in drupal 6 and they are asking about progess on this module.

If we're talking about an additional week or two to get the bells and whistles fine. But clearly several more months would not be good. People will start to explore other alternatives if that is the case and that would in my opinion be a shame after all the effort that is going in to this.

It's the module maintainers call and you'll hear no more from me on this issue.

how about rather than making demands, you help out? http://drupal.org/node/448000 is the place.

Definitely will try to now that I know where to go. Thanks for pointing this out to me. I never realized what some of the issues were.

There's no reason why my quick solution can't be used while helping out the XML sitemap project. I'm not sure why people seem so split on the issue like you can't do both. I was just trying to help out everyone and myself included. Just remember if you use my module you have to edit the code to make it work for you (node types, tax categories, etc) - just look at the function that chooses which to content to add to the sitemap.

If I wasn't so busy I would be helping get XML sitemap ready for 6 - but I don't have the time at the moment.

mikestefff: I would love if you reposted your code and ideas in #448000: Big, Over-Arching Sitemap Architecture Discussion (tm).

An update for everyone following this thread: there are now three active maintainers for this project instead of one, which should 1, help move things a little faster and 2, increase the quality of our code since we're trying to peer review before committing. Earnie is now helping Kiam a little more actively in the 6.x-1.x-dev version with getting bugs fixed for a stable release soon. I've been added as the maintainer of the 6.x-2.x-dev version, which is a complete rewrite with considerations for performance, scalability, and reliability.

@dave reid: just posted a link to it and some other rantings about why i think this module is way overkill to what is needed. keeping such detailed tabs on every node, etc, is wasteful and unnecessary, I believe. Read Google's docs on sitemaps. No database should be used for this, unless there is something I am missing, which is very possible.

I took mikestiffs code and edited it to provide sitemap links to about 90% of my pages. Rest are too hard, as they're views with dynamic arguments, but this will work for most of my links, so I'm happy for now. I consider it a viable solution for those in need (and with programming experience) while we wait for a feature full sitemap to get completed. I consider mike's posting useful, even though he got flamed, because he resolved my issue for this.

I think the 80/20 principle (Parento principle) which Dries has talked about should be applied to this module. Spend 20% of your time making this module work for 80% of the easiest links to add into a sitemap. Put out a stable but featureless release. This should satisfy at least 80% of the people at least in the short term.

Then spend the other 80% of the time creating a more robust featurefull version for everyone...but I think right now, Drupal 6 needs something to generate sitemaps, quick and easy...even if it doesn't index everything.

I'm in the same boat as the maintainer of this module though, I don't have enough time to work on all my projects either. Not blaming you on this, I personally just needed a solution.

This modules been in dev for like half a year...is that normal?

@Kiam, @Earnie, and Dave: I am maintaining another software development project (not drupal related) in my spare time, so I can understand your situation, where time is the major limiting factor. You have been and are doing a great job!

I would rather have a well designed, maintainable, scalable, and flexible module that will stand up to future developments like new modules, or that can be ported more easily to new drupal versions than a hastily thought out version that currently meets my needs, but has to be tinkered with each time changes are made in other places.

@everyoneWaiting: Maybe we should let the development just go on, and not make the developers spend their time reading and responding to comments that do not help in the development process.

Also, for a summary of the current state of the module, where we are going to be headed, and to help sponsor the 6.x-2.x rewrite, please read http://blog.davereid.net/content/state-of-drupal-xml-sitemap-2009.

@Dave Ried, thanks for the write up. Really lets you know where this project is going.

Thanks for everyones hard work on getting a 6.x beta version out, looking forward to testing it on my new sites.

i am glad to see the beta releases! thank you!

In the next beta release, the modules will not copy their data from their database table to a central table, but they will use their table to directly populate the cache files used to serve the sitemap content; the cache files will be created in a different time, and if the cache files have not been created for any reasons, the code will try to serve a sitemap by asking to the modules to output the first 100 links of the sitemap chunk being requested. In that way, the code will be able at least to offer a partial content of the sitemap, even in the case it is not able to give the full content; this would also resolve the PHP timeout issue that some users noticed.

Viewing the sitemap will not cause the cache files to be generated; the cache files will be generated in a different time, and I am actually implementing the code for sitemap.php, a file that should be used in the same way cron.php is (it is called by the cron command), and that would uniquely create the cache files for each chunk of the sitemap.

I set #468142: Extimated time of arrival as duplicate of this report.

See also #465734: Use sitemap chunks per module to prevent the rebuilt of all the cached files, #439632: Viewing the sitemap should not also update it, #467442: Provide a system to generate the cache files before the sitemap is viewed, and #463564: Remove the central database table.

In the next hours I will commit all the changes I have done in the past days.
Take in consideration that the code could contain some errors that don't allow it to work in all configurations. Please, report any errors you should find.

The difference with the past code is that the creation of the cache files will not happen when the sitemap is being request, but at a different time; actual code contains a file that must be invoked in the same way cron.php is. In the future, the tools page will also allow to create the cache files, and the cache files will only be created for the modules that need to updated their cache files.
I cannot do it tonight, and soon I will not be able to work on this project as I am doing now because a vacation I planned to do. My plan for the next days is to give to the community the code I developed in these past days, and to fix some bugs, if there would be any (which is not something I can actually exclude). What I cannot do in the next two weeks will need to wait my return, or another maintainer to do it. I am not sure what I will be able to do on the project code for the next two months, but I will not be surely able to do as much as I am doing now (which could mean I am not able to write a single line of code).

-- Kiam
Kiam la luno renkontas la sunon.

The next development snapshot that Drupal.org will make available removes the limits on the number of links the code outputs when the cache files are not found, or it's not possible to open them.
This means that the code is able to output the sitemap chunks even when the cache files have not been created, yet; the operation can take a little more time than serving the content from the cache files, but at least it outputs the content of the sitemap.

The beta5 version has been already created; for a list of the features it adds, see the the release page for the beta4 version. The beta5 version adds an update function that was missing from the beta4.

First, thanks to everyone for the discussion and work on this--I just want to say that its been an education to me (and a very edifying one) in the inner workings of the Drupal community and how the work actually gets done.

I just thought you might like to hear, from a (relatively) outside point of view, that the way the Drupal community works, and the social skills (as well as the technical expertise) involved is impressive.

I have one question, since I'm not that technically savvy.

I have what I believe is a relatively small site--about 1500 nodes--and I've been waiting for this module so I could upgrade to 6.x. Am I correct in thinking that for a site my size it should be reasonably safe to go ahead at this point with 6.x 1.0 beta5 ? Or am I jumping the gun?

Obviously, I'm not asking for guarantees--I'm not that new. But I am trying to build my readership, and so want to make sure that upgrading at this point isn't incredibly stupid.

I'll be very appreciative of any feedback...

Thanks,

Ken
the daily mull

I am in a similar situation with my site and like you I am concerned about building readership. I think the answer to your question may depend how current users are finding your site. If it is via Search Engines primarily and you are using version 5 of the XML SiteMap module than you may want to stay with Drupal 5.x for the time being to limit the downside risk of losing the ability to attract the search engines.

Of course this all depends on your risk tolerance. To put this into better context "If u are the kind of person who wears a belt and suspenders and still worries about their pants falling down, then don't use the version 6 beta. [sorry, just couldn't help myself]. Just wait for a stable version of XML SiteMap in the 6.x-1.x version and then switch over.

Also consider what is to be gained in Drupal 6.x. If Drupal 5.x is providing all the functionality that you need, then relax and stay with it. It will be a round for a long time.

Your site like mine is relatively small so I am not sure if you'll experience the performance issues mentioned in http://drupal.org/node/482550.

Good luck.

Thanks.

Both comments were very helpful.

Ken

I find the development of the module a little confusing: first the problems are fixed, and then they reappear because of the code being rolled back without to first apply the changes to the old code.

Inter-dev updates are also said to not be supported (see #485304: Fatal database update error). This means that you need to uninstall the development snapshot 6.x-1.x-dev before to install the newer 6.x-1.x-dev; I let you imagine what happens to whom has 10,000 nodes that appear in the sitemap, but he is forced to uninstall the module, and see the sitemap being reset.

Hi, I'm having some issues with the beta5 version and I'm not sure if I should 1) post it here, 2) post a new thread, 3) live with it to see if beta6 fixes the problem, or 4) Troubleshoot it myself. The issue is that I've added additional links to the sitemap, but they all appear with a lastmod date of 1970, a changefreq of yearly, and a priority of 1.0. Additionally, when I add a link, it gives the following message:

array_shift() [function.array-shift]: The argument should be an array in /public_html/includes/form.inc on line 1332.

I went in to the xmlsitemap_additional table and the "previously_changed" and "changed" fields were all blank. I filled in the fields, which caused the lastmod and changefreq to update, but the priority did not update. Is this a known problem? I have run cron several times, emptied my cache, and deleted the xml file in my files/xmlsitemap folder. Thanks, any feedback would be appreciated.

As a side note, features or tasks approved by more than one maintainer have been removed in the last commit done. This means the code returned to the situation existing one month ago.
I am not sure if this can be called progress, and I am not sure if putting the blame on just one maintainer could be of any help. One could make note that the others maintainers didn't do anything, not even to make a comment to say what was wrong, or to express an opinion about how the code was being developed, when (and if) the code was not developed in the correct way. Not doing anything doesn't always mean to not be guilty.

@#222: I would rather open an issue report for the errors you are seeing.
I am not sure if the next beta version will fix the problems you are seeing, as it doesn't seem to fix the bugs the recent commits have re-introduced. That would be a reason more to open a different report, anyway.

As of June 17 Dev, still getting this one:
Unknown column 'xt.tid' in 'where clause' query: SELECT t.* FROM term_data t LEFT JOIN xmlsitemap_taxonomy xt ON xt.tid = t.tid WHERE xt.tid IS NULL LIMIT 0, 200 in /usr/home/hoslo/public_html/sites/all/modules/xmlsitemap/xmlsitemap_taxonomy/xmlsitemap_taxonomy.module on line 26.

Looking fantastic otherwise, thanks!

I'm using a server running mySQL6 which as a maximum key length bug especially for utf-8

http://bugs.mysql.com/bug.php?id=32915

It totally took out my back up and migrate file.

I'm getting the following, so I was wondering if it might be related?

user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap DROP KEY loc in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 417.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap ADD INDEX loc (loc) in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 434.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap CHANGE status `status` TINYINT unsigned DEFAULT 1 in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 520.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap ADD INDEX status (status) in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 434.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap DROP PRIMARY KEY in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 386.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap ADD PRIMARY KEY (id, type) in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 374.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: SHOW COLUMNS FROM xmlsitemap LIKE 'language' in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql.inc on line 349.
user warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'type for db_type_placeholder LIMIT 0, 1' at line 1 query: SELECT * FROM xmlsitemap WHERE type = unsupported type for db_type_placeholder LIMIT 0, 1 in /home/www/www.graphnetwork.com/htdocs/sites/all/modules/xmlsitemap/xmlsitemap.module on line 460.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap ADD `priority_override` FLOAT DEFAULT NULL in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 298.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap CHANGE lastmod `lastmod` INT unsigned NOT NULL DEFAULT 0 in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 520.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap ADD `node_type` VARCHAR(32) DEFAULT NULL in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 298.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: ALTER TABLE xmlsitemap ADD INDEX node_type (node_type) in /home/www/www.graphnetwork.com/htdocs/includes/database.mysql-common.inc on line 434.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: UPDATE xmlsitemap SET node_type = (SELECT type FROM node WHERE nid = xmlsitemap.id) WHERE type = 'node' in /home/www/www.graphnetwork.com/htdocs/sites/all/modules/xmlsitemap/xmlsitemap_node/xmlsitemap_node.install on line 82.
user warning: Table 'gnw.xmlsitemap' doesn't exist query: SELECT COUNT(id) FROM xmlsitemap WHERE status = 1 in /home/www/www.graphnetwork.com/htdocs/sites/all/modules/xmlsitemap/xmlsitemap.module on line 330.

@Bensbury: Those update function SQLs can only be from the 6.x-2.x-dev version. If you previously used the 6.x-1.x version, as the project page and the release notes state, there is currently no upgrade path from 6.x-1.x to 6.x-2.x supported. You must disable all xml sitemap modules, uninstall them at admin/build/modules/uninstall, and make sure to delete all files in the module directory before unpacking the new version of the module.

Oops sorry.
I've been trying all the different versions.

The latest 6.1 version gave me a taxonomy_sitemap error and I have been trying all the different version to try to find one that works.

Because there are these different versions, which is the best one to go for?
I assume as the 6.1 are being discontinued I should try the 6.2 and wait it out if it doesn't work?

I have to agree with #234, unless you're following this module daily, it's hard to tell what to use

What would you suggest, I use to submit a sitemap.xml to google tomorrow?

this where it gets confusing....

so download 6.x-1.x-dev for now, then update to later Beta6 ?

the update path is there?

thanks

#234

I've read this article and started to use 6.x-2.x

- http://blog.davereid.net/content/state-of-drupal-xml-sitemap-2009

Gloriousness!
It worked.

Seriously look at the release notes.... because you have to clean out the old database info and it has specific instructions.

Now for submission adventures.
Cheers Earnie.

#234: 6.x-2.x is working well for us, despite it being in beta. We have a site with 60,000 nodes.

Translation update is complete.

@earnie: What is holding the final 1.0 release back?

Since upgrading to 1.0 I now get errors when submitting to any searchengine.

Details
Type xmlsitemap
Date Monday, 24 August, 2009 - 21:30
User administrator
Location http://nafof.org.uk/admin/settings/xmlsitemap/tools
Referrer http://nafof.org.uk/admin/settings/xmlsitemap/tools
Message Error occurred submitting sitemap to Google: 0.
Severity error
Hostname 78.146.204.228
Operations

I upgraded by disabling, uninstalling, deleting, installing new module, enabling & configuring as before; I was previously using beta6 I believe.

@richard.e.morton: If you have any bug reports, please create a new report. This report is titled XML sitemap 6.x-1.x-dev progress.

Congratulations!