Architectual problem: Mass updating links

db_query("UPDATE {xmlsitemap} SET priority = 0.7 WHERE type = 'node' AND priority = %d AND id IN (SELECT nid FROM {node} WHERE type = 'story')", 0.6);

The AND priority = %d part should not be required, especially when the query is executed right after the priority has been changed; how much nodes will have their priority already set to the new priority?
The AND id IN (SELECT nid FROM {node} WHERE type = 'story') part would not be necessary if the table would have a field containing the content type of a node. It would be set when the node is first added to the table containing the sitemap links; successive updates of the node content would not change the content type (or at least, normally a node has a content type that doesn't change).

We can't store everything.

That is true; you cannot store everything in a single table, especially if the data contained in the table come from very different sources.
A unique table cannot even try to have fields that can used from all the different modules that access it. That is the reason I didn't use a unique central database table. If I want to add a new field for a specific module, I can do it in its own table, without to waste space for the other modules that don't know what the new field can be used for.
The difference is also visible in the time got to run queries in the central database; if the table contains 20000 rows, but just 5000 are rows containing nodes data, any query that needs to filter the node rows will take a lot more time than if it would be executed in a table that contains only node rows.

I am not proposing anything, and I could not propose anything as I am not considered in the rank of doing that.
My note was merely about the fact to adopt a solution for a problem that could be resolved easier if a different approach about the database structure would have been taken.

There aren't any problems with the user roles, also because the sitemap contains reference to users, not to user roles.

Actually, if the 6.x-2 code would adopt an approach similar to the one used in 6.x-1 code for the database schema, it would show that what I was thinking of having two co-maintainers to develop two completely different branches is right.

EDIT: My code already saves {term_data}.vid in {xmlsitemap_taxonomy}, and that is the field that allows xmlsitemap_taxonomy.module to delete all the taxonomy terms from the database table when a vocabulary is deleted.

A different approach would be using cron tasks; if I understand correctly the 6.x-2 code, the cron tasks are only used to add rows for new content, but what do they do when there aren't any more nodes, taxonomy terms to add to the central database table? They stop to do anything, when they could update the tables rows that need to be updated. It's all to see how the rows are filtered to find the ones that really need to be updated, but it's a waste of resources having the cron tasks that stop to do anything.

You've completely rewritten 1.x and have lost sight of the goal of creating a version 1.x that would simply work as it was already written.

As far as I remember, my task was to continue the development of the Drupal 6 version from the point it has been interrupted from somebody else; in particular, the task given me from Darren Oh was to fix the problem with the exhausted memory, and the PHP timeouts that somebody noticed.
I don't think I didn't do that task.

This is why you have trouble with more than one maintainer per branch.

Other projects where there are more than one maintainer, there aren't two co-maintainers developing different branches for the same Drupal version. In the case of XML sitemap, this is not something I chose; it was the choice of somebody else, who decided to use a different base code, rather than proposing patches to the existing code.

The default priority for nodes of type 'story' is 0.6. The user goes and changes the default value to 0.7. We should probably update all the existing content with type 'story' with the new priority.

To compare the 6.x-2 code with the code used in the previous branches, the previous branches resolve that problem by not inserting in the database rows the actual value for the content type priorities, and inserting a special value (-2.0).
I am not saying that what the 6.x-1 code does is correct, or is perfect; the problem you noticed would not exist with the code developed before I started to co-maintain the project, and that I kept.

I didn't read this whole thing (will try to later) but real quickly on this point:

The default priority for nodes of type 'story' is 0.6. The user goes and changes the default value to 0.7. We should probably update all the existing content with type 'story' with the new priority.

For better or worse, this is not how core works. If you make 3 stories and then decide to make stories turn revisions on by default, the 3 stories you created do not get the revision checkbox applied. IMO, that's what your big "Reset and start over" button is for.

D. use batch_api for all time intensive tasks.

Maybe the two possibilities can be used in combo; in some situations, the module should not change the data of the already existing content, while in other situations it would plan a batch operation.

Between the two possibilities, there is still the possibility to update the data with cron tasks, which are used only for a specific situation.

D. use batch_api for all time intensive tasks.

for the love of god NO!!!!!, please do not do this!! This is what caused my 2-week-long tirade against this module in the first place.

Try performing a batch API process when you have 100K nodes and you will soon feel my pain. #437804: For development: Large sample data has 25K nodes or so and that's enough to show you what it feels like. Hint: It feels a lot like sitting there for over 8 hours not being able to close your laptop or your browser.

@webchick: I'm not sure what the problem is, but this is what batch api was build for!? Tasks that would fail if you run them in only one step or tasks that overload the database if only one request, etc. that times out PHP...

I also have tried to use xmlsitemap on 120.000 nodes sites (over 2 years ago) and I'm much more worried about a failed installation, "half" created xmlsitemap, failed cron and other issues like I've had in 5.x of xmlsitemap - only caused by sql and php timeouts and overloaded servers.

Job_queue does not provide a multistep batch process. I also asked somewhere else for a batch api that works with cron, but it's still not available and I do not see someone is working on it. I would also be happy to change to a cron based batch api, but until we have such a new batch api we are simply lost. Your 8 hours "pain" could only reduced by executing multiple tasks in one batch step (for e.g. do more than one node update in one batch step).

I have also used batch api in linkchecker and I do not see any other *reliable* way to scan all nodes if you must. And I'm pretty sure that linkchecker could be more intensive on the first run than xmlsitemap as it need to scan the content of all nodes for link... load each node, scan for links, save all found links in DB, etc. I'm happy not to have the build issues xmlsitemap have - if the xml file is requested first... nevertheless the load and time it takes to complete a monster site scan is high. If you could come up with a better solution for such issues I would also be happy to learn from the solution.

A really simple solution is just something like sleep(1000) in hook_cron() in between loop executions. That'd help your database not get pounded. There are ways of checking the timing of cron so that your process doesn't eat up enough processing time that other processes die. Check feedapi_cron() and feedapi_cron_time(). There's http://drupal.org/project/job_queue. Seriously, we can solve this issue.

There is just simply absolutely no way that an 8+ hour long process should ever be forced to be initiated and run through the browser, regardless of what the batch API was/was not built to do. Browser are notorious flaky and crash-prone, Internet connections die, laptops get closed, cats unplug ethernet cables, etc. Populating an extremely large sitemap has to be done from a command-line/cron process on the server.

There is just simply absolutely no way that an 8+ hour long process should ever be forced to be initiated and run through the browser, regardless of what the batch API was/was not built to do.

I agree; causing a user to be blocked on a page for too much time is not something user-friendly.
The batch API has been thought to avoid a browser timeout, not to keep the browser busy more than it needs to.

Well, but job queue doesn't help much here... see #373050: Loop job_queue jobs until half cron time has been used that is a real problem. Additional it exits at least after 120 seconds execution time (all counted with the suggested Core standard settings). You cannot do much in 120 seconds - really not! And job_queue does not provide a batch api for endless tasks. You need to WAIT months to get all completed #386612: Prevent possible cron failures caused by too many link checks. Normally cron is configured to run every hour. I'm sure you can calculate the months for completion yourself... and nobody really like to analyse why xmlsitemap or linkchecker do not finish within an foreseeable future. Your 8 hours batch api may become months! I also have this request #383980: Provide option to force immediate Analysis of links... well I have been punished by building an high performance module based on job_queue and the people who do not understand why it takes time to finish all stuff if cron is in the game.

We really need a better job_queue module that works like batch api in the background (without javascript) and runs endless until all work is done. +++ by me.

and runs endless until all work is done

It would run endless if there would not be any timeouts on the server side, which is normally not true.

The intention of "endless" does not mean in a single request without a time-out. Processes without time-outs could easily become zombies (I don't like zombies). I mean multiple requests, for e.g. fired by wget/curl with endless redirects of cron.php to cron.php until all jobs are completed, but every cron request need to take less than 240 seconds or only 120sec like job_queue use today. It need to implement the same logic like batch_api in the browser, but with redirects on command line.

I'm not sure if this is possible on command line, but otherwise on every normal site cron is only executed once per hour (recommended core configuration, see http://drupal.org/cron) and cron have a maximum execution time slot of 120 seconds per hour defined by job_queue... how long would take xmlsitemap to build the sitemap on a site having 100.000 nodes? I don't know, but I would expect years to build up from ground and not only 8 hours like today (unverified myself).