Out-of-memory error reindexing site via Drush

This is happening on xmlsitemap_node.module line 198:

    $node = node_load($nid, NULL, TRUE);

Despite the fact it is telling PHP to flush all of the cached nodes, it's still running out of memory.

It seems that I'm ultimately running into a PHP bug:
http://stackoverflow.com/questions/1145775/why-does-this-simple-php-scri...

:(

Maybe adding an extra command for just building the indexes and being able to set a limit would work around this?

'xmlsitemap' has 8521 records, 'node' has 65,641, the memory limit is 256mb (268435456 / 1024 / 1024).

Roughly 98,000 =-)

Doing a bit better:

PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 106 bytes) in /[path]/sites/all/modules/platform/filefield/field_file.inc on line 295
PHP Stack trace:
PHP   1. {main}() /usr/local/drush/drush.php:0
PHP   2. drush_main() /usr/local/drush/drush.php:37
PHP   3. drush_dispatch() /usr/local/drush/drush.php:78
PHP   4. call_user_func_array() /usr/local/drush/includes/drush.inc:23
PHP   5. drush_xmlsitemap_rebuild() /usr/local/drush/includes/drush.inc:0
PHP   6. batch_process() /[path]/sites/all/modules/platform/xmlsitemap/xmlsitemap.drush.inc:74
PHP   7. _batch_process() /[path]/includes/form.inc:2538
PHP   8. call_user_func_array() /[path]/includes/batch.inc:189
PHP   9. xmlsitemap_rebuild_batch_fetch() /[path]/includes/batch.inc:0
PHP  10. module_invoke() /[path]/sites/all/modules/platform/xmlsitemap/xmlsitemap.inc:439
PHP  11. call_user_func_array() /[path]/includes/module.inc:450
PHP  12. xmlsitemap_node_xmlsitemap_links() /[path]/includes/module.inc:0
PHP  13. node_load() /[path]/sites/all/modules/platform/xmlsitemap/xmlsitemap_node/xmlsitemap_node.module:198
PHP  14. node_invoke_nodeapi() /[path]/modules/node/node.module:788
PHP  15. content_nodeapi() /[path]/modules/node/node.module:673
PHP  16. content_load() /[path]/sites/all/modules/platform/cck/content.module:422
PHP  17. _content_field_invoke() /[path]/sites/all/modules/platform/cck/content.module:253
PHP  18. filefield_field() /[path]/sites/all/modules/platform/cck/content.module:1253
PHP  19. filefield_field_load() /[path]/sites/all/modules/platform/filefield/filefield.module:283
PHP  20. field_file_load() /[path]/sites/all/modules/platform/filefield/filefield_field.inc:114
PHP  21. _field_file_cache() /[path]/sites/all/modules/platform/filefield/field_file.inc:50
Drush command could not be completed.                                [error]

This time it completed about 55,000 (xmlsitemap) before failing. I should note that this time it did actually complete xmlsitemap_node's turn and failed on xmlsitemap_taxonomy.

Like I said, I'll see if I can come up with a command to do it in batches... and rewrite some of our own code to use db_fetch_array instead of db_fetch_object.

Would there be a problem rebuilding the entire xmlsitemap_node_xmlsitemap_links() process to just use an array rather than the full node? It isn't using much data anyway, only a few fields.

Yeah, I saw that, was building up a list of the (at the time) five fields that were needed then my stomach dropped when I saw node_access() :-|

Here's a small patch that lets you pass either a node object or just the bare nid to taxonomy_node_get_timestamps.

FYI this is related to #511956: Integrate Rebuild into Drush.

Here's a patch that adds two new Drush tasks:

• xmlsitemap purge - single command to purge the existing data.
• xmlsitemap build - the same as 'rebuild' only it doesn't purge first.

These are modeled off the 'rebuild' task.

Here's an alternative version that just adds an option "--no-purge" to the existing "xmlsitemap rebuild" command.

The rationale for adding this is to let you repeatedly run the command "drush xmlsitemap rebuild --no-purge" until it completes building all of the sitemap entries, which can be useful for compiling the sitemap for the first time.

The patch does not resolve the problem which is that the sub-modules still process all records and disregard anything that has been previously processed. For example, the taxonomy_node module uses the following query:
SELECT n.nid FROM {node} n WHERE n.nid > %d ORDER BY n.nid

The %d value is passed in from the batch so it can build them e.g. 500 at a time, but the process itself starts from scratch each time.

In effect, while it doesn't specifically purge the data before continuing, because every time it's executed it starts over from the beginning so will fail at roughly the same place each time.

Here's another patch that adds another argument, "module". By default, when this option is not passed it works the way it does currently and builds everything; if you pass in either "node", "term" or "menu" it will build those data structures using the built-in xmlsitemap submodules, otherwise it calls xmlsitemap_link_info on the module name passed in.

Tiny tweak to the last patch, wraps check_plain around the argument and indicates which modules were executed at the end.

Success at last!

I added this alias on my server:

alias drush='php -d memory_limit=1024M /usr/local/drush/drush.php'

I was then able to successfully build the sitemap:

XML sitemap files rebuilt in 1055409.41 ms. Peak memory usage: 332.75 MB.

Thanks to a co-worker for suggesting the "-d memory_limit=1024M" argument on php.

Feel free to keep or can the patches as you see fit.