Hi We started using the node export module and we had it working. We are not sure what changed but now we have recursion errors. No matter how much memory we give it it will fail.

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 139282 bytes) in /all/modules/node_export/node_export.module on line 851, referer: [removed]

We have tried this on may instances of drupal different nodes and content types and we even get this problem when exporting just one node. We have not been able to figure out the problem can someone help?

CommentFileSizeAuthor
#9 modules.txt13.4 KBericdfields
#4 vardump.txt14.18 KBdeverman

Comments

danielb’s picture

danielb commented
Priority: Critical » Normal
deverman’s picture

deverman commented
Issue tags: +export, +Feeds, +data not importing, +recursion

After investigating this further we have found that the node being passed to the function node_export_node_encode() contains a reference node object to itself so this function doesn't handle reference variables well and keeps looping over itself until it runs out of memory.

The array contained an entry like this:

  ["0"]=>
  object(stdClass)#1 (42) {
    ["nid"]=>
    string(5) "44606"
    ["type"]=>
    string(4) "feed"
    ["language"]=>
    string(2) "en"

This is from a var_dump so each object is in ["0"]. We seem to have partially fixed the problem by changing the code to ignore ["0"]. In a previous post someone had a similar problem and disabled a module. I don't know what module on our system is causing this interference.

danielb’s picture

danielb commented

Can you show me a dump of the whole node? You can change the field values if the information is sensitive. Do you know if you're using node reference or user reference cck, etc... ?

deverman’s picture

deverman commented
StatusFileSize
new vardump.txt14.18 KB

I have attached a vardump just after the function node_export_node_encode is called. You will see it is a couple node with in a nodes (the same node id) and then the text "* RECURSION *" as the var_dump function realized it was looping through the same node since the object must be stored as a php reference variable. Yes we are using node reference module, no we are not using user reference cck. The node reference module wasn't used on this specific content type and when we do have a node reference they don't reference themselves. This problem happens on ALL content types we have.

As mentioned above we were able to get things partially to work by modifying the function to ignore the ["0"] in the array. We still seem to be missing some data after import but can't tell if this is related or this is a limitation of the module.

danielb’s picture

danielb commented

That is so weird, I've never seen that before. You have to find out what module is doing that. What is the purpose of embedding a node within itself?

punjabi’s picture

punjabi commented

Same issue here! I tried bumping up the memory in PHP.ini, from 32M to 64M, and then from 64M to 128M, still the same problem PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 9968393 bytes) in /var/www/referer/drupal/sites/all/modules/node_export/node_export.module on line 860

deverman’s picture

deverman commented

I am not sure which module it is. We just did something hacky like this to prevent the recursion:

Line 851:
if ( $key == "0" && $iteration > 3) continue;

Line 857:
if ( $iteration == 3 ) {
//var_dump($var);
//exit;
}

Sorry that is all I have time for.

danielb’s picture

danielb commented
Status: Active » Postponed (maintainer needs more info)

I need to know which module is causing the recursion so I can figure out the best solution with some testing. If possible, can the people encountering this problem at least give a list of their enabled modules, and try to determine if the problems exists on all content types, or just particular ones??

ericdfields’s picture

ericdfields commented
StatusFileSize
new modules.txt13.4 KB

i'm experiencing this too. attached is my module list.

ericdfields’s picture

ericdfields commented

As a follow-up, file export mode was set on the default base64 encode. We don't really need this as our /files directory is under source control, so I changed it to local file export. This fixed the problem for me.

It could have been related to our 'video' content type, which has attachments upwards of 10+MB. Our image content types which just have JPEGS exported fine. Video was the only hangup.

HylkeVDS’s picture

HylkeVDS commented

My closedquestion module is also causing recursion in this method.

In ClosedQuestion the node object gets a Question object that has (and needs) a reference to the node object. I tried making this reference protected, but that didn't solve the problem.

from: http://php.net/manual/en/language.types.array.php:
protected variables have a '*' prepended to the variable name.

So you could check for that * and skip those fields, but regardless, you should do some sort of recursion check to avoid endless loops.

danielb’s picture

danielb commented

I don't know how to do a recursion check. I guess I could store each object temporarily and test subsequent object against that, but then how do you add it back in during import?

danielb’s picture

danielb commented

There is some info here on spotting references: http://www.php.net/manual/en/language.references.spot.php
Perhaps something like that could be done?

As each object/array is processed, add a unique id for internal use, then check subsequent object/arrays for the presence of such an id. If it's already there, then do not store the object/array but rather something with the id to tell the import script to reference the object with that id.

danielb’s picture

danielb commented

How the heck do I find HylkeVDS's closedquestion module to test this? Is it gone?

I don't see why you'd need to reference the node object in the node :/ Not even nodereference puts in a reference, just the nid. I guess it does happen though.

HylkeVDS’s picture

HylkeVDS commented

ClosedQuestion can be found here:
http://wmmrc.wur.nl/drupal-modules/closedquestion/about

Any object oriented code is very likely to have reference loops. Just think of a double-linked-list where every item references the previous and next item. So this reference back to the node object is just one example, the problem is bound to happen in other places as well once more code starts using OO. Calling node_load is not always an option, as that returns a clone of the original cached object, and there might have been modifications since then.

You could take a look at how serialize() handles references, as that function is loop-safe and manages to put those references back.

Is there hook for node-modules to tell node_export how to export a node? As the module itself will know what data needs to be copied.

danielb’s picture

danielb commented

Is there hook for node-modules to tell node_export how to export a node? As the module itself will know what data needs to be copied.

There are many hooks that enable modules to modify the node before export and upon import, however it is not intended that a typical node module would need to use these, as such a module would normally attach the necessary data upon node_load.

danielb’s picture

danielb commented

I found some more info that might help
http://webreflection.blogspot.com/2009/09/php-serialization-and-recursio...

He has a suggestion that leverages serialize() that might be a solution.

danielb’s picture

danielb commented

-- removed --

danielb’s picture

danielb commented

Tested that on a some sample code and it seems to work, I'll try to implement this approach for each node object that gets sent into the encoding process, and hopefully we'll wrap this issue up.

danielb’s picture

danielb commented
Category: support » feature
Status: Postponed (maintainer needs more info) » Fixed

Committed this to development version

I don't know if the place I put the code on the import side of things is correct, let me know how it goes

HylkeVDS’s picture

HylkeVDS commented

There are many hooks that enable modules to modify the node before export and upon import, however it is not intended that a typical node module would need to use these, as such a module would normally attach the necessary data upon node_load.

Yes, but on node_load the module will make the node fully-functional, which includes building those recursive object structures that are not needed for export. For export only the raw data is needed. So unless there's a hook that tells the module "prepare this node for export" the module will not know the node is being loaded for export and will thus populate the node with lots of stuff not needed for export.

In other words: When exporting a ClosedQuestion node, you can totally ignore the "question" field of the node, because it is only needed when viewing a question. Is there a way to tell node_export that it can ignore that field?

danielb’s picture

danielb commented

I don't agree with your distinction between a fully functional node and a node object populated for any other purpose, such as saving or exporting. If you're attaching other data for convenience, that could probably done on another node hook or the data could be acquired at the time it is needed. In any case that stuff will be ignored on the import side so it isn't a problem.

There is a way to modify the nodes before export, using one of the available hooks (most are documented in node_export.api.php), but I really wouldn't bother. This problem with recursion should now be taken care of.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Useful Idiot’s picture

Useful Idiot commented
Component: Node Export » Node export

So what is the fix for this issue exactly? I do not see any sort of clear fix. Any help is much appreciated.