I'm importing (or rather trying to) about 2.7 million records into a drupal setup. The records are fairly simple, I'm using the 'node_save' method to store them into the database and everything works fine except that the procedure is leaking memory like crazy.
It looks like node_save is triggering a problem somewhere deeper in PHP but I can't seem to find out if that really is what is going on.
It almost looks as though php is losing track of references to copies of the $node objects. It seems to make no difference if I re-use the same instance or allocate new ones, no matter which way I go eventually all the memory in the system is consumed.
When I dump the variables that php knows about it looks like everything is ok (but clearly it is not, top tells a totally different story).
The rate at which it goes is about 10 K per node saved.
This is running drupal 5.x on a 64 bit machine, the php version is 5.2.0-8+etch9
My plan to deal with this right now is to batch the records in groups of 10,000 or so and then to load each batch in turn but that is quite a nasty way to solve the problem, if anybody could shed light on this then I'd be very happy.
