Mass deletion caused excessive memory use

Hey ... just to be clear, the problem here isn't time related -- with something close to 41,000 URLs connected I expected it to take some time :-) The problem is memory related, from the first click to delete the node, the server became nearly 100% unresponsive (occasional read-a-byte from terminal) --- classic memory exhaustion, 30+ minutes later and I had to give up and pull the plug .... it might have worked itself out but I doubt it, syslog entries showed that processes were being killed for lack of memory.

So, my guess is that there is some dynamic memory being allocated based upon some DB information that boost requests and you and/or drupal core wasn't expecting such an explosion of URLs. I really do not believe it was the deletions themselves causing the problem (FWIW: the db, hosted on a separate server, was inactive during this incident, this wasn't a load on the DB). The server in question has 4GB physical RAM and 4GB of swap and it is a dedicated web server for (testing) GATE Village.

The top level node reads a single block of Quicktabs which itself calls 8 other quicktabs, each of which loads between 5 and 15 nodes (depending on what it is highlighting). The various options just indicate to quicktabs what the view of the information should be. The action I took was to delete the top-level node via the normal Drupal edit form.

WRT to Khalid's article, he is absolutely correct, quicktabs that pre-load can cause a great deal of end-user performance pain (which of course has nothing to do with the pain of deletion). I have a rule, I'll only pre-load if the contents of the nodes/blocks being loaded are themselves very small (twitter-sized) and don't themselves create dynamic content, otherwise I'll use AJAX. So the example includes some that do pre-load and others that do not -- as I say, none of that is relevant to the problem at hand, all of the cached files are ".html".

"Example" in the name is deliberate, it is a big ugly page of various quicktabs approaches ... not really an appropriate page for use in a consumer-based web site. The various URLs are found out via crawling (external not via Boost) and I want to cache them all because they do take too long to create.

http://www.gatevillage.net/quick-tabs-example if you are curious.

What else would you like to know?

I was remote at the time, the only thing I know for sure is that there was memory exhaustion, but given that all the swap was used I imagine the hard drive was working furiously :-) I'll try to reproduce sometime soon but it isn't a particularly convenient thing to do, even on the test box; I'll need to schedule some time.

Food for thought, I don't think it is a mass deletion per se that is the problem but the expansion of all of the possible URLs that must be deleted once the delete node function was hit .... again, perhaps not Boost at all but somehow Boost / Drupal needed to understand that the cached URL /quick-tabs-example?quicktabs_1=0&quicktabs_2=0& etc should be deleted (or marked for deletion?) / cleared from cache once the real node /quick-tabs-example is deleted ....

The cached files did not get deleted (but the DB entry for the node was), or at least I don't think any did and there was plenty of time (30 minutes) to do so (manual deletion took roughly 1.5 minutes) ...

Of course, until I do reproduce it I suppose it is possible that it wasn't related to Drupal at all (but that is very unlikely as it is a dedicated box).

It is a dev branch that was updated, I downloaded it and tested with it but haven't put it into production yet.

I don't see how quicktabs is related to the problem at all, it just happens to be the module that is creating all of the URLs because of the embedded quicktabs .... I've so far been unable to reproduce it, we'll have to leave it to gremlins for now ... seems I might have a few :-)

I'll mark it as fixed, as it does seem to fix it ... (in fact, I came to that solution myself, more or less by accident). I'm still struggling with boost and multisite setup (the cache isn't getting cleared at all for all but the main site). Not yet sure of the details so no official issue as yet until I determine it isn't user error.

Now see perhaps I missed that in the documentation, it makes so much sense when I see it written like that :-) but no, I was not running it for each of my web sites ... I am doing so now, however, and the indications are that it is working as advertised, thanks very much. I will open another ticket in the (probably not-so-soon) future because this approach won't scale as there is only one boot-crawler allowed at a time but I'll see what I can do to offer a solution rather than just a ticket when I do so.

Anyway, I digress, I just wanted to say thanks for the great module and support! and let you know my troubles have abated.