Cron & Advanced Cache

When you have a complicated website you may have quite a few cck fields. In my case I also use the subproducts addon to the e-commerce package. This brings up a situation where the subproducts module creates variations on a parent product during the cron jobs. With many cck fields this translates into about 6mb per new node it creates, and that is after I made adjustments to greatly reduce the memory being chewed up during the node creation.

The biggest use of space I determined was that the node_load module was caching every node that was being created.

Before I changed some of the module it was also caching all the parent products and their existing children just in the figuring out stage of what to do next. Under this circumstance even a 500mb limit wasn't enough memory to do the job.

This is a specific instance of broader issue as I see it. That being, when a user goes from page to page you are typically loading 1 or perhaps 2 nodes. When cronjob is running it can routinely load many nodes since bulk processing is what you would normally delegate to a cronjob.

Given this, and since how much memory a node can use in some cases could be quite large, I propose that the node_load cache (and perhaps other caches) should be completely bypassed for cronjobs. Or, at the very least, be optionally bypassable.

In my case I created a global variable called cron_cronjob_is_running and when set to TRUE I just skip all the node_load caching code... and the problem was solved. Clearly there must be a more elegant solution than a global variable but it was fine as a quick solution to a problem that needed to be solved quickly.