Autorepair may run out of memory by loading all nodes/comments at once

no, sorry, it's not my server it's this module. My server resources are fine, memory usage < 10%, disk is 20% full, plenty of CPU, only 5% utilization, 8 cores, disk space and unused ram.

I'm telling you that your module made a wild memory request via mySQL and crashed my server. To fix it, after a hard reboot, it completely locked me out of the server, I had to uninstall your module as there were locked up tables that never released and were empty, warnings listed above after the hard reboot to the server.

Your module was the cause of hard "out of memory" and I already showed you the mysql error file shows an infinite loop out of memory errors until a hard crash but those out of memory errors were caused by your module's initial > 1MB query.

No, my.cnf, skip-innodb.

As I already said the linkchecker tables via myphpadmin were corrupted, locked after the massive memory usage request by your code, which I also list via the logs above, clearly I already checked the logs and why you have what I've got.

I'm telling you the tables in the database were "in use" plus "did not exist" so the massive memory request coming from your code not only hard crashed the server for 3 hrs but also corrupted the tables.

I'm not renabling this module on a live production server that has a lot of traffic, you need to check your own sandbox with a large database of links to check. But frankly I suspect something is not error checked in your code on mysql queries and possibly an loop situation that doesn't exit upon fail and I suspect it has something to do with content URLs are aliased, redirected or something along those lines. Possibly even after broken links are fixed on the 2nd pass via cron.

I don't have the time to debug your code to figure out the bug for you.

May I suggest you don't blame the messenger here? I wouldn't be bothering except this is clearly a dangerous bug to crash a server to complete lock out, hard reboot.

Well, I did do a quick vim of linkchecker.module and I do see an issue with auto_repair for 301s, line 424 and that would correspond with a 1045540 byte mysql query followed by a loop of 128800 byte queries.

If there was an error and a slew of 301s came back (I don't believe we have that many 301s yet, we do reference economic statistical sites so it's possible that a major site was moved, thus affecting many nodes).

Anyway, just doing a quick scan of the code I see no error checking for number of nodes to process in that while and then for loop starting at line 424. While you're limiting the number of links to check in cron, on line 424 I see no limit to 100 nodes. I see all nodes which match resulting 301 to be processed and that I think could cause a fatal out of memory error, esp. with nested loops.

Like I said, I can't hang out and debug this because it's not critical to our site and I'm busy rewriting other modules working on a major site upgrade, so that's my best guess on a quick scan.

Don't shoot the messenger.

Yes, but again, that's not the issue, it's your code. You have nested loops, trying to process an unbounded number of nodes, at once. Beyond adding memory checking code before the loops, the loop structure should be broken down and rewritten so you process one node at a time to repair the broken link. Iteratively and probably set to a cron. That pretty much solves trying to use beyond system memory.

That's the bottom line, you don't write nested loops with the potential to be editing and rewriting an infinite number of node content database entries at a time, all resulting in multiple memory mallocs. This is a programming technique issue.

I'm not using this code frankly due to performance, load on a production site, doing my own MySQL queries in a dev. site. I already pointed to the coding flaws and it's your project so you should be looking at programming techniques that will keep you out of memory allocation trouble, as I'm outlining. These are very important concepts to learn so it's up to you to rewrite your nested loops to be iterative to update the database and demalloc your memory, as well as check system memory in your routine.

linkchecker.module, line 424

$res = db_query("SELECT * FROM {linkchecker_nodes} WHERE lid = %d", $link->lid);

Gee, what could possibly go wrong? Hints: LIMIT, resource and free up memory after insert.

I am not using your code as repeatedly stated, hence I am not rewriting your code. As a developer, may I suggest you consider realizing there is a problem instead of continually getting pissy at the bug reporter. Looking anywhere else but linkercheck.module, line 424 and below won't fix your code.

The only reason I reported this is to warn others they could 100% crash their high traffic production site due to these coding errors.

Denial is an amazing thing and yes, it will never get someone a paycheck in software engineering. It did and will crash a server under the conditions I have outlined repeatedly and perhaps you need a good sandbox to play in considering your never ending insults to the bug reporter.