Failing performance of node_access_rebuild [#163862]

Reposting into a separate issue from here: http://drupal.org/node/157942#comment-279723

This is a real showstopper for upgrading sites with lots of nodes (I have >30,000) and OG enabled. Rebuilding node_access causes each of the nodes to be loaded and re-saved, timing out / crashing the update script.

Most common case is when OG is the only access module. For this case nodes not member of any group can be re-generated by a single SQL query and access-controlled nodes run through usual grants hooks.

Would be nice to commit this patch or at least let people know it exists.

Comment	File	Size	Author
	node_og_rebuild_0.patch	1.76 KB	dkruglyak

Comments

Comment #1

ShutterFreak commented 2 August 2007 at 07:31

Subscribing to this issue.

Comment #2

yched commented 2 August 2007 at 09:21

Status:

Reviewed & tested by the community

» Needs review

@dkruglyak : You've been around for some time, I think you should know by now that you're not supposed to RTBC your own patches.
Plus I'd be very surprised if such a module specific bunch of code went into core...

You should probably think of ways to let these changes happen in OG module, or, if the current node access API does not allow it, make suggestions about _generic_ changes ?

Comment #3

dkruglyak commented 2 August 2007 at 14:55

OK, my bad. I marked it RTBC since it was well tested in my upgrade test runs.

Unfortunately the way D5 node access system works, there is no way around without patching core. As written now, rebuild involves loading and saving every node and naturally chokes on relatively medium sized sites with just thousands of nodes.

So the solution is to check for special cases whenever much of those calculations are unnecessary. I presume we can offer a hook to let modules override the entire rebuild process for speed... In that case OG module would have to check on its own that it is the only access module enabled. Optimizations with multiple access modules enabled could get really murky, possibly beyond our hook system...

D5 bottom line: either add a hook (not a chance) or commit this patch (your concerns) or leave the patch here for those who want it.

Comment #4

damien_vancouver commented 6 August 2007 at 21:26

This patch should definitely be committed... so at least it is there in 5.3+ if it comes out.

Without it, there is simply no way to upgrade a d4.7 site that has organic groups to d5. It times out every time and I mean every time. To make matters worse, there are 3 node_access_rebuild() calls in a row that happen in the update process.

I got my 4000+ user site with 130+ groups upgraded, but only because I had the knowhow to rip the og.install file apart and make it so only one of those rebuilds ran, in its own update. A "regular" user who doesn't have development skills would simply never figure it out. Also, these types of regular users are probably also unlikely to successfully install a patch (the obvious rationale to me for committing this to 5.x).

I have the old version of that huge site saved (the upgrade was hellish and took about 100 dry runs before we were ready to do it for real). I'll test this patch on that upgrade and report back - if that site makes it through its update.php then I would say it is a strong argument for committing this. Otherwise, just having this issue here with the correct title will help people who are stuck doing their upgrade.

Now that the new version and accompanying security vuln are out, I'm sure there will be more people considering the 4.7 -> 5 upgrade who maybe had no reason to move off 4.7.6 before.

will report testing results back in the next day.