Node_access_rebuild needs tons of memory
ChrisKennedy - January 12, 2007 - 03:04
| Project: | Drupal |
| Version: | 5.x-dev |
| Component: | node system |
| Category: | bug report |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | closed |
Description
node_access_build() requires a ton of memory when rebuilding the node_access table because it causes node_load() to load *every* node and keep them cached. On my site with 3800 nodes this easily swamped my 24MB of php memory. This patch tells node_load() not to cache the nodes.
| Attachment | Size | Status | Test result | Operations |
|---|---|---|---|---|
| node_access_rebuild_0.patch | 642 bytes | Ignored | None | None |
#1
Dries have the exact same patch a +1 at http://drupal.org/node/123705. that one is a dupe of this one. Since Chris, Dries, and I all reviewed this, I set to RTBC.
#2
While this is useful, this is not adequate to make this function usable. http://drupal.org/node/124727 . Note that this must be backported to 5.x .
#3
@chx - thats a nice feature, but lets not get in the way of this patch. You are overstating the issue when you say that this function is not usable. With this patch i was able to rebuild groups.drupal.org. Without the patch, I wasn't. This patch deserves to go in, and your comment just muddies the water, IMO.
#4
Committed - http://drupal.org/cvs?commit=59253
#5
Portable.
#6
I am sorry if I muddled the waters -- my comment began with 'this is useful' and just wanted to draw attention to a related issue. groups.drupal.org is a rather small site btw in this context.
I am happy that my muddling have not stopped this issue from being committed and that's a good thing.
#7
COmmitted to 5.
#8
#9
I may be wrong, but in the description for this bug, I believe that node_access_build() should be node_access_rebuild(). I hunted in the API docs for some time trying to find node_access_build(), lol to no avail. Just a minor change.
#10
Yes, it was a typo and cannot be changed.
#11
I'm building a site with 80 000 nodes, and the node_access_rebuild_0.patch didn't seem to fix the problems I had with node_access_rebuild. So I wrote a command line script that does (see the attachment). It solves also the following problems:
The rebuild process takes several minutes (at least), and during that time most of the nodes get "permission denied" because of this line run:
db_query("DELETE FROM {node_access} where nid=%d",$node->nid);
So I made the script to delete "per node".
I also got around the php.ini time limit settings by calling the script from command line which has different php.ini than for apache2.
So.. its a bit of a hack, but it does the trick.
#12
My approach to a similar problem was creating a script which updates just one node each time and using javascript calls himself to update next node. The script is called rebuild_permissions.php:
<?phprequire_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
user_authenticate('admin', 'admin');
$actual=db_result(db_query("SELECT nid FROM {node} WHERE nid > %d ORDER BY nid",$_GET['node']));
if ($actual>0) {
$sencer=node_load($actual);
node_access_acquire_grants($sencer);
}
?><html>
<head>
<script type="text/javascript">
<!--
function delayer(){
window.location = "rebuild_permissions.php?node=<?=$actual?>"
}
//-->
</script>
</head>
<body <?=($actual>0 ? " onLoad=\"setTimeout('delayer()', 500)\"" : "") ?>>
<?php
if ($actual>0) echo "doing...".$actual;
else echo "Done";
?>
</body>
</html>
The output isn't necessary... But I like to know its working.
I already posted it in the mailing list, but I think it may be useful to other users.
#13
I've been having the same problems and until we've got a server to support drupal 6 I decided on a quicker hack than sjs or enboig: i.e. when it happens, finish the job, don't start again. The code below is designed to be incorporated into a page (with php permissions) that only admins have access to and works with old versions of MySQL. It simply abstracts from the node_access_rebuild() function. If it fails to finish, it could be run again.
<?phpif (count(module_implements('node_grants'))) {
$result = db_query("SELECT n.nid FROM {node} as n left join {node_access} as a on n.nid=a.nid where a.nid is null");
$total = $granted= $left = 0;
while ($node = db_fetch_object($result)) {
$total++;
$loaded_node = node_load($node->nid, NULL, TRUE);
if (!empty($loaded_node)) {
node_access_acquire_grants($loaded_node);
$granted++;
}
}
$left= db_result(db_query("SELECT count(*) FROM {node} as n left join {node_access} as a on n.nid=a.nid where a.nid is null"),0);
drupal_set_message("granted $granted nodes from $total, $left left");
}
?>
Chris
#14
Thank you Chris for that most sweet script. I have been struggling with this issue for weeks thinking it was a problem with my Db or a configuration problem. The core permissions rebuilder looked as if it was working even though it hung for hours at 90%....after all the documentation said it could take a long time... I loved how easy it was to plug it in a page. I clicked preview and watched it click away! Totally awesome. Thanks again.
Jon
#15
Simply beautiful! It is a shame your script is not included in the drupal scripts directory.