node_access_build() requires a ton of memory when rebuilding the node_access table because it causes node_load() to load *every* node and keep them cached. On my site with 3800 nodes this easily swamped my 24MB of php memory. This patch tells node_load() not to cache the nodes.

CommentFileSizeAuthor
#11 node_access_rebuild.php_.txt2.3 KBsjs
node_access_rebuild_0.patch642 bytesChrisKennedy

Comments

moshe weitzman’s picture

moshe weitzman commented
Status: Needs review » Reviewed & tested by the community

Dries have the exact same patch a +1 at http://drupal.org/node/123705. that one is a dupe of this one. Since Chris, Dries, and I all reviewed this, I set to RTBC.

chx’s picture

chx commented

While this is useful, this is not adequate to make this function usable. http://drupal.org/node/124727 . Note that this must be backported to 5.x .

moshe weitzman’s picture

moshe weitzman commented

@chx - thats a nice feature, but lets not get in the way of this patch. You are overstating the issue when you say that this function is not usable. With this patch i was able to rebuild groups.drupal.org. Without the patch, I wasn't. This patch deserves to go in, and your comment just muddies the water, IMO.

ChrisKennedy’s picture

ChrisKennedy commented
Status: Reviewed & tested by the community » Fixed

Committed - http://drupal.org/cvs?commit=59253

ChrisKennedy’s picture

ChrisKennedy commented
Version: 6.x-dev » 5.x-dev
Status: Fixed » Patch (to be ported)

Portable.

chx’s picture

chx commented

I am sorry if I muddled the waters -- my comment began with 'this is useful' and just wanted to draw attention to a related issue. groups.drupal.org is a rather small site btw in this context.

I am happy that my muddling have not stopped this issue from being committed and that's a good thing.

drumm’s picture

drumm
he/him
Location NY, US
commented
Status: Patch (to be ported) » Fixed

COmmitted to 5.

Anonymous’s picture

(not verified) commented
Status: Fixed » Closed (fixed)
jlmeredith’s picture

jlmeredith
Location Denver, CO
commented
Priority: Normal » Minor
Status: Closed (fixed) » Fixed

I may be wrong, but in the description for this bug, I believe that node_access_build() should be node_access_rebuild(). I hunted in the API docs for some time trying to find node_access_build(), lol to no avail. Just a minor change.

ChrisKennedy’s picture

ChrisKennedy commented
Priority: Minor » Normal
Status: Fixed » Closed (fixed)

Yes, it was a typo and cannot be changed.

sjs’s picture

sjs commented
StatusFileSize
new node_access_rebuild.php_.txt2.3 KB

I'm building a site with 80 000 nodes, and the node_access_rebuild_0.patch didn't seem to fix the problems I had with node_access_rebuild. So I wrote a command line script that does (see the attachment). It solves also the following problems:

The rebuild process takes several minutes (at least), and during that time most of the nodes get "permission denied" because of this line run:
db_query("DELETE FROM {node_access} where nid=%d",$node->nid);

So I made the script to delete "per node".

I also got around the php.ini time limit settings by calling the script from command line which has different php.ini than for apache2.

So.. its a bit of a hack, but it does the trick.

enboig’s picture

enboig commented

My approach to a similar problem was creating a script which updates just one node each time and using javascript calls himself to update next node. The script is called rebuild_permissions.php:

<?php
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
user_authenticate('admin', 'admin');
$actual=db_result(db_query("SELECT nid FROM {node} WHERE nid > %d ORDER BY nid",$_GET['node']));
if ($actual>0) {
 $sencer=node_load($actual);
 node_access_acquire_grants($sencer);
}
?><html>
<head>
<script type="text/javascript">
<!--
function delayer(){
   window.location = "rebuild_permissions.php?node=<?=$actual?>"
}
//-->
</script>
</head>
<body <?=($actual>0 ? " onLoad=\"setTimeout('delayer()', 500)\"" : "") ?>>
<?php
if ($actual>0) echo "doing...".$actual;
else echo "Done";
?>
</body>
</html>

The output isn't necessary... But I like to know its working.
I already posted it in the mailing list, but I think it may be useful to other users.

chris55’s picture

chris55 commented

I've been having the same problems and until we've got a server to support drupal 6 I decided on a quicker hack than sjs or enboig: i.e. when it happens, finish the job, don't start again. The code below is designed to be incorporated into a page (with php permissions) that only admins have access to and works with old versions of MySQL. It simply abstracts from the node_access_rebuild() function. If it fails to finish, it could be run again.

  if (count(module_implements('node_grants'))) {
    $result = db_query("SELECT n.nid FROM {node} as n left join {node_access} as a on n.nid=a.nid where a.nid is null");
	$total = $granted= $left = 0;	
    while ($node = db_fetch_object($result)) {
	  $total++;
      $loaded_node = node_load($node->nid, NULL, TRUE);
      if (!empty($loaded_node)) {
        node_access_acquire_grants($loaded_node);
		$granted++;
      }
    }
  $left= db_result(db_query("SELECT count(*) FROM {node} as n left join {node_access} as a on n.nid=a.nid where a.nid is null"),0);
  drupal_set_message("granted $granted nodes from $total, $left left");
}

Chris

stieglitz’s picture

stieglitz commented

Thank you Chris for that most sweet script. I have been struggling with this issue for weeks thinking it was a problem with my Db or a configuration problem. The core permissions rebuilder looked as if it was working even though it hung for hours at 90%....after all the documentation said it could take a long time... I loved how easy it was to plug it in a page. I clicked preview and watched it click away! Totally awesome. Thanks again.
Jon

mikhailian’s picture

mikhailian commented

#11
sjs - May 15, 2008 - 17:15

Simply beautiful! It is a shame your script is not included in the drupal scripts directory.