Out of memory with large taxonomies

I can confirm...when I try to get to settings page:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 21999200 bytes) in /home/sporbcom/public_html/includes/form.inc on line 2070

Are you sure this is not a duplicate of #630312: Runaway javascript on admin page ?

I'm having the same problem. I've a large number of vocabularies and terms, with about 12 content types and maybe 6 menus - and excluding the navigation/admin one - all have only about 10 items. The settings page is a massive memory hog. I had to increase my php memory_limit to 240MB temporarily in order to view it.

This is unrelated to any javascript problems. For a start javascript shouldn't cause php memory limit errors, and secondly I can reproduce it with javascript disabled.

I still have the problem as well in version 6.x-1.1

I can confirm this problem with 6.x-1.1 as well. Jumped to 240M in php.ini to get view it but I *also* get the Javascript error from the #630312 issue when I try to save the settings page (but not when I try to view it). Related issues?

+1 i'm seeing this with 1.1

In my case it is around 8000 tax in three levels.

It happens in the settings page because of the call to taxonomy_get_tree(). A workaround could be to add a checkbox to vocabularies edit form to be able to 'expose' them to menutrails. By default the checkbox is disabled and there's also a message that warns to not use vocabularies with a large taxonomy set.

"needs work" state is only for when there is a patch attached which doesn't full solve the problem, but there's no patch here yet. Resetting to active.

Same problem here: 8 vocabularies, a little over 2000 terms.

+1 seeing this problem with 1.1 though reverting to 1.0 fixes it.. but i neeeed the new features..

This might be interesting for some: I found out that I can actually see the settings page (and change settings) by creating a user, giving that user the admin role (using adminrole module) and then masquerading as that user. I wonder why this is, since the admin role created by adminrole module can do EVERYTHING except run update.php.

I have my memory_limit at 126M which gives me Fatal Error as User 1.

anyone get anywhere with this issue?

Here's a patch implementing what I described in #8.

@jonhattan: Thanks for the patch. I tested it and found that some particularly large vocabulary will not show up on the settings page and having only like 3 vocabularies shown on the settings page (one being very large) makes that error come back. I also have one very large vocabulary which _does_ get shown on the settings page, but I get a browser timeout when trying to open the collapsed fieldset. Do you think this can be improved by listing all available vocabularies on admin/build/menu/trails with a link to a separate page specifically for that vocabulary and its terms and then using a pager for the terms (because even on a separate page there might be timeouts when people have around 5000 terms)? I don't think this can be done another way when dealing with such a great number of terms. Unfortunately I don't know enough PHP to help you out here.

3 vocabs of 1000 terms is more or less equivalent to a single vocab of 3000. So the memory limit problem may still be present if you enable those 3 vocabs for menutrails. The problem with the collapsible fieldset seems to be this other issue #630312: Runaway javascript on admin page. Not server related.

I don't see the point in using vocabularies with more than a few dozen of terms to set menutrails. Perhaps I'm blind in this point. I'm not interested in solving this usecase.
By the way, the only scalable approach I think of is to not load the whole terms tree and use an approach based on autocompletion in admin/build/menu/trails. This also implies a relative big change to the module.

Lastly, using a pager probably need a custom implementation of taxonomy_get_tree() to accept a limit.

@jonhattan: I was thinking the same thing - I don't see how anyone would wanna set menutrails for such big vocabularies. But the problem right now is that people with huge taxonomies still need to set menutrails for node types and can't. However, there are temporary solutions in this thread and very likely not that many people with this problem, so imo there's no need to rush this.

The patch in #14 allowed me to access the administration page...site has ~1000 terms in 7 vocabularies.

I might be missing something.. I have rather small taxonomies that i want to work with menu trails..
But very large taxonomies that are needed for other things..
How do i disable the large taxonomies for menutrails?

What were the changes between 1.0 & 1.1 that caused this to break?

Would it be beneficial to have a large taxonomy to test against for future releases?

For active sites having taxonomies of over 1000 terms shouldn't be unusual.

Looks like the patch in #14 provides the ability to set the vocabulary used to set menu trails.

Surely this would be useful for performance reasons even if you weren't dealing with a large taxonomy.

@jonhattan, post #14: +1 - worked for me beautifully, thanks :)

The problem as I see it, is the enormous number of complete menu trees that must be sent to the browser, to populate the select lists for each and every menu trail term.

Loading 1000 taxonomy terms isn't so bad. But sending 1000 menu tree select lists to the browser for those terms is an insane amount of data... and 99.9% redundant. You're getting O(N^2) exponential growth defined by: ( content-types+taxonomy terms ) * size of menu tree.

The real solution to this scalability problem is to only send the menu tree ONCE, then do "lazy" population only when each menu select list is actually clicked. This eliminates the multiplier effect entirely and improves the situation to O(N) linear scalability. Unfortunately this approach requires Javascript, so I'm not sure what issues that raises.

Subscribing.

I don't actually use Menu Trails for taxonomy at all, so I've just commented out the operative code in menutrails.module, lines 266-285, in lieu of an actual solution:

  /*
  foreach ($vocabs as $vocab) {
    // Tagging gets out of hand too fast, so we disallow.
    if ($vocab->tags != 1) {
      $form[$vocab->vid]['menutrails_terms'] = array(
        '#tree' => TRUE,
        '#type' => 'fieldset',
        '#collapsible' => TRUE,
        '#collapsed' => TRUE,
        '#title' => t('Categories: @vocabulary', array('@vocabulary' => $vocab->name)),
      );
      $terms = taxonomy_get_tree($vocab->vid);
      foreach ($terms as $term) {
        $form[$vocab->vid]['menutrails_terms'][$term->tid] = array('#type' => 'select',
          '#title' => t('Parent item for @term', array('@term' => $term->name)),
          '#default_value' => isset($term_trails[$term->tid]) ? $term_trails[$term->tid] : NULL,
          '#options' => $options,
        );
      }
    }
  }
  */

Patch #14 worked for me, thanks.

Tried both #14 and #25. Was able to reduce PHP memory limit significantly to enable settings page to be shown with either option.

PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72352486 bytes) in /includes/common.inc on line 2976

Yep, I've done the same, this made it work.

I think we should make the Settings page a "little" smaller ;)

No need to duplicate huge select boxes, custom_breadcrumbs module's settings page is better way.

The patch in #14 worked for me as well.

I can confirm that this problem could also be caused by the menu tree as I had menu trails working on my site - removed it for a while as I was trying to get i8n menus working - instead decided to build another menu for my 2nd language, and only now on reinstalling menu trails have I started getting out of memory errors. The size of my taxonomies has not been changed, just this extra menu has been added.

Unfortunately my server provider has set a maximum PHP memory limit which I am unable to exceed, so there's nothing much I can do about it! - Strange thing is, the error is actually showing that it tried to allocate a number which is way under the actual limit...

+1 more for #14
I'm also now hitting problems due to a huge menu (thousands of items repeated in a selectbox a few dozen times in the admin page) but that's another issue.
For now, I am surviving with the nice opt-in option by jonhattan

It's needed to clean up variables in hook_uninstall(). Will do it later today

subscribe

subscribe

The path form #14 is still the solution for big taxonomy.
Thank You

Any reason it's not RTBC & already in this module? That issue was from May 30, 2010 which is nearly 2 years ago now.

Patch #14 just saved my life, thanks a lot!

The cleanup can be done on commit, this is likely outdated now though:)