Memory timeout with entity tokens and token module

I have the same problem with some field_collections

In some situations I get this error from apache. There is a recursion that seems to go forever

Maximum execution time of 120 seconds exceeded in /var/www/kalimera/includes/bootstrap.inc on line 1821

In other situations the field-collection is saved but with some fields missing.

Here i put a node module that causes this type problem.
I made the necessary adjustments for this module to fit in other Drupal installations.
Anyway, I might miss something and maybe still needs few lines commented.

Still not working in the latest dev release.

All I can say until now is that the problem disappears if I use another theme than bartik.

I'm seeing this problem too.

So I can work around the problem by reverting the code base to 7.x-1.0-beta8, hitting the website which gives WSOD because the rules code can't load the EntityAPIControllerExportable class as it doesn't exist. Then upgrading the code to either -beta9 or -dev and the site runs fine until you manually clear the cache or install a new module..

Still present in beta10

Adjusting the title accordingly.

Subscribe

Subscribing

Subscribe

Same issue.

Seems this could be an issue with views & civicrm: http://forum.civicrm.org/index.php/topic,20521.30.html

Everyone subscribe here, please try to help solving the issue by identifying the modules or module combination triggering it.

Reducing priority, as it doesn't seem to apply for most users.

Ok so looking at the backtrace on the second page of the post I linked above it seems that the issue is around calling views_fetch_data in views_plugin. hook_views_plugin gets called during a theme registry rebuild and thus when civicrm's hook_fetch_data gets called it recursively triggers another rebuild by calling the l() function.

I'm not actually sure whose at fault here; is it core for allowing reentrant theme registry rebuilds? Is it entity API for calling fetch_data during plugin discovery or civicrm for calling l()?? I don't know enough about the preconditions on the functions involved to say; anyone with more knowledge want to pipe up?

I think this is related to the above issue. I'm getting out of memory errors when entity_token is enabled. The following error occurs on: admin/config/people/accounts/settings I assume it is due to the site taxonomy structure which contains additional fields. Look at it trying to allocate 42megs! o_O
Note: Current crash is with dev, but same problem occurs with beta10.

The xdebug crash report:

The section of code in common.inc where the out of memory occurs:

  $prefix = isset($elements['#prefix']) ? $elements['#prefix'] : '';
  $suffix = isset($elements['#suffix']) ? $elements['#suffix'] : '';
  $output = $prefix . $elements['#children'] . $suffix; // <---Line 5670

  // Cache the processed element if #cache is set.
  if (isset($elements['#cache'])) {
    drupal_render_cache_set($output, $elements);
  }

  $elements['#printed'] = TRUE;
  return $output;

My error in #16 was corrected by applying the patch in #1068990: Optimize and add caching for theme_token_tree()

Patch file here: http://drupal.org/node/1068990#comment-4825584

I find it hilarious, but it is really a problem: The HTML for the Account Settings page(admin/config/people/accounts) on my project is 42 megabytes. In #16, I thought it was nonsense.

The problem is caused by the fields I've added, plus that the Token Tree is loaded five times on the Account Settings page. The User Profile contains five node references, which contain nodes, which contain authors, which point to user profiles, which goes to infinity. One of content type has node references to three of the other content types. The max depth stops it from actually rendering infinity, but the damage is already done and the resulting page is 42MB.

I'm posting here, because again, it may be related to the problem other people are having. I don't think any of the content type, user profile, or taxonomy fields are unreasonable. There are only 27 fields.

Dave Reid has started looking into the UI issue in #1229568: Token UI 2.0 Perhaps this is a problem in Token, made completely obvious by Entity Tokens? Thoughts? I'm tempted to open a new issue, but still think this is all related to a base issue.

I'm not sure how to find out how much memory some of my pages are using, but I've had to increase the memory allocation to PHP.

I've applied the patch from here, but I'm still finding that to load some of the pages to edit content types is taking ages (and Firefox throws me a warning about the script taking a long time...).

If Entity Token is the problem, then this is a real worry because we need that module to get Rules to work.

@gmak: Entity Token isn't the problem. The problem is the Tree Table UI in the Token module. It loads four layers of the token tree, which once you install Entity Token becomes very complex. In my current project this results in a 42MB(seriously) Account Settings page, resulting in an extremely long load time for HTML and later JS.

I wrote an admin page for the token module allowing you to adjust the tree depth. Changing the setting from the default of 4 to 2, significantly reduces the load time and data size. The patch to Token is available here: http://drupal.org/node/1058912#comment-4830326 I doubt the patch will be applied to Token, but it will work until a better solution can be determined.

The Token Tree UI problem is separate from the original reported issue. I've filed an issue against Token here: #1252840: Node references + Entity Tokens creates HUGE Token Tree UI table HTML

subscribe - same issue here

@JamesSharpe was right about the issue for me. It was related to CiviCRM. Updating to 4.0.5 fixed the memory timeout that was throwing the entity error.

subcribe, same problems here. But I'm not sure if its entity. But got a lot of error messages from time to time.

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 83 bytes) in /www/htdocs/sitename/includes/theme.inc on line 444

(I'm on a D6->D7 migration install)

If I execute update.php my drupal installation is completely inaccesible because of this problem. Dont know how to fix this, just how to restore the database ;)

Warning: Invalid argument supplied for foreach() in /www/htdocs/w006fdd1/v3/sites/all/modules/entity/entity.info.inc on line 138

Notice: Undefined index: required in /www/htdocs/w006fdd1/v3/sites/all/modules/entity/modules/field.info.inc on line 64

Notice: Undefined index: required in /www/htdocs/w006fdd1/v3/sites/all/modules/entity/modules/field.info.inc on line 64

After that sometimes I get an
The website encountered an unexpected error. Please try again later.

Simon

@gmak: Entity Token isn't the problem. The problem is the Tree Table UI in the Token module. It loads four layers of the token tree, which once you install Entity Token becomes very complex. In my current project this results in a 42MB(seriously) Account Settings page, resulting in an extremely long load time for HTML and later JS.

Exactly. Setting to postponed here unti it's fixed in token.

1. I have already fixed multiple token trees getting output on the Account Settings page.
2. We have added caching and optimization to theme_token_tree() to the best of my ability. If it's still not good enough stop complaining and write a patch.
3. I have started http://drupal.org/project/token_tweaks to allow users to change the maximum depth level in the tree and do other things like disable tokens.

The simple fact is that Entity tokens set the token tree up for failure by overloading it with more tokens and chaining than it was ever anticipated. A better question is why does Rules depend on Entity tokens?

Are most of these patches in this thread applied to the dev release already?

I'm getting massive issues with Rules where even setting my php.ini memory limit to 2 GB is not enough.

Please see http://drupal.org/node/1322682

I thought the issue was the Rules module until I read my Apache Error Log where I saw the memory limit.

My site is randomly crashing now and these 3 lines randomly appear across my Apache Error Log.

[Thu Oct 27 00:15:41 2011] [error] [client XX.XX.X.XX] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 44 bytes) in /var/www/html/root/sites/all/modules/entity/entity.module on line XXX

or

[Thu Oct 27 00:15:41 2011] [error] [client XX.XX.X.XX] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 44 bytes) in /var/www/html/root/sites/all/modules/entity/includes/entity.wrapper.inc on line XXX

or

[Thu Oct 27 00:15:41 2011] [error] [client XX.XX.X.XX] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 44 bytes) in /var/www/html/root/includes/common.inc on line XXX

2GB should be enough :)

If your problem is token related, pls try http://drupal.org/project/token_tweaks, it might help you. A big thanks to Dave Reid for creating this :)

I found I was able to resolve this problem by installing the Token Tweaks module and visiting
/admin/config/system/tokens and setting the maximum depth to 2. The default is 4, but I was still running out of memory when it was set to 4.

Even without token module enabled, Entity has this issue as of 2013, 1.0 and 1.1 version checked. In an installation running smoothly with 128MB, enabling it causes the memory to rise up to 300MB and beyond. The only effective solution seems to be disabling the module and do not use any dependent module (e.g. View Bulk Operations) until this issue will be fixed.

@chirale:

You might have a different issue.
This one is specifically for memory issues relating to tokens and entity token.

If you are not using these token modules I would recommend opening a new issue with as much information as you can provide about the problem.

I've encountered this issue, however specifically for taxonomy reference fields.

I have a field which is a reference to the tags vocabulary, my site has well over 10,000 tags. I'm attempting to use the field_tags token to list the tags taxonomy terms on the current node for analytics.

I tracked where the memory exception is stemming from, the taxonomy_allowed_values() function. This function loads the entire taxonomy tree, which on my site intermittently errors out while iterating in the foreach, see https://api.drupal.org/api/drupal/modules!taxonomy!taxonomy.module/funct....

This problem isn't solved by token_tweaks as my problem is not depth, but rather attempting to load all of the tags terms on the whole site in EntityMetadataWrapper::label() when $this->optionsList('view') is called.

I back-traced the error up to where entity_token defines taxonomy term reference fields as a list type, I then traced down and found that the problem begins when $wrapper->label is called in _entity_token_get_token(). The label method in the EntityMetadataWrapper instance calls $this->optionsList('view') which calls entity_metadata_field_options_list() which calls taxonomy_options_list() which calls taxonomy_allowed_values() - note that these are not all direct calls, but rather call_user_funcs.

The memory issue really stems from Drupal core taxonomy not checking if there is an absurd number of values in taxonomy_allowed_values() before fetching the tree, to be fair taxonomy_allowed_values() is only intended to be ran when used for populating a form field. Other entity reference fields, which are identified as lists in entity_token, could potentially have this same issue, however I do not have data to prove any instances of that and would be dependent upon their defining modules doing something similar to taxonomy_allowed_values().

I've attached a patch which solves this particular problem by bypassing calling $this->optionsList('view') and instead proceeding to the else statement in EntityListWrapper::label(). The token successfully outputs the expected values with the else logic in a performant way.

I'm sure there may be a better solution, however I would imagine it would require substantial work/changes to the module and I'm not familiar enough with the system and potential repercussions to provide a better solution at this time.

I have the fatal error when trying to access the rules interface. My memory limit is 128M and i cannot change it.
On my test server all is fine with 512M memory_limit.

I tried both token tweaks (set to 1 or 2), and the patch here provided on comment #32.
Unfortunately none of them worked, i still have the White screen with fatam error memory limit...

Most of people here mentioned it worked with one of these solutions, do you know why it doesnt' for me and what could be done.
Is it something related to rules?

@Weblights Once you've got more than a couple contrib modules in Drupal 7 you always need more than 128M. If you cannot change the limit to 256M, you need different web host.

You should be able to run a decent size website off 128MB RAM.
If not you have some issues somewhere.

Thanks for your replies.

The site and rules work fine.
My problem is actually only in accessing the rules Interface!

That's why i tried token tweaks and the patch.
Any idea why those didn't worK?

Token tweaks works for me.

It does depend though on your site and what entity types and fields you have as to how many tokens need to be included in the tree.

For one of my sites I had to go down to 2 levels of tokens.

If you are still having trouble at 1 level that seems a bit odd unless you have a lot of entity types.

Is there a way to know the number of entities of my site?

The number of entities shouldn't be a concern.
It is the number of entity types (nodes, users, field collection, profile2, etc.) and number fields on those that can easily make the tokens go through the roof.

Ok, must be that i have too much content types and field types
+ a too wide range of fields on my CT
+ on taxonomies
+ on users.
According to you, that's why even a 1 level token tree cannot prevent drupal to crash when trying to provide the rules interface, with php memory_limit set to 128M.

With pathauto i can disable Entity token to make it accessible, with rules this is impossible.
I guess providing you with the list of modules won't help.

Seems a bit strange still that even 1 token tree level doesn't help.
Thanks if you have more advice, but seems like i'll stay stuck there.

Well I can't say for sure without actually seeing and debugging your site but if you do have masses of fields that's a possible cause.

It's also possible that there are completely unrelated things on your site that are using up decent portions of memory.

Even with a list of modules it comes down to site configuration, your server, etc.

subscribe -

We hit a tipping point - all of the sudden, just a couple more fields were added and the site basically instantly stopped from being editable / functional. I was VERY generous with memory to even get the site to a point where I could remove fields through the UI (1024M locally ;-)) but it would kill the other developer's (and hosted sites) with less flexible resources.

Symptoms on our site of the memory issues:
The the initial content type edit page and ANY pages loading the token tree / replacement patterns interface to throw memory errors by either white screening or throwing an out of memory PHP error.

Pages/editable interfaces w/ noticeable impact:

• content type edit (tab one) to support auto_nodetitle
• pathauto patterns config page
• any node edit with a user with edit metadata (metatags module)
• any field edit page that leveraged filefield_paths (heavily reliant on tokens)

Architectural work-around (ish):
I had to do a massive re architecture to a site to reduce/reuse # of fields. This is normally best practices for Drupal, but we were constrained by a distribution model and the use of very portable features. Luckily, it was a pre-production distribution but unluckily most was built before the nice features field base / field instances split was released.

@Weblights - I used the architecture module to expose the fields and did a very manual field inventory and re-use design using the reports. I also looked directly in the DB.

What we did:
Completely rearchitected the feature structure. Now we are using deploying Field Bases separately from field instances and you can more flexibly re-use fields (if you are using features) than you could have done in the past which really helped our distro constraints. Note: this architectural approach is really dependent on your feature architecture and site constraints.

Would like to have (maybe this should go in a different queue?):
Either a new window / Ajax / some type of interface for me to pick my tokens without having to wait for the large amount of tokens to load when doing common activities. I did a look in the core issue queue, but I'm not sure if I'm looking for the right key words. Has anyone seen a similar request for me to upvote / +1?

Thanks Aimee Degnan for these info.
What you are suggesting though goes too far for me, as my site is running smoothly, except for accessing interface of pathauto and rules.

I tried something: deactivating Views Rule module fixes the fatal error problem fro accessing rules interface.
But it obviously breaks my rules ;-)

At least one path to start, but i think i will rather just move to another host...
Note: on my test server, 512 M allow me to access Rules interface, but not pathauto

I think what Aimee is suggesting in #42 is good... some sort of token finder rather than it showing all tokens at once... ?? Though maybe that only solves the problem when loading a page/form that has token/replacement patterns shown on the page. Perhaps there is just a general issue of token bloat that needs to be resolved.

On my site, the front page is a view of nodes with a lot of fields. On my local site, I have no problems, but on the dev (QA) site, admins run out of memory, but non-admins are fine. The Apache log always shows the error as being in entity_extract_ids().

I'd be happy to try token_tweaks, but I can't get to the modules admin page to enable it.

environment_indicator 7.x-1.1	extlink 7.x-1.12	rules_admin 7.x-2.3
reroute_email 7.x-1.1	faq 7.x-1.0-rc2+8-dev 2012/06/16	rules_scheduler 7.x-2.3
admin_devel 7.x-3.0-rc4	field_collection 7.x-1.0-beta5	select_or_other 7.x-2.17
admin_menu 7.x-3.0-rc4	field_collection_table 7.x-1.0-beta1+2-dev 2012/06/11	serial 7.x-1.2
admin_menu_toolbar 7.x-3.0-rc4	field_group 7.x-1.1+67-dev 2013/05/16	superfish 7.x-1.9
advanced_help 7.x-1.0	field_hidden 7.x-1.3	taxonomy_manager 7.x-1.0
auto_entitylabel 7.x-1.2	field_validation 7.x-2.3	token 7.x-1.5
backup_migrate 7.x-2.7	field_validation_ui 7.x-2.3	admin_page_fieldsets 7.x-1.0-rc1+14-dev 2013/01/09
block_class 7.x-1.2	globalredirect 7.x-1.5	no_index 7.x-1.0-rc1+14-dev 2013/01/09
calendar 7.x-3.4+1-dev 2012/09/25	hierarchical_select 7.x-3.0-alpha5+9-dev 2012/06/16	quick_list 7.x-1.0-rc1+14-dev 2013/01/09
cer 7.x-1.x-dev 2013/04/16	hs_taxonomy 7.x-3.0-alpha5+9-dev 2012/06/16	util 7.x-1.0-rc1+14-dev 2013/01/09
commentsblock 7.x-2.2	imce 7.x-1.7	views 7.x-3.7
comment_goodness 7.x-1.4	imce_wysiwyg 7.x-1.0	views_ui 7.x-3.7
computed_field 7.x-1.0-beta1+2-dev 2012/02/14	libraries 7.x-2.1	views_bulk_operations 7.x-3.1
conditional_fields 7.x-3.x-dev 2012/11/12	link 7.x-1.1	views_field_view 7.x-1.1
ctools 7.x-1.3	mailsystem 7.x-2.34	workflow
date 7.x-2.6	mimemail 7.x-1.0-alpha2	workflow_access
date_api 7.x-2.6	mimemail_compress 7.x-1.0-alpha2	workflow_admin_ui
date_popup 7.x-2.6	og 7.x-2.2	workflow_cleanup
date_views 7.x-2.6	og_access 7.x-2.2	workflow_revert
devel 7.x-1.3	og_context 7.x-2.2	workflow_rules
devel_node_access 7.x-1.3	og_ui 7.x-2.2	workflow_views
diff 7.x-3.2	og_extras 7.x-1.1+5-dev 2012/08/14	workflow_extensions
email 7.x-1.2	pathauto 7.x-1.2	wysiwyg 7.x-2.2
entity 7.x-1.1	phone 7.x-2.x-dev 2013/05/24	child_nodes
entity_token 7.x-1.1	profile2_page 7.x-1.3	node_to_word
entityform 7.x-1.1	profile2 7.x-1.3	submitted_by 7.x-1.x-dev 2013/04/29
entityreference 7.x-1.0	r4032login 7.x-1.5	tta_custom
entity_view_mode 7.x-1.0-rc1	rules 7.x-2.3

I see that entity_extract_ids error sometimes too but not sure if it was in context of this problem.

I also put this into a custom module:

/**
 * This is needed to prevent infinite recursion on the pathauto patterns
 * configuration page. Code is from:
 * http://drupal.org/node/1058912#comment-4801902
 */
function mymodule_preprocess_token_tree(&$variables) {
  if (array_intersect(array('node', 'user'), $variables['token_types'])) {
    $variables['recursion_limit'] = 2;
  }
}

+1 for the token_tweaks module mentioned in #26: https://drupal.org/project/token_tweaks

token_tweaks is an ok workaround to stop the site from breaking but is not a proper fix, becuase it has the side effect of making the token listings around the site next to useless when you have to set it to 1 or 2 levels.

You're right. I also submitted a patch here: #1058912: Prevent recursive tokens

#47 worked a treat for me :-)

while the token UI stuff sounds ideal, since #47 works, can this not be applied as an interim work around until the token UI features are working?

it could even be easily made as a configurable "load limiter" option which could be disabled altogether or have a selectable number of recursion levels to limit - just a thought, as this issue clearly causes a lot of people a lot of frustration

My issue to give an idea of the scope, had this on an empty site.
https://www.drupal.org/node/2407939

+1
for the token_tweaks module mentioned in #26: https://drupal.org/project/token_tweaks

The issue in #50 has been closed and will not be fixed. The function in #47 doesn't really work for me. When a site has in excess of 200 node and field_collection fields, the token tree never manages to finish rendering (at least in FF). Some other mechanism is required.

I think this issue is related with this one: #1940710: SQL crashing

I had this problems when using the entity_email module: https://www.drupal.org/project/entity_email

Check under "Providing A UI For Browsing Tokens" in https://www.drupal.org/documentation/modules/token.

I experienced this error, Fatal error: Allowed memory size of 268435456 bytes exhausted in sitename/includes/theme.inc on line 2085, while using Entity Registration module.

As other in this issue stated, I also was able to resolved this problem by installing the token tweaks module and setting the maximum depth level to 2 in the tree.

Thanks,
the module https://www.drupal.org/project/token_tweaks indeed also solved my "Fatal error: Allowed memory size of .... bytes exhausted" problem

#47 still works well for this Drupal Token issue, thanks!

This and https://www.drupal.org/project/token_tweaks to limit the token depth to 2 for example.

Thanks #47 really worked for me too.

Not much to review here. There is no generic solution to this issue, except workarounds mention in this thread.

I was running into this on a particularly large/complex site. I sort of threw everything at it, including using token tweaks (So I'm not sure if this patch is what fixed it), but I patched the token module to cache every row of the token tree. Probably not normally a good practice, but recursive algorithms that generate a lot of output like this need some sort of incremental caching.

#47 worked like a charm. I've been dealing with this for months, unable to edit my content types except on every fourth or fifth try. This needs to be on the project page itself.

Working on a real solution to this issue. Not yet production-ready as it is missing some features and needs to have proper caching and security checks implemented.

The module works by asynchronously loading each level of the token tree when it is requested instead of building and serving it all at once. Works with the memory limit set to unrealistically low levels (e.g. 16M).

This also resolves performance issues in the browser as the DOM tree no longer has to deal with loading and scripting tens of thousands of nodes.

Module is up here: https://www.drupal.org/project/fast_token_browser

Kind of new at all this. How and where does one insert a module (a la #47) into the site to help with this Token timeout issue?

Google for Drupal apply patch. In the meant time you can also simply use https://www.drupal.org/project/token_tweaks to fix the issue.

@mtdaveo See https://www.drupal.org/docs/7/extend/installing-modules

The short version: go to the module project page, download and extract the archive to yoursite.root/sites/all/modules. Enable it in the admin interface. Make sure to back up your site files and database first!

RE: Token Tweaks module...is it possible/safe/wise to install 7.x-1.x-dev into my 8.6.14 environment? I'll wait for a bit, then maybe try inserting #47 per #69.

Thanks, y'all!

mtdaveo, no, it probably won't even be possible to enable it

@napacker - Thanks, but I know how to install regular modules. My question was how to do something like #47.

@wizonesolutions - That's what I figured. Thanks.

To whomever is interested, the module fast_token_browser mentioned in comment #66 did the trick to stop the memory errors for me.

token_tweaks doesn't have a tagged release and neither has security coverage, both of which fast_token_browser has.

As mentioned in #63, I don't think this problem can be fixed in entity module.

Fast Token Browser sounds good, but it's Drupal 7 version. Why do people still post/refer to 7.x modules? Is there a way to use them in 8.x or are many people still using core 7.x? If so, why? (Honest questions.)

Trying my hand at #47 above. My first foray into coding here. Can anybody help?

Ha ha. How do I upload/show error screenshots?

@mtdaveo porting code from Drupal 7 to 8 might be a bit challenging if you are new to Drupal.

I suggest you reach out for support in Drupal's Slack group where help should be more effective.

Please follow the instructions here: https://www.drupal.org/slack

Fast Token Browser sounds good, but it's Drupal 7 version. Why do people still post/refer to 7.x modules?

Because this issue here is for 7.x.

@leymannx Ha ha. Oh, man. So it is. Just now looked at info panel at top right and just above this comment box. Thanks for being kind with the obvious.

FYI: I found the culprit that was jamming up my token search. It was the FontAwesome module. I unchecked the default option to "Allow CSS psuedo elements" and my token tree loaded up right quick. For the first time in...6 weeks? Good God, man.

I'm not using that module much yet, but likely will soon. Do you think it's okay to toggle that option on/off depending on my need to access tokens?