Prevent recursive tokens

We don't do any thing specific towards token integration, so I'm assuming the issue is more related to the way token.module collects field data ?

Token does nothing with fields currently in D7...

WHEN I REVERT REFERENCES MODULE back to dev version prior to 11 Feb 2011 everything returns to normal

That would incriminate this patch : #1005364: support hook_entity_property_info(). Do you have the "Entity API" module enabled ? If so, could you try disabling it (while keeping noderef enabled) ? If that solves the issue, we'd probably bump this over to http://drupal.org/project/entity

OK - thanks for helping narrowing this down.

I'll report this in #1005364: support hook_entity_property_info() and will ping @fago about it. I'd rather not revert the commit just right now, until we get other reports of the issue.

Meanwhile, the "entity API" module is obviously central to many setups, including yours, so disabling it is not an option. I'd suggest you comment out the 'property_type' line in both node_reference_field_info and user_reference_field_info(), before re-enabling entity API.

I took a look at this.

>Also I had to disable Entity Tokens.
I think that Entity tokens + token module causes the problem. If entity tokens is enabled and the property info is given, it will provide tokens for us, which results in lots more available tokens. Then token module comes and recursively builds the token tree for its widget, which can become increasingly huge the more chained token possibilities you have - and this is exactly what you get if you have reference fields...

Let's take an example having quite some fields, so let's say there are 100 tokens for a node. If we have 10 node references in there, there will be 100x10 tokens in the next level - so on the fourth level we would have 100x(10^4) tokens. Of course this eats lots of memory - while I must say ~524MB is really impressive!

I did some testing on my development site with only a few fields. I had 34,5MB memory usage on page where the token tree widget is shown. When I added another node-reference field it went up to 40MB.

I have a similar issue which is not directed to the references module but might be directly related to the solution found here because of the Entity module API mentioned.

These are already enabled with no issues:
References 7.x-2.x-dev
Node Reference 7.x-2.x-dev
User Reference 7.x-2.x-dev

My goal was actually to install the Rules module: Rules 7.x-2.0-alpha5

Rules requires Entity and Entity tokens. I already had Token 7.x-1.0-beta1 enabled.

The trouble seems to begin here:
I enabled:
Entity API 7.x-1.0-beta7
with no issue. Then I enabled:
Entity tokens 7.x-1.0-beta7
and got this error:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 24 bytes) in /home1/xxx/public_html/xxx/sites/all/modules/token/token.pages.inc on line 145

I have looked for the property_type (to comment out) and cannot find where it is.

Meanwhile, the "entity API" module is obviously central to many setups, including yours, so disabling it is not an option. I'd suggest you comment out the 'property_type' line in both node_reference_field_info and user_reference_field_info(), before re-enabling entity API.

I looked in entity.api.php and many other files, where are these reference fields?

I am not sure how to proceed. I hope this was the right place to post this. Please advise.

Ultimately I want to set defaults for user reference fields. i.e. when a user adds a content node that the "contact" is by default the uid entering the content for that node. I cannot test the suggestion because I can't find the property_field.

@oceanman : look in node_reference and user_reference modules

@issue: see #1068990: Optimize and add caching for theme_token_tree()

I was looking in entity and not the references module. Thanks for redirecting me. Got it.

@fago Thanks for the link. I have had no success.

After commenting out 'property_type' in these modules and enabling them again
Node Reference 7.x-2.x-dev
User Reference 7.x-2.x-dev

I patched the token.pages.inc file with the token.patch from #1068990
I disabled both the entity and entity token modules.
I then enabled the entity AIP module with no issue.
I then enabled the entity token module and got this error:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 159063 bytes) in /home1/xxx/public_html/xxx/includes/theme.inc on line 1738

Is there any kind of fix for this memory error using the entity token module?

Thanks @amsri

I have those both commented out but the errors are still coming when I enable the Rules module.

This is the error on a white screen after I enable Rules:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 7 bytes) in /home1/xxx/public_html/xxx/includes/menu.inc on line 3393

After I disable the Rules module I got this error:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 11 bytes) in /home1/xxx/public_html/xxx/includes/registry.inc on line 42

Same type of memory error but not from the same file. Any ideas for these errors are coming from?

I face the same problem, but needed a long way to pinpoint the error to references.
I applied #1068990: Optimize and add caching for theme_token_tree(), which didn't help.
I reversed #1005364: support hook_entity_property_info(), (as also described in #19) which resolved the issue.
For any further details, please see #1005364
I don't use Rules, so I cannot confirm Oceanman's #20. If his comments are resolved, this issue could be closed?

I have gone back and looked at my files and discovered a mistake on my part. I had

// Support hook_entity_property_info() from contrib "Entity API".
'property_type' => 'node',

and not this as amsri indicated in #19

// Support hook_entity_property_info() from contrib "Entity API".
//'property_type' => 'node',

After performing several (3) clear cache I am not getting the white screen or any other errors. Therefore, I guess this is resolved.

I did try to reinstall Token module again and when it is enabled I get this list of errors but they do not appear to be related to the References module. But, since fago referred to Token in #15 I thought I would share them below. If anyone has come across these in the process, please let me know.
* Notice: Undefined index: date-field in _token_build_tree() (line 736 of /token/token.module).
* Warning: Invalid argument supplied for foreach() in _token_build_tree() (line 736 of /token/token.module).
* Notice: Undefined index: datestamp-field in _token_build_tree() (line 736 of /token/token.module).
* Warning: Invalid argument supplied for foreach() in _token_build_tree() (line 736 of /token/token.module).
* Notice: Undefined index: datetime-field in _token_build_tree() (line 736 of /token/token.module).
* Warning: Invalid argument supplied for foreach() in _token_build_tree() (line 736 of /token/token.module).

Oceanman, the errors are from the new Date module.
I suppose we can close this message. The solution is: revert the patch in issue #1005364: support hook_entity_property_info()
Let's continue over there.

This likely has to be fixed in token module by preventing building trees of recursive tokens, like can easily happen with references. Transferring to my issue queue.

Adding performance tag

Thanks for putting this in your issue queue. Do you have an ETA for a fix/path/update, or can you point me to a likely place in the code to start looking if I want to try to fix it myself? Very interested in getting this resolved ASAP and am happy to support you in whatever way possible. Thanks!

Commenting out 'property_type' in these files fixed the issue for me. Thanks a ton!

sub

Subscribing

Curious about the status of this. It's now even affecting the admin page for Google Analytics for me. It takes about 1.3GB of RAM to load the page.

After reviewing #24 and #25, I agree with amsri: trigger of the error is References, but root cause is Token.
I've added the following temporary fix to token.module:

   // If $token_type is an entity, make sure we are using the actual token type.
   if ($entity_token_type = token_get_entity_mapping('entity', $token_type)) {
     $token_type = $entity_token_type;
   }
+  // avoid massive memory consumption, as described in d.o. issues 1005364 / 1058912
+  if ($token_type == 'user' || $token_type == 'node') {
+    $options['depth'] = min($options['depth'], 2);
+  }

Is there any progress on this? I still get WSOD on many pages even with the fix in #35

subscr

#35 helps

Marking as a D7 stable release blocker.

This seems like a nasty bug, and is a result of loading tokens without care to the actual required replacements. Here would be my approach (and not sure if feasible ATM):

For token discovery, stop when a chained referenced type token is discovered.

For theming, only display the base tokens, with a AJAX option to load the chain referenced type tokens help

For rendering, preprocess required replacements and load only the required replacements. This would potentially require major refactoring, but would result in a minimal replacement load. No object spl / entity tracking would be required as if the user dumbly does some type of recursive loop, the input token chain is finite and should be safely loaded.

i.e.
[obj1:title]
[ob1:field1:chaining-token{ob2}:field3:chaining-token{ob1}:field1:chaining-token:{ob2}author]

where {ob1} & {ob2} are only added to reference the token object currently referenced at that point in the chain.

Since only the single chained token is supplied, then we have the following loads:
2 * base obj loads
2 * referenced obj loads

This would lead on nicely to treat entity author tokens, et al, as chaining tokens, preventing the loading of these unless the input specifically requires these.

Love to know if you think this is feasible!

Converting the token UI to perform AJAX loads of sub-trees would be a separate issue that we can tackle post-1.0 (but I do like that idea - just never had time to do such an implementation). For now we're just looking for a solution that doesn't require a complete refactor and can get us a stable release.

I've added simple recursion protection with http://drupalcode.org/project/token.git/commit/55634cb - that prevents cases like [comment:parent:parent] or deeper from getting generated in the token trees. This will not be enough but it at least is a start.

Subscribe. Hitting this on the content type field settings pages. [Edit: Also on realname settings page, and any other settings page where token replacements are shown.]

This started happening after I upgraded to 7.x-1.0-beta3 (July 26). beta3 seems to include git commit 9e0dda4, which still doesn't fix the problem for me :-(

[Edit 2: I found that half the problem is that I am using profile2, and have about 25 extra profile types for my site. That means a heck of a lot more tokens. I've implemented the solution found a few posts earlier, and will attach a diff for those interested in a quick/interim fix.]

Diff attached (don't have time to create a proper patch). Definitely not the route to go for good, but this at least lets me view admin pages on my site again.

The better way to do this without hacking the module is:

function mymodule_preprocess_token_tree(&$variables) {
  if (array_intersect(array('node', 'user'), $variables['token_types'])) {
    $variables['recursion_limit'] = 2;
  }
}

Good suggestion :)

This should work better for me in the long run, anyways, especially as I load up my site with more profile2 profiles.

sub

This patch adds an admin section for token which allows the max depth to be adjusted. It replaces the theme $variable. Maybe not the best way to do it, but it allows me to continue working on my project. Set it to 2, and everything works. Leave it at the default, and PHP runs out of memory or JS locks up the browser. The best fix is a UI change, but thats for 7.x-2.x. I am purposefully not changing the status of this issue.

[subscribe]

I'm not sure if here is a good place, but I badly need to resolve my problem.

I need Rules looping some list and creating new entities (nodes). Debug says all evaluation is OK.
But only first node is created, and debug informs in the "Reaction" section that it is "Not evaluating reaction rule [My Rule] to prevent recursion".

Is this due to exaggerated recursion prevention in Drupal 7.7?

Oh, that makes everything that shows a token list work _much_ better! Thanks!

Michelle

subscribe

Unfortunately, using newest beta4 version didn't solve my problem.

This issue is about preventing recursive tokens - not limiting the tree depth. While the latter is indeed a temporary solution, let's continue on-topic. If you have problems related to *specific* tokens being recursive the in token help UI I need details about what tokens they are, what module provides the tokens, etc.

Removing the blocker tag since we did implement basic protection. It can be improved but it is no longer a 7.x-1.0 blocker.

Understood & agreed. New issue for #1252840: Node references + Entity Tokens creates HUGE Token Tree UI table HTML created.

subscribe

For those of us that don't know the inner workings of Token, could you possibly clear something up? The patch in #49 works to make things that list tokens available load in a reasonable time. If I'm understanding correctly, it's just a work-around and not a permanent solution. Is the permanent solution for that what will be worked on in the new issue from #56 or is that still being worked on in this issue? And when you say "we did implement basic protection" is that something since yesterday? I got the dev snapshot yesterday and it was still very slow before this patch.

To be clear, I'm not trying to be demanding on getting this fixed or anything. Just trying to clarify what needs to be done to have Token not be horribly slow without having to have an un-official patch applied to it. :)

Michelle

Subscribing

@Michelle: This issue is half about recursive tokens, and half about token tree, but the OP's issue is only about recursive tokens. The problems are related, but are better handled separately.

The permanent solution for token tree UI (your issue) will be discussed in #1252840: Node references + Entity Tokens creates HUGE Token Tree UI table HTML. My patch in #49 isn't a permanent solution because it doesn't stop the problem from occurring, it only allows you do something about it. The correct fix, a UI change as far as I can tell, must make the problem never occur. What you need to do? A patch is much better than manually editing the module, so you'll need to apply my patch until another solution is worked out.

"we did implement basic protection" Dave Reid refers to is #43 on July 26, 2011 http://drupal.org/node/1058912#comment-4780906 where he implemented some basic checks to stop token recursion. In fact, is anyone still encountering the recursion issue? I think #43 may have effectively solved it.

Subscribe

subscribe

Can the patch in #49 be used for the latest token module?

I am having a smiliar issue with my site which has a 7.8 core, and 7.x-1.0-beta6 modules. When I enable the entity token, the alias patterns page gives me a out of memory error (requires more than 128MB); when I disable the entity token, this same page works and uses only 45 MB memory.

I do have reference fields in my several content types. I think my issue should be related to this thread. And I need to get my development going with entity token and rules. So I wonder if the patch provided here can be used in my case. Thanks!

@ouyang1512 The patch in #49 was deprecated in favor of http://drupal.org/project/token_tweaks

Thanks very much! This really helps!!! Now I am able to access the patterns page and the memory usage is devel_boot()=3.77 MB, devel_shutdown()=37.83 MB, PHP peak=44.25 MB after I set the depth to 2.

What is the likelihood of getting Token Tweaks module incorporated into the main Token module where it can get a little more attention? It seems the issue of unmanageable trees is going to become increasingly commonplace and so dealing with it is becoming increasingly essential and not an afterthought.

If not that then maybe at least roll a Beta or 1.0 release of Token Tweaks. it's never had any bugs. (Many are allergic to dev code on production sites.) Personally, I'd also like to also see Token Tweaks mentioned on the main Token page on d.o.

I think development of an admin interface for managing WHICH modules generate tokens in the trees might be a good candidate for development in a separate module as this function wouldn't be as essential for keeping Token functioning and there's also likely to be a variety of opinions about how that's done and might take some time to work out.

No, token tweaks will not be accepted ever into Token. The proper solution to help this issue is #1334456: Enable ajax on-demand loading of sub-trees in the Token help.

Ah, now that's the way. Nice to see such an elegant solution in the works...

What is the band-aid of the day for this problem? Token Tweaks is not preventing it from occurring.

I'd like to put a temporary fix in place because this is preventing my user logins.

I am using Entity API, but not Entity Tokens, OG, CAS & CAS Attributes (where my Token recursion is currently coming from I believe), Pathauto, and Real Name.

We hit an enormous recursion with Token and Pathauto, taking all available memory and having MySQL to disconnect.

Using Token 7.x-1.x-dev (2011-Dec-13) (from 7.x-1.0-beta7) help reduce a lot memory usage when loading a content type. In my case from 512Mb to 128Mb PHP memory limit.

But Token 7.x-1.x-dev (2011-Dec-13) and Pathauto 7.x-1.0 are not playing nice together when trying to load the Patterns tab. This is where all memory is taken. Even when pushing back the memory limit from 128 to 512Mb, I experience memory issues. And with a still larger value, MySQL kills the connection.

Any clues?

#46 This saved it for me!

This should be documented in the module Read me.

Thanks.

@xmacinfo, you better use #65:

The patch in #49 was deprecated in favor of http://drupal.org/project/token_tweaks

I have hit a recursive real names token bug too, and I have followed advice and installed the token_tweaks module. Originally I posted this issue on the OG module issue list http://drupal.org/node/1440284 and have found it may be token and real name related. Can anyone give me any advice? I have attached the error message. At the end of my tether with this bug.

More testing required on my part tomorrow. Appears to be a real name module bug in disguise. Will post my findings.

Turned real name module off and all fine for me now. Don't know if it's a real name bug or token bug but here's the issue in real name module http://drupal.org/node/1369824

I have a dev sit with d7.10, Views 7.x-3.0, Panels 7.x-3.0-alpha3, and @font-your-face 7.x-1.6. This was working great.

I created an identical site for staging which also worked great, then I updated to d7.12, Views 7.x.3.3, Panels 7.x.3.0, and @font-your-face 7.x-2.1 . With this combo update the memory usage error occurs at and number of places, for example admin/structure/panels , admin/structure/views, admin/structure/nodequeue, admin/structure/mini-panels. Reverting back Views, Panels and fyf still leaves me with the memory usage error.

Does this mean the issue began with d7.12 update? I should have tested it more thoroughly before rushing forward with contrib updates...

My apologies if this issue is better posted elsewhere, its just that after reading through this line of posts, they seemed most relevant to the issues I've experienced...

My issues were caused by the real name module but it looked as if it was the organic groups field access module as the issue both didn't appear until this module was enabled and went away when it was disabled. Took me ages to figure out it was token and real name module causing the problem. I was receiving function recursion level too deep messages rather than out of memory messages though. Have you tried increasing the allocated memory just to check that it's recursion?

Thanks, increasing allocated memory did the trick

You can use hook_tokens_alter in a "hacky" way, instead of using Entity Tokens.

/**
* Changing the language of a Term token.
* Implements hook_tokens_alter().
*/
function mymodule_tokens_alter(&$replacements,  $context){
	if(isset($context['data']['entity'])){
		if($context['data']['entity']->type == 'page'){
			if(isset($replacements['[node:field_term]'])){
				$term = $context['data']['entity']->field_term['und']['0']['tid'];
				
				global $language;
				$term = taxonomy_term_load($term );
				$term = i18n_taxonomy_term_name($term,$language->language);
				$replacements['[node:field_term]']  = check_plain($term);
			}
		}
	}
}

Token Tweaks Module is a great way to sircumvent this issue until lazy loading is implemented. It still makes the pages load slowly but there is no memory exhausted error and no javascript error. It works with the token-7.1.0 with entity reference and automatic nodetitle. If you happen to have the exact same issue as I had.

Lazy loading? Is there an issue for this that I could check out?

Patch from #35 attached. I've upped the depth from 2 to 3.

This problem is solved by the token_tweaks module: https://drupal.org/project/token_tweaks

I've tested the patch from #85 and it did not work until I reduced the recursion depth to 2. Here's a new patch, vastly superior to the old patch. :-)

Also, token_tweaks did not solve my problem because it sets recursion depth to 4, which evidently is too high (at least on my system).

@koppie: The Token Tweaks module should be seen as temporary. I prefer doing a registry_alter than adding another module to the mix. Thanks for working on a patch.

#86 worked for me.

#88 worked for me. Thanks!

If it works, can we get an RTBC? Doesn't seem right to do it myself (since I submitted the patch).

#86 is not something to be committed, it is something you add in custom code on your own site.

#88 is not a viable solution because you are limiting all users to 2 levels when not all users have this problem. IMO it cannot be committed.

Limiting to 2 levels can in some cases make token UI completely useless so is not really a great solution for all.

If you want to solve this problem by limiting the number of levels that get printed, use the https://drupal.org/project/token_tweaks> token tweaks module.

Note that it has a settings page, where you have to go and configure the level you want to restrict to.
So there you can configure it to stop at 2 levels.

#88 Limiting to only two levels will not be something that we all would like to have. I suggest we move it back to needs work.

We are repeating ourselves.
Let's close this issue as a "Won't fix / Works as designed".

The answer is:
- If you don't want to modify the existing code, use Token tweaks module. (It is not hinted anywhere, but you can set the depth on admin/config/system/tokens. See #1316152: Better info on modules page., which adds this info to the modules page.)
- If you don't want to install another module, use the patch from #85, and test which depth doesn't break your site anymore.

I'll update the summary.

Update summary with latest status.

Added error message.