We dropped converting field names with underscores to dashes in the official field token issue as it was too much trouble when using the direct field name works just fine. This has a problem that all field tokens are duplicated by Entity tokens. This contributes to the token overload that is causing memory problems.

Files: 
CommentFileSizeAuthor
#29 token-situation.gif515.87 KBMXT

Comments

Why are you using underscores? The token system makes use of dashes for separation, so should the field-tokens.

>And now everyone will have broken tokens if the names switch.
So why are you going with different token names then, if you know entity-tokens already does it with dashes for more than a year? I don't see any cause why it should be switched now.

Underscores are perfectly valid names and as part of the field tokens issue it was decided that it is not worth converting variable token names with underscores from this_format to this-format, especially when underscores may be used with other things like custom date formats (#1173706: Date tokens do not handle all date types e.g. [current-date:my_date_format]).

I never stated an official position or standard on field token names until we officially committed Field tokens to Token.module. Entity tokens went ahead and did its own implementation and now I have to recommend that no one use Commerce or any Rules-dependent module.

Uh... subscribe.

Subscribe.

Priority:Normal» Major

Bumping to major since #1272570: Drop dependency on entity_tokens.module was won't fixed.

sub...

To reiterate: I'm proposing changing the entity_tokens_token_info_alter() to list the tokens under the underscore, but have the token *replacement* support both underscores and dashes.

+1

i'm just seeing that entity tokens has been using [node:field-name] for a quite some time.
token takes the [node:field_name] approach since http://drupal.org/node/691078?page=1#comment-4878092

it is not quite clear to me when exactly the move from dashes-only to underscores as in field names was made. some reference points:
http://drupal.org/node/691078#comment-2577624
http://drupal.org/node/1222592

well, in the end it might just be about standardizing on one approach

Thanks dasjo for looking this up. Indeed eaton mentions in #691078-4: Field tokens that the convention is using dashes, though I'm not aware of any documention of that either.

@Dave Reid:
So, in #967136-11: Content type tokens you were arguing that we even have to convert underscores to dashes in type-names:

>Where has the core token API used dashes for data-type names?

There are no core token types that are more than one word. I suspect that if I asked eaton that if we did, it would use dashes, since *every single* token in core uses dashes in-between words (and the unfortunate naming of the 'term' token type would probably be 'taxonomy-term' to at least mostly match the corresponding entity type).

As you said there - all core tokens use dashes and I've followed that convention. And now suddenly, I should convert my dashes to underscores on an arbitrary basis, because token module decides to? Sry, I know you are the maintainer of the core token system as well as of token.module, but still token.module is not core and I'm not keen on changing everything any time token.module comes up with something new.

I guess we need a mediator here...

Btw: As far as I can see Dave Reid provided one valid reason for his policy in #2 yet #11 tends to be a good argument too.

This seems like a serious communication issue. Please resolve this issue in a civil and timely manner (i.e. not forcing a certain solution...). The community will be very grateful!

Title:Entity tokens duplicates field tokens created by token moduleField token conflict between entity_tokens and token module

I apologize to fago and any other entity maintainers - I have not done a very good job of making a rational, detailed argument as to what the problem is, why the decision was made, and solutions on how to move forward. I'll be re-posting a better reasoning later today.

It's been a week... Just wondering if anything has come of this?

Michelle

Thanks Dave. Any update on the reasons?

I think dashes would be preferable, because it's the way it works for other tokens and it stays consistent inside a single token - so users don't have to remember where to use what.

Anyway, we could also add a setting to entity-tokens which allows people to disable entity-token provided field-tokens. Not sure what would be a good place for adding it? Just "admin/config/system/entity-tokens" maybe.

If we'd have dashes in token.module's field tokens, entity-tokens would give precedence to token.module's field tokens anyway. Still, I guess there might be difference in implementations of nested field-tokens though, any suggestions how we should handle that?
Just merge possible tokens? Or maybe we can agree on a setting that allows users to choose which module should provide field tokens at all? Thoughts?

-1 to a preference between the two. I could see a situation where I site builder needs two modules, but each module depends on a different field token provider. It would make the modules incompatible for potentially no other reason.

Personally, I think token module should be the one providing field tokens in all cases. It makes sense: token module provides tokens for core things that core doesn't define tokens for, and other modules provide their own tokens for non-core functionality.

Ideally Token module would provide most of the tokens, but at the moment most of the functionality needed for field tokens is missing from Token and the issues to get that fixed aren't moving. In the meantime, Entity Tokens *is* working pretty well, so is the only functional solution.

I agree with fago that it seems wrong to change the pattern this late in the game and I don't know why it was changed. It also breaks all tokens being used now, or that have been in use since before the change was made. I'm going to assume Dave had a good reason for doing it, even though we don't know what it is. Anyway, the only place work is going on is here and the only thing works is this, so I suggest that fago just go ahead and do whatever will make Entity Tokens work without duplicating Token tokens so we can use the module. If we wait for Token to make changes, I don't know how long that might be. Token is all tied up on questions about architectural concerns related to integrating it into D8 later, and history tells us that issues like that can linger for a very very long time.

This (or at least I think it is this) issue is causing a lot of problems for me. I need the Entity Tokens module for Rules.

Whenever I enable Entity Tokens and try to either edit or add a new content type the system goes into a long process (loop?) that can take a minute or more. It appears to be building some kind of token list as the tokens are visible while it is processing.

Whenever I make a backup (using Backup and Migrate), if I first clear cashe and run cron, the resulting file is ~400 KB. If I fail to clear cashe it will produce a file that is about 1.5 MB. I suspect this is related but cannot be certain.

As of yesterday I can no longer edit or create a new content type without getting an “out of memory error”. My php memory is limited by the hosting company to 160 MB.

I can manage fields and manage display in content types (with Entity Tokens enabled), so at the moment, when I need to add or edit a content type, my only work around it to disable Rules and Entity Tokens, make the changes and then re enable the modules.

Thanks for all your great work and if there is anything I can do to help isolate or resolve this please let me know.

Sorry I forgot to add that the problem is the same on 7.x-1.0-rc1 and 7.x-1.x-dev (Jan 16, 2012)

@tgriswold, none of that is relevant to this issue. This is an issue about the pattern to use for the tokens.

So Dave promised in #13 to provide further insight into his reasoning. So far nothing seems to have happened on this front or was there some discussion behind the scenes? Atm everything seems blocked and nobody is pushing for a solution. :(

The upgrade to Entity release 7.x-1.0-rc2 apparently breaks using entity-tokens.
FYI, I did run update.php.

A release process should NEVER break things (big complaint)!!!
There's not even an upgrade path available.
@fago, please help us out this misery!

Issues for dependent modules:
- auto_nodetitle: "dashed" entity-token replacements don't work anymore.
For empty fields, the tokens got replaced with the token string "[.. : ..]".
Switching to regular "underscored" tokens solves this issue.
- rules: a rule containg a form action "load form element" is now broken: "Error: Unkown action load_element".
- Also in the Rules issues queue, some people reported similar problems with the Entity RC2 codebase (reported about DEV = RC2).

I don't oversee all reported issues (in dependent modules), but this seems critical to me.

It seems that when Entity Tokens have been activated, the 'normal' tokens do not work any more even when you deactivate and deinstall Entity Tokens.

I have tried to use the [node:group_content_access] when I had both that one and [node:group-content-access].

Please guys resolve this!

In the "Replacement patterns" area I have tons of duplicated token fields and also the browser often crashes trying to load all the tokens tree!

Thank you very much for resolving this

Title:Field token conflict between entity_tokens and token moduleEntity tokens duplicates field tokens created by token module
Priority:Major» Critical

per #22 and #23

Was there any move by either fago or Dave Reid to solve this problem? I cannot believe this poison is still lingering in the veins of loads of drupal sites.

Priority:Critical» Major

How is this critical?

Priority:Major» Critical

Well, #24 reports it crashing his browser. Not sure why his comment was unpublished, though.

In addition, figuring this out sooner than later will reduce the amount of headache needed for a merge (the only thing that waiting will do is allow people to build more sites on top of this functionality).

@cweagons
I believe the token list is crashing his browser because it isn't retrieved with ajax, its loaded all-at-once.
Depending on the size of your site and the number of content types or entity definitions that expand token sets, this list could theoretically contain hundreds, if not thousands or more options.

StatusFileSize
new515.87 KB

My situation in the image in attachment.

The image is only one of the "Replacement patterns" section in Pathauto patterns settings page, so, to try to understand the CRITICAL situation, you have to:

  1. multiply a similar tree for all the "Replacement patterns" sections presents in that same page (I have: "content paths", "taxonomy term paths", "user paths", "forum paths")
  2. in the image I have expanded only one field tree "step_page" > "nodes with delta 1": try to think that ALL THE OTHER fields have a similar situation under them...
  3. I'm only at 60% in developing the architecture of my site: other fields/relations will be added... Will my browser hold the load?

I hope you get the idea of my frustration.

Thank you for resolving this

I believe the token list is crashing his browser because it isn't retrieved with ajax, its loaded all-at-once.

Uh, I have no idea what you mean, but in any case, browser crashing == bad.

The image is only one of the "Replacement patterns" section in Pathauto patterns settings page, so, to try to understand the CRITICAL situation, you have to:

To be clear, this is not critical because you have a long list of crap to scroll through. This is critical because it crashes browsers. They are related, but the browser crashing is what would make this critical.

I'd also like to note that I've gotten a few tab crashes lately in the Chrome Beta channel on Mac because of huge token lists.

This issue makes me sad. A breakdown in contrib cooperation and community communication, unsolved for 9 months and counting.
If there's anything I can do to help, please let me know.

Per #13 we're waiting on a comment from Dave Reid to explain the rationale to use underscores vs dashes. Once that is done I think we'd all appreciate it if Dave and Fago could reach an agreement on which should be the "official" separation character. I would then request that all module maintainers adjust their modules to support the new standard, leaving the other one as a deprecated option that (see Dave's suggestion in #8).

I pinged Dave Reid on twitter and got the impression that he isn't interested in discussing this further.

I'll add an explanation for the rational and my thoughts sometime in the next 48 hours.

@DaveReid: thanks :)

#28 I believe the token list is crashing his browser because it isn't retrieved with ajax, its loaded all-at-once.

Not sure it's been mentioned in this issue: there's the token tweaks module which may be able to help with this problem (-> limit the token tree depth).

#23 It seems that when Entity Tokens have been activated, the 'normal' tokens do not work any more even when you deactivate and deinstall Entity Tokens.

This may be because as of #909728: fix property-rename problems, which started using underscores for caching (property) token values; my guess is that this "overrides" values provided by the token module; If that's the case, disabling entity_token and clearing the cache should help (I think). Either way this probably needs a separate issue...

disabling entity_token and clearing the cache should help (I think).

I tried to disable entity_token: it was a nightmare!

To disable it, you first have to disable all dipendents modules:

  1. rules,
  2. commerce,
  3. commerce_ui,
  4. rules_scheduler,
  5. commerce_customer,
  6. commerce_price,
  7. commerce_line_item,
  8. commerce_order,
  9. commerce_checkout,
  10. commerce_product,
  11. commerce_product_reference,
  12. rules_admin,
  13. commerce_product_pricing,
  14. commerce_cart,
  15. commerce_customer_ui,
  16. commerce_line_item_ui,
  17. commerce_order_ui,
  18. commerce_payment,
  19. commerce_payment_ui,
  20. commerce_product_pricing_ui,
  21. commerce_product_ui,
  22. commerce_tax,
  23. commerce_tax_ui

Ok I did it, then I disable entity_tokens, I cleared the cache, and tried to re-enable all modules: the nightmare started!:

I encountered the following error:

WD php: FieldException: Attempt to create an instance of field commerce_customer_address on bundle billing that already has an instance of that field. in field_create_instance() (line 490 of /var/www/my-site/modules/field/field.crud.inc).
Cannot modify header information - headers already sent by (output started at /var/www/drush/includes/output.inc:37) bootstrap.inc:1239 FieldException: Attempt to create an instance of field commerce_customer_address on bundle billing that already has an instance of that field. in field_create_instance() (line 490 of /var/www/my-site/modules/field/field.crud.inc).
Drush command terminated abnormally due to an unrecoverable error.

I think it was due to: http://drupal.org/node/858722

Final result: my site crashed.

Fortunately I had my database backup, but now I'm at the initial point.

You can't just disable and re-enable entity tokens.
As you can see, it requires you to disable pretty much every other module you have installed, and they can't just be re-enabled (disabled modules are broken in D7 so a module suite like Commerce can barely be uninstalled, let alone disabled and re-enabled, that's a fantasy).

My advice is to forget that the "disable" option even exists, unless it's a small module (doesn't provide entity types, fields) or you're planning to uninstall right after.

While it's true that while field tokens was being originally developed for Token module, I had started off by converting field machine names (which by default contain underscores) to use dashes because it seemed nice for consistency with the rest of the explicitly-defined tokens provided by core (non of them are defined by user input). The field token patch that landed in Token module ultimately did not convert underscores to dashes for the following reasons:

  1. As I started working with more machine names (as was the case with #1173706: Date tokens do not handle all date types), I started to question if I would have to do this conversion *every single time* we encountered something with underscores was actually something that is sustainable or reasonable. As we add more and more support for tokens based on user-entered input, we have to do more and more conversion...
  2. ...which requires additional code and complexity to peform the conversions both in hook_token_info() as well as in the actual token implementation hooks.
  3. As nice as it was for consistency, I haven't had any reports that using a mix of underscores and dashes in the token names has been a problem for end users (aside from #1343228: Confusion between field names with dashed and underscores which is confusion about the token conflict itself). I don't think it's really that big of a deal.

This said, even if Token and Entity tokens are fixed to use the same field token namespace, this really doesn't solve the root problem at hand: that Entity tokens floods the amount of available tokens to the point where the Token UI was unprepared to handle it. And it was ignored and shoved off (like Field tokens itself not being available in Drupal 7 core) to again be my problem to fix. I have even debated using hook_system_info_alter() in token module to un-declare Entity tokens as a dependency for Rules so that it can be disabled.

While I understand that Entity tokens had to do what it needed to do, I have had severe reservations about the project and its lack of cooperation with the Token module by pushing code and doing its own thing without concern for other projects. Should I expect anyone adding token integration to their modules to warn or ask me to review it first? Hell no, but now this is why I monitor every single issue on Drupal.org that mentions the word 'token' so that I can try and prevent any other major problems. Would it be courteous and responsible for projects that do major efforts with tokens to coordinate and cooperate with the Token system maintainers? Yes. I would do the same if I was in on either side of the situation, but that's just me.

Side note that this is a little ironic since Entity API at one point converted all property names using underscores to use dashes, but starting with 7.x-1.0-beta1 and #909728: fix property-rename problems it actually changed it to not modify machine names, except for the token integration.

I should also note that this is not the first time that I've encountered issues with the Entity API module and its token integration:

It's not only been the token integration, but also that Entity API allowed exportable entities to have strings/machine names as entity IDs which is went against assumptions that entity IDs are numeric in core and contrib:

I'm stating this because it's part of the reason why I have such hurt feelings about this and why I'm not willing to budge. I feel strongly that this module has made broad changes which have had serious repercussions for other modules that interact with entities and for Drupal core. Some of those changes have been for good (and are making their way into core - yay!). But all of those types of changes need community consideration and review. If this module was more of an experiment on how to improve entities and wasn't a real-world dependency for major modules like Rules, Commerce, and VBO I would not thing this is as serious.

I chatted with @davereid about this on IRC tonight. It seems that the biggest blocker to solving this problem and improving the site builder UX for token.module on sites with lots of tokens is a patch for #1334456: Enable ajax on-demand loading of sub-trees in the Token help.

I'm not putting my hand up to write that patch, I have enough work on my plate as it is. I will commit to trying to help get this issue resolved and moving forward.

For what it's worth, my recommendation is:

  • If it isn't already, make Token a dependency of Entity Token, and keep the minimum required version current (hook_requirements) to encourage/push developers to use the correct tokens,
  • Deprecate the tokens provided by the Entity Token module in favor of those provided by Token,
  • Ultimately deprecate most if not all of Entity Token.

I think I've just run into this issue today while integrating Fill PDF with Rules :( - Webform Rules's chained tokens aren't chaining in the data selector. They used to.

The problem here is that Rules is so complex that it isn't trivial to jump into Xdebug and try to figure something out. And so then I get tempted to put PHP in text boxes - double :( - but I'll just make do somehow with what I've got.

Looking forward to positive resolution on this.

It's not only been the token integration, but also that Entity API allowed exportable entities to have strings/machine names as entity IDs which is went against assumptions that entity IDs are numeric in core and contrib:

I've no idea how this play into this issue. However, for completeness: Entity IDs need not be numeric, it's clearly stated in the docs that this is only a requirement for fieldable entities and I consider this flexibility as an important feature of the entity api. Having you and others consistently pretending something else, does not help to reduce confusion and so bugs (because people wrongly rely on numeric ids). So please, stop.

This said, even if Token and Entity tokens are fixed to use the same field token namespace, this really doesn't solve the root problem at hand: that Entity tokens floods the amount of available tokens to the point where the Token UI was unprepared to handle it.

If the problem with entity tokens is that it provides tokens, it is guilty. I'm afraid, but it's clearly no solution to don't provide tokens because token UI cannot deal with that. That's something that needs to be fixed there, yes.

Still, it would help if we would not have duplicated entries for each field - yep. But not entity token has started introducing that duplication...

And it was ignored and shoved off (like Field tokens itself not being available in Drupal 7 core) to again be my problem to fix.

I've never put the burden of doing token UI on you, but hey - you took over writing that (thanks!). In contrast entity tokens could have taken over field tokens if you'd have wanted it so.

Would it be courteous and responsible for projects that do major efforts with tokens to coordinate and cooperate with the Token system maintainers? Yes. I would do the same if I was in on either side of the situation, but that's just me.

Indeed, I should have done that. Though, when I've wrote that (2010 Mar), this was a quite experimental, the whole path forward for d7 field tokens was not clear at all (at least for me), and you weren't even appointed as token system maintainer. As it was just a minor thing for my area of work I've not started reaching out, no. I was prepared to accommodate with whatever core evolves to, but obviousuly that did not work out for the token module, which started taking over implementing core stuff when core was already stable.

Still, the implementation tries to be friendly in that it *only* adds tokens if there are none yet, what remains true until today. For that to work out, entity tokens and the whole entity properties metadata API tries to mimic the structure of core tokens. Again, that worked out well with core, but not with differently-structured (dash vs underscore) tokens introduced by token module for core once core was already stable. At that point, having token.module ignore what entity tokens already did has not helped the situation either.

Still, if we manage to fix the dash-problem, enity tokens gives token.module precedence due to the way entity tokens is built.

However, once token module tokens take precedence, I fear there might be some side-effects with people using nested-(entity)tokens that are not available in the token module variants. To avoid this we'd need some additional functionality in entity-tokens to handle that case as well. I'd be happy to work on that, although it could take some time as I'm busy with other things.

@dashes-underscores:
Honestly, I don't care much and I'm tired of that issue. So if it helps the situation we can change entity tokens to explicitly use underscores for field tokens, while supporting *both* during token replacements for compability.

For an upcoming project, I'd really like to use 1.0 of Entity API. How can we resolve this issue? What can I do to help?

Right now I'm looking at converting a reasonably large site to Drupal. But I need this bug fixed before I can continue. How can we get people to agree on a solution and get this problem resolved? It's been going on way too long already.

on a sligthly bigger site i am now working on, the form with tokens is really a big pain :)
jQuery('td.token-key').length = 15981

node/X/devel/token runs out of memory ...

Is there any way we can help to solve this ? Is the way @DamienMcKenna suggested viable ?

thanks :)

Try http://drupal.org/project/token_tweaks - this can help stop Token getting carried away and makes the tree less memory hungry.

Also FYI Token module v7.x-1.2 and newer have had an optional popup for displaying the tokens that completely removes the need to do the token loading during the regular page load, modules just need a little tweak to support it.

Just a comment: I had problem with Entityforms tokens, I guess this issue address that problem. I used Entityform tokens in Rules to create user entity and it would generate error. could that be the case of duplicate tokens?

What options do we have as site builders and developers moving forward? Module dependencies are putting us in a tough spot.

I have certain custom entities that have administrative fields that I don't want exposed as tokens, yet I'm already using Rules, which depends on Entity API. Are my choices simply:

1. Tell the client not to use and just ignore all the duplicates tokens and tokens that don't make sense?
OR 2. Uninstall Rules and manually write in my own hook implementations to replace the functionality Rules was providing so I can disable Entity Tokens?

Or, and I know this is not the proper forum, but could the dependency be removed from Rules?

I'm all for helping to resolve the issue. I won't be in Sydney but hopefully I can make it to Portland. If we locked a few of us in a room at DrupalCon for a few hours could we hash/hack out a solution?

Reread some comments above and came back on this: #1272570: Drop dependency on entity_tokens.module So the dependency won't be broken.

Unfortunately I've not found time to work on this yet as d8 entity stuff is keeping me busy. Anyway, we have to solve it in a BC-compatible way so there for it to block a long overdue 1.0 release.

@DamienMcKenna could you elaborate a bit on what's that little tweak you mentioned in #49? Thanks

@esclapes, per #1684984-8: Support opening token browser in a jQuery UI dialog on demand, you can use a '#dialog' => TRUE in your theme('token_tree') call to display the Token browser in a modal.

Token maintainer has explained how this code need to work aka how tokens need to be generated and Entity API maintainers need to follow their decision.

Is there any chance that this issue get finally solved now?

In rules, for action using send mail with variable, if I used [entityform:field_email] as a value in To, errors are generated and email is bounced back. I have to use [entityform:field-email] .

However, I try to display [entityform:field_email] vs [entityform:field-email] in email body, no differences between them, exactly as what user input. I was puzzled about why this [entityform:field_email] couldn't be used for To in email.

After that, I switch to use those tokens with dash instead of underscore though I still don't know when and how they'll behave differently in different context. Thanks

Token and Entity Token are generating different results on some field types

Entity Token
Company Tel No [entityform:field-tel-no:?]
Field "field_tel_no". The following properties may be appended to the token: number (The phone number.), country_codes (The country code for a given phone number.), extension (The extension for a given phone number.)

Token
Company Tel N [entityform:field_tel_no]
Phone number field. Also known as Personal Tel No.

For File Type module, both also fail to display

Entity Token
Reference Documents [entityform:field-file:?]
Field "field_file". The following properties may be appended to the token: file (The file.), description (The file description), display (Whether the file is being displayed.)
I tried to use [entityform:field-file:file] or [entityform:field-file:description], htting error
Fatal error: Call to a member function value() on a non-object in public_html/sites/all/modules/entity/entity_token.tokens.inc on line 297

Token
Reference Documents [entityform:field_file] File field.

@jetwong98: for the fatal error, try applying the patch at #1440928-44: Entity tokens for multi-value fields produce a fatal error