We dropped converting field names with underscores to dashes in the official field token issue as it was too much trouble when using the direct field name works just fine. This has a problem that all field tokens are duplicated by Entity tokens. This contributes to the token overload that is causing memory problems.

CommentFileSizeAuthor
#29 token-situation.gif515.87 KBMXT
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

fago’s picture

Why are you using underscores? The token system makes use of dashes for separation, so should the field-tokens.

>And now everyone will have broken tokens if the names switch.
So why are you going with different token names then, if you know entity-tokens already does it with dashes for more than a year? I don't see any cause why it should be switched now.

Dave Reid’s picture

Underscores are perfectly valid names and as part of the field tokens issue it was decided that it is not worth converting variable token names with underscores from this_format to this-format, especially when underscores may be used with other things like custom date formats (#1173706: Date tokens do not handle all date types e.g. [current-date:my_date_format]).

I never stated an official position or standard on field token names until we officially committed Field tokens to Token.module. Entity tokens went ahead and did its own implementation and now I have to recommend that no one use Commerce or any Rules-dependent module.

geerlingguy’s picture

Uh... subscribe.

cweagans’s picture

Subscribe.

Dave Reid’s picture

Priority: Normal » Major

Bumping to major since #1272570: Drop dependency on entity_tokens.module was won't fixed.

wjaspers’s picture

sub...

jherencia’s picture

Dave Reid’s picture

To reiterate: I'm proposing changing the entity_tokens_token_info_alter() to list the tokens under the underscore, but have the token *replacement* support both underscores and dashes.

crispinbailey’s picture

+1

dasjo’s picture

i'm just seeing that entity tokens has been using [node:field-name] for a quite some time.
token takes the [node:field_name] approach since http://drupal.org/node/691078?page=1#comment-4878092

it is not quite clear to me when exactly the move from dashes-only to underscores as in field names was made. some reference points:
http://drupal.org/node/691078#comment-2577624
http://drupal.org/node/1222592

well, in the end it might just be about standardizing on one approach

fago’s picture

Thanks dasjo for looking this up. Indeed eaton mentions in #691078-4: Field tokens that the convention is using dashes, though I'm not aware of any documention of that either.

@Dave Reid:
So, in #967136-11: Content type tokens you were arguing that we even have to convert underscores to dashes in type-names:

>Where has the core token API used dashes for data-type names?

There are no core token types that are more than one word. I suspect that if I asked eaton that if we did, it would use dashes, since *every single* token in core uses dashes in-between words (and the unfortunate naming of the 'term' token type would probably be 'taxonomy-term' to at least mostly match the corresponding entity type).

As you said there - all core tokens use dashes and I've followed that convention. And now suddenly, I should convert my dashes to underscores on an arbitrary basis, because token module decides to? Sry, I know you are the maintainer of the core token system as well as of token.module, but still token.module is not core and I'm not keen on changing everything any time token.module comes up with something new.

dddave’s picture

I guess we need a mediator here...

Btw: As far as I can see Dave Reid provided one valid reason for his policy in #2 yet #11 tends to be a good argument too.

This seems like a serious communication issue. Please resolve this issue in a civil and timely manner (i.e. not forcing a certain solution...). The community will be very grateful!

Dave Reid’s picture

Title: Entity tokens duplicates field tokens created by token module » Field token conflict between entity_tokens and token module

I apologize to fago and any other entity maintainers - I have not done a very good job of making a rational, detailed argument as to what the problem is, why the decision was made, and solutions on how to move forward. I'll be re-posting a better reasoning later today.

Michelle’s picture

It's been a week... Just wondering if anything has come of this?

Michelle

fago’s picture

Thanks Dave. Any update on the reasons?

I think dashes would be preferable, because it's the way it works for other tokens and it stays consistent inside a single token - so users don't have to remember where to use what.

Anyway, we could also add a setting to entity-tokens which allows people to disable entity-token provided field-tokens. Not sure what would be a good place for adding it? Just "admin/config/system/entity-tokens" maybe.

If we'd have dashes in token.module's field tokens, entity-tokens would give precedence to token.module's field tokens anyway. Still, I guess there might be difference in implementations of nested field-tokens though, any suggestions how we should handle that?
Just merge possible tokens? Or maybe we can agree on a setting that allows users to choose which module should provide field tokens at all? Thoughts?

cweagans’s picture

-1 to a preference between the two. I could see a situation where I site builder needs two modules, but each module depends on a different field token provider. It would make the modules incompatible for potentially no other reason.

Personally, I think token module should be the one providing field tokens in all cases. It makes sense: token module provides tokens for core things that core doesn't define tokens for, and other modules provide their own tokens for non-core functionality.

KarenS’s picture

Ideally Token module would provide most of the tokens, but at the moment most of the functionality needed for field tokens is missing from Token and the issues to get that fixed aren't moving. In the meantime, Entity Tokens *is* working pretty well, so is the only functional solution.

I agree with fago that it seems wrong to change the pattern this late in the game and I don't know why it was changed. It also breaks all tokens being used now, or that have been in use since before the change was made. I'm going to assume Dave had a good reason for doing it, even though we don't know what it is. Anyway, the only place work is going on is here and the only thing works is this, so I suggest that fago just go ahead and do whatever will make Entity Tokens work without duplicating Token tokens so we can use the module. If we wait for Token to make changes, I don't know how long that might be. Token is all tied up on questions about architectural concerns related to integrating it into D8 later, and history tells us that issues like that can linger for a very very long time.

tgriswold’s picture

This (or at least I think it is this) issue is causing a lot of problems for me. I need the Entity Tokens module for Rules.

Whenever I enable Entity Tokens and try to either edit or add a new content type the system goes into a long process (loop?) that can take a minute or more. It appears to be building some kind of token list as the tokens are visible while it is processing.

Whenever I make a backup (using Backup and Migrate), if I first clear cashe and run cron, the resulting file is ~400 KB. If I fail to clear cashe it will produce a file that is about 1.5 MB. I suspect this is related but cannot be certain.

As of yesterday I can no longer edit or create a new content type without getting an “out of memory error”. My php memory is limited by the hosting company to 160 MB.

I can manage fields and manage display in content types (with Entity Tokens enabled), so at the moment, when I need to add or edit a content type, my only work around it to disable Rules and Entity Tokens, make the changes and then re enable the modules.

Thanks for all your great work and if there is anything I can do to help isolate or resolve this please let me know.

tgriswold’s picture

Sorry I forgot to add that the problem is the same on 7.x-1.0-rc1 and 7.x-1.x-dev (Jan 16, 2012)

KarenS’s picture

@tgriswold, none of that is relevant to this issue. This is an issue about the pattern to use for the tokens.

dddave’s picture

So Dave promised in #13 to provide further insight into his reasoning. So far nothing seems to have happened on this front or was there some discussion behind the scenes? Atm everything seems blocked and nobody is pushing for a solution. :(

bobslee’s picture

The upgrade to Entity release 7.x-1.0-rc2 apparently breaks using entity-tokens.
FYI, I did run update.php.

A release process should NEVER break things (big complaint)!!!
There's not even an upgrade path available.
@fago, please help us out this misery!

Issues for dependent modules:
- auto_nodetitle: "dashed" entity-token replacements don't work anymore.
For empty fields, the tokens got replaced with the token string "[.. : ..]".
Switching to regular "underscored" tokens solves this issue.
- rules: a rule containg a form action "load form element" is now broken: "Error: Unkown action load_element".
- Also in the Rules issues queue, some people reported similar problems with the Entity RC2 codebase (reported about DEV = RC2).

I don't oversee all reported issues (in dependent modules), but this seems critical to me.

edvanleeuwen’s picture

It seems that when Entity Tokens have been activated, the 'normal' tokens do not work any more even when you deactivate and deinstall Entity Tokens.

I have tried to use the [node:group_content_access] when I had both that one and [node:group-content-access].

MXT’s picture

Please guys resolve this!

In the "Replacement patterns" area I have tons of duplicated token fields and also the browser often crashes trying to load all the tokens tree!

Thank you very much for resolving this

dddave’s picture

Title: Field token conflict between entity_tokens and token module » Entity tokens duplicates field tokens created by token module
Priority: Major » Critical

per #22 and #23

Was there any move by either fago or Dave Reid to solve this problem? I cannot believe this poison is still lingering in the veins of loads of drupal sites.

tim.plunkett’s picture

Priority: Critical » Major

How is this critical?

cweagans’s picture

Priority: Major » Critical

Well, #24 reports it crashing his browser. Not sure why his comment was unpublished, though.

In addition, figuring this out sooner than later will reduce the amount of headache needed for a merge (the only thing that waiting will do is allow people to build more sites on top of this functionality).

wjaspers’s picture

@cweagons
I believe the token list is crashing his browser because it isn't retrieved with ajax, its loaded all-at-once.
Depending on the size of your site and the number of content types or entity definitions that expand token sets, this list could theoretically contain hundreds, if not thousands or more options.

MXT’s picture

FileSize
515.87 KB

My situation in the image in attachment.

The image is only one of the "Replacement patterns" section in Pathauto patterns settings page, so, to try to understand the CRITICAL situation, you have to:

  1. multiply a similar tree for all the "Replacement patterns" sections presents in that same page (I have: "content paths", "taxonomy term paths", "user paths", "forum paths")
  2. in the image I have expanded only one field tree "step_page" > "nodes with delta 1": try to think that ALL THE OTHER fields have a similar situation under them...
  3. I'm only at 60% in developing the architecture of my site: other fields/relations will be added... Will my browser hold the load?

I hope you get the idea of my frustration.

Thank you for resolving this

cweagans’s picture

I believe the token list is crashing his browser because it isn't retrieved with ajax, its loaded all-at-once.

Uh, I have no idea what you mean, but in any case, browser crashing == bad.

The image is only one of the "Replacement patterns" section in Pathauto patterns settings page, so, to try to understand the CRITICAL situation, you have to:

To be clear, this is not critical because you have a long list of crap to scroll through. This is critical because it crashes browsers. They are related, but the browser crashing is what would make this critical.

I'd also like to note that I've gotten a few tab crashes lately in the Chrome Beta channel on Mac because of huge token lists.

bojanz’s picture

This issue makes me sad. A breakdown in contrib cooperation and community communication, unsolved for 9 months and counting.
If there's anything I can do to help, please let me know.

DamienMcKenna’s picture

Per #13 we're waiting on a comment from Dave Reid to explain the rationale to use underscores vs dashes. Once that is done I think we'd all appreciate it if Dave and Fago could reach an agreement on which should be the "official" separation character. I would then request that all module maintainers adjust their modules to support the new standard, leaving the other one as a deprecated option that (see Dave's suggestion in #8).

bojanz’s picture

I pinged Dave Reid on twitter and got the impression that he isn't interested in discussing this further.

Dave Reid’s picture

I'll add an explanation for the rational and my thoughts sometime in the next 48 hours.

DamienMcKenna’s picture

@DaveReid: thanks :)

bforchhammer’s picture

#28 I believe the token list is crashing his browser because it isn't retrieved with ajax, its loaded all-at-once.

Not sure it's been mentioned in this issue: there's the token tweaks module which may be able to help with this problem (-> limit the token tree depth).

#23 It seems that when Entity Tokens have been activated, the 'normal' tokens do not work any more even when you deactivate and deinstall Entity Tokens.

This may be because as of #909728: fix property-rename problems, which started using underscores for caching (property) token values; my guess is that this "overrides" values provided by the token module; If that's the case, disabling entity_token and clearing the cache should help (I think). Either way this probably needs a separate issue...

MXT’s picture

disabling entity_token and clearing the cache should help (I think).

I tried to disable entity_token: it was a nightmare!

To disable it, you first have to disable all dipendents modules:

  1. rules,
  2. commerce,
  3. commerce_ui,
  4. rules_scheduler,
  5. commerce_customer,
  6. commerce_price,
  7. commerce_line_item,
  8. commerce_order,
  9. commerce_checkout,
  10. commerce_product,
  11. commerce_product_reference,
  12. rules_admin,
  13. commerce_product_pricing,
  14. commerce_cart,
  15. commerce_customer_ui,
  16. commerce_line_item_ui,
  17. commerce_order_ui,
  18. commerce_payment,
  19. commerce_payment_ui,
  20. commerce_product_pricing_ui,
  21. commerce_product_ui,
  22. commerce_tax,
  23. commerce_tax_ui

Ok I did it, then I disable entity_tokens, I cleared the cache, and tried to re-enable all modules: the nightmare started!:

I encountered the following error:

WD php: FieldException: Attempt to create an instance of field commerce_customer_address on bundle billing that already has an instance of that field. in field_create_instance() (line 490 of /var/www/my-site/modules/field/field.crud.inc).

Cannot modify header information - headers already sent by (output started at /var/www/drush/includes/output.inc:37) bootstrap.inc:1239 FieldException: Attempt to create an instance of field commerce_customer_address on bundle billing that already has an instance of that field. in field_create_instance() (line 490 of /var/www/my-site/modules/field/field.crud.inc).

Drush command terminated abnormally due to an unrecoverable error.

I think it was due to: http://drupal.org/node/858722

Final result: my site crashed.

Fortunately I had my database backup, but now I'm at the initial point.

bojanz’s picture

You can't just disable and re-enable entity tokens.
As you can see, it requires you to disable pretty much every other module you have installed, and they can't just be re-enabled (disabled modules are broken in D7 so a module suite like Commerce can barely be uninstalled, let alone disabled and re-enabled, that's a fantasy).

My advice is to forget that the "disable" option even exists, unless it's a small module (doesn't provide entity types, fields) or you're planning to uninstall right after.

Dave Reid’s picture

While it's true that while field tokens was being originally developed for Token module, I had started off by converting field machine names (which by default contain underscores) to use dashes because it seemed nice for consistency with the rest of the explicitly-defined tokens provided by core (non of them are defined by user input). The field token patch that landed in Token module ultimately did not convert underscores to dashes for the following reasons:

  1. As I started working with more machine names (as was the case with #1173706: Date tokens do not handle all date types), I started to question if I would have to do this conversion *every single time* we encountered something with underscores was actually something that is sustainable or reasonable. As we add more and more support for tokens based on user-entered input, we have to do more and more conversion...
  2. ...which requires additional code and complexity to peform the conversions both in hook_token_info() as well as in the actual token implementation hooks.
  3. As nice as it was for consistency, I haven't had any reports that using a mix of underscores and dashes in the token names has been a problem for end users (aside from #1343228: Confusion between field names with dashed and underscores which is confusion about the token conflict itself). I don't think it's really that big of a deal.

This said, even if Token and Entity tokens are fixed to use the same field token namespace, this really doesn't solve the root problem at hand: that Entity tokens floods the amount of available tokens to the point where the Token UI was unprepared to handle it. And it was ignored and shoved off (like Field tokens itself not being available in Drupal 7 core) to again be my problem to fix. I have even debated using hook_system_info_alter() in token module to un-declare Entity tokens as a dependency for Rules so that it can be disabled.

While I understand that Entity tokens had to do what it needed to do, I have had severe reservations about the project and its lack of cooperation with the Token module by pushing code and doing its own thing without concern for other projects. Should I expect anyone adding token integration to their modules to warn or ask me to review it first? Hell no, but now this is why I monitor every single issue on Drupal.org that mentions the word 'token' so that I can try and prevent any other major problems. Would it be courteous and responsible for projects that do major efforts with tokens to coordinate and cooperate with the Token system maintainers? Yes. I would do the same if I was in on either side of the situation, but that's just me.

Side note that this is a little ironic since Entity API at one point converted all property names using underscores to use dashes, but starting with 7.x-1.0-beta1 and #909728: fix property-rename problems it actually changed it to not modify machine names, except for the token integration.

Dave Reid’s picture

I should also note that this is not the first time that I've encountered issues with the Entity API module and its token integration:

It's not only been the token integration, but also that Entity API allowed exportable entities to have strings/machine names as entity IDs which is went against assumptions that entity IDs are numeric in core and contrib:

I'm stating this because it's part of the reason why I have such hurt feelings about this and why I'm not willing to budge. I feel strongly that this module has made broad changes which have had serious repercussions for other modules that interact with entities and for Drupal core. Some of those changes have been for good (and are making their way into core - yay!). But all of those types of changes need community consideration and review. If this module was more of an experiment on how to improve entities and wasn't a real-world dependency for major modules like Rules, Commerce, and VBO I would not thing this is as serious.

skwashd’s picture

I chatted with @davereid about this on IRC tonight. It seems that the biggest blocker to solving this problem and improving the site builder UX for token.module on sites with lots of tokens is a patch for #1334456: Enable ajax on-demand loading of sub-trees in the Token help.

I'm not putting my hand up to write that patch, I have enough work on my plate as it is. I will commit to trying to help get this issue resolved and moving forward.

DamienMcKenna’s picture

For what it's worth, my recommendation is:

  • If it isn't already, make Token a dependency of Entity Token, and keep the minimum required version current (hook_requirements) to encourage/push developers to use the correct tokens,
  • Deprecate the tokens provided by the Entity Token module in favor of those provided by Token,
  • Ultimately deprecate most if not all of Entity Token.
wizonesolutions’s picture

I think I've just run into this issue today while integrating Fill PDF with Rules :( - Webform Rules's chained tokens aren't chaining in the data selector. They used to.

The problem here is that Rules is so complex that it isn't trivial to jump into Xdebug and try to figure something out. And so then I get tempted to put PHP in text boxes - double :( - but I'll just make do somehow with what I've got.

Looking forward to positive resolution on this.

fago’s picture

It's not only been the token integration, but also that Entity API allowed exportable entities to have strings/machine names as entity IDs which is went against assumptions that entity IDs are numeric in core and contrib:

I've no idea how this play into this issue. However, for completeness: Entity IDs need not be numeric, it's clearly stated in the docs that this is only a requirement for fieldable entities and I consider this flexibility as an important feature of the entity api. Having you and others consistently pretending something else, does not help to reduce confusion and so bugs (because people wrongly rely on numeric ids). So please, stop.

This said, even if Token and Entity tokens are fixed to use the same field token namespace, this really doesn't solve the root problem at hand: that Entity tokens floods the amount of available tokens to the point where the Token UI was unprepared to handle it.

If the problem with entity tokens is that it provides tokens, it is guilty. I'm afraid, but it's clearly no solution to don't provide tokens because token UI cannot deal with that. That's something that needs to be fixed there, yes.

Still, it would help if we would not have duplicated entries for each field - yep. But not entity token has started introducing that duplication...

And it was ignored and shoved off (like Field tokens itself not being available in Drupal 7 core) to again be my problem to fix.

I've never put the burden of doing token UI on you, but hey - you took over writing that (thanks!). In contrast entity tokens could have taken over field tokens if you'd have wanted it so.

Would it be courteous and responsible for projects that do major efforts with tokens to coordinate and cooperate with the Token system maintainers? Yes. I would do the same if I was in on either side of the situation, but that's just me.

Indeed, I should have done that. Though, when I've wrote that (2010 Mar), this was a quite experimental, the whole path forward for d7 field tokens was not clear at all (at least for me), and you weren't even appointed as token system maintainer. As it was just a minor thing for my area of work I've not started reaching out, no. I was prepared to accommodate with whatever core evolves to, but obviousuly that did not work out for the token module, which started taking over implementing core stuff when core was already stable.

Still, the implementation tries to be friendly in that it *only* adds tokens if there are none yet, what remains true until today. For that to work out, entity tokens and the whole entity properties metadata API tries to mimic the structure of core tokens. Again, that worked out well with core, but not with differently-structured (dash vs underscore) tokens introduced by token module for core once core was already stable. At that point, having token.module ignore what entity tokens already did has not helped the situation either.

Still, if we manage to fix the dash-problem, enity tokens gives token.module precedence due to the way entity tokens is built.

However, once token module tokens take precedence, I fear there might be some side-effects with people using nested-(entity)tokens that are not available in the token module variants. To avoid this we'd need some additional functionality in entity-tokens to handle that case as well. I'd be happy to work on that, although it could take some time as I'm busy with other things.

@dashes-underscores:
Honestly, I don't care much and I'm tired of that issue. So if it helps the situation we can change entity tokens to explicitly use underscores for field tokens, while supporting *both* during token replacements for compability.

cosmicdreams’s picture

For an upcoming project, I'd really like to use 1.0 of Entity API. How can we resolve this issue? What can I do to help?

jstuckle’s picture

Right now I'm looking at converting a reasonably large site to Drupal. But I need this bug fixed before I can continue. How can we get people to agree on a solution and get this problem resolved? It's been going on way too long already.

mojzis’s picture

on a sligthly bigger site i am now working on, the form with tokens is really a big pain :)
jQuery('td.token-key').length = 15981

node/X/devel/token runs out of memory ...

Is there any way we can help to solve this ? Is the way @DamienMcKenna suggested viable ?

thanks :)

apemantus’s picture

Try http://drupal.org/project/token_tweaks - this can help stop Token getting carried away and makes the tree less memory hungry.

DamienMcKenna’s picture

Also FYI Token module v7.x-1.2 and newer have had an optional popup for displaying the tokens that completely removes the need to do the token loading during the regular page load, modules just need a little tweak to support it.

monsoon’s picture

Just a comment: I had problem with Entityforms tokens, I guess this issue address that problem. I used Entityform tokens in Rules to create user entity and it would generate error. could that be the case of duplicate tokens?

KeyboardCowboy’s picture

What options do we have as site builders and developers moving forward? Module dependencies are putting us in a tough spot.

I have certain custom entities that have administrative fields that I don't want exposed as tokens, yet I'm already using Rules, which depends on Entity API. Are my choices simply:

1. Tell the client not to use and just ignore all the duplicates tokens and tokens that don't make sense?
OR 2. Uninstall Rules and manually write in my own hook implementations to replace the functionality Rules was providing so I can disable Entity Tokens?

Or, and I know this is not the proper forum, but could the dependency be removed from Rules?

I'm all for helping to resolve the issue. I won't be in Sydney but hopefully I can make it to Portland. If we locked a few of us in a room at DrupalCon for a few hours could we hash/hack out a solution?

KeyboardCowboy’s picture

Reread some comments above and came back on this: #1272570: Drop dependency on entity_tokens.module So the dependency won't be broken.

fago’s picture

Unfortunately I've not found time to work on this yet as d8 entity stuff is keeping me busy. Anyway, we have to solve it in a BC-compatible way so there for it to block a long overdue 1.0 release.

esclapes’s picture

@DamienMcKenna could you elaborate a bit on what's that little tweak you mentioned in #49? Thanks

Simon Georges’s picture

@esclapes, per #1684984-8: Support opening token browser in a jQuery UI dialog on demand, you can use a '#dialog' => TRUE in your theme('token_tree') call to display the Token browser in a modal.

hass’s picture

Token maintainer has explained how this code need to work aka how tokens need to be generated and Entity API maintainers need to follow their decision.

Is there any chance that this issue get finally solved now?

jetwodru’s picture

In rules, for action using send mail with variable, if I used [entityform:field_email] as a value in To, errors are generated and email is bounced back. I have to use [entityform:field-email] .

However, I try to display [entityform:field_email] vs [entityform:field-email] in email body, no differences between them, exactly as what user input. I was puzzled about why this [entityform:field_email] couldn't be used for To in email.

After that, I switch to use those tokens with dash instead of underscore though I still don't know when and how they'll behave differently in different context. Thanks

jetwodru’s picture

Token and Entity Token are generating different results on some field types

Entity Token
Company Tel No [entityform:field-tel-no:?]
Field "field_tel_no". The following properties may be appended to the token: number (The phone number.), country_codes (The country code for a given phone number.), extension (The extension for a given phone number.)

Token
Company Tel N [entityform:field_tel_no]
Phone number field. Also known as Personal Tel No.

For File Type module, both also fail to display

Entity Token
Reference Documents [entityform:field-file:?]
Field "field_file". The following properties may be appended to the token: file (The file.), description (The file description), display (Whether the file is being displayed.)
I tried to use [entityform:field-file:file] or [entityform:field-file:description], htting error
Fatal error: Call to a member function value() on a non-object in public_html/sites/all/modules/entity/entity_token.tokens.inc on line 297

Token
Reference Documents [entityform:field_file] File field.

klonos’s picture

@jetwong98: for the fatal error, try applying the patch at #1440928-44: Entity tokens for multi-value fields produce a fatal error

RoSk0’s picture

Issue summary: View changes

Three years past...
Any hope that we will have some decision made on this?
Just as from my point of view entity_tokens has great support of field components,rather then token fields support. Why can we drop field support from token and drop conversion of underscores from entity_token in the same time? This will be just great!

tsvenson’s picture

Is anything decided for Drupal 8 yet in regards to this? Could be a way to find a solution to this issue maybe.

Alternatively, would it be possible to offer a configuration option for this? One where I can decide which to use "-" or "_".

csc4’s picture

Just lost my site to bloated Token tree timing the browser out caused in part by Entity Token duplicating with - and _ can there really not be a fix - surely the suggestion to make it a configuration option so those who need the duplication can keep it while those who don't can't halve their Token trees is a good pragmatic way forward?

Raphael Apard’s picture

A quick dirty fix :

/**
 * Implements hook_token_info_alter().
 */
function YOURMODULE_token_info_alter(&$info) {
  // Remove duplicate tokens
  foreach($info['tokens'] as $entity_type => $tokens) {
    foreach($info['tokens'][$entity_type] as $key => $token) {
      // remove duplicates
      if(strpos($key, "field_") !== FALSE) {
        unset($info['tokens'][$entity_type][$key]);
      }
    }
  }
}
dsnopek’s picture

So, can someone at least comment on which set of tokens is preferred? Like, should site builders choose the '_' ones over the '-' ones when the same tokens are available? Or vice-versa?

giupenni’s picture

News?
What must choose the site builder? '_' or '-' ?

halefx’s picture

Wherever both are available, use the "-"

ExTexan’s picture

@Raphael,

Does your "fix" determine whether both "_" and "-" versions of each token exist (it doesn't appear to), or does it just remove all "_" ones? I don't know enough about the inner workings of tokens to know - is that a valid assumption?

@halefx,

Can you elaborate on why it's better to use "-" tokens? I seems to recall that, in some cases, when I select a "-" token, then look at it after saving the form, it has been changed to the "_" versions by the module in question.

fago’s picture

Can you elaborate on why it's better to use "-" tokens?

- tokens are the ones exposed via entity tokens, thus rely on the values of the Entity property info system / the entity wrappers. The other ones are implemented by token module.

If it helps, I could see us adding variable to entity tokens that adds #63 - so people can configure to remove the other tokens. I'd be fine to have the option to support excluding tokens for fields with entity tokens also. Of course, it's a dangerous option for existing sites, but for new sites it might be quite useful.

joelpittet’s picture

Here's a slightly improved version of @Raphael Apard's code from #63.

It works with things that are not fields and finds only duplicates as asked by @ExTexan in #67 In my case it found 153 duplicates.


/**
 * Implements hook_token_info_alter().
 *
 * @see https://www.drupal.org/node/1272560
 */
function MODULE_token_info_alter(&$info) {
  // Remove duplicate tokens.
  foreach ($info['tokens'] as $entity_type => $tokens) {
    foreach ($info['tokens'][$entity_type] as $key => $token) {
      // Remove duplicates with '_' with duplicates by '-'.
      if (strpos($key, '_') !== FALSE) {
        // Check for duplicate with '-'.
        $duplicate_key = str_replace('_', '-', $key);
        if (isset($info['tokens'][$entity_type][$duplicate_key])) {
          unset($info['tokens'][$entity_type][$key]);
        }
      }
    }
  }
}

I put in a counter and just did a breakpoint to get the count:


function MODULE_token_info_alter(&$info) {
  $duplicates = [];
  foreach ($info['tokens'] as $entity_type => $tokens) {
    foreach ($info['tokens'][$entity_type] as $key => $token) {
      if (strpos($key, '_') !== FALSE) {
        $duplicate_key = str_replace('_', '-', $key);
        if (isset($info['tokens'][$entity_type][$duplicate_key])) {
          $duplicates[] = $key;
          unset($info['tokens'][$entity_type][$key]);
        }
      }
    }
  }
  // BREAKPOINT HERE.
  $duplicates = $duplicates;
}

Also, this won't work if your module is run before entity/token. I wrote an update hook in my MODULE.install file:

/**
 * Implements hook_install().
 */
function MODULE_install() {
  // Give commerce.module a higher weight than token and entity so we can use
  // hook_token_info_alter() to override them.
  $weight = db_select('system', 's')
    ->fields('s', array('weight'))
    ->condition('name', array('entity', 'token'), 'IN')
    ->orderBy('weight', 'DESC')
    ->execute()
    ->fetchField();
  db_update('system')
    ->fields(array('weight' => $weight + 1))
    ->condition('name', 'MODULE', '=')
    ->execute();
}

/**
 * Set weight of the MODULE module to 1.
 */
function MODULE_update_7100() {
  // Give commerce.module a higher weight than token and entity so we can use
  // hook_token_info_alter() to override them.
  $weight = db_select('system', 's')
    ->fields('s', array('weight'))
    ->condition('name', array('entity', 'token'), 'IN')
    ->orderBy('weight', 'DESC')
    ->execute()
    ->fetchField();
  db_update('system')
    ->fields(array('weight' => $weight + 1))
    ->condition('name', 'MODULE', '=')
    ->execute();
}


kristiaanvandeneynde’s picture

I just stumbled upon this issue and found that this leads to a confusing UI for views content panes (comes with ctools). It lists the tokens by label, not token id. This leads to undistinguishable duplicate entries in the dropdown.

I'm also in favor of dropping (i.e. hiding) the Field tokens provided by Token in favor of those (chainable ones) provided by Entity Token.

joelpittet’s picture

My tricks in #69 break imagecache_token... going to patch that module up so metatags work with og images.

joelpittet’s picture

@fago yes, option to choose one or the other would help a bit, though modules like commerce may have hardcoded the tokens into checkout messages and rules.

joelpittet’s picture

#2538658: Entity Token Support for hyphenated field names Here gets that working with entity_tokens

paulwdru’s picture

Hi,

Quite often, the Value returned from a underscored token is DIFFERENT from a hyphenated token.

I applied both underscored and hyphenated tokens in different scenario since different value is returned.

In fact, this is very confusing and makes Drupal insecure with this messy usage of tokens.

DamienMcKenna’s picture

@paulwdru: "insecure"? No. A definite annoyance that on a large site can make the token browser unusable? Definitely :-\

Five years in and Entity API development for D7 has all-but dried up, is this a "won't fix"?