Convert Token API into plugins

Posted an inital patch to get the conversation going.

Fixed a few obvious fatals.

Using a plugin bag to instantiate all known plugins.

Need to handle token_info as well. Fixes TokenTest too.

Hm. I'm not too sure about this. Making it plugins like this seems very arbitrary. And it's very common for contrib to add more tokens so it's not quite clear to me if that would make sense then. We could possibly make the token types plugins, but we still need to define the info/metadata, and more closer to views_data and field definitions (which both *use* plugins but are not themself plugins) than something else. And sadly, I don't think we can still pull that off. We're done to *one* beta blocker and after that is done, we'll have two weeks to deal with beta targets and then it's over for 8.0.

Another problem in 8.x was that token API in 8.x saw _zero_ improvements, nobody was reviewing and RTBC'ing any of the various issues that for example tried to bring it closer to typed data (which would make more sense than to invent yet another data type system).

That said, 90%+ of the tokens are related to entities and fields. In #2326949-37: Move entity-type-specific schema information from the storage class to a schema handler, I had the idea that we could do the same as we already do for views_data and possibly soon for the sql schema there.

Meaning, we'd add a "token" entity handler that entity types can optionally provide. just like views_data, in a first step we can just move it and later on possibly think about an optional default implementation.

While the result will be similar to those first paches (a class per token type with the default tokens), it would feel more logical to me to provide an additional integration point for entities instead of replacing the info hook completely. For discovery, we'd go through all entity types with a token handler, and for generation, we can call that handler first and then the hook.

Thoughts?

Hi Berdir,

Thanks for your detailed feedback.

I have to concede it's probably too late in the cycle to be messing with the token system, However, it think this is a pretty simple change, and bring tokens inline with the rest of D8 architecture.

Making it plugins like this seems very arbitrary. And it's very common for contrib to add more tokens so it's not quite clear to me if that would make sense then

We have hook_tokens_alter and hook_token_info alter still. This allows contrib to add more tokens. Is that not sufficient?

it would feel more logical to me to provide an additional integration point for entities instead of replacing the info hook completely.

I'm not across the work being done on entity handlers, so I can't really comment on that. However, wouldn't it make more sense to have a universal token solution, rather than an entity-specific one?

That sounds more like a 9.0.x issue now?

Maybe it could still be done in a 8.x.0 point release, by adding the ability to declare tokens as plugins in addition to the hook?

Tentatively reviving by forward-merging this patch across 3.5 years :D

However, I rather think that converting node.tokens.inc is premature for this issue. Maybe we can start by adding the plugin API, and leave core hook implementations to be converted in a follow-up?

@cburschka: Cool you picked this up.

Looking into all of this, i feel i strongly agree though with some of the points @Berdir rose in #14:

> ... any of the various issues that for example tried to bring it closer to typed data (which would make more sense than to invent yet another data type system).

Strong +1... see some of the related issues.

> Meaning, we'd add a "token" entity handler that entity types can optionally provide. just like views_data, in a first step we can just move it and later on possibly think about an optional default implementation.

Makes sense to me. Maybe just typed data (as readonly-pseudofields).

So i'd rather unify tokens and typed data than bandaid tokens. And maybe this should mature in contrib first...
(I won't start this right now but feel free to ping me if someone does.)

This is a deprecation, so won't get into 8.8.x or 8.9.x at this late stage.

Re-rolling for 9.0.x

We're not quite ready to remove the hook_token_info implementations yet!

I still don't think that this direction makes much sense, it moves the code into classes, but it doesn't really address any of the underlying issues with the token system.. that you have to manually traverse into nested tokens, re-defining all the things that already are defined as fields, ..

I'd much rather spend time on getting some if the improvements in token.module into core, like automatically generating tokens based on field definitions, as well as finding a way to remove the limit of only supporting a single data item per type. E.g. rules could never use the token system because you for example have the current and original node during an entity save hook.

Or alternatively, instead of injecting typed-data based information into the token system, we could possibly directly use it. ctools has a typed data traversal API that I'm also using in pathauto. See \Drupal\pathauto\Entity\PathautoPattern::getContexts(). We used to have a generic UI where you could add relationships and arbitrary conditions, but it was too complex to use for 90% of the use cases, so it's hidden away ;)

That would have the same issues as the serialization system though, that everything that you want to expose needs to be exposed as typed data, which is often awkward/slow, e.g. the long-standing issue of exposing image style URLs.

OK fair enough. I'll leave this alone.

Are there any issues for automatically generating tokens off field definitions? Or if we are proposing we use typed data directly, we should probably have an issue in the ideas queue?

If your goal is to move that code out of hooks into classes, then maybe the most straight-forward way to do so would be to try and get #2237831: Allow module services to specify hooks moving, based on the ideas from @catch, then you could just implemnt a a service for the token info + tokens hooks and use DI and so on without having to change anything about the token API.

Found these issues about the topics that I mentioned: #1222592: Architecture RFC: Field token architecture, #1920688: Support multiple instances of the same token type in token replacement and #1869582: Leverage the Typed Data API for token replacements

As I get the main question is does it makes sense to make field tokens with typed data or maybe better to propose token module to core?
Anyway it will be great to package remains of this hooks to plugins or tagged services to make faster at least

1. +++ b/core/lib/Drupal/Core/Token/TokenTypeCollection.php
   @@ -0,0 +1,53 @@
   +    foreach ($this->definitions as $plugin_id => $definition) {
   +      if (!isset($this->pluginInstances[$plugin_id])) {
   +        $this->initializePlugin($plugin_id);
   ...
   +    $this->configurations[$instance_id] = $configuration;
   +    parent::initializePlugin($instance_id);
   
   +++ b/core/lib/Drupal/Core/Utility/Token.php
   @@ -374,7 +394,13 @@ public function getInfo() {
   +        foreach ($this->getTokenTypes() as $token_type) {
   +          $this->tokenInfo += $token_type->getTokenInfo();
   
   @@ -406,4 +432,18 @@ public function resetInfo() {
   +      $token_type_bag = new TokenTypeCollection($this->tokenTypeManager);
   +      $this->tokenTypes = $token_type_bag->getAll();
   

   Any reason to load all types all the time?
   I think it should use contexts like https://www.drupal.org/node/1938688

2. +++ b/core/lib/Drupal/Core/Token/TokenTypeInterface.php
   @@ -0,0 +1,123 @@
   +   * @see hook_token_info()
   +   * @see hook_tokens_alter()
   
   +++ b/core/lib/Drupal/Core/Token/TokenTypeManager.php
   @@ -0,0 +1,31 @@
   +    $this->alterInfo('token_info');
   

   token(s)_info_alter could be painful for transition

I fully support @berdir's reasoning in #33/#35. Let's put the effort into #1869582: Leverage the Typed Data API for token replacements instead!

This would be a minor-only change (with a new plugin API and deprecations of the old ones). Since 8.9.x and 9.0.x are now in beta, I'm moving this to 9.1.x. Thanks!

To echo #36 , we need to ensure that the following systems interlink in a way that makes sense:

• Typed Data API: Enables a system for retrieving, storing, and discovering Typed Data in Drupal
• Context: Provides route, request, and build-specific Entities containing Typed Data to rendering systems
  • Plugins should have context provided by default on instantiation, and have that cleanly separated from context mapping and definitions
• Token: Allows run-time replacement of strings with Entity values from arbitrary sources
  • The default Token Plugin Deriver should generate tokens based on Typed Data
  • The default arbitrary source for Token Entity values should be Context

That allows Plugins implementing token replacement to automatically have most of the work handled by the architecture of the system, instead of relying on a developer or end user to assign contexts to token types. It also would greatly reduce and simplify much of the context and token-related code in core.

This involves also touching the Plugin system, in addition to creating a TokenPluginManager and ecosystem.

#40: Fully agree, thanks for elaborating. Is this a +1 for #37?