Token Drupal 8 port

Added the YAML-key 'type' to the file now. Strange, my Git tracks the deletion of the old .info-file, but does NOT record the creation of the new .info.yml-file. Any good hints are welcome!

Anyhow, attached you will find the compressed file itself, and below is the code. Sorry, currently no Git patch.

name: Token
description: Provides a user interface for the Token API and some missing core tokens.
core: 8.x
type: module
files:
  - token.test

... yep ("another user")!
Testing is another labour-intensive area, indeed. And, the existing Token test looks, at first glance, quite extensive already. How much effort, do you think, is it to convert it from D7 to D8?

Firstly, your patch for git will not include files that you don't "add" first. When you get a "diff" you must first add all the files you created to git. It's a bit weird. I messed up a few times.

Now down to business. Here is a patch with a few preliminary changes. I have not tested it yet and I doubt it works fully yet.

If you want to get into it a bit more there is a guide here: https://drupal.org/node/1911346

It is worth a read and possibly the 50min to watch the video.

Token is required by quite a few modules, so we should get the ball running.

Also just a side note, patches will not test against the branch until the branch can pass all tests. So you should do testing using the testing module from core until we have a stable build. Then we can start doing testing on the patches.

Tag isn't really appropriate.

Newer Patch.

Updating tag to match what other contrib modules seem to be using.

If someone wants to take over on this, go for it. I am tied down with lots of other conversions at the moment.

Working on this at DrupalSouth sprint, using github as sandbox: https://github.com/larowlan/token

We're ready to look at merging this back into the token repo.
Current sandbox is http://github.com/larowlan/token.

Patch against 7.x-1.x is attached, setting do not test cause its a) for 8.x and b) testing on contrib for d8 is borked because of #2171683: Remove all Simpletest overrides and rely on native multi-site functionality instead

We've used clean commit messages like so in the sandbox, so merge is better than patch™ - eg
Issue #1962358 by grom358: Fixed TokenNodeTestCase

To merge - apologies if this is obvious

git remote add github https://github.com/larowlan/token.git
git fetch github
git checkout -b 8.x-1.x github/8.x-1.x
git push origin 8.x-1.x

Screenshot of tests passing:

Now we need manual testers.

Note no work done on the field formatter integration.

Note token patch above needs #2198041: Comment::urlInfo no longer includes the comment-{cid} fragment against core

JFYI, I'm working on larowlan's sandbox getting up to date with core latest changes and converting routes to controllers.

Added token.local_tasks.yml and other D8 changes upgrades in the sandbox.

I have prepared a clean install of D8 with devel and token (from github) to test this. I've only reviewed the workflow of content types and content. Here are the steps I followed:

I've created a content type and when I tried to add a field (text area) I got a WSOD with this error message:

PHP Fatal error:  Cannot use object of type Drupal\\field\\Entity\\FieldConfig as array in /var/www/d8/modules/token/token.module on line 673, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP Stack trace:, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   1. {main}() /var/www/d8/index.php:0, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   2. drupal_handle_request() /var/www/d8/index.php:15, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   3. Drupal\\Core\\DrupalKernel->handle() /var/www/d8/core/includes/bootstrap.inc:1594, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   4. Drupal\\Core\\HttpKernel->handle() /var/www/d8/core/lib/Drupal/Core/DrupalKernel.php:284, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   5. Symfony\\Component\\HttpKernel\\HttpKernel->handle() /var/www/d8/core/lib/Drupal/Core/HttpKernel.php:65, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   6. Symfony\\Component\\HttpKernel\\HttpKernel->handleRaw() /var/www/d8/core/vendor/symfony/http-kernel/Symfony/Component/HttpKernel/HttpKernel.php:66, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   7. call_user_func_array() /var/www/d8/core/vendor/symfony/http-kernel/Symfony/Component/HttpKernel/HttpKernel.php:126, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   8. Drupal\\Core\\Controller\\HtmlPageController->content() /var/www/d8/core/vendor/symfony/http-kernel/Symfony/Component/HttpKernel/HttpKernel.php:126, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP   9. Drupal\\Core\\Controller\\HtmlPageController->getContentResult() /var/www/d8/core/lib/Drupal/Core/Controller/HtmlPageController.php:53, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP  10. call_user_func_array() /var/www/d8/core/lib/Drupal/Core/Controller/HtmlPageController.php:76, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP  11. Drupal\\Core\\Controller\\FormController->getContentResult() /var/www/d8/core/lib/Drupal/Core/Controller/HtmlPageController.php:76, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP  12. Drupal\\Core\\Form\\FormBuilder->buildForm() /var/www/d8/core/lib/Drupal/Core/Controller/FormController.php:81, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP  13. Drupal\\Core\\Form\\FormBuilder->prepareForm() /var/www/d8/core/lib/Drupal/Core/Form/FormBuilder.php:235, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP  14. Drupal\\Core\\Extension\\ModuleHandler->alter() /var/www/d8/core/lib/Drupal/Core/Form/FormBuilder.php:787, referer: http://d8.local/admin/structure/types/manage/content_type/fields
PHP  15. token_form_field_ui_field_edit_form_alter() /var/www/d8/core/lib/Drupal/Core/Extension/ModuleHandler.php:406, referer: http://d8.local/admin/structure/types/manage/content_type/fields

Ignoring the error, the text area field is created correctly.

I try to create a node and is created correctly.

I go to node/[node id] and I can view the node correctly.

After creating the node I go to node/[node:id]/devel/token to see if the tokens are correct. Most of them look correct but I see some weird things (maybe because I'm not enough used to the new version of the module).

I see some tokens like this: [node:author:created:custom:?] (note the question mark at the end). They don't have any value and I don't know what are they. [node:author:created:custom] exists and have the correct value.

The rest of the tokens that appear look correct, but I can't find the tokens for the custom fields I created. The body, title and the rest of the data is there, but not the custom fields, maybe due to the WSOD found before.

I don't find anything else on the devel/token page.

I try to delete the node and it works fine.

I try to delete the content type and is deleted correctly.

Aside from things being needs work currently, I'm a bit concerned that it seems like the branches have diverged significantly since the fork point (https://github.com/larowlan/token/commit/ad58414f207e85d17d14d5928e0a11f...) with 20 commits in 7.x-1.x that do not seem to be reflected in the 8.x code. Shouldn't these be brought forward before I can merge this in?

Things I've noticed:

• admin/help/token page missing.
• Field help text issues: "This field supports tokens" appears twice and missing browse tokens dialog link.
  
• "This field supports tokens." missing on block titles. Token replacement inside block titles doesn't seem to be working.
• The 'Token' view mode not created for entity types that are created/discovered after Token module is enabled.

token.routing.yml:
I think the TokenCacheController should probably be using requirements:_csrf_token: 'TRUE'?

token.services.yml:
Looking at examples in core, it seems that arguments: [cache_token] should actually be arguments: [token]? I don't see anything else using cache_ in the arguments. Confirmed that this makes a cache_cache_token table instead of the desired cache_token. And looks like the service name should be cache.token instead of cache.cache_token?

Created a pretty large pull request at https://github.com/larowlan/token/pull/15 that fixes quite some stuff, most tests (except two, something with block label validation and different comment URL's) are now green but they're pretty lacking..

Also merged in the 7.x-1.x branch, had a lot of weird merge conflicts, tried to get the dialog UI with CSRF protection loaded, which works now and it loads, but it looks broken, there's some theming missing.. could use some help there ;)

Thanks for porting. I'm testing on latest github code.

Doing manual test on Google Analytics. Is it intentional that the "Browse available tokens." popup is not shown in center of viewport? It is shown on right/bottom. It also scrolls outside of the viewport if the page get's longer.

Got some errors if form is saved with an unknown token. You just need to enter a token like [current-user:role-ids2222] in custom dimensions and hit save and you get this:

Strict Standards: Only variables should be passed by reference in modules\token\token.module on line 635
Catchable fatal error: Argument 2 passed to Drupal\Core\Form\FormBuilder::setError() must be of the type array, string given, called in modules\token\token.module on line 635 and defined in core\lib\Drupal\Core\Form\FormBuilder.php on line 817

Hello maintainers? Are you there? :-)

Can we commit the latest code to D8 and create a DEV, please? I would be happy if my tests no longer fail and test robot will be able to download the latest well working code...

I was trying to wait until at least the UI functionality was semi-working, but I do have plans to merge it to the proper 8.x branch soon.

Do you mean the missing link in #21 or what else is not working? It's a bit easier to follow up with minor corrections if the large stuff has been committed, isn't it? Can someone write a summary, what is currently known to be broken? As noted... The stuff I found is not really blocking development of my modules, but the missing branche make all tests fail on d.o :-(

#697760: Replace getInfo() in tests with native phpDoc + annotations (following PHPUnit) need to be added.

@Dave: Do you have an idea how far away these "soon" is?

I guess this is caused by anything inside token:

Drupal\Component\Plugin\Exception\PluginNotFoundException: The "view_mode" entity type does not exist. in Drupal\Core\Entity\EntityManager->getDefinition() (line 243 of drupal8\core\lib\Drupal\Core\Entity\EntityManager.php).
Drupal\Core\Entity\EntityManager->getDefinition('view_mode')
Drupal\Core\Entity\EntityManager->getHandler('view_mode', 'storage')
Drupal\Core\Entity\EntityManager->getStorage('view_mode')
token_install()
call_user_func_array('token_install', Array)
Drupal\Core\Extension\ModuleHandler->invoke('token', 'install')
Drupal\Core\Extension\ModuleHandler->install(Array, 1)
Drupal\simpletest\WebTestBase->setUp()
Drupal\google_analytics\Tests\GoogleAnalyticsCustomDimensionsAndMetricsTest->setUp()
Drupal\simpletest\TestBase->run()
_simpletest_batch_operation(Array, '4', Array)
call_user_func_array('_simpletest_batch_operation', Array)
_batch_process()
_batch_do()
_batch_page(Object)
Drupal\system\Controller\BatchController->batchPage(Object)
call_user_func_array(Array, Array)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1)
Drupal\Core\StackMiddleware\PageCache->handle(Object, 1, 1)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1)
Stack\StackedHttpKernel->handle(Object, 1, 1)
Drupal\Core\DrupalKernel->handle(Object)

I have D8 Beta 1 running, and would love to help. Are there any releases to test at this point?

The current state can always be found here: https://github.com/larowlan/token

Based on https://travis-ci.org/larowlan/token, tests are not very happy right now, due to the url() changes, among other reasons.

Thanks--
Just pulled the latest from git, installed on my D8 Beta 2 setup, flushed caches. So far, no anomalies detected.

I know what you mean about changes to core--a theme I downloaded won't display its included info in the HTML head now in Beta2, but did in 8.0 due to the ever-changing code.

If there is anything specific to test at this point, I'd be happy to help.

Great!! waiting for a semistable release that I can try on my dev site! Good work!

Is it possible to get a dev release so that we can tag issues? It would make the meta game a little easier. It might also get more people involved.

+1 on the dev release here.

On installing token in drupal 8 beta 4, I get a white screen of death (MAMP 3.0.7.2 / PHP 5.6.2).

php_error.log:
PHP Fatal error: Call to undefined function current_path() in .../modules/token/token.module on line 32

If I comment (remove) the function "token_help" containing the call to current_path(), everything works fine so far.

+1 on the dev release here.

+ 1 for dev release. thanks for making the effort

I've opened the dev release back up so we can file issues for it for fixing things. Will wait on patches to be submitted back.

Here's the current state on the github repo

1. index 0000000..62e4acc
   --- /dev/null
   
   --- /dev/null
   +++ b/.travis.yml
   

   I guess we should remove this from the patch?

2. +++ b/README.md
   @@ -0,0 +1,3 @@
   +# Token [![Build Status](https://travis-ci.org/larowlan/token.svg?branch=8.x-1.x)](https://travis-ci.org/larowlan/token)
   

   And this?

@Dave: What is your plan exactly? Get the current patch in asap and then continue in issues, get some reviews first to identify missing parts, ...?

Yes, I'd like to review this within the next day or so for anything major, learn from the port, and start filing issues for anything outstanding or regressions, like you said.

Thanks to everyone that's been working on this for so long, and sorry I've taken a while to make the effort to get this merged back in.

1. +++ b/src/Controller/TokenCacheController.php
   @@ -0,0 +1,54 @@
   +    return $this->redirect($request->attributes->get(RouteObjectInterface::ROUTE_NAME), $request->attributes->get('_raw_variables', array()));
   

   Wow, this 'redirect to the next destination' code makes me incredibly sad. Not a comment on porting code, but D8DX.

2. +++ b/token.drush.inc
   @@ -0,0 +1,22 @@
   +<?php
   +
   +/**
   + * @file
   + * Drush integration for the Token module.
   + */
   +
   +/**
   + * Implements hook_drush_cache_clear().
   + */
   +function token_drush_cache_clear(&$types) {
   +  if (function_exists('module_exists') && module_exists('token')) {
   +    $types['token'] = 'drush_token_cache_clear_token_info';
   +  }
   +}
   +
   +/**
   + * Clear caches internal to Token module.
   + */
   +function drush_token_cache_clear_token_info() {
   +  token_clear_cache();
   +}
   

   This appears to be new. We should have a ticket to add this to D7. Are there any new features I should know about that we're missing currently from the D7 code that are in the D8 code?

3. +++ b/token.libraries.yml
   @@ -0,0 +1,24 @@
   +token:
   +  version: VERSION
   

   This seems wrong since the version of our contrib library shouldn't be tied to the core version.

4. +++ b/token.tokens.inc
   @@ -957,12 +978,9 @@ function token_tokens_alter(array &$replacements, array $context) {
   -          // http://drupal.org/node/1185842 was fixed in core release 7.9.
   -          if (version_compare(VERSION, '7.9', '<')) {
   -            $account = user_load($node->uid);
   -            $name = format_username($account);
   -            $replacements[$original] = $sanitize ? check_plain($name) : $name;
   -          }
   +          $account = user_load($node->uid->value);
   +          $name = $account->getUsername();
   +          $replacements[$original] = $sanitize ? String::checkPlain($name) : $name;
   

   I think this can just be removed since it was fixed in D8 as well.

5. +++ b/token.tokens.inc
   @@ -975,12 +993,9 @@ function token_tokens_alter(array &$replacements, array $context) {
   -          // http://drupal.org/node/978028 was fixed in core release 7.7.
   -          if (version_compare(VERSION, '7.7', '<')) {
   -            $account = user_load($file->uid);
   -            $name = format_username($account);
   -            $replacements[$original] = $sanitize ? check_plain($name) : $name;
   -          }
   +          $account = user_load($file->uid->value);
   +          $name = $account->getUsername();
   +          $replacements[$original] = $sanitize ? String::checkPlain($name) : $name;
   

   Same here, can just all be removed.

Ok, I'll see if I can do the same. I noticed quite a few things as well from scanning through the patch, there's still a hook_menu() will old stuff in it, and some controller classes look completely bogus right now..

There are two open pull requests for known issues, one is templates/UI/token browser is currently broken. Another is that menu link related tests are currently completely broken, that all changed faster than we could keep up with it. There's some work from kim but not complete.

TokenCacheController is exactly one of those, probably hasn't been touched in at least a year, that is very old code, CSRF protection is built into the router and just needs a flag to be enabled (including adding it to the generated link, neat stuff).

I don't really get to where the code is trying to redirect to (seems like the current route?), the easiest option I think would be $this->redirect('<front>'), that redirects to the frontpage, and an eventually passed along destination should be picked up and win over that automatically.

/D8-defend-mode-off :)

@berdir sent fixes for the above to the github repo.

Here it is, less the .travis.yml file

Also as a formatted-patch to retain individual commits from there (or commands at #15 if merging is preferred).

"Interdiff" for my changes is visible on https://github.com/larowlan/token/pull/59.

There are a few bigger topics that need work I think:

- The whole theme/rendering of the token browser. My suggestion would be to keep the browser modal link as the only theme function, convert the nested table theme functions to a service/function that sets up the render cache and then uses #type/pre_render callbacks for the real work, allowing to rely on the render cache properly (which is currently disabled as those functions it used are gone). I think that would also mean that overrides it used to do for POST are no longer relevant. Also, zero tests :)

- Field tokens, basic output works, but generic references/nesting is missing and there is zero test coverage. Also related is the token view mode, that could probably use an event now for new entity types to create them for entity types that are added alter on.

- Menu (link) tokens are very dead right, as the tests show. With all the refactorings that happened there, I'm not exactly sure how they are supposed to work now, there are multiple different types of menu links, which are then exposed as a common thing through plugins and derivates. So we probably need to run tokens on that, but I'm not sure what the use cases there actually are.

- Various helper functions and info functions, especially token_get_info() should converted to a service, either by replacing the core Token service, or adding a new service that wraps the one from core, or a mix of both. That would allow to write clean tests for a lot of complex code there, and remove the need for all those drupal_static()'s, which makes the code very ugly right now.

- token cache clearing is broken right now, but that's actually a core bug, resetInfo() calls Cache::invalidateTags() incorrectly. Which proves that none of the calls that are supposed to call it actually happen, e.g. because the field hooks are misnamed.

I might find time to work on some of those things, but I'd love to get some feedback/thoughts from Dave on those first.

See #2420025: Token::resetInfo() uses invalid cache tag structure for the mentioned core cache clear issue.

The whole theme/rendering of the token browser.

I think this sounds like a good plan, but maybe it would be good to merge first and then take this on. It looks like POST requests are still being ignored in Drupal 8: https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Render%21... - #2367555: Deprecate EnforcedResponseException support

Field tokens, basic output works, but generic references/nesting is missing and there is zero test coverage.

This sounds exactly like the D7 current branch, so I'm not sure why this is something to address (aside from zero test coverage - in the D8 code I assume?)

Waiting until it is merged was the plan, yes. POST requests are still ignored, but that is largely irrelevant when we (almost) only render the token tree in a separate ajax (GET) request?

Oh yes, that would be true. Yeah, the POST ignoring only really matter when it was rendering in the page and not in the modal. I still think it would be nice if the token tree still had it's own independent theme function, because it's used at admin/help/token to show all available tokens on the site. If that's still possible with the tree output being in a service/function instead of theme function, then that's ok.

FYI both the Metatag and FileField_Paths modules depend upon this; at work we're working on porting both of those modules, so I'm hoping we'll be able to throw some time into this too.

Opened a new pull request to fix some comment tests: https://github.com/larowlan/token/pull/61

If this happens to be merged to d.o before someone merges that and provides an updated patch, it is easy enough to just get the changes as a patch: https://github.com/larowlan/token/pull/61.patch

This sounds exactly like the D7 current branch, so I'm not sure why this is something to address (aside from zero test coverage - in the D8 code I assume?)

I don't know the exact state of the 7.x port, that is possible yes, I never remember if I end up using rules field tokens or those from token.module ;) Test coverage definitely, references also possible now that I think about it, otherwise there would be some code for that somewhere. We did a bunch of specific references in custom code now, simply by setting the type of a field token to e.g. node, and 3 lines to pass the sub-tokens through. Should be relatively easy to make that generic, although we should probably do it as a sub-token then, similar to how entity field api/typed does that now, only downside is that it exposes 'entity' to the UI. It would look like this then:
- [node:some_file_field] => field view
- [node:some_file_field:target_id] => 5
- [node:some_file_field:entity:fid] => 5
- [node:some_file_field:description] => the file description.

i guess the way to do this would be to have token types for each field type, should be possible to get all that out of the property definitions.

But as mentioned above, I have no plans to work on this before it is merged, and not before there are tests for the existing field token functionality.

Yeah, the tricky part has always been that works great for single-value fields, but not for multiple-value fields.

Here's an updated patch that contains the changes from https://github.com/larowlan/token/pull/61 and https://github.com/larowlan/token/pull/62.

This was green after PR 62, for the first time ever :)

Providing two patches, one against 8.x-1.x, another against 7.x-1.x, that possibly makes more sense to review, due to commits in 7.x-1.x, e.g the different README.txt.

No formatted-patch, I suggest to simply merge from the repository if you want to keep the commits.

So, all failing tests are, once again, subfolder things, a lot of tokens are /checkout/some/path. In some cases, the test expectations seem to be wrong (absolute url definitely needs to have it), in others, probably not, like path/alias/arg(n). I'm not sure if we should actually continue to support tokens like that, given that core tries to remove the internal path thingy from the API wherever possible.

Created a pull request to fix that, should be green on d.o together with that: https://github.com/larowlan/token/pull/63

Here's a patch for that. Interdiff is visible in the pull request.

There are some whitespaces issues but I think that can be fixed on commit.

If instead of the megapatch you want to keep history:

git clone --branch 8.x-1.x http://git.drupal.org/project/token.git
cd token/
git remote add larowlan git@github.com:larowlan/token.git
git rebase larowlan/8.x-1.x
git pull --rebase larowlan 8.x-1.x

Merged and pushed the current 8.x-1.x using proper history. I'm going to leave this issue now at needs work for now to review the regressions from D7.

We should get separate tickets filed for the current regressions, then we could close this out.

looks like we already have some test fails, fixed here: #2430797: request attribute _system_path was removed

Also created a few follow-ups:
- #2430821: Token view modes are only created for entity types that exist when token is installed
- #2430823: Token tree rendering uses a lot of theme functions and can not be render cached
- #2430829: Move helper functions to a service
- #2430827: [meta] Improve field support and add test coverage

The other points from my comments have been addressed already I think.

Cool, let's close this in favor of all the open issues then!