Add a UI for browsing tokens

Placeholder for an actual comment. Or Subscribing

Popup modal dialog FTW.

So a update, I think one problem I did find is that - with descriptions (which is fine), is this interaction is unprecedented - so one behavior we should try to mimic is that of collapsing. I see a certain problem, with beign able to drill down but not beign able to move up quickly, so I think exposing the other tokens might be valuable - but this is by no means a set opinion.

So it's basically a generic combined tree/table control? How many levels we need? Can we do it based on existing fieldsets http://groups.drupal.org/node/19500#comment-76033 or should we create a new UI element?

- Correct, this is hybrid "tree/expandable table" control then. It would be nice to have it as genric UI element.
- what are the proposed actions user can to with this widget? Perhaps I am mis-reading the mocks but the tokens 1.0-style guidance "here are the codes you can insert into a text field: [author-name], [node-title] ..." are lost in the new UI. How the user should know he needs to mash together active tree elements [node:author:last-login:short] in order to get the preferred token code? Or is there a planned JS trick to automatically paste the complete token code to the textfield from the active row?

subscribing

Introducing a new tag for feature freeze: API clean-up.

Yeah the mockup in #7 is heading down the right path. If we take a shortcut and don't provide a way to see chaining it will drastically decrease the number of users who will realize this new ability exists.

Wondering if there is any movement on this? / subscribing.

Task #1: #675492: Allow exchangable ACDBs for autocomplete. Please review!

Why not do it like Facebook's @mention UI? No list below the text field at all -- just start typing and it autosuggests.

subscribe

Moving on up.

@#8 "Or is there a planned JS trick to automatically paste [...] to the textfield [...]?"
@#9 "That's what I was thinking."

two points:
1- downgrade gracefully when JS is disabled (therefore design without entirely relying on JS: autocompletion, onclick events, etc)
2- if a page with 10-15 textfields/textareas provided by the same module wishes to show up only ONE table of tokens (since it might be huge):
2a-- to which textfield/textarea will be the token pasted?
2b-- forced to use one huge table for each tiny textfield?!

another important point:
3- Note that collapsible fieldsets lost their collapsibility within vertical tabs (and BTW lost their titles as well)

I truly believe that the better approach should be find out a logical (human readable) way to display that bunch of data
I also believe that it can be implemented in contrib (kind of token_ui or token_helper module) AND IF it is ready before D7 release it might even get into core (a couple of theme functions won't hurt anyone IMO, otherwise long live in contrib apply for D8)

Currently I have no solution for this
but this was my first shot

the algorithm to build this table was:
1- in this case the consumer module only matters about [comment:*] and [site:*] tokens (the old D6 scopes for 'comment' and 'global')
2- fetch all the tokens
3- pick selected types and bundle them in an array ready for theme('table', ...)
4- for each one see what chains they have and take note of the type and the key of the chain (i.e. for [comment:parent:*] take note that the chain is 'parent' but its type is 'comment')
5- pick those chains and bundle them in another array ready for theme('table', ...)
6- for the current chains at hand repeat 4-5-6 until all chain's keys have been marked as considered
7- output the table of 'Tokens' with token like this [comment:cid], [comment:parent:*] to remark what is a chain an what's not (also prepend to chain descriptions the bold word 'chains')
8- output the table of 'Chains' with chain fragments like this *:parent:cid, *:node:created:*

the result is that chains that appeared first are nearest the top while, lets say for instance, date chains appear bellow node chains (depending on the case they might be even, but it is just an example)

I think the planned jQuery UI 'Tree' element would be good for us here. :)

#735354: User token types should *not* be nested is a big blocker for any kind of token UI or browser so it could use a quick review.

temporary you may use the type name as the group name, e.g "Comment tokens", "User tokens", and would have "Current-user tokens"

that's how I didn't hit that one

  foreach ($token_list as $type => $tokens) {
    $rows[] = array(array('data' => drupal_ucfirst($type) .' '. t('tokens'), 'class' => 'region', 'colspan' => 3));

so, it is not so blocker that issue, you may work and enhance the section header later on

Here's a *working* token UI I'm developing for Token D7. Work in progress of course, but feedback is welcome. :)

Here's a *working* token UI I'm developing for Token D7. Work in progress of course, but feedback is welcome. :)

EDIT: Thanks Guru Mediation. :P

Live demo up at http://www.davereid.net/d7/examples/token, will post a token issue patch shortly. It's currently blocked since there isn't any good jQuery tree/table plugin that is also licensed with GPLv2. I contacted the author of the one I liked best to see if he would dual-license MIT/GPLv2.

Really like this, it'd look great with a slide effect similar to the default way we display fieldsets.

I really like how TreeTable looks and functions. As an added bonus, it's actually not that bad when JavaScript isn't available too. Can't click on the text to copy in the token, obviously.

I looked for a similar solution that uses jQuery UI and found jQuery UI Tree, which unfortunately doesn't look as nice. But, it would be nice to have something that is future proof through the upgrade to jQuery 1.4 and jQuery UI 1.8.

..... What does SimpleTest use? We have something similar to this effect there, don't we?

Ehh, how is it diffrent from what I purposed in #5? Eitherway, this is not Drupal 7 stuff obviously.

Bojhan: The difference is that Dave made an implementation of it.

Got a response from Ludo, the author of the treeTable jQuery plugin and he's open to dual licensing his plugin with MIT/GPLv2.

By the way, you're free to relicense MIT to GPLv2 yourself, MIT is fully compatible with the GPL and specifically allows you to sublicense work covered by MIT (IANAL).

Great work! The tree table looks very smooth in the demo and perfectly suited.

Didn't know that, but I'd also feel much better just having the authors permission and approval, plus its fun to let someone know that you really like and appreciate their work. :)

@Rob Loach : Well if I could, I would :). Eitherway good work, I do think putting the arrows inside the table will make more sense.

I've made an official patch against the Token module for D7 and it wouldn't be too hard to do the same against core, although I want to know for sure before I put effort into that.

Follow #738372: Token UI browser + jQuery treeTable for all the patchy token UI goodness.

This is no longer a microproject or related to D7UX anymore.

I am interested in taking this on. Would this mean moving Token module into core?
Or would this just be added into token.inc?

tagging, I know this one will be involved.

I thought core already had a browser built-in?

RE #40 - Where? There are only a few token functions in token.inc to do replacement, and no UI is provided. That's why there is still the contrib Token module.

I have just added an issue summary, since there wasn't one and I believe the current idea would be to adopt the Token contrib module's token browser (which was discussed above and then adopted into the contrib module as noted in #36 above). So, anyone want to make a patch that takes the contrib Token module's UI for tokens and puts it into core?

Another proposal

Explanation:

• Text box with an arrow next to it
• When arrow is clicked, display a rules data selector-like dropdown that results in a graphical token.

@Gaelan: Do we have time to build something like that? How about we start with patching it with Dave Reid's popup token browser from the Token module?

@DaimenMckenna: I guess we might not. D9 maybe?

It's too bad if D8 doesn't have a token browser because we don't have time to work on your proposal when there is an already working implementation somewhere else (that has it's own set of issues but working nevertheless).

I don't have time to work on it but would be happy to review a patch for this.

I am working on this.

Here is a preliminary, completely untested patch. What I did is to take the bits of the Token contrib module that are related to the token browser UI, and ported them (hopefully) (somewhat) to Drupal 8.

A couple of notes:
- The theme functions need to be converted to Twig presumably.
- There was some caching stuff that I ripped out because caching is different in D8. There are @todo notes in there to indicate where this needs to be put back in.
- I am not sure how to test it. The existing Token contrib module does a bunch of form alters to use tokens... where do we currently want to use it in core?
- There are two PNG files in the patch. I'm not sure that will work, so I also attached them separately.

This is probably about as far as I can take it at the moment... setting to "needs review" so at least the patch will not drop totally into obscurity. If someone has an idea for where this should be used in Core, I could add it there and see if it actually works, and perhaps even add a test... at least for the non-dialog method of displaying tokens. I don't think we can easily test a pop-up dialog using our testing framework.

Dave Reid in IRC suggested adding this to the user email settings form in Core. So I did and it works pretty well! Here's a new patch (was missing a function from the Token project and now it has this added to user.admin.inc).

Things to fix:
- Needs caching as noted in @todo comments.
- Do we need token validation or is that a different issue?
- No tests.
- I noticed that the first time I clicked a token to insert it into a field, it went where I expected (where my cursor was). But then if I clicked another token, it went to the end of the textfield, not where my cursor had been. I don't know how to fix this. daveried wasn't aware of this being an issue in the d7 Token project but it's exactly the same JS code so I expect it is.
- I wasn't sure whether this should really be a separate module or part of system.module... if separate (as in this patch), maybe it should be enabled by default in standard.profile?

Anyway, this is probably ready for a review. nod_ might work on the JS bug?

Couple of questions

• Shouldn't the module be called Token UI?
• Do we need the insert into textfield functionality? it's kinda weird to have 1 table that messes with several textfields. not having it would take away some tricky code. That'll probably be a pain for a11y as well.

The real question around the JS is do we want the tableTree plugin in core? It has perf issues for big tables (on desktop!) so we can't use it as-is, it'll be terrible on mobile.

I'd argue that we'd want a proper list instead of a table, that would make the JS pretty trivial and that'd scale much better. Also this stuff needs to be able to work on mobile, the current table will likely be problematic.

All in all, it's possible to get the JS in shape and in time, need some UX love and fixing the perf issue somehow.

RE #50 -

- We could call the module Token UI, but we have other modules that provide UIs that are not called "something UI", and we currently don't have a Token module, so I decided to go for the "one word if possible" philosophy of core module naming. But I don't care. The other question is, should it just be part of system.module or separate? I don't have a strong opinion.

- I personally think the "insert into field" functionality is a useful UI (minus the current bug that it loses your cursor position)... I have no idea about accessibility though, that's a good point. But having to copy/paste the tokens (or try to type them from a list) does not seem like a friendly UI to me.

- Without the expanding/contract/tree functionality, the full list of available tokens would be LARGE and take up a huge amount of screen real estate. So I think we do need the expand/contract capability.... Is there another UI widget we can use that will accomplish that? I don't know that the table functionality is necessary, so a different tree widget that uses text items instead of a table would likely be workable.

The current name supposes you need to enable the token module to enable the token functionality. I don't care either way, just pointing out a possible misunderstanding.

Yep, copy paste kinda sucks for that, agreed. I don't see an issue adding the clicky thing after that got in initially and we figured out the a11y implications and probably other things we haven't thought about yet.

Oh yeah totally I do want collapsible things. It is a must have here. I'm questionning the library and markup we're using since right now it's too slow. didn't feel like a strong use case for a table either, which is why a simple nested list might be workable. Pinged Bojhan about it, hopefully he'll agree :)

I do not see 3 days sufficient to fix or even attempt to fix the UX, this has gotten designs before those are implemented in the current token UI - and I consider that good enough to iterate upon (adding search, improving the "token" in a field alá facebook, fixing the descriptions). I am more concerned that we might end up exposing this everywhere, something I'd love to avoid.

I do not care if the markup is a nested tree, the only semantic reason I could see for a table is that we have data associated to rows.

Another thing to consider with this patch: I took all of the UI components from the contrib Token module... I am not sure we actually need the hook_menu piece, which provides a pop-up window (I think we just want the expandable list/table piece, at least for now?) and the autocomplete (not sure exactly what that is being used for in contrib Token, if anything?)... so those (and accompanying functions) can probably come out of the patch, which will further simplify it.

Bojhan/#53 - what do you want to avoid exposing everywhere? The tree table or the token browser?

FYI the popup dialog is a replacement for the normal token browser that can *greatly* improve performance in many cases as the large tree is neither a) compiled, b) displayed.

@jhodgdon what is the difference between the two, I guess I am just afraid to find a fieldset below every field :) that could potentially have tokens. However I am sure that worry can be mitigated by a nice toggle or something.

@bojhan: Install the latest version of Token and Metatag on a D7 site to see the popup interface in action.

Turns out we got time now. Better do it right from the start.

The more I see it, the more we need to have tokens in a modal all the time. A link below whatever textarea, input field. Like it's done for metatags in D7. But not the "end of page table" there is on the user page after this patch.

It'd solves the context issue of the "insert into textarea" thing and it'd makes the performance issue less bad.

@Bojhan now that we don't have to rush it as much, can we try out a few things (on g.d.o or something, not this issue) ?

Yeah, this needs some design explorations first. Anyone up to laying out the problem space in a wiki post on gdo/usability? There's enough source materials in the first part of this thread to make a nice briefing of sorts: the problem, the challenges, the constraints and a couple of use cases should trigger some interesting sketches.

Well, for as much as this can be split up in a plumbing part and a UI part it would be good to push forward getting the plumbing in place. We should not 'reset' the effort already put in here. As with the modal/dialog patch, it got committed without a first implementation. Maybe the same can be done here?

@yoroy: We have time to at least decide the best approach to including this:

1. Where should it be added - as its own module or to system.module?
2. Is there an existing library or code pattern that could be used to simplify the code?

I'll be happy to help on the patch, but I'd like consensus on #1 and feedback on #2 before poking it with TextMate.

Good, thank you. Both of those are code/architecture and should be hashed out in here then. Please continue :-)

RE #62 item #1:

I think this should be its own module. In many/most cases, using tokens is a site set-up task, and like the Views UI, turning off the token browser module should be an option for an optimized production site (once you've set up the token-related stuff, you should be able to turn off the browser). Also, some people may simply not need the browser. However, I think we should turn on the module in the Standard install profile.

RE: Design...

- Are modal pop-ups accessible? If so (or if we can make them accessible), I think having a modal pop-up is a fine idea, and it would be easy to change the User Email setup to use them (provide a link). The code is already there... I agree that having a large area on every token-using form for the browser is ... clumsy at best. :) If people are interested in seeing the modal, I can make a new patch that uses the modal for the user email fields, and we can try it out. I can also provide screen shots. Let me know if this would be useful.

- I'm also wondering about the modal dialog... if I open the modal, insert a token or two, and then close it, and then I need some more tokens for another text field, could it remember the status (which items were open/closed)? That would be killer!

- Regarding what format is used for the actual browser section (whether modal or on the main page)... The tree table is nice, because it has columns (token/description), but if we had a flat list, I think that would also be fine (after all there are really only two columns that are relevant). So if a flat list tree browser is more efficient and we already have a JQuery library that does it (which??), re-jiggering for that would be fine. If someone points me towards which library, I'd be happy to work up that patch too and we can compare.

Design:

1. Modal accessible, yes, in time. It will be made thanks to jquery ui work (they changed their roadmap to fix accessibility things for us).
2. Technically pretty easy. It's a UX thing though and the interaction need to be thought through.
3. I don't know of a library, Collapsible nested lists are pretty trivial to implement, maybe based on the collapsible.js thing since the accessibility has already been figured out for it.

As for the columns if the visual thing is an issue, we can make a lists look like a table, that was already done on the responsive table issue as a proof of concept.

@nod_: could you better explain what you mean for "we can make a list look like a table"?

By using the CSS display:table and associated display values (like table-cell, table-row, etc.). As far as I know it's only a visual thing, I'm actually not sure how screen readers handle that.

Unrelated: @falcon03 I'm really happy you're around making sure our stuff is accessible. It's awesome work, thanks! if something needs fixing feel free to ping me, I'll be happy to help out.

nod_, Bojhan, jenlampton, and I have discussed the UI for this a bit in IRC and an etherpad. I'll add this to the issue summary too, but here are some notes:

Design considerations

• Efficient (code/load time - current treeTable implementation breaks with lots of tokens)
• Accessible
• Usable
  • Efficient to find the token you are looking for ("scannable")
  • Search capability - which may or may not be different from autocomplete
  • Hierarchical structure of tokens reflected in hierarchical-looking UI (intuitive)
  • Screen real estate (keeping mobile in mind too!)
  • Maybe different modes depending on the scope of the token list?
    - Flat list if there are only a few
    - Tree list and/or hierarchical autocomplete and/or search if the list is longer
    - Not loading the entire list initially if it's really long (page load time)
• Reusable UI pattern that can be used in Rules, Panels, Blocks, Layouts, Views, Etc. for both tokens and "token-like things".
• Naming conventions / standards for "token-like things"

Other notes

- TableTree/current implementation is apparently not accessible -- see #922022: The token tree UI is not accessible to keyboard only users on the Token project 7.x
- Token project was exploring other UI possibilities but apparently didn't get too far -- see #1229568: Token UI 2.0

Plan

- Spin off auto-complete and search capaiblity to a separate issue.
- Spin off adaptable modes depending on size of token list to a separate issue
- Token naming should also be separate
- Make this issue only about building a tree-based browser for long lists of tokens
- Use an expandable list instead of the tableTree of the current patch. Core doesn't apparently have a current UI widget we can use for that. nod_ may write one or we could use an existing JQuery component?

Add an issue summary with current proposed resolution, which is to adopt the contrib Token module's UI.

I did a search for jQuery tree libraries. It seems that the main actively-developed and available one for now is:
http://www.jstree.com/
(MIT/GPL licensed; accessibility unknown).

The jQuery UI people are also working on a tree plugin but they don't have it done yet apparently:
http://wiki.jqueryui.com/w/page/12138128/Tree

And I also found this one, which is basically an Aria-compatible enhancement of a UL list (sounds promising, and MIT licensed so that is OK):
download: https://github.com/filamentgroup/jQuery-Tree-Control/blob/master/js/jQue...
demo: http://filamentgroup.com/examples/jquery-tree-control/
info: http://filamentgroup.com/lab/jquery_aria_tree_control_from_book_designin...

nod_ also volunteered to write one but if we can use one someone else developed, that would probably be better?

@jhodgdon: Good research! One thing to point out, though, shouldn't it be an expandable tree-table hybrid because it needs to show columns of data too (description, etc)? There's already work going on in a somewhat-related VDC issue: #1834576: Improve details on the Views listing page.

RE #70 @DamienMcKenna - thanks for the link to that other issue. The TreeTable plugin that Token contrib is currently using is not workable (inefficient and not accessible) so we are looking into other possibilities, and right now probably it will be a list. Not sure.

By the way I am working on the next iteration...

Here's a proof of concept patch, minus JavaScript... I need some help with that, not familiar with how Drupal does attach stuff?!? Anyway this patch will put up a nested UL list of tokens that you can browse. The links are not working either. I have to run out for a while but I thought I'd put it up in case someone can help with the JS.

@jhodgdon: A list could work if there were appropriate tooltips to provide the detail listed in the current multi-column display. That said, we could conceivably handle it in two steps: 1. get a basic tree list in place, 2. improve the UI to show the other details.

RE #75 - missing information: The previous patch is only missing the token's name. I've added that now... could be formatted better but at least it's all there.

The other thing that was not working in the patch on #74 was the JS, which I have working now.

Here's another patch (proof of concept only!)... it does work this time!

One note on the patch in #77 - the Tree JS thing is not very smart -- it couldn't handle having a DIV inside each level of the item lists. So I (temporarily) made a copy of theme_item_list that omits the divs except on the outer list (so the inner lists are just nested UL LI UL LI UL LI etc., not UL LI DIV UL LI DIV UL etc.).

That should be fixed in the JS, which shouldn't be very hard actually.

I forgot, here's a screen shot of the pop-up dialog for the token browser.

@jhodgdon: Nice work :) I'll give it a review later.

I took a look at it (via the Menu -> Configuration -> Account Settings -> email message fields) and thought it was really quite nice.

My only concern right now is that with the default theme it takes up a lot of screen space so you don't have a lot of space left on the window to see around it. That said, I think it's definitely a step in the right direction.

@jhodgdon: do you think the patch is ready for an accessibility review?

@nod_: oh, I didn't know this css capability! I'll look for a demonstration of this feature in action and I'll test it with my screen readers, although I don't think it is "parsed" as a table for accessibility...

Unrelated: thank you very much! Developers' availability is very important for me, since actually I don't have the needed skills to roll my own patches (especially for JavaScript, which I don't know at all ;))! Again, thank you very much!

RE #82 - well, the patch is more of a "proof of concept" rather than a full patch, but ... yes probably an accessibility review would be useful since we attempted to use a tree plugin that said it was full ARIA/accessible. I am unsure about the dialog pop-up too -- if that is not accessible, we should start thinking about how to make it so and/or an alternative.

Note that the current patch lacks the capability of inserting the token into the field. It just lists the available tokens in a pop-up expandable dialog.

And it has only been added to the email section (vertical tabs) of admin/config/people. To test:
- Install the patch.
- Enable the new Token core module
- Navigate to admin/config/people
- Click the link where it says "Browse tokens that can be used in e-mail messages".

I wonder if we could use a series of nested HTML5 Detail and Summary elements like we did to replace fieldsets.
#1168246: Freedom For Fieldsets! Long Live The DETAILS.
An example of nested elements: http://jsbin.com/egefop/14#html,live

Hm. What is that "example of nested elements" link supposed to do? In my browser (Firefox 13 on Linux), it's just a paragraph followed by a single bullet list. Clicking the links does nothing.

RE #84 - I made a patch for collapsible fieldsets, but my usual browser (Firefox 13 on Ubuntu, which is a bit out of date) doesn't support them, sigh. Guess it's time to update Ubuntu, which I've been putting off. [Aren't the new display elements supposed to give you something usable in all browsers though if yours doesn't support HTML5??!? Maybe that hasn't been done yet...]

Anyway, I also have Chrome available, which does support them, and they do seem to be a good choice for this. The patch is somewhat nicer (lots smaller!) and the UI seems OK to me. Here's a screen shot and a patch in case anyone else wants to offer an opinion. The token list is a bit bulkier than with the UL lists, but to me the distinction between the expandable items and token items is clearer. And there's no additional JS to load, since these display elements are in core.

So... I would be inclined to go for #86. Any other opinions? The patch needs a bit of polish but it's not horrible as it is.

I am a little bit confused, why we are changing UI's. I thought the idea was only to change the underlaying code.

nod_ says that the tree table UI is not workable due to performance reasons, so we are exploring other options.

Also regarding the UI problem I'm having, I filed #1858600: "Filter log messages" details on Recent log messages page applies bogus floats if anyone is interested.

The test failure is from #937284: DEADLOCK errors on MergeQuery INSERT due to InnoDB gap locking when condition in SELECT ... FOR UPDATE results in 0 rows.

Things that will need to be fixed in this patch:
- Doing drupal_render() in the theme function is not going to work because it doesn't trigger the proper JS and CSS being added to the page.
- The theme functions need to be Twig templates.
- See #1858600-5: "Filter log messages" details on Recent log messages page applies bogus floats - the JS needed in some browsers for the details elements is being properly triggered in the dialog (whether it's a pop-up or in its own page).
- There are some additional @todo notes in the code about caching.

Great progress here!

Tagging it per Roy Scholten's request.

Realistically I am not going to have time to work on this again until January at the earliest, sorry! Is there someone else who can pick it up from here and take it to completion please?

#86: token-browser-proof-concept-v3.patch queued for re-testing.

Is someone going to take this over? Is it too late to get this into core anyway?

Doing a little bit more research on 'drill-down' UIs and came across jQuery Mobile which supports the iOS 'Listview' display, which I think actually may fit very well for our purpose. A sample can be viewed at http://jsfiddle.net/YmcM4/3/embedded/result/ (you have to click the 'Run' button on the top left-ish to see it).

It's kind of nice since we get filtering ability, a very structured way of drilling into more specific tokens (and seeing if the current token offers sub-tokens). It could play well with AJAX since clicking on a list item could load the new "page" via AJAX and shove the generated output into the same request. It works well in a modal. I dunno. I might play with this a little bit more.

The potential problems I see with this so far is that it requires the entire jQuery Mobile library, and for desktop browsers, the UI doesn't really work in a non-modal context.

Available types (first page):

Current date tokens (incomplete):

Node tokens (incomplete):

Looks awesome! If jq mobile the only thing that stops - we need it in core

We don't have a good solution, so we've missed our window for D8 at this point. Let's continue to let contrib find the right UI and get it ready to merge into Drupal 9.

@davereid: Maybe I'm the only one, but that UI doesn't look good to me :p

Also, #sadpanda :(

I definitely need to reiterate that the UI in #100 makes sense only used in a modal, so imagine it in a modal only. :) And it also could play well with a theoretical autocomplete functionality that can be used in text fields since the same AJAX that is used to build the 'sub pages' dynamically could also be used to respond with JSON for sub-tokens.

Given so much has changed in core's JS, are there any similar nested-tree UI widgets elsewhere in core? Maybe we should start there - find a UI widget that fits this use case and others, get it added to core and then add this use case on top of it?

There are non in core that I know of. We would still need to find an external library and add it to core.

I can discuss it with some of the Spark crew this weekend at the SprintWeekend in Boston, see if they have any ideas.

FWIW: I've used http://www.jstree.com/ before and it was pretty decent.

RE #108 - does JSTree satisfy the design criteria in the issue summary?

The website has an unfinished jsTree v1.0 but on github there's a v2 in progress with lots of commits recently.

update with current plan

As there is consensus that we can keep adding features to Drupal 8 after the initial release, as long as we don't break backwards compatibility, I am re-opening this issue.

The attached patch takes a slightly different approach than the previous one. Instead of porting the Token module's behavior, I decided to lay properly tested foundations first. It is relatively easy to change how the overview is rendered later on (like adding JavaScript), as opposed to changing the reusability after we first wrote the code to be usable for one specific case only.
Next to that, the biggest difference is that since the previous patch was made, the menu routing system has been removed and the token API is now a service. This has its consequences for the code.

This patch is still a work in progress. The token overview page at /token/overview works and most new methods are tested, but some aren't yet. I am not posting an interdiff, as it would be too big to be useful.

I think it would be nice if you adress how you wish to continue on the steps that @jhodgdon already made in https://drupal.org/comment/6793062#comment-6793062.

I fear that your focus on a nice engineering effort, without vision on how to tackle the UI part will eventually catch up on us - when real users use it. The lack of a good UI was the primary reason why it didn't go into D7. If you wish to get it in, you should figure out how to make it part of tackling this problem. This is not a followup task, this issue is about introducing a UI - therefore it should be good.

1. +++ b/core/lib/Drupal/Core/Utility/Token.php
   @@ -67,6 +67,16 @@ class Token {
   +   *   An array of token types, or NULL when the types are not set.
   ...
   +   * @var array|null
   

   I think the NULL part can be dropped, both from the @var and from the description. Unless they have a default, variables can simply be not set, and in this case that is what happens. The variable is never explicitly set to NULL.

2. +++ b/core/lib/Drupal/Core/Utility/Token.php
   @@ -305,5 +315,44 @@ public function setInfo(array $tokens) {
   +    if (is_null($this->globalTypes)) {
   

   In accordance with the above, an isset() would be fine here, I think.

3. +++ b/core/lib/Drupal/Core/Utility/Token.php
   @@ -305,5 +315,44 @@ public function setInfo(array $tokens) {
   +    if (array_key_exists('type', $token_info['types'][$type])) {
   

   I think an isset() would be more standard here. As far as I'm aware having the key set to NULL is not a valid option here.

Didn't get further than that with a detailed review. Looks good though, from a cursory look. I agree with the assessment of putting this into D8. Even if we don't expose it in the core UI, having something in core that contrib can build on that is usable, performant and accessible would be very prudent, IMO.

I'm SO SORRY for being late to the party, but here's one addition:

There is one UI option that i could not find in the above thread but should be considered: The rules data selector broser.

IMHO it has some advantages over the other options:
* there is working code to steal
* usability is proven

It might not be the best thinkable but it's there already in terms of code and UX.

I'm working heavily with the token browser and the rules data selector browser.
And everytime i use the token browser i wish i could have the data selector browser.

Just my 5 cents...

@alex.rutz: I thought that the Rules team were working on a complete overhaul of their UI? And I wouldn't say the "usability is proven" just because Rules has used that UI for a few years.

...and we never "steal" - we "borrow" :P

Yhea, I think you are assuming some usability there - we actually did do Rules testing a while ago for Drupal Commerce and the Data selector part is where most of the problems occurred.

I fear that your focus on a nice engineering effort, without vision on how to tackle the UI part will eventually catch up on us - when real users use it.

The main problem with previous efforts is that the code was procedural and did not come with unit tests, and also lacked in reusability; there was no possibility to just render the overview regardless of the context.

I understand that UX is extremely important, but the 'engineers' are the ones who will have to maintain this code and ensure its quality, and who will need to implement it in different contexts. That is why I added a very simple token overview to demonstrate that the code works.

However, any UX and performance advice is not only appreciated, but also actively requested. So if you are experienced in either of these two fields, please share your knowledge here, so we can incorporate it in the patch.

With regards to @jhodgdon in #68, it seems the suggested usability and performance improvements are targeted at two specific parts of the UI:

1. The token overview: the list of tokens that is used as a reference when you don't know what is available, or what exactly you need.
2. Form input helpers, such as an autocomplete feature for text input fields that is useful when authors (roughly) know which token they want to use.

If you agree on this, then I propose we make sure these two UI improvements are dealt with in different issues.

I understand that we wish to have a solid architectural basis, obviously I always support that kind of thinking. I think that jhodgdon outlined it very nicely:

1. Improve the interaction for getting the token browser (e.g. modal, or a collapsed display).
2. Show a teaser list, but not only when the number of token > N.
3. Improve the default token naming (could be separate issue)
4. Improve and build upon the tree based browser currently in contrib (its interaction is rather poor).
5. Allow for auto-complete and/or some kind of search capability.

I'd consider 1,4,5 necessary to get this in and 2,3 improvements that could greatly enhance the usability. I can deal with UI improvements, being in a separate issue - I cannot deal with another half-assed UI in core, that no one cares enough about to fix.

Thanks for your feedback! There's one last thing I'd like to address again at this stage: I believe the autocomplete and the token browser/overview are two different things, because autocomplete applies to input elements, whereas a link to open the token browser does not have to be attached to a particular input element per se. What are your thoughts on this?

Well, ideally you want to be able to both autocomplete in the token field and search the token overview. They are indeed, two separate UI's in rules its current interface they are presented side by side - I expect this not to be the pattern we pursue for core. Instead the browser will be launched either by modal or fieldset.

I discussed the inclusion of these changes in Drupal 8.1 or later with @catch and he said "API additions and new features are allowed, FAPI array changes are under discussion".
This means that as long as we develop a solution that does not break APIs and does not add any magic to existing token API implementations (including forms elements that accept tokens), it will be considered.

Well, ideally you want to be able to both autocomplete in the token field and search the token overview. They are indeed, two separate UI's in rules its current interface they are presented side by side - I expect this not to be the pattern we pursue for core. Instead the browser will be launched either by modal or fieldset.

I agree that the ideal situation is to have both. What I am trying to establish is whether both additions are technically related or not, so we can split them up into multiple issues and work on smaller patches that are easier to review.

On IRC Bojhan agreed that the autocomplete/suggestions and the browser/overview are two separate solutions. I created #2168793: Add token autocomplete support to form elements to deal with the autocomplete, so we can continue working on the browser here.

Please follow progress on: #2346973: Improve usability, accessibility, and scalability of long select lists

Changing to Select2 may render this issue obsolete. Select2 has better ways of handling tokens.

@tkoleary: I think that statement ignores the complexity we've had to face from exposing a UI in contrib, with all the nested information.

The nested information does not need to be exposed, and in fact is a barrier to usability because of information overload.

Here's how it might look with Select2 in site email config UI: http://invis.io/2T1HGEWZJ

The select2 features this might utilize are:

• Multi-value select boxes (for the pills inside the text area)
• Placeholders
• Loading Array Data (possible substitute for using ajax)
• Programmatic Access or Reacting to external value changes (for resetting the pill when a suffix is added)
• Drag drop sorting (a nice-to-have if we want users to be able to move the tokens around the text area)
• Responsive Design - Percent Width (for mobile)

Here's the documentation for reference:
http://ivaynberg.github.io/select2/index.html

The nested information does not need to be exposed, and in fact is a barrier to usability because of information overload.

Here's a linear mockup of how it might look with Select2 in site email config UI:

http://invis.io/2T1HGEWZJ (hold down the shift key to expose what's clickable in the mockup)

The "add suffix" placeholder should only appear when there are children, and I'm not sure that's the right nomenclature.

I hallway tested this on a few members of the Acquia support staff who use this UI commonly and all found it a big improvement. My hypothesis is that if a user comprehends the use of tokens they are likely to have a good idea which one they are looking for to start typing and the category labels help refine that.

Select2 features this could potentially utilize are:

• Multi-value select boxes (for the pills inside the text area)
• Placeholders
• Loading Array Data (possible substitute for using ajax)
• Programmatic Access or Reacting to external value changes (for resetting the pill when a suffix is added)
• Drag drop sorting (a nice-to-have if we want users to be able to move the tokens around the text area)
• Responsive Design - Percent Width (for mobile)

Here's the documentation for reference:
http://ivaynberg.github.io/select2/index.html

@tkoleary: That would work for two or maybe three levels of depth, but what about accessing data from entities tied into the current entity via reference fields? It's an interesting idea, but it would simply need a way of digging into the detail further.

@DamienMcKenna

There seem to be a few different methods in Select2 for loading data. Maybe one of them could be used? Pinging Nod, he made the select2 patch for language selection.

I like the idea for tokens in textarea but we can't do that only with select2. The prototype is showing what CKEditor widgets are. We can't implement the design without taking control of everything in the textarea, which means implementing most of what a WSIWYG editor does.

Short of putting ckeditor in those textareas and having token widgets the design is not doable. The input search field looks good though.

@nod

I figured there might be a problem with getting the tokens to display as pills, but in fact that is really a nice-to-have and the usability improvement is not really diminished if we just have them appear in square brackets as they do now.

(PS. I have not read the whole thread in detail, just searched the page for "role" and "permission")

Out of curiosity;

Performance-wise, would it be useful to have role-based ability to load whole sub-trees?

All users could be served all tokens down to level 2 or 3 on the initial page load.

Users with role X could then be permitted to load whole sub-trees below that limit with one click (on the desired level/group), using AJAX inline load so that the whole page does not refresh.

Users without that role would need to click on each level and wait for the same AJAX to load one level leeper on each click, yet still without loading the same page.

That would combine elegancy and flexibility with performance concerns.

"Add a UI" is a feature request... so this really does need to be postponed to 8.1.x at this point, and/or contrib, in spite of the fact that it would be really nice to have in core.

#132:

Short of putting ckeditor in those textareas and having token widgets the design is not doable.

We're actually attempting to do that in #2912947: CKeditor Enabled Breaks Functionality, and could use some help.

Autocomplete in textareas:
https://stackoverflow.com/questions/6157992/twitter-style-autocomplete-i...

CKEditor 4 has an official autocomplete plugin now, which seems suitable for tokens.

https://ckeditor.com/cke4/addon/autocomplete

Coming from the recent usability review on #85494: Use email verification when changing user email addresses, as part of implementing a token browser, it would be great if we could have a specific action to look at how we transform the current UI of the user notification emails in Account Settings.

Currently there is just a basic list of available tokens below each of the text areas, in the usability review we identified this as a potential issue for site builders (some of the tokens may not be very clear as to what they do by name alone).

So as part of this work, we should have a specific piece of work to look in detail at the implications for that interface.

I'd love to get a UX proposal on how such a UI could work. See also the related token.module issues.

However, I would recommend that you don't focus too much on the current cor use case but more the existing token browser in terms of complexity. The number of tokens that core currently exposes on that page is _tiny_ compared to how it really looks for use cases like metatag, pathauto and so on. Almost all non-trivial sites have the token module enabled, so just designing something for core isn't going to actually help many users.

I love the direction this thought process is going. Let's see if we can iron out some use cases. (without considering available implementations)

Using Tokens
As a content creator, I want a UI that helps me discover what tokens I can use (for things like url paths).
As a developer, I want to be able to have my custom created tokens appear in the UI when they should.

Understanding Tokens
As a content creator, I want a short description of what tokens are so I can understand why I'm using them.
As a content creator, I want to see a preview of how my token will be rendered.

Using Tokens
As a content creator, I only want to be able to see tokens that will work in my current context.

One of the most frequent "bug" reports on the token_filter module is people trying to use tokens outside of their available context, like using a block token inside a node or vice versa.

The existing token UI in contrib supports that, it's the responsibility of the module embedding the UI to show that. If it's hard for that module (I can see that being the case for a filter description as you don't have that context) then even more so for the UI itself.

@Berdir yes it does.

As @cosmicdreams is asking for user stories "without considering available implementation" I added it to the list as for me it's a must-have.

The fresh approach to user stories is worth going through if we can get enough input, the outcome will be more robust that way.

As a separate issue, once there's a token browser in core maybe field types could be extended to add an option indicating whether the values can have tokens? Then the system could automatically provide the token browser link and process the value during rendering. That would IMHO save some of the complexity that site builders have with wanting to use tokens in different fields.

Ok, so it looks like I derailed things with #157, and on further reflection @Berdir you're absolutely right with #156.

@DamienMcKenna that would be nice - is there an existing effort to have a token browser in core?