Fix HTML escaping due to Twig autoescape

node/1/translations - added issue today, just made it a child of this one: #2305799: HTML tags (raw) shown in Status column on Translations page

I noticed this on some Views UI pages today too, but hadn't filed issues, will hold off.

and here is another one: User info on "Search results" page - /search/node?keys=

The issue summary here could do with some instructions for people who want to help with this initiative. I'm seeing people attempt to use SafeMarkup::set() as HEAD does and then get slapped down for it.

Yes. I'm also unclear on whether this issue is a "fix it all up in one fell swoop" or if we're supposed to file child issues.

The permissions page also has double escaped strings.

One more: on the page with adding new field, autogenerated field name is double-escaped.

Filed patch for #9 #2309929: HTML double-escaping in field forms

So... These problems are all over the admin UI now. Can't we fix the root problem that caused them to appear? Or can someone ***** PLEASE ***** update the issue summary so we know what the real cause and correct fix is?

I guess if you sneak it in as usage of StringTranslationTrait::t() it gets passed review?

$this->t() is used over 2000 times in core, which is pretty much SafeMarkup::set().

If the string needs to be translated anyway, then is it really so bad to use SafeMarkup::set() as that's what translation is going to do?

I can't help with how to fix it, but I can help people understand why we shouldn't just revert the patch that caused a very visible problem. I've added a non-technical explanation to the issue summary.

#2289999: Add an easy way to create HTML on the fly without having to create a theme function / template

Adding #2310241: Uninstall page showing raw HTML output

Wrong way round... Made #2310241: Uninstall page showing raw HTML output child of this.

Added #2314513: Add test coverage for DiffFormatter not double escaping   to children list

Go!

Another url with double escaped strings that I found was in the install process #2317281: Double escaping of install errors.

#2321229: Over escaping html on field help text form element description

Another child issue #2319667: Simpletest Module Double escaped HTML in hook_requirements

Just curious, has something changed in last commits in Drupal 8.0.x? All the issues with Twig Autoescapes are seems to be fixed.
I checked with #2309215: HTML double-escaping in revision messages, #2309929: HTML double-escaping in field forms & #2319667: Simpletest Module Double escaped HTML in hook_requirements with the latest code checked out and all seems to be fixed automatically.

Can any one confirm on these?

Thanks!

Confirm that, that's very strange!!!

@andypost, so this means I'm not wrong. Something do happened in the last commits. Maybe the recent one. Strange!
Get the core committers, ha ha... :-)

Probably #2273277: Figure out a solution for the problematic interaction between the render system and the theme system when using #pre_render

@andypost, as you have seen also, a fresh install brakes all of them.

Another related issue: #2346611: Wrong characters on field edit page

Tried SafeMarkup::set(), but it's not working with conditionals.

I just want to mention here that it looks like this ongoing issue will fix a lot of these cases:

#2324371: Fix common HTML escaped render #key values due to Twig autoescape

So please test with the patch there before creating new child issues. Thanks!

Guess this is related: #2322439: Titles in a user's activity tab displays as just text and not a link., adding.

Removing related issues that are actually child issues. I added the missing relationship to a couple of them.

Do we need a fresh run through all of these to verify them now that #2324371: Fix common HTML escaped render #key values due to Twig autoescape was committed?

Yes.

Okie doke.

This is should be active now that #2324371: Fix common HTML escaped render #key values due to Twig autoescape is in.

To clarify, I un-postponed so people don't think that this is a postponed issue that needs fixing elsewhere. This issue "Needs issue summary update" as the active task and it's meta tasks need to be verified and closed if they are no longer valid.

I actually wonder if we should just close this out and open up individual issues for things we find?

@webchick: I worked in a few child issues and, for me, it was very helpful that we had this one as a parent issue, as I could easily access proposed solutions and check other similar issues' solutions as a reference.

Fair enough. The first step is go through all of the issues mentioned in the issue summary and sidebar and see if they are still issues anymore.

I've went through the child issues and they are all still valid in the context of not being fixed by that render #key values commit.

I was pouring over whether they are the 'correct' solution or not. We are trying to avoid SafeMarkup::set(), some people just side step the issue by using #markup which in some cases is the same but different way of doing SafeMarkup::set().

'#inline_template' also should be discouraged, but it's many times the only viable solution when you are dealing with table cell content with new tags. (#type table we are manipulating the data to fit the table markup, instead of marking up the data in X list format in a specific template. ) This pattern seems to be very consistent, which is good, but also a PITA for both manipulating markup and SafeMarkup::set.

2¢

Found #2406903: HTML double-escaping in views debug messages.

Added #2409477: Various ! placeholders throughout core are now showing escaped HTML (especially <a href="!..."> examples) (caused by the recent changed introduced by #2399037: String::format() marks a resulting string as safe even when passed an unsafe passthrough argument).

Added #2409477: Various ! placeholders throughout core are now showing escaped HTML (especially examples) (caused by the recent changed introduced by #2399037: (change notice, etc.) String::format() marks a resulting string as safe even when passed an unsafe passthrough argument).

So looking into this issue I noticed that simply changing the "!url" to "@url" totally mangles any GET request that you can make.

For example:
"admin/config/media/image-styles/add?a=b&b=c"

becomes:
"admin/config/media/image-styles/add%3Fa%3Db%26b%3Dc"

It looks like the string is marked as "unsafe" and escaped. Perhaps there should be a way to mark variables as urls or some special use case for urls specifically. However for now, setting the escape character to "@" in a t() string is NOT the way to go.

I'm not sure how this is supposed to work as templates are nested, some templates are going to be outputting the rendered markup from child templates.

I've created a custom theme and every place I want to output markup from a variable it is always being escaped.

For a specific example, I have a view and without any custom templates it renders fine. However when I copy views-view-fields.html.twig to my custom theme without changing it at all, the view content is then displayed as entirely escaped markup.

From this thread I'm guessing its because field.content is the pre-rendered markup for each field.

Can someone point out how you are supposed to output markup in twig templates please? Maybe using my specific example as a use-case.

Thanks

@Adam Clarey the nesting is not the issue, the issue is that by default views_view_fields is rendered as a theme function, not a Twig template:

#2348747: Convert theme_views_view_fields() to 100% Twig: remove the theme function
#2363423: views-view-fields.html.twig gets escaped

So am I to assume this is a bug that is a bug that is being fixed? The only thing I can gleam from the long issue thread is that if I want to use a template in my own theme, I need to add '|raw' to the end of the field so it will output the raw markup instead of the escaped markup.

Obviously this isn't ideal

@Adam Clarey if you can please test the latest patch from #2363423: views-view-fields.html.twig gets escaped and comment on that issue with your results that would be super helpful!

You can ignore the first issue, I probably shouldn't have included that link (or not first anyway), that's just how we discovered the issue.

The patch didn't help i'm afraid.

Putting the template views-view-fields.html.twig in a custom theme only outputs the sanitized markup.

To output as actual markup the twig variables require '|raw' adding to theme, eg field.content|raw

Please, #2363423: views-view-fields.html.twig gets escaped is the issue for these comments. If you can post there the steps you used to test that would be super helpful! Thanks.

#2511780: Node revision log HTML is escaped

#2610236: Views - "No Results Behavior" for individual fields escapes HTML

#2637550: Quick edit adds markup to node titles which is encoded and displayed in modal titles

Got the same problem as mentioned in #59.
Related with #2610236: Views - "No Results Behavior" for individual fields escapes HTML.
Running on 8.0.6.
When using Views Custom text field, whatever you put in the text field, whether it resolves to empty or not, the returned value is not going to be empty, unless the text field is left completely empty (kind of useless and defeats the purpose of the field).
See the very good inline comment in Custom.php line 67:

<?php
// Return the text, so the code never thinks the value is empty.
?>

So whether your pattern in the text field evaluates to empty or not, it doesn't matter, returned value is not going to be empty.

When setting up a No results text, the value then ends up in the wrong case, in my opinion, again, with a great inline comment in FieldPluginBase.php line 1308:

<?php
      // If the string is not already marked safe, it is still OK to return it
      // because it will be sanitized by Twig.
?>

ok, so is it going to be the recommended practice for such field customization or is this perhaps a buggy limit case?
 
I would be very happy to provide whatever additional information or screenshots, but the test case is exactly the same as #2610236-8: Views - "No Results Behavior" for individual fields escapes HTML with the Custom text field.
 
Of course, overriding the views field theme with {{ output|raw -}} worked fine, as mentioned in #59 and at #2584655-13: Views field "Global > Custom Text" incorrectly escapes HTML.
Let me know if there's anything I can do to help on this.
Cheers!

This is causing some small problems for Metatag: #2649592: Write tests to cover apostrophe handling in meta tags

Added #2850496: Double escaping in Rest UI

Revision log messages when using Views (base table content_revisions) field has escaped HTML shown to user.