if t() string has no translation or fallback language, text should have lang attribute

Should it be in English or in the default language chosen by the user?

The fundamental problem with trying to implement this is that when t() or format_plural() is invoked, we don't know if HTML is generated. We cannot just output HTML wrapped text if what's generated is for an email or an XML format like RSS. We don't have that information, so we'd risk breaking all emails and RSS feeds generated by the site if we'd do. The localization client module has the same issue over at #218021: Make it possible to translate by clicking on elements.

Yes, to implement this, we'd either need to return a structure from t() instead of a string, so people can use just the string or the whole markup or alternatively we could pass in some information to t() that adding markup is ok (or rather that it is not ok, since that is less common).

Consider this as well:

$form['item'] = array(
  '#title' => t('Pick a number'),
  '#type' => 'select',
  '#options' => array(t('One'), t('Two'), t('Three')),
);

Now which of the four t()'s would be safe to return something like:

<span class="untranslated" lang="en">....</span>

To me it seems like only the first. The other three are option values, that are escaped for inclusion in the select box, so the select box would literally include the span tag. It seems to me like although we can add an extra argument to t(), this is not something I'd consider easy developer experience as they need to track how the strings are used in obscure places like this...

$form['item'] = array(
  '#title' => t('Pick a number'),
  '#type' => 'select',
  '#options' => array(t('One', array(), array('wrapper' => FALSE)), t('Two', array(), array('wrapper' => FALSE)), t('Three', array(), array('wrapper' => FALSE)))),
);

Well, not nice. I don't really have better ideas, since as you can see the kind of ways we use the output of t() can be very different from line to line.

You can try this for yourself easily. Just apply this simple patch and see how your Drupal site looks:

Ok, so it would be adding an $option to the array passed to t():
t($string, array $args = array(), array $options = array())

Currently this supports the following elements:
- 'langcode' (defaults to the current language): The language code to translate to a language other than what is used to display the page.
- 'context' (defaults to the empty context): The context the source string belongs to.

The best case described so far is to add a 'nomarkup' element to this which would check to see that no SPAN tags have been added or run the output through PHP's strip_tags() to yank out the HTML.

Well, while that sounds technically true I doubt this would generally be accepted as a trade-off. It puts lots of burden on the developer to figure out where the output of t() will be used eventually. Sometimes it is pretty impossible to tell if you are in a generic API function.

Is there a way that this could be perhaps handled by a contrib module? I'm just trying to see if there is any way to address this.

@mgifford: I don't think there is a contrib way. I'd love to be able to solve this issue, and mark all separate language items proper on the page but we need to consider the developer experience closely. Let's consider you have an API function somewhere like this:

function mymodule_status_of_service() {
  if (some_logic_to_do()) {
    return t('Enabled');
  }
  return t('Disabled');
}

Since we don't know whether this data is going to be used in a form item (escaped), text email (looks ugly with markup) or straight on a webpage (would be ok with markup), we don't know if that t() is allowed to add markup or not. It is really an API responsibility question. We should not spice up all of our API functions with arguments to pass to t() for markup, so I can tell this function that the output should or should not contain markup, right?

I don't really see a good way to do this, even if we add a nomarkup option to t() which I think people would be freaked out about, that would still not give us the tools we need to solve this, since often we don't even know what to pass in that option.

I do agree it would be great to solve this, but I'm at a complete loss of ideas. If I'd have had an idea in the past four years (since the issue at #218021: Make it possible to translate by clicking on elements was submitted), I'd have done it for my l10n_client contrib module honestly. It would be a very powerful thing.

Interesting option.. Could possibly play into modules like http://drupal.org/project/l10n_client

Well, the punctuation would need to be there if the text is RTL, which is what the markup tells the browser. I agree it would be good to have these wrappers to mark up RTL/LTR differences too, that unfortunately does not get us any closer to a solution :|

As discussed at A11ySprint. So much of the world doesn't use Drupal in English.

Any good ideas?

Guess we need to write up an issue to help assess whether or not it's returning HTML or not. Alternatively, do some tag sniffing as part of processing the string.

I think the problem is when should this markup be added (if required?

Perhaps an API function should return untranslated, but translatable, strings, and then at the point that these strings are being output, or added to a render array, they should be translated. This would require separation of translation into two calls. This is more easily understood, but still a extra burden on developers.

// Indicate a string is translatable.
translatable('Disabled');

// Translate a string.
translate($string);

Edit: Perhaps the second function could mean translatable and translate. The first function would declare a string translatable only, the second would declare a string translatable and perform the translation, so that literals can still be used and simple strings need only be passed into one function before output.

Obviously the function names used are for demonstration. In this case API functions would declare a string to be translatable, but the t() function could be used to translate non literal strings.

Another problem, which is exposed by Gábor above, is the need to set the lang and dir on specific render elements

$element['#lang'] = 'en';
$element['#dir' = 'ltr';

What if all fields on my form are in english, except one? Or, a more complicated case:

$element['#type'] = 'radios';
$element['#options'] = array(
array(0, 'English', array('#lang' => 'en', '#dir' => 'ltr')),
array(1, 'French', array('#lang' => 'fr', '#dir' => 'ltr')),
);

Great to see progress. Would be so neat if this patch could be brought in to address this on a more basic level.

@Hanno, you've added directionality via unicode, but not the language definition. I was quite hopeful to see this example here where they equated:
<Q lang="he" dir="rtl">...a Hebrew quotation...</Q>
and
‫״...a Hebrew quotation...״‬

Unfortunately I couldn't find a way to do that in English.

Having the untranslated string display in left to right I think would be an improvement for sighted users.

Ultimately we need to insert LTR & English around strings that haven't been translated. The unicode solution also won't work for XML I don't think.

Think we might have to go back to API function changes as per @Everett's note above.

Thanks @Hanno for your persistence on this issue. Language of parts is an important part of WCAG 2.0AA and one that isn't well supported anywhere.
http://www.w3.org/WAI/WCAG20/quickref/#qr-meaning-other-lang-id

It's too bad that the language tags in unicode have been depreciated. But yes, it's unlikely that AT will support it if they don't already..

Interesting the option of inserting the depreciated unicode and then swapping it with JS. It shouldn't cause a problem I would assume.

I like the idea of handling every untranslated string by an anonymous users as an error. I think that many sites would see it this way anyways.

I do wonder what the performance implications might be for any of these solutions.

adding tags.

Why are you using json wrappers when the same chars can be represented in PHP strings just as well (and in much less CPU time :)? Also, are you envisioning all strings in themes be wrapped in this decoding code? How do you think this could be feasible to implement?

Awesome that you're making some progress!
This is at least a task, so the feature freeze shouldn't stop you IMHO.

I had some problems applying the patch so just refreshed it.

In looking at the code though I want to confirm that from the source perspective like:

Actually, it seems I can't paste in the code here as the first character after the title="

It seems that this is being stripped out along with everything following it by the filters we're using.

I think the problem comes because the tests are looking for absolute strings, but the strings are different. Your patch appends characters which we can't see but that are there. I think the tests just need to be modified to allow for "\xF3\xA0\x80\x81\xF3\xA0\x81\xA5\xF3\xA0\x81\xAE\xE2\x80\xAA" . $string . "\xE2\x80\xAC\xF3\xA0\x81\xBF" as right now it's looking for $string in most cases.

It's a brilliant approach I think. It's only going to be wrapped around untranslated strings when in another language so for most sites it isn't something anyone is going to know about.

It will make some things harder to troubleshoot, but really like the approach you've developed here.

EDIT: There was a duplicate.

But thought I'd add here that I do believe that non-English sites seeking to meet WCAG 2.0 AA will need this. Will be useful to not have to hack core to be able to implement this. Not sure if there's any way to do this.

Is the next step here still:
more discussion to settle on an implementation?

Is there anyone we can ping that has not had a look at this yet that might be interested or have some expertise?

#40: no-translation-1165476-40.patch queued for re-testing.

Partly I think it's a matter of getting the patch to pass the bot. I've just re-queued it so perhaps it will work.

There are also open concerns by Gábor Hojtsy that haven't been addressed. Updating the issue with a summary of the concerns and responses would be useful.

I like that approach. Would probably be easier to get into core anyways. Any thoughts on how you're do that?

So postponing on #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols)

Also recategorizing as a bug. The screenshots in #16 should be pretty convincing.
Even in a fully RTL translated scenario, it's practically impossible not to have a few untranslated strings left, if it is desired at all.

Not sure if this really needed to be postponed.
The question ist: How far can we get without using the 4-byte characters?
It might be worth getting that part fixed and do a followup, whenever #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols) lands, which is quite unclear atm.

Unicode control characters are forbidden in phrasing content per the HTML 5 candidate:

Text nodes and attribute values must consist of Unicode characters, must not contain U+0000 characters, must not contain permanently undefined Unicode characters (noncharacters), and must not contain control characters other than space characters. […]

The "control characters" are used somewhat ambiguously in normal speach, but the spec perhaps refers just to the "Control Codes". The Unicode glossary defines those as in the ranges U+0000..U+001F and U+007F..U+009F, which would make this approach at least non-forbidden.

Nice to see progress in #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols)

Strings can fallback to other languages than English, e.g. Arabic (Egypt) can fallback to Arabic, thus it should have lang="ar" & dir="rtl" (so the writing direction need to respect the writing direction of the string language used)

Amsterdam2014 is the master sprint tag for AMS :)

@Hanno, can you re-roll your patch in #64? It would be great if this could be brought into D8.

I'd like to get this in but might need to wait to 8.1.

That should be Needs Work. Hanno's patch is from 4 years ago...

This came up as a random triage target for the #bugsmash initiative. Seems like this is still a bug, and it's still a tricky problem. At the bare minimum, we need to re-roll this for modern core (probably target 10.1.x for now, worry about backports later). I haven't exhaustively read the whole history in here to know if there's more needed (although I also see it's tagged for "Needs tests" and the existing test coverage in here is both very spares (only changes 2 lines) and is changing existing coverage, not adding new coverage.

Read all comments. I was concerned too about what Gábor described in the first 12 comments, as t() misses the context for when something is rendered in html (and where) or other formats. But maybe now with delayed translation with TranslatableMarkup is worth a revisit. Still think this would be a DX issue and a big change only doable in a major release.

For the current attached patch, looks like that covers only what is described in #2336491: if t() string has fallback language in another text direction, bidi should be added, so probably that should be moved there.

Also a concern: #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols) was closed, but not sure about the rest of engines we support (core and contrib).

After discussion in Slack with @dww and @Gábor Hojtsy, looks like there's agreement that this is a legitimate and important request, but cannot be realistically fixed unless a major re-architecture haul happens.