Enable translation fallbacks

Seems like this would be great for country-specific translations - i.e. it's a valid use case to have en-gb as a language for UK spellings, but there's only going to be a handful of strings changed compared to core, other languages are likely to have much the same issues.

I'll better make this work this way:

- import the most specific language
- then import the less specific one, without overriding strings already loaded

That would allow the "French (Quebec)" translation to only override the "French (Standard)" one on an handful of strings.

subscribing

This is a reroll, locale.inc was altered a bit.

It seems the method for querying was switched from using db_query to execute() just before the meat of the first hunk.

Re-rerolled, also got rid of a db_fetch_object() since we can foreach over result sets now.

I can't get this to work, but maybe I'm doing things in the wrong order. Perhaps you could provide some steps on how to test this?

I think he idea of language fallbacks is ok though you cannot always make the assumption of the fallback being the two letter code. I.e. my fallback chain may be like en_CA -> en_US -> en
Other times you may want no fallback at all, as processing fallbacks may slow down page rendering.

So maybe the first step here would be to add an (optional) 'fallback' field to the language table. Then we can use it for localization, content or whatever..

Processing fallbacks may slow down page rendering.

The fallback is pre-processed. There is no impact on the page rendering.

@Damien,
> The fallback is pre-processed. There is no impact on the page rendering.
Yes, you're right. Not if we use it for importing translations only.

However there may be other uses for language fallbacks. What about real time fallbacks for string translation or if we end up implmementing translation for user defined strings, or searching for alternate content? Another case in which we can use fallbacks is.

Anyway, I have to admit this patch as it is, is an important improvement on what we currently have. I'd just like to see a better generic implementation of fallbacks so we can use it later for other things.

Patch re-roll against latest HEAD, but fails a number of 'locale' simpletests, so leaving at CNW.

Can you post the simple tests too and I'll test them in conjunction with the other issue? I can confirm the existing tests pass with the new patch.

Re-roll against latest HEAD. All tests pass.

this needs to wait until 8.x

In a custom solution I built on Drupal 6 I also too the approach of 'topping up' a small set of localised strings with strings from a more fully-localised language. In my case, I had been asked to allow the fallback language to be (linguistically) unrelated to the primary language, the typical use case being that the fallback language would be a language of wider communication (regional, trade, national languages, etc) that would typically also be comprehensible to speakers of the primary language. I kept track of which strings were actually in the primary language, so that updates to the fallback language could be incorporated, and to make it easy to see which strings had not yet been localised in the primary language.

I never felt completely happy with a solution that had basically contaminated a localisation set with strings that were in a different language.

For Drupal 7 I proposed, and have had some success in testing, a different solution which modifies the operation of the locale() function. I had one approach but Gábor suggested a smarter one: the use of an object, pretending to be an array, that is assigned to $conf['locale_custom_strings_$langcode'] during the bootstrap. This probably has an impact on page-load time initially, but it does then allow the results to be cached in the standard locale cache, so that there should be no ongoing extra burden.

My thoughts are initially documented in the following issues, though I haven't yet posted the complete working module anywhere - I can do so if anyone is interested in this approach. I'd also be interested to read thoughts on which of my two design options is best (or at least, better) practice - as described in the second of these issues:

http://drupal.org/node/1261034
http://drupal.org/node/1266882

I had a similar problem after enabled a custom language for de-CH (D7). The frontend will appear in english instead of german as expected. I implemented a solution, that patches the locale.module. The query in line 666
respectively 680 will set the the cached locale_t to TRUE... and the the source string will returned in line 720.

...
   return ($locale_t[$langcode][$context][$string] === TRUE ? $string : $locale_t[$langcode][$context][$string]);
...

I changed this behaviour by implementing a i18n_fallback function:

....
return ($locale_t[$langcode][$context][$string] === TRUE ? i18n_fallback($string, $context, $langcode) : $locale_t[$langcode][$context][$string]);
}

function i18n_fallback($string, $context, $langcode){
    $parts = explode('-',$langcode);
    if(count($parts)>1)
    {
        array_pop($parts);
        return locale($string, $context, implode('-',$parts));
    }
    return $string;
}

the fallback will be now de-CH -> de -> en (as source). It may be potentially possible to check some mappings in the fallback function, too.

Patch for D8, fallback language can be specified on the language edit form.

[4:49pm] GaborHojtsy: attiks|away: hum, that sounds like way last minute to me 
[4:49pm] GaborHojtsy: attiks|away: however Jose has a lookup rework patch that would allow contrib to swap out the lookup class 
[4:50pm] GaborHojtsy: attiks|away: so it would be very easy to do a contrib with that patch that supports this
[4:52pm] GaborHojtsy: attiks|away: http://drupal.org/node/1813762
[4:52pm] Druplicon: http://drupal.org/node/1813762 => Introduce unified interfaces, use dependency injection for interface translation => Drupal core, locale.module, normal, needs review, 32 comments, 16 IRC mentions

In short I don't think this fits into Drupal 8, it was posted too late, but it should be possible to do in contrib relatively elegantly once this mentioned patch from Jose lands. Help there :)

For D8 the other patch seems the way to go, since that patch cannot be back ported switching to D7 to fix it as well.

Our use case:
International site localized by region and language, 99% of the translations will be the same as the Drupal language, but some strings will be specific for a certain region.

@attiks: you might want to look at http://drupal.org/node/347351#comment-5905198 above for Drupal 7 :)

@Gábor Hojtsy I saw that one, but it tries to solve it in a different way, we'll explore both approaches to find to most flexible.

I'm sorry I've missed this one for too long, not it seems to be too late for D8 :-( Anyway, these are some mixed thoughts about how to move on with this one:

1. As @Gábor Hojtsy says, this is the patch that should allow a clean replacement of t() function for Drupal 8, so we may not be late after all, provided this one lands eventually in D8 core (it is a cleanup & consistency patch with quite possitive feedback so far, so it should at some point) #1813762: Introduce unified interfaces, use dependency injection for interface translation

2. Though I'm not sure I like the per-string fallbacks for translation as it can get you a paged mixed in a few different languages, I'd encourage and support building it as an add on module. But what we really need, first of all is:

3. A plain simple generic module for getting us language fallbacks which provides some configuration UI (which language falls-back to which one) plus a simple api function like language_fallback($langcode), or if you want it more D8 style language_fallback()->get($langcode). Maybe building it for D7 first could be a good idea.
This one can be used for string translations and for some many other things.

4. After thinking more about the problem, I don't think the first idea in #11 (adding fallback field to each language) is the best one. And the reason is this: we may want to define fallbacks for not enabled / not configured languages too.
Say you have 3 defined languages and then, for each other language in the world, you want to define to which of the 3 it falls back too.
So I'd make a separate setting that may be in the variable system in D7, in a configuration object in D8 and has its own settings page. (Which doesn't mean for configured languages you cannot add this option in the form, just not stored with the language itself but somewhere else).
Also for resolving long fallback chains, like 'lang 1 -> lang2 -> lang3 ...' we'll be able to do it faster because all our fallbacks are stored in one place and we don't need to load every language object to see what it falls back to.

5. Keep in mind the problem of the browser fallback language needs to be resolved differently as a browser has its own list of preferred languages and the process should go like this:
Try: browser-lang-1 -> browser-lang-2 -> browser-lang3 ....
(now we don't have the thing available in any of these, this is where our fallbacks kick in but NOT from browser-lang3. It should start from browser-lang-1 -> ....)
I'm really pissed off by all the sites in the world, that are a lot, that don't respect the preferences I have stated in my browser and do stupid assumptions like "oh, you are in japan, then you want to see the japanese translation". People travels a lot these days... (which doesn't mean geo-ip doesn't have its use, just it's not to override the user's language preferrences)

And these are some of the reasons why I think we need simple 'language_fallback' module that doesn't do anything else, and then other modules using that fallbacks do to useful stuff.

Some feedback on #31

2/ Your fallback language is in essence the same language, but without a region/dialect specifier, you're not going to define en as a fallback for fr. This is about string translations, not about content translation, because that can be solved in other ways.

4/ You lost may here, how are you going to now the language if it isn't enabled in Drupal, or you talking about browser language?

5/ Browser fallback (or actually mapping) is already possible in D8 thanks to #365615: Language detection not working correctly for most Chinese readers (and add a user interface for all browser language mappings), this allows you to map any browser language to a Drupal language. And you're right sites should stop trying to determine your language from ip addresses.

The main problem with using locale_custom_strings_ for this, is that we will be duplicating a lot of code from locale, we have a working implementation, but I need to see if the performance is acceptable.

@attiks,

2/ ... you're not going to define en as a fallback for fr. .... This is about string translations, not about content translation, ....

That's the point I'm trying to explain (may have failed though): I want a module just defining language fallbacks not making any assumption on what it's going to be used for. (Then the string translation module can use it for its business)

4/ You lost may here, how are you going to now the language if it isn't enabled in Drupal, or you talking about browser language?

I would like to be able to map a language, from wherever it comes (browser, web service..) to a language actually defined/configured/enabled in my Drupal.

5/ Browser fallback (or actually mapping) is already possible in D8 thanks to...

That's cool, I didn't know about this one, thanks. Nice, we've got an UI to reuse for any other language fallback :-)

Now, why can't this be used too (though it would need to be a bit repurposed and moved to language module), to define
generic language fallbacks, that can be used for any other thing?

The main problem with using locale_custom_strings_ for this...

I'd like locale_custom_strings to be moved out of core. If the "introduce unified interfaces" patch lands we don't need it anymore in Drupal core.

#33

2/ + 4/ I think we're talking about the same thing and we probably can abstract the browser mapping interface to be used for other things as well, now we only need someone to do it ;-)

FYI: I created http://drupal.org/project/language_fallback

Hi attiks,

Thanks for creating a 'real' project through a refactoring of my code. I never got around to doing that, and I'm glad that there is a felt need to do something with what I had started to put together. I didn't realise 'locale_custom_strings_XX' could be set using variable_set() - this is much cleaner than setting values in $conf using hook_boot(). Storing the value of the fallback language code in the object rather than separately in the variables table is also a great improvement that is made possibly by doing it this way.

You've removed the cascading of fallbacks, which would mean it would not be possible for, say, one dialect of German to be a minor variant of another dialect of German which itself is a major variant of standard German without having to fully populate both the dialects with almost identical information. Was this more of a performance or a principle based decision?

When I found myself manipulating these objects to mimic multi-dimensional arrays, I found that it was blowing both my own mind and also seemed difficult to grasp for most of the people I tried to describe it to. In particular I was asked to comment copiously on why I did things like just returning [NULL] for the offsetSet() and offsetUnset() functions in these two object implementations. Obviously it makes perfect sense to do this in this situation but people coming in and trying to understand the code might not immediately get it. So - even if my level of commenting was excessive - I'd suggest including at least a bit more explanation than is currently in the code you have written.

I'm happy to be a co-maintainer of this in D7 if needed/wanted.

@martin_q: I created an issue in our project. See #1877880: Cascading of fallbacks