I've just been doing some thinking about how my project will make the best use of localise.drupal.org etc in Drupal 7.

Up till now, it has been possible for Drupal site admins to choose the language code to represent a language on their system. This is a good thing, since the two-letter code that is mainly used is not enough to cover the approx 6000 languages in the world.

Typically in my project we will use the three letter code defined by ISO 639-3 (aka 'Ethnologue code', from the Ethnologue, an authoritative source on the languages of the world: http://www.ethnologue.org/codes/default.asp ). We will be hosting some of the first ever online material in a number of languages, so I expect we will also be making the first contributions to localise.drupal.org for those languages. However, we will also be making use of some more major languages and I plan for us to be contributing to localise.drupal.org what our localising staff provide in those languages too.

I'm thinking that I will automate, or semi-automate, the process of submitting to ldo the localisation strings that we have sourced. I assume therefore that I will need to make a mechanism for specifying which language on ldo the strings taken from a language on my site refer to (i.e. to map 'fas' on my site to 'fa' on ldo, and so on).

But where will the mechanism be for specifying the language relationship on the download client that we worked on at the sprint? What I'm saying is that if I create 'fas' on my site, I want to be able to tell the system to go and download strings from the 'fa' section on ldo. I don't recall that we planned to allow for that kind of mapping.

This is making me wonder whether it is time for Drupal to fix an authoritative list of language codes for all the languages of the world. The ISO 639-3 specification is an obvious candidate, but there may be others. In any case, if there was a system which covered all the languages that were needed, then there would be no need for any mapping between local and ldo language codes.

What do you think?

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Gábor Hojtsy’s picture

Drupal is really driven by browser's notion of languages. The reason being is that whetever meta information we output about languages or whatever language negotiation/selection we do, the browsers give us 2 letter language codes with optional country/region suffixes. While you can easily switch your own site to 3 letter language codes, Drupal lacks the mapping ability to interface with browsers or the localization files (currently shipped in .po files) to map to your 3 letter codes.

martin_q’s picture

Agreed - it lacks that mapping ability!

OK, I can see there is a case for the existing system, so I understand that this will probably persist.

But Drupal core does allow site admins to change those codes on their own site, so isn't that mapping ability going to become more and more necessary, somewhere in the process? Otherwise allowing the codes to be changed will reduce the value of l.d.o - sites that change the codes will be incompatible with the localisation server.

So either:
- changing the codes should be prohibited (not ideal)
- or the localisation server is rendered less useful (not ideal)
- or some kind of mapping is required

...or am I missing something?

Gábor Hojtsy’s picture

This ability to set a custom code was there since we have a web UI for language setup. In fact, for a long time, there were no predefined language codes to choose from if I remember right. So I'd say that this use case did not come up yet, but was the same problem with .po files not being imported in Drupal 6 if you had a different language code compared to how the po files were named coming from CVS. It is not a new problem with l.d.o.

Pasqualle’s picture

I think this functionality belongs to a contrib module, and not an issue for l.d.o.
The simple question: what happens if someone tries to switch the language code at translation export, import and remote submission? Is it even possible without hacking core (and related contrib modules)?

Gábor Hojtsy’s picture

Project: Drupal.org infrastructure » Localization update
Version: » 6.x-1.x-dev
Component: localize.drupal.org » Code

Once you set up a language in Drupal, changing the language code is prohibited. At the start though, you could set up languages with any language code needed. For the client mapping, I believe this issue belongs to l10n_update module. But I'm not sure it will be high priority.

Jose Reyero’s picture

Priority: Normal » Minor

Agree with Gabor this can be a nice client side feature, though not hight priority atm.

Also we don't want to clutter the UI, so I'm thinking maybe a patch using a hidden variable for language mapping would be acceptable.

Gábor Hojtsy’s picture

Title: Localisation, minority languages and 2- vs 3-letter language codes » Map local language codes to remote language codes

Retitling.

Gábor Hojtsy’s picture

Ludo.R’s picture

Subscribing.

I think this feature would enhance the custom languages.
In my case, i would like to use the language-COUNTRY code like 'en-US' 'en-UK', 'fr-FR' or 'fr-BE'.

But i cant at this moment because my custom languages dont get their "primary" translations.

penyaskito’s picture

I'm interested on this issue.

Any hints for working on a patch?

penyaskito’s picture

Ok, I have something working for my use case.

I'm defining my languages using codes as #580260-9: Map local language codes to remote language codes says: es-CO for Colombia Spanish, fr-BE for Belgian French and so on.
If the language es-CO file is not found, we try to find the 'base' language files (es).

I don't know what's the better way of making this more generalist. Maybe add a field at the language creation form for adding a 'base' language?

penyaskito’s picture

Status: Active » Needs review

Forgot to change status.

Jose Reyero’s picture

Version: 6.x-1.x-dev » 7.x-1.x-dev
Status: Needs review » Needs work

Though penyaskito's patch is some improvement I don't think it's generic enough to handle any language mapping and also it makes too many assumptions about which is the default. It may not always be a regional variant.

A quick solution could be using some hidden variable -we can build an UI later- and we also need to keep track of whether we've already downloaded the needed translation. For instance if we've already downloaded 'es' for 'es_ES', we don't download it again for 'es_CO', just reimport it again. This 'es' language should be at least a disabled language so we can handle it properly.

When sending back translations to the server the system needs to be aware of these mappings too.

Setting version to 7.x which is where we should be adding new features, maybe to backport later.

Sutharsan’s picture

For reference: In #347351: Enable translation fallbacks a core solution for translation fallback was prepared.

torotil’s picture

Here is a re-implementation of comment#11 for recent l10n_update versions.

Sebastien M.’s picture

Version: 7.x-1.x-dev » 7.x-2.x-dev
Status: Needs work » Needs review
FileSize
8.15 KB

This patch allows to specify for each language the language code to use to build remote or local uri.
A new field has been added at the end of language form edit (and add) and display the new language code in the language table summary.
This patch has been created using the "7.x-2.0+9-dev" release.

Sebastien M.’s picture

Can someone review my patch ?
Many thanks

Sutharsan’s picture

Issue tags: +Needs re-roll

Thanks for the patch, but it no longer applies.

Sebastien M.’s picture

FileSize
8.19 KB

Patch updated based on the current DEV branch.
Thanks

Sebastien M.’s picture

Issue tags: -Needs re-roll
Sutharsan’s picture

Status: Needs review » Needs work

Sorry, for responding so late. This approach is much too complicated in my opinion. What about a simple text area for the mapping?

Sebastien M.’s picture

In my opinion "What about a simple text area for the mapping?" is not enough user friendly.
I've done something like in D8.

Sutharsan’s picture

The user is the system administrator, usually the site builder. The same who enters key value pairs for select list fields.

Sebastien M.’s picture

Except for field "list", I never saw such UI to specify corresponding values.
That's why I worked to enrich the language form by adding a new field to specify the matching language code for update.
From my point of view, a "simple textarea" is an awful approach.

torotil’s picture

FileSize
732 bytes

Here is an update of #15 for the 1.2 release. I’m not sure whether I will get around to move all sites to the 7.x-2.x patch before the D7 EOL.