split contrib translations into their own directories and projects

merlin and i were discussing the problem of how translations get packaged inside the corresponding contrib modules. this makes it nearly impossible for translators to provide working translations inside the contrib releases (since contrib maintainers never coordinate their releases with the translators).

furthermore, not everyone who downloads a given contrib cares about all the translations of that contrib. the use case for translations seems to be: "i'm building a site and i want the interface in language xy... is there a core translation? what contribs have xy translations?". not "i'm building a site using views, i wonder what languages it's translated into?...".

the solution to both of the above problems (plus the original one in this thread) is simple: split out the translations of each contrib into their own projects, with their own directories in CVS.

a) each language/contrib pair gets a unique project node.

b) we'd use project taxonomy stuff to group the core translation and any contrib translations for a given language together. so, just like we've got modules/category for grouping related modules, we'd have translations/category for each language. all the download pages would work seemlessly, with no changes in code. we'd just have to create a new taxonomy term for each language, and associate all the existing translations with the right term (which we would automate). then, there'd be all the built-in goodness of taxonomies (RSS feeds, specific pages, etc) for all translations (core and contrib) for a given language (not to mention the "browse by category" downloads pages).

c) we'd still need to move things around in CVS, almost exactly as killes suggests. i'd just propose that core translation files are all in their own subdirectory, too. either called "core" or "drupal" (to match the drupal core project name), like so:

translations/xy/core/core_po_files
translations/xy/contributions/modulename/po_files

or, perhaps:

translations/xy/drupal/core_po_files
translations/xy/contributions/modulename/po_files

i think this would be more clear than having everything in the root of /translations/xy, and it'd vastly simplify the problem of the packaging script. keeping each contrib in a separate project with a different directory (not directly under the core translation files) means that each translation can be tagged/released/packaged independently, instead of trying to do it all within the core translation. as killes points out, this is a nightmare since core and contribs use different version numbers and different CVS branching/tagging conventions. if we do it my way, the translators can officially release the xy-views-5.x-1.3 translation of views 5.x-1.3 whenever the translation to xy is complete.

i'll grant this will make a lot more project nodes. however, we've already got a ridiculously large number of projects, and the monolithic project dropdown in the issue queues is already basically unworkable. so, we need to fix that UI regardless, and if separate projects for translations help push these improvements, all the better. with a sane, enforced naming convention for directories in cvs and the corresponding project names, i don't think it'll be a real disaster. basically, the "project short name" should be "xy_modulename". we could add some custom validation and form_alter() stuff to make it required (that'd be trivial, and i've been meaing to add a project_druapl_org.module for such local customizations, anyway).

i also suspect some translators might get confused by the project node taxonomy UI, but i've got plans to make that better, too. again, if this task is what drives the improvements, great.

finally, for end-users, it'd be nice to provide "meta-packages" of all the latest 5.x translations for a given language. this is *exactly* the same problem as the installation profiles/distribution question. so, again, if the translations help push the infrastruture to support this kind of operation, all the better, since we'll then have the functionality for the distros...

needless to say, all these separate project nodes mean fine grained CVS access control. it'd also mean separate issue queues for the contrib translations themselves (which could be useful, e.g. "sorry, that should be 'tchau', not 'tchaou' in line 123 of the pt-br translation of views-5.x-1.3...").

it also provides translators the ability to release their translations of popular contribs when they're ready, instead of being at the mercy of when the contrib maintainer tags/releases.

if we're going to be moving stuff around in CVS, changing the packaging scripts, etc, etc, i'd just as soon go all the way and solve all these other problems while we're at it.

I really like this solution(s). One of the problems I've now is that sometimes I translate a module and I can't even send the translation the the project directory because I don't have access to it.

With this solution the translating project would have more flexibility to add the translations and users in my language would know which modules are already translated to my language.

ok, great, people agree this is would work and solve a bunch of problems. :) changing title to reflect the new scope.

in my mind, the only open question is: what should we use for the taxonomy terms for each language?

a) english version of the language name: "Turkish"
b) native character set version of the language name: "Türkçe"
c) country-code: "tr"
d) a+b: "Turkish (Türkçe)"
e) b+a: "Türkçe (Turkish)"

my vote would probably be (e). thinking of how http://drupal.org/project/Translations will look when it defaults to category-sorting, maybe we should use "Türkçe [Turkish]" as the taxonomy term, so that the page looks like:

русский [Russian] (2)
Türkçe [Turkish] (3)
...

hehe, though i wonder how well the ORDER BY is going to work on those taxonomy terms. ;)

any other suggestions?

the only other question is what subdirectory the core translations should be moved into -- "core" or "drupal"? i could argue either way...

thanks,
-derek

First of all, I think the new directory structure makes sense. I’d choose core for the Drupal core translation files.

I am ok with e if you figure out how to order by the English name.

Is there a reason why we can’t sort on the localized name? There are at least three different sorting schemes (english full name, localized name, language code) and the one we chose is pretty indifferent. Users will find their translation even if the page is in random order.

Good ideas! I would vote for core translations being in the 'drupal' directory, and (d) for presentation/sorting, since this is how languages are displayed on the locale module interface (well, there the 'original' language names are not English, but translated to the current locale, but you get what I mean probably).

Sure, projects and project nodes will explode, but fortunately translations will increase too. If there won't be standalone projects for contrib translations, there won't be matching translations for their releases. Some logic would be nice in place to merge the corresponding translation files with the module releases, so that people still download all at once if need be, and need not search for the n number of downloads (module download and downloads for each language) separately.

We should also keep in mind theme translations, since themes could and do have translatable strings. Install profile translation is another issue, but since install profile hosting is not supported in drupal.org cvs as far as I know, this is just something to think about for the future still.

We should also keep in mind theme translations, since themes could and do have translatable strings. Install profile translation is another issue, but since install profile hosting is not supported in drupal.org cvs as far as I know, this is just something to think about for the future still.

ok, good point. my new directory layout proposal:

translations/xy/drupal/core_po_files
translations/xy/modules/modulename/po_files
translations/xy/themes/themename/po_files
...

basically, take "contributions" out from under the language, and just mirror the directory layout for the rest of the repository. if we ever add install profiles/distributions, it'll fit right in... this also matches what people see at http://drupal.org/project (the downloads tab).

re: sorting by (d): yeah, that seems to be the best for our needs. lots of our languages don't even have a native "translation" of their language name on their project pages, and not all languages need a separate charset, etc. so, for now, having the english version first, and optionally including the native version, seems best. glad to hear it's also consistent with the locale UI internally. ;)

subscribing

Gabor is supposed to be working on this for his SoC project (see http://drupal.org/node/128616). I hope he assigns himself to this task at some point soon. ;)

Well, sure. I have written you (as well as my other mentors) some thought provoking mails, but it seems it did not arrive. Will try to resend them to your drupal.org user email address.

Hi, I am a translator and just have a few questions/comments hope you can consider， thanks.
1. a) each language/contrib pair gets a unique project node.

Will these projects be auto-generated or require translators to create them manually one by one? How does the ownership work, each project has its own maintainer? I'd prefer core translation project maintainers have access to all contrib projects of that language.

2. How will contrib translation version work with module version? Keep them the same all the time? If a module has a new release, but no string changes, should the translation project follow suit also submit a new release just to keep up with the version number? And is there an easy way to check if a module's new release involves string changes?

3.
+1 to Gabor's comment on packaging contrib translations back to the contrib module download, so users only need to download once.

4. On taxonomy terms, I'd prefer option d (i.e. sorting on English words).

Posted some deep thoughts on the matters here: http://groups.drupal.org/node/4531 (I am not entirely sure I should copy it here, as it looks into different directions, so please comment there for now).

Just wondering... what is the current status of this? You might want to have this sorted out (if it isn't already, I'm not entirely sure. :x) by D6rc1, so translators know what to do. :)

There will be an online translation server, but it will definitely not be ready by RC1. Some teams already use this server though. Look at http://drupal.org/project/l10n_server

Changes in D6:

- there will be a 'translations' folder instead of a 'po' folder, D6 contribs need to accomodate to this
- the core templates will be different (ie. different file names and string groupings), but advice will be given on how to merge old strings

So the directory and file structure is going to stay as it is for D6, only replacing "/po/" with "/translations/"? Contrib will still use lang.po? Should this issue then be marked wontfix? Or will the structure change with the release of l10n server?

Although this is going slightly off topic: (Some) translation teams are running l10n server on their own sites then? But the plan is (still) to, eventually, have it running on g.d.o for all languages, right?

The server will only be stable and will only have a good UI if actual people use it for actual translation work. That's what a few teams (maybe parts of the teams?) do. So I get testers, feedback, requests, the module gets better, more performant, etc. The guys found bugs which would have brought down the central server so better now then later :)

The plan is to run a central server, and move all translation work out of CVS for the time being. So the move from /po to /translations would not have been needed at all, if all would have been ready in time. Unfortunately I was swamped with Drupal 6 and other stuff, so it is not ready yet. I am active in the issue queue though, bugs get fixed and I have nicer plans for the UI, so it will get better. The central server would sync source code with the releases and would push translation packages for download to drupal.org. The sync part is close to ready, translation packages are generated, but their placement on drupal.org and their exact structure and scope is not yet defined.

So there is work on fixing bugs, improving the UI and improving drupal.org integration. Once these are adequately covered, the localization server setup will be pushed to somewhere at *.drupal.org and translation files will not be in CVS.

Just to be completely sure:
So the directory and file structure is going to stay as it is for D6, only replacing "/po/" with "/translations/"? Plus some changes in how core .po's are named.
Contrib will still use lang.po? As in da.po, as opposed to module.po (e.g. l10n_server.po).
Should this issue then be marked wontfix? Or fixed or bydesign, depending on how you see the issue.

1. Yes, contrib file placement is the same except the './po' subdirectory renamed to './translations'. So as long as we use CVS still, Drupal 6 ports of projects should have their .po files in './translations'. Projects with submodules should have their submodules in subdirectories (they will have .info, .pages.inc, admin.inc, .tpl.php etc files anyway, if following the core lead). All submodules can have a './translations' subfolder. In fact they should have their strings distributed. Common strings among the submodules should be organized into the .po file of the root module of the project (ie. which is always enabled). This way when you install the root module, you get the common strings and when you install more submodules, only then you get those translations loaded. Core translation packages will also be organized to place their translations directories under each module, theme and install profile.

2. Po files are loaded by language name, so you need to end your PO file name in "$lang.po". You can have an arbitrary prefix though. So you can have l10n_server.it.po for Italian, or it.po. The use of prefixes comes in handy when the it.po file required by the module would be too big in itself (ie. the module has too many stuff to translate in its .admin.inc, .pages.inc, et files). Because importing big files might time out, you can break out the .po files to modulename-module.it.po, modulename-admin-inc.po, etc. This is not required, and will likely not happen with anything but the biggest modules. However, it is fully supported.

3. Well, it is not fixed yet. It is by design for now, but it will be fixed with a different approach later :) Because the approach is different, this issue is probably best left by-design. I'll try write up a doc about Drupal 6 translations soon.