Add drush make command for pulling core installer translations from l.d.o

On A) core translations will probably have translations releases, but that is not going to happen soon. Until then, we need to package these on a best effort basis, pulling the translation at the time of packaging IMHO.

On B), there will be no "tarball", for a defined core version per language, since we are talking about one .po file per language per Drupal version, so we do not package that further. It's just a .po file. We can make up a file naming scheme for that on l.d.o. The translations are hosted at http://ftp.drupal.org/files/translations/ at the moment in a sensible directory structure. Only core files are produced at the moment.

For C), we have a languages.xml file in the root of the translations files (link in the point above), and that lists all languages available. Then the packaging script can traverse the file structure looking for the known filenames as we define for B) and look whether any of those are there. Package what's available.

What needs to happen to make this feature a reality?

D) in the drush make workflow, core translations would normally go in a distribution build .make file, not a profile .make file, since they need to be installed relative to the drupal root, not the profile root.

however, the drupal.org packaging system fully controls the distribution build .make file for drupal core. therefore we'd need another customization in drupal-org.make files as opposed to standard .make files -- namely, that they would include a 'translations' array that the packaging system would look for, and use to insert the necessary translations into the distribution build .make file that it constructs.

with that approach, we'd also need to make an adjustment to drush_make_d_o.drush.inc and drush_make_d_o.convert.inc to add support for handling the new translations array element in drupal-org.make -- otherwise the .make file would fail to verify/build.

I'm working on getting drush dl command to support packaged install profiles (please can we call them distributions soon). We really want to get localization working under plain old `drush dl` and `drush enable` as well. Subscribe.

*bump* any update on this?

A) That's ok, we can accept that (we can't do better).
B) Out of the scope of this issue (obviously), but still needs to be solved.
C) is unnecesarry.

kika: basically we'd probably just need some code which parses translations[] = foo

*nudge*

Not really sure what to say - the next step is basically just to do it.

subscribe

subscribe

Sorry for hijacking the topic -- I do not know where this discussion belongs.

I feel this issue is dependent on high-level decisions on future of the Drupal translation packaging. l.d.o is great effort for getting community together, but we need to decide the packaging method and format.

There seems to be two main ways to go:

OPTION 1. Rely on our VCS (CVS, soon Git) and allow doing (semi)automatic language commits based on l.d.o database/files. Something like a button [publish a release] in l.d.o what language admins can use when they feel doing it.

This launches a script in the background what commits language files to standard translations VCS repo. For contrib modules it's trickier: should the script package as patch in the issue queue of respective module?

Non-automated version of the same flow is going on right now.

Pros:

- drush / make already support language projects downloads (although it's buggy #639158: Download translations extracts just a single file) and does not need reengineering. drush make for d.o just needs to accept project[fr] = 1.2. This way installer profile can import all required core .po files, including profile/*profilename* ones.
- all language-packing scripts what convert .po flat package format to .po directory format will stay the same
- Old-school manual VCS commits can still be used

Cons

- Can automatic commits be done from the security perspective? Also consider upcoming Git migration.
- Does not really solve two critical translation distribution problems:
-- asymmetry between contents of VCS (flat .po) and final packaging (directory .po) format (there seems no way to get your translations from (other) repo via drush make CVS/SVN/Git backend?)
-- Core and contrib translation/distributing workflows are really different.

OPTION 2. Keep translations off the VCS, modify drush / make tools to get the translations from elsewhere, with non-standard package signatures and version handling.

Pros:

- Lot of new opportunities to simplify our messy translation distributions:
-- simpler-to-grasp and symmetrical packaging format (flat .po files with version, project and language info in filename OR .info files for translations?)
-- re-think core vs contrib translation packaging flows

Cons

- Time and resources and backward compatibility (this does not feel being in D7 core timeframe at all).

I'd go for Option 1 for now but keep discussing about translation packaging future.

subscribe

suscribe

subscribe

subscribe

Here is an initial patch modeled after the work in l10n_update and using localize.drupal.org as a default l10n server for downloading translations.

• Translations can be downloaded for any given makefile using the --translations=ar,es flag (where a comma-separated list of languages is provided). Any project (core, profile, theme, module) with sufficient metadata (core version, project name, version) can have its translations downloaded and placed in its translations/ directory.

• Projects can make use of either the l10n_path or l10n_url keys in a makefile to specify an alternate localization server if they choose to (see l10n_update for more information on what these keys should be). By default, localize.drupal.org is used.

• README.txt updated and a translations.make test makefile and test md5 hash entry have been added.

Bad copy & paste in README. Rerolled patch with fix.

Better failure and smarter logic for when to not check for translations.

you are awesome

why _url and not _server?

w00t

This seams to import the translations into drupal-root/translations and is not available during the install precess, or am I missing something?

Correct, as I stated above the translations are downloaded into the translations<code> of each project. For a module, this would be for example <code>sites/all/modules/token/translations and for drupal core this is just translations.

This may be an issue as the installer will only offer language options if there are translations in profile/[profile]/translations... I will look into whether downloading translations to both translations and profile/default/translations will work for Drupal core.

Core translation file need to be placed in core module folders, too!?

I've committed a workaround for this issue here:

http://drupal.org/cvs?commit=416508

To summarize the challenges:

• localize.drupal.org is currently treating all of Drupal core as a single translation project (e.g. http://localize.drupal.org/translate/languages/fr/translate?project=drupal) and all of the translations for Drupal core are packaged into a single .po file. (For what it's worth, I think this is a good thing).
• The Drupal installer, however, will not expose a language as available during install unless it lives in profiles/[x]/translations. What's worse, the translation file must be named fr.po, not drupal-6.19.fr.po (the latter is an acceptable format for all other project translation files).
• There is some interesting handling of language translations during install. In particular, the profiles/[x]/translations file is imported very early on and affects strings during the install process, while strings in, say, modules/system/translations only take effect after installation is finished.

The current workaround is this:

1. Download Drupal core translation from localize.drupal.org

2. Create two copies of the .po file for the core project, e.g.:

   profiles/default/translations/[language].po
   modules/system/translations/drupal-6.19.[language].po
   

   This second copy allows other l10n_update or another install profile to properly import Drupal core strings.

The main problem with this approach is

1. It's confusing.
2. It's wasteful. profiles/default/translations would ideally contain only the strings needed to get through the install, not a full copy of all strings from Drupal core.

I think the proper solution for this is to have the Drupal installer make an exception for the core project and look for core translations at translations at the Drupal root.

Looking for feedback on this one.

localize.drupal.org is currently treating all of Drupal core as a single translation project (e.g. http://localize.drupal.org/translate/languages/fr/translate?project=drupal) and all of the translations for Drupal core are packaged into a single .po file. (For what it's worth, I think this is a good thing)

No, this is a very BAD idea. You missed a few more important things.

1. Gabor have build the po importer to run small chunks of translatable strings. Why? Very simple! Many cheap hosters have run time limits. Big monster files will fail importing if you have only 30s. We have seen this with russian core translation import very first 3 or more years ago. I have not seen this very often myself as I have a timeout of 240s and only imported German, but others have also complained failures in German if time windows are small. This IS the MAIN reason why we have the BATCH functionality and why autolocale 5.x have never been released officially!
2. If you have a file per module like we have in translation releases - you have the small files in the modules folders + a few in systems module folder ("deep" core). I do not enable all modules of my core installation. Very often many modules will never ever been enabled on a system. We do not like to clutter the translation database tables with data that never get's used. There have been some speed tests made in past where the number of existing strings have compared to page run time and it was shown that less strings are much better.
3. Memory usage use much lover if smaller files are used. There are cheap hosters with limitations...

The only reason why you lay a monster file in a drupal system module folder having all translatable strings is only that there is no button for re-importing all files and if you do not have this button you need to select every single modules file .po by hand. This is bad and I complained in many places, too, but this is part of l10n_client (however i wish this would be in core). Overall - this is no reason to make one big file if this will cause major import failures to many many people.

Can someone explain why the splitted download is no longer available on l.d.o? In past it was... but now the selection form is no more accessible as in past.

I agree with your points about performance and problems with expensive import processes.

If I understand correctly localize.drupal.org is meant to mirror Drupal.org projects and releases, and since Drupal core is a single project this is why the translations for the project have been consolidated into a single translation project as well. Whether it will be possible to package these translations into distinct module .po files is a question for Gabor and Jose I believe.

Let's see if we can get them to look into this issue and give us some direction here.

#30: Split downloads are still available at http://localize.drupal.org/translate/languages/[langcode]/export

You need to be a member of the respective translation team though - anonymous access doesn't seem to exist anymore.

subscribing

subscribing

Nice to see the --translations=.. working already in drush_make 6.x-2.0-beta9.

Any progress on getting support for "translations[] = nl" included?

projects[] = nl

dmitrig01: Thanks, seems very logical :)

But maybe it should be more explicit in the documentation, what about this patch.

Subscribing.

Subscribe

I'm going to make this issue as fixed for now, and if there is an issue standing (hass?), then another can be opened.