This patch extends the l10n_packager module for packaging all projects/releases per language, works , on cron, has some admin ui, etc, etc..

Just in case, here's the full module,

https://devseed.svn.cvsdude.com/sandbox/drupal-6/l10n_tools/l10n_packager/

It is a remake of our old l10n_download module for l.d.o

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Gábor Hojtsy’s picture

Status: Active » Needs review

Marking as needs review. However, there was a pre-existing issue at #568994: Implement translation package export in l10n_packager which outlined some requirements, so not sure whether to mark that as duplicate or this as duplicate and ask to move the patch there.

Gábor Hojtsy’s picture

Status: Needs review » Needs work
FileSize
25.73 KB

I've applied and looked at the patch. Tried to clean up the code (missing CVS ID tags, bad phpdoc, etc), and found some issues:

- not all L10N_PACKAGER constants are used, so I'm not clear on the purpose of them (I've even removed a pointless one from the top of the admin.inc)
- the language export is now only one of the packager tasks, so used more specific names there and moved to this new admin area
- localize.drupal.org needs a way to not run this on Drupal's cron but instead on its own cron from a specific machine. (this is explained in detail at #568994: Implement translation package export in l10n_packager), so I've added an option to disable running on cron; you've already had a way to disable cron by setting the interval to 0, but we'd need to be able to set the interval, but it will be applied to the non-Drupal-cron run

I've also fixed various smaller issues, but this still needs work:

- the purpose of all items is still not clear to me, needs more time to study
- the manual packager / mark to package feature's project selector is not consistent with the rest of the modules, and on the manual export, the release picker is a second form step (which has a patch suggestion to Ajaxize)
- all files are saved into the same directory without a further structure to help distribute the number of files (we have 33 languages already with 5000+ releases, so that is already 165.000 files, and we do plan to keep older release dumps there for some time, so this can grow pretty much; also we could grow quickly to 100 languages)

Looks generally good, but needs more cleanup.

Jose Reyero’s picture

Status: Needs work » Needs review

Done some improvements over the latest one:

- Export directory is now either relative to Drupal install or absolute, so this is more flexible and doesn't need to be inside site files anymore
- Added a configurable file path (and file name). There are some variables now to build a directory tree and to name the file.
Example (default): '%core/%project/%project-%release.%language.po'

Hey Gabor, if you think this looks good, can we just commit it and move on with smaller patches?

I know there are some more things we need to do, like keep track of which is the latest release for repackaging and think about #595644: Thinking about the release / translation workflow but I think in current state you could already do some manual packaging and see how the whole thing works...

Jose Reyero’s picture

FileSize
27.63 KB

Sorry, forgot to post the patch

Gábor Hojtsy’s picture

Tagging.

Gábor Hojtsy’s picture

Status: Needs review » Fixed
FileSize
29.93 KB

Ok, worked a lot more on this patch:

- sticked the export to all-in-one format, since the file name template was forcing that anyway; also set it to use the compact format for export, since Drupal can import that and it takes way less disk space / bandwidth
- found a NOTICE issue in the compact export due to the above
- figured that your last change code was a bit overcomplicated; turns out the l10n_community code was in error of not always setting the uid_approved and time_approved; it was only set when an existing previous string was there but a new one was added not when a new one was added fresh (@todo: needs update function before deployed)
- moved settings to not be the default packaging tab
- packaging form now contains both language and project rebuild forms at once
- fixed project autocomplete description and code to properly handle that a title is actually input there
- modified schema to have a "status" column instead of "package" to be more clear on what does that mean
- other minor changes like use of a nonexistent $download variable where $files should have been used

We definitely need to do in the short term:

- have an outside CLI packager, so we can make the packaging independent of the Drupal cron run; drupal.org needs to have this running only on one of the webnodes (where the filesystem is local) and also, this will take huge amounts of time, so we need to possibly run it more often initially at least
- add upgrade path to l10n_server to fix the badly recorded uid_approved and time_approved records so the current latest time SQL will work properly; something like UPDATE l10n_community_translation SET uid_approved = uid_entered, time_approved = time_entered WHERE is_suggestion = 0 and uid_approved = 0; basically solving the problem that initial "instant" translations (those which were never suggestions originally) were recorded without approval data while further instant translations had this approval data

I'm highly interested also in:

- writing tests for the packager
- possibly renaming the 'checked' column, and/or try to better document what it means exactly
- fix the compact export mode to not export empty translations since that could help to reduce the file size again dramatically for incomplete languages

I'm interested also in:

- fixing the project / release selector to use the Ajax code suggested for the export screen (we should fix this at both places)

Because the attached patch is now committed (and all existing tests still pass), we can close this down and work on the above in follow up patches. Thanks again for your continued help!

Gábor Hojtsy’s picture

FileSize
885 bytes

Here is the update path I'm adding in.

Gábor Hojtsy’s picture

Results of the first run (which I took the liberty to run via a browser, so it was slower then just running from www1 probably):

The directory sites/default/files/l10n_packager/5-x has been created.
The directory sites/default/files/l10n_packager/5-x/drupal has been created.
Repackaged release drupal-5.19 for Afrikaans. Created file drupal-5.19.af.po.
Repackaged release drupal-5.19 for Amharic. Created file drupal-5.19.am.po.
Repackaged release drupal-5.19 for Armenian. Created file drupal-5.19.hy.po.
Repackaged release drupal-5.19 for Bosnian. Created file drupal-5.19.bs.po.
Repackaged release drupal-5.19 for Catalan. Created file drupal-5.19.ca.po.
Repackaged release drupal-5.19 for Chinese, Traditional. Created file drupal-5.19.zh-hant.po.
Repackaged release drupal-5.19 for Croatian. Created file drupal-5.19.hr.po.
Repackaged release drupal-5.19 for Czech. Created file drupal-5.19.cs.po.
Repackaged release drupal-5.19 for Danish. Created file drupal-5.19.da.po.
Repackaged release drupal-5.19 for Dutch. Created file drupal-5.19.nl.po.
Repackaged release drupal-5.19 for English (British). Created file drupal-5.19.en-gb.po.
Repackaged release drupal-5.19 for Estonian. Created file drupal-5.19.et.po.
Repackaged release drupal-5.19 for Filipino. Created file drupal-5.19.fil.po.
Repackaged release drupal-5.19 for Finnish. Created file drupal-5.19.fi.po.
Repackaged release drupal-5.19 for French. Created file drupal-5.19.fr.po.
Repackaged release drupal-5.19 for Greek. Created file drupal-5.19.el.po.
Repackaged release drupal-5.19 for Hebrew. Created file drupal-5.19.he.po.
Repackaged release drupal-5.19 for Hindi. Created file drupal-5.19.hi.po.
Repackaged release drupal-5.19 for Hungarian. Created file drupal-5.19.hu.po.
Repackaged release drupal-5.19 for Japanese. Created file drupal-5.19.ja.po.
Repackaged release drupal-5.19 for Kannada. Created file drupal-5.19.kn.po.
Repackaged release drupal-5.19 for Latvian. Created file drupal-5.19.lv.po.
Repackaged release drupal-5.19 for Lithuanian. Created file drupal-5.19.lt.po.
Repackaged release drupal-5.19 for Lolspeak. Created file drupal-5.19.xx-lolspeak.po.
Repackaged release drupal-5.19 for Malayalam. Created file drupal-5.19.ml.po.
Repackaged release drupal-5.19 for Mongolian. Created file drupal-5.19.mn.po.
Repackaged release drupal-5.19 for Nepali. Created file drupal-5.19.ne.po.
Repackaged release drupal-5.19 for Norwegian Bokmål. Created file drupal-5.19.nb.po.
Repackaged release drupal-5.19 for Norwegian Nynorsk. Created file drupal-5.19.nn.po.
Repackaged release drupal-5.19 for Persian (Farsi). Created file drupal-5.19.fa.po.
Repackaged release drupal-5.19 for Polish. Created file drupal-5.19.pl.po.
Repackaged release drupal-5.19 for Portuguese, Brazil. Created file drupal-5.19.pt-br.po.
Repackaged release drupal-5.19 for Portuguese, Portugal. Created file drupal-5.19.pt-pt.po.
Repackaged release drupal-5.19 for Punjabi. Created file drupal-5.19.pa.po.
Repackaged release drupal-5.19 for Romanian. Created file drupal-5.19.ro.po.
Repackaged release drupal-5.19 for Slovak. Created file drupal-5.19.sk.po.
Repackaged release drupal-5.19 for Slovenian. Created file drupal-5.19.sl.po.
Repackaged release drupal-5.19 for Spanish. Created file drupal-5.19.es.po.
Repackaged release drupal-5.19 for Tamil. Created file drupal-5.19.ta.po.
Repackaged release drupal-5.19 for Telugu. Created file drupal-5.19.te.po.
Repackaged release drupal-5.19 for Test. Created file drupal-5.19.test.po.
Repackaged release drupal-5.19 for Turkish. Created file drupal-5.19.tr.po.
Repackaged release drupal-5.19 for Ukrainian. Created file drupal-5.19.uk.po.
Repackaged release drupal-5.19 for Welsh. Created file drupal-5.19.cy.po.
The directory sites/default/files/l10n_packager/6-x has been created.
The directory sites/default/files/l10n_packager/6-x/drupal has been created.
Repackaged release drupal-6.13 for Afrikaans. Created file drupal-6.13.af.po.
Repackaged release drupal-6.13 for Amharic. Created file drupal-6.13.am.po.
Repackaged release drupal-6.13 for Armenian. Created file drupal-6.13.hy.po.
Repackaged release drupal-6.13 for Bosnian. Created file drupal-6.13.bs.po.
Repackaged release drupal-6.13 for Catalan. Created file drupal-6.13.ca.po.
Repackaged release drupal-6.13 for Chinese, Traditional. Created file drupal-6.13.zh-hant.po.
Repackaged release drupal-6.13 for Croatian. Created file drupal-6.13.hr.po.
Repackaged release drupal-6.13 for Czech. Created file drupal-6.13.cs.po.
Repackaged release drupal-6.13 for Danish. Created file drupal-6.13.da.po.
Repackaged release drupal-6.13 for Dutch. Created file drupal-6.13.nl.po.
Repackaged release drupal-6.13 for Estonian. Created file drupal-6.13.et.po.
Repackaged release drupal-6.13 for Filipino. Created file drupal-6.13.fil.po.
Repackaged release drupal-6.13 for Finnish. Created file drupal-6.13.fi.po.
Repackaged release drupal-6.13 for French. Created file drupal-6.13.fr.po.
Repackaged release drupal-6.13 for Greek. Created file drupal-6.13.el.po.
Repackaged release drupal-6.13 for Hebrew. Created file drupal-6.13.he.po.
Repackaged release drupal-6.13 for Hindi. Created file drupal-6.13.hi.po.
Repackaged release drupal-6.13 for Hungarian. Created file drupal-6.13.hu.po.
Repackaged release drupal-6.13 for Japanese. Created file drupal-6.13.ja.po.
Repackaged release drupal-6.13 for Kannada. Created file drupal-6.13.kn.po.
Repackaged release drupal-6.13 for Latvian. Created file drupal-6.13.lv.po.
Repackaged release drupal-6.13 for Lithuanian. Created file drupal-6.13.lt.po.
Repackaged release drupal-6.13 for Lolspeak. Created file drupal-6.13.xx-lolspeak.po.
Repackaged release drupal-6.13 for Malayalam. Created file drupal-6.13.ml.po.
Repackaged release drupal-6.13 for Mongolian. Created file drupal-6.13.mn.po.
Repackaged release drupal-6.13 for Nepali. Created file drupal-6.13.ne.po.
Repackaged release drupal-6.13 for Norwegian Bokmål. Created file drupal-6.13.nb.po.
Repackaged release drupal-6.13 for Norwegian Nynorsk. Created file drupal-6.13.nn.po.
Repackaged release drupal-6.13 for Persian (Farsi). Created file drupal-6.13.fa.po.
Repackaged release drupal-6.13 for Polish. Created file drupal-6.13.pl.po.
Repackaged release drupal-6.13 for Portuguese, Brazil. Created file drupal-6.13.pt-br.po.
Repackaged release drupal-6.13 for Portuguese, Portugal. Created file drupal-6.13.pt-pt.po.
Repackaged release drupal-6.13 for Punjabi. Created file drupal-6.13.pa.po.
Repackaged release drupal-6.13 for Romanian. Created file drupal-6.13.ro.po.
Repackaged release drupal-6.13 for Slovak. Created file drupal-6.13.sk.po.
Repackaged release drupal-6.13 for Slovenian. Created file drupal-6.13.sl.po.
Repackaged release drupal-6.13 for Spanish. Created file drupal-6.13.es.po.
Repackaged release drupal-6.13 for Tamil. Created file drupal-6.13.ta.po.
Repackaged release drupal-6.13 for Telugu. Created file drupal-6.13.te.po.
Repackaged release drupal-6.13 for Test. Created file drupal-6.13.test.po.
Repackaged release drupal-6.13 for Turkish. Created file drupal-6.13.tr.po.
Repackaged release drupal-6.13 for Ukrainian. Created file drupal-6.13.uk.po.
Repackaged release drupal-6.13 for Welsh. Created file drupal-6.13.cy.po.
Repackaged release drupal-5.20 for Afrikaans. Created file drupal-5.20.af.po.
Repackaged release drupal-5.20 for Amharic. Created file drupal-5.20.am.po.
Repackaged release drupal-5.20 for Armenian. Created file drupal-5.20.hy.po.
Repackaged release drupal-5.20 for Bosnian. Created file drupal-5.20.bs.po.
Repackaged release drupal-5.20 for Catalan. Created file drupal-5.20.ca.po.
Repackaged release drupal-5.20 for Chinese, Traditional. Created file drupal-5.20.zh-hant.po.
Repackaged release drupal-5.20 for Croatian. Created file drupal-5.20.hr.po.
Repackaged release drupal-5.20 for Czech. Created file drupal-5.20.cs.po.
Repackaged release drupal-5.20 for Danish. Created file drupal-5.20.da.po.
Repackaged release drupal-5.20 for Dutch. Created file drupal-5.20.nl.po.
Repackaged release drupal-5.20 for English (British). Created file drupal-5.20.en-gb.po.
Repackaged release drupal-5.20 for Estonian. Created file drupal-5.20.et.po.
Repackaged release drupal-5.20 for Filipino. Created file drupal-5.20.fil.po.
Repackaged release drupal-5.20 for Finnish. Created file drupal-5.20.fi.po.
Repackaged release drupal-5.20 for French. Created file drupal-5.20.fr.po.
Repackaged release drupal-5.20 for Greek. Created file drupal-5.20.el.po.
Repackaged release drupal-5.20 for Hebrew. Created file drupal-5.20.he.po.
Repackaged release drupal-5.20 for Hindi. Created file drupal-5.20.hi.po.
Repackaged release drupal-5.20 for Hungarian. Created file drupal-5.20.hu.po.
Repackaged release drupal-5.20 for Japanese. Created file drupal-5.20.ja.po.
Repackaged release drupal-5.20 for Kannada. Created file drupal-5.20.kn.po.
Repackaged release drupal-5.20 for Latvian. Created file drupal-5.20.lv.po.
Repackaged release drupal-5.20 for Lithuanian. Created file drupal-5.20.lt.po.
Repackaged release drupal-5.20 for Lolspeak. Created file drupal-5.20.xx-lolspeak.po.
Repackaged release drupal-5.20 for Malayalam. Created file drupal-5.20.ml.po.
Repackaged release drupal-5.20 for Mongolian. Created file drupal-5.20.mn.po.
Repackaged release drupal-5.20 for Nepali. Created file drupal-5.20.ne.po.
Repackaged release drupal-5.20 for Norwegian Bokmål. Created file drupal-5.20.nb.po.
Repackaged release drupal-5.20 for Norwegian Nynorsk. Created file drupal-5.20.nn.po.
Repackaged release drupal-5.20 for Persian (Farsi). Created file drupal-5.20.fa.po.
Repackaged release drupal-5.20 for Polish. Created file drupal-5.20.pl.po.
Repackaged release drupal-5.20 for Portuguese, Brazil. Created file drupal-5.20.pt-br.po.
Repackaged release drupal-5.20 for Portuguese, Portugal. Created file drupal-5.20.pt-pt.po.
Repackaged release drupal-5.20 for Punjabi. Created file drupal-5.20.pa.po.
Repackaged release drupal-5.20 for Romanian. Created file drupal-5.20.ro.po.
Repackaged release drupal-5.20 for Slovak. Created file drupal-5.20.sk.po.
Repackaged release drupal-5.20 for Slovenian. Created file drupal-5.20.sl.po.
Repackaged release drupal-5.20 for Spanish. Created file drupal-5.20.es.po.
Repackaged release drupal-5.20 for Tamil. Created file drupal-5.20.ta.po.
Repackaged release drupal-5.20 for Telugu. Created file drupal-5.20.te.po.
Repackaged release drupal-5.20 for Test. Created file drupal-5.20.test.po.
Repackaged release drupal-5.20 for Turkish. Created file drupal-5.20.tr.po.
Repackaged release drupal-5.20 for Ukrainian. Created file drupal-5.20.uk.po.
Repackaged release drupal-5.20 for Welsh. Created file drupal-5.20.cy.po.
Repackaged release drupal-6.14 for Afrikaans. Created file drupal-6.14.af.po.
Repackaged release drupal-6.14 for Amharic. Created file drupal-6.14.am.po.
Repackaged release drupal-6.14 for Armenian. Created file drupal-6.14.hy.po.
Repackaged release drupal-6.14 for Bosnian. Created file drupal-6.14.bs.po.
Repackaged release drupal-6.14 for Catalan. Created file drupal-6.14.ca.po.
Repackaged release drupal-6.14 for Chinese, Traditional. Created file drupal-6.14.zh-hant.po.
Repackaged release drupal-6.14 for Croatian. Created file drupal-6.14.hr.po.
Repackaged release drupal-6.14 for Czech. Created file drupal-6.14.cs.po.
Repackaged release drupal-6.14 for Danish. Created file drupal-6.14.da.po.
Repackaged release drupal-6.14 for Dutch. Created file drupal-6.14.nl.po.
Repackaged release drupal-6.14 for Estonian. Created file drupal-6.14.et.po.
Repackaged release drupal-6.14 for Filipino. Created file drupal-6.14.fil.po.
Repackaged release drupal-6.14 for Finnish. Created file drupal-6.14.fi.po.
Repackaged release drupal-6.14 for French. Created file drupal-6.14.fr.po.
Repackaged release drupal-6.14 for Greek. Created file drupal-6.14.el.po.
Repackaged release drupal-6.14 for Hebrew. Created file drupal-6.14.he.po.
Repackaged release drupal-6.14 for Hindi. Created file drupal-6.14.hi.po.
Repackaged release drupal-6.14 for Hungarian. Created file drupal-6.14.hu.po.
Repackaged release drupal-6.14 for Japanese. Created file drupal-6.14.ja.po.
Repackaged release drupal-6.14 for Kannada. Created file drupal-6.14.kn.po.
Repackaged release drupal-6.14 for Latvian. Created file drupal-6.14.lv.po.
Repackaged release drupal-6.14 for Lithuanian. Created file drupal-6.14.lt.po.
Repackaged release drupal-6.14 for Lolspeak. Created file drupal-6.14.xx-lolspeak.po.
Repackaged release drupal-6.14 for Malayalam. Created file drupal-6.14.ml.po.
Repackaged release drupal-6.14 for Mongolian. Created file drupal-6.14.mn.po.
Repackaged release drupal-6.14 for Nepali. Created file drupal-6.14.ne.po.
Repackaged release drupal-6.14 for Norwegian Bokmål. Created file drupal-6.14.nb.po.
Repackaged release drupal-6.14 for Norwegian Nynorsk. Created file drupal-6.14.nn.po.
Repackaged release drupal-6.14 for Persian (Farsi). Created file drupal-6.14.fa.po.
Repackaged release drupal-6.14 for Polish. Created file drupal-6.14.pl.po.
Repackaged release drupal-6.14 for Portuguese, Brazil. Created file drupal-6.14.pt-br.po.
Repackaged release drupal-6.14 for Portuguese, Portugal. Created file drupal-6.14.pt-pt.po.
Repackaged release drupal-6.14 for Punjabi. Created file drupal-6.14.pa.po.
Repackaged release drupal-6.14 for Romanian. Created file drupal-6.14.ro.po.
Repackaged release drupal-6.14 for Slovak. Created file drupal-6.14.sk.po.
Repackaged release drupal-6.14 for Slovenian. Created file drupal-6.14.sl.po.
Repackaged release drupal-6.14 for Spanish. Created file drupal-6.14.es.po.
Repackaged release drupal-6.14 for Tamil. Created file drupal-6.14.ta.po.
Repackaged release drupal-6.14 for Telugu. Created file drupal-6.14.te.po.
Repackaged release drupal-6.14 for Test. Created file drupal-6.14.test.po.
Repackaged release drupal-6.14 for Turkish. Created file drupal-6.14.tr.po.
Repackaged release drupal-6.14 for Ukrainian. Created file drupal-6.14.uk.po.
Repackaged release drupal-6.14 for Welsh. Created file drupal-6.14.cy.po.

Now moving on to #593282: Set up syncing of l10n_packager files to ftp.drupal.org so that we can test with a client once the code is there. For later packaging, we still need the cli script explained above, since running from a browser is just too time consuming due to the file system setup.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Gábor Hojtsy’s picture

Issue tags: +localized install

Changed tag.