Closed (fixed)
Project:
Project
Version:
5.x-1.x-dev
Component:
Releases
Priority:
Critical
Category:
Task
Assigned:
Reporter:
Created:
7 Jan 2008 at 21:07 UTC
Updated:
4 Feb 2008 at 22:41 UTC
Jump to comment: Most recent file
Actually, I hoped this day will never come, but here we are. The localization server is not yet ready for prime-time, so we need to make project module generate Drupal 6 tarballs nicely. Drupal 6 translations are in a different source format (file names, string placement) then Drupal 5 translations though, and they need to be put into a different package structure, so this needs some work here. I am volunteering to do the work with some guidance.
I studied the code in project-release-nodes.php where package_release_contrib() is already split into translations and not translations. What we need here is to branch translations into Drupal 6 translations and earlier translations. I wonder whether it is practical to do the branching inside this function still, and depend on $version's matching to 6.x or earlier.
To compare the earlier and the D6 workflow, this is what we have done before D6:
- keep installer.po as it is
- merge all other .po files into a $langcode.po (taking general.po by precedence)
- stuff a README.txt with the status of the $langcode.po file
Now with Drupal 6 we need to:
- mimic the core directory structure, but have 'translations' subfolders in each place
- move files to these subfolders based on their names
- move and rename the installer.po file as well with the same rules
So the good news is that we don't need to merge anything, the "bad" news is that we need to build a directory structure. Files in a translation (based on the drupal.pot project):
general.po modules-contact.po modules-ping.po modules-trigger.po
includes.po modules-dblog.po modules-poll.po modules-update.po
installer.po modules-filter.po modules-profile.po modules-upload.po
misc.po modules-forum.po modules-search.po modules-user.po
modules-aggregator.po modules-help.po modules-statistics.po root.po
modules-block.po modules-locale.po modules-syslog.po themes-chameleon.po
modules-blog.po modules-menu.po modules-system.po themes-garland.po
modules-blogapi.po modules-node.po modules-taxonomy.po themes-pushbutton.po
modules-book.po modules-openid.po modules-throttle.po
modules-color.po modules-path.po modules-tracker.po
modules-comment.po modules-php.po modules-translation.po
The pattern is that modules-* and themes-* files go that modules and themes subfolder's translations folder, while common files go to the system module and installer.po goes to the install profile. All files should have the language code added to the filename.
We use the following script to generate the described package format with the Hungarian translation:
// $Id: build.php,v 1.1.2.1 2007/11/06 21:27:52 goba Exp $
/**
* @file
* Build a Drupal 6 ready package out of files in this folder.
*/
// Clean up build directory.
`rm -rf ./build`;
$files = glob('*.po');
foreach ($files as $file) {
if (!strpos($file, '-')) {
if ($file == 'installer.po') {
// Special file, goes to install profile.
$target = 'profiles/default/translations/hu.po';
}
else {
// 'Root' files go to system module.
$target = 'modules/system/translations/'. str_replace('.po', '.hu.po', $file);
}
}
else {
// Other files go to their module or theme folder.
$target = str_replace(array('-', '.po'), array('/', ''), $file) .'/translations/'. str_replace('.po', '.hu.po', $file);
}
$target = 'build/'. $target;
// Create target folder and copy file there.
mkdir(dirname($target), 0777, TRUE);
copy($file, $target);
}
// Build package.
unlink('hu-6.0-dev.tar.gz');
`cd build && tar -cvzf ../hu-6.0-dev.tar.gz * && cd ..`;
// Clean up build directory.
`rm -rf ./build`;
With all this run, the desired package format for 6.x translation packages materializes as attached.
May I ask for some starter help in integrating this logic into the packager script?
| Comment | File | Size | Author |
|---|---|---|---|
| #16 | package-drupal-6-translations-no-subdir.patch | 2.85 KB | dww |
| #16 | fr-6.x-1.x-dev.tar_.gz.no-subdir.txt | 168.59 KB | dww |
| #13 | package-drupal-6-translations_3.patch | 7.75 KB | dww |
| #13 | fr-6.x-1.x-dev.tar_.gz.txt | 168.63 KB | dww |
| #11 | package-drupal-6-translations.patch | 7.69 KB | gábor hojtsy |
Comments
Comment #1
dwwAll looks reasonable to me. I'm not sure what info/help you need...
Personally, I'd put all this 6.x-specific translation logic into a separate function, and then inside package_release_contrib(), as soon as you get past the totally common, shared parts, figure out if it's a) 6.x or later translation, b) 5.x or earlier translations, or c) something else (module, theme, etc), and call the appropriate function to setup the directory structure we want to tar. then, we'd have 3 separate functions for the 3 kinds of things we package, to keep them separate, since they have pretty different needs and logic. then, control will return to package_release_contrib() for the last few code blocks to actually invoke tar, call package_release_update_node(), etc. i think putting the logic for all 3 into the same function won't help, since there won't be much shared code, and it'll be more confusing to follow. sound good?
is that all the input you need from me? i read the post twice and it looks like you've got a firm grasp on everything -- this question about where in package-release-nodes.php to put the logic seems like the only open question. just let me know if there's anything else.
thanks!
-derek
Comment #2
gábor hojtsyThanks for the info, this is what I looked for.
Comment #3
dwwYikes, people have already started trying to tag and release D6 translation packages. :( See http://drupal.org/node/208454. I'll at least email the translators and devel lists about this, but maybe mention should go out in the RC2 announcement, too?
Comment #4
dwwFYI: http://lists.drupal.org/pipermail/translations/2008-January/000527.html
Comment #5
gábor hojtsyHere is my first attempt (untested):
- I removed README.txt exclusion from youngest file counting, as I have seen no reason why it was there for translations.
- broken out the preparation tasks to d6 and pred6 functions
- made sure that if these return FALSE, we return from the original function as well
- the d6 function first does stats, while the .po files are still in the same folder, so it is easier to stat them
- then it calculates the target place for each file
- removing fuzzy translations was done as part of the merge step in pred6, now we do it as we copy the file
- only the found .po files and *.txt files are packaged, as before
- because the directory structure needs to be defined, I use mkdir() with recursive directory creation (last param TRUE); this only works with PHP 5, so let me know if I'd rather do it PHP 4 friendly... we might also need to take care of the mode we are creating the directories with, I used 0777 which is far from the safest
As said, this is untested, but I read and reorganized the code multiple times to be hopefully as clean as possible. Note that the D6 code has no error condition (I am not sure on what we would fail, except where some drupal_exec() fails, which is already dealt with), so it does not need the $version and $view_link params, which is only required in the preD6 packager to display the error properly.
One thing I am thinking about is that we possibly need to rename *.txt with a pattern to $name.$uri.txt or something along the lines. We suggest that people extract this package to their Drupal root. So a translation README would overwrite the Drupal readme, which we are not fond of :) Same for INSTALL.txt, UPGRADE.txt, etc.
Comment #6
gábor hojtsyNote that the patch is against HEAD, which is I assume the correct version to work with now.
Comment #7
dwwSorry it's taken a while to review this -- I'm out of town right now and have very limited net access... So far, this is looking quite good. A few replies and concerns.
- Yes, HEAD is the right branch to patch against for this.
- README.txt exclusion was legacy code for the weird practice of ignoring the translation's own file and trying to put translation status there.
- Translation status has never worked since the new release system, and I believe it was probably broken before that, too, since I just cut and pasted the identical code from the original packaging script for this. :( See http://drupal.org/node/92915 and http://drupal.org/node/100652. Frankly, it seems evil to keep saving status into a file called "README.txt". Why not a) allow languages to include their own language-specific readme, and b) put the translation status into something like STATUS.txt?
- This script is so d.o-specific, and d.o is definitely PHP5, that I certainly don't care about that. ;)
- So long as the script explicitly sets the right umask (0022), creating the dirs with 0777 is fine, since they'll end up 0755 which is what we want. the d.o copy has been locally patched to do this for months, so I just committed that "upstream" so we're explicit about it. That said, to be extra safe, I'd check the return value of mkdir() and wd_err() about it if it fails. I know that re-introduces the args you left out, but given how cursed d.o seems to be, anything that can go wrong in this script will go wrong, and I'd rather see an explicit message about it.
- s/package_release_contrib_pred6_translation/package_release_contrib_pre_d6_translation/ please. ;) "pred6" is kinda hard to read at first glance, IMHO.
- I agree we should rename *.txt to *.[langcode].txt during packaging.
Thanks for working on this. I agree it's tragic that we'll have to limp along for another core release cycle with CVS-based translations, but such is life. ;) There are only so many hours in the day, as we're both painfully aware...
Cheers,
-Derek
Comment #8
gábor hojtsy- Used STATUS.txt instead. Note that the mentioned issues talk about the missing stats support for translations at large, not that the stats does not work in the generated tarballs. If there is some problem here with the packager, we should be able to solve it locally.
- added wd_err() to all possible places where the packaging could fail
- renamed function
- added code to rename .txt files
Comment #9
gábor hojtsyThe same preg_match() call was used inside a body of a condition which was already only executed if that preg_match() returned true. (Removing some code duplication).
Comment #10
dwwCool, thanks. I'll take a look a little later today. However, the thing about README.txt being empty (http://drupal.org/node/100652) is still relevant. I think the code that's currently trying to populate README.txt (which your patch just copies) doesn't in fact work. I'd have to test to be sure, but it seems to result in empty files on d.o, as far as I can tell...
Comment #11
gábor hojtsydww: msgcat and msgfilter seem to work, or the packaging to $uri.po would not work in D5. So what is not working is
$msgfmt --statistics $uri/$uri.po 2>> $uri/STATUS.txt
I looked into running this on my machine and it looks like it is working:
$ msgfmt --statistics general.po 2>> STATUS.txt
This put the stats of general.po to the STATUS.txt file. So it should work in D5 as it is in principle. It might be some server issue on drupal.org?
I also tried the analogue of the Drupal 6 version I came up with:
$ msgfmt --statistics *.po 2>> STATUS.txt
Unfortunately I got some errors here:
includes.po:28: duplicate message definition...
general.po:137: ...this is the location of the first definition
msgfmt: found 1 fatal error
Looks like it tries to merge the files as well for the stats. So we might need to run msgfmt for each .po file individually in the foreach loop. Attached is a version which generates STATUS.txt information for each .po file individually in the 6.x case. This is not a problem in the 5.x case, as it is running for a single .po file already.
Comment #12
dwwPatch seemed fine, so I tried testing this on d.o (not actually overwriting real tarballs nor updating release nodes, only the error reporting integration was kept in place). I tried on a few of the existing 6.x translation releases and the script always exited with "Unable to generate directory structure in [uri] translation in version 6.x-1.x-dev, not packaging".
I see the problem, stay tuned for a quick re-roll...
Comment #13
dwwThe problem is that we try to call mkdir() every time we hit a target file, even if that directory already exists. Re-roll attached. This seems to work correctly. I've attached an output tarball from running a copy of this on d.o.
The only thing that's still broken is STATUS.fr.txt is empty. :( But, that seems to be broken independently of this patch, so we could try to fix it over at http://drupal.org/node/100652 instead of holding this up.
Comment #14
gábor hojtsyChecked the tarball, and it looks good. So I'd say go with this and solved the status issue over there!
Comment #15
dwwCommitted to HEAD. Was about to deploy on d.o, and then Gabor and I discussed a new minor problem. The tarball shouldn't include a language code subdir, or it's harder to install these.
Comment #16
dwwLike so. Patch and resulting fr-6.x-1.x-dev tarball from a test run on d.o attached.
Comment #17
gábor hojtsyBoth the patch and the tarball looks good.
Comment #18
dwwCommitted and deployed on d.o.
I regenerated the tarballs for all of the 6.x-1.x-dev translation tarballs.
Everything seems to look great, except that 1 of them has a .po error, although that seems to be a problem in the translation, not the packaging: http://drupal.org/node/208231
Comment #19
Anonymous (not verified) commentedAutomatically closed -- issue fixed for two weeks with no activity.