Problem/Motivation
When adding a new language on admin/config/regional/language
, the language files are not downloaded. If eg. the site is installed in English and French language is added, no French translation is downloaded. This is caused by two factors:
- Drupal does not report a reasonable version number to work with (git checkouts, alpha releases, etc. all report as Drupal 8.x). Git checkouts will never report a proper version number. Only a server side fallback solution would help with this unless we want to manually update a list of prior core versions in the code. See #2113957: Build server side version fallback system for translations. In the meantime, git_deploy was ported to Drupal 8 and can identify dev versions to the commit :)
- Drupal core / locale module does not fall back on prior releases from project update data if a dev version is encountered. That is what this issue is here to resolve.
Proposed resolution
- Fall back from dev versions to prior core releases based on project data.
- Add tests.
Remaining tasks
- Tests
User interface changes
None.
API changes
None.
Related
- #1804688: Download and import interface translations
- #1914070: Improve version fallback for install language.
- localize #2113957: Build server side version fallback system for translations
- localize #2113955: Rely on proper server side version fallback for translations
- #1963092: Convert .info file rewriting in packaging plugins to deal with D8 .info.yml files
Comment | File | Size | Author |
---|---|---|---|
#67 | interdiff-2030537-60-67.txt | 2.29 KB | Sutharsan |
#67 | locale-update-2030537-67.patch | 8.71 KB | Sutharsan |
#60 | interdiff-2030537-52-60.txt | 9.86 KB | janoka |
#52 | interdiff-2030537-39-52.txt | 4.71 KB | janoka |
#52 | locale-update-2030537-52.patch | 6.14 KB | janoka |
Comments
Comment #1
penyaskitoTagging.
Comment #2
YesCT CreditAttribution: YesCT commentedI can confirm, that when I add spanish, I have 0% UI strings translated. I thought #1804688: Download and import interface translations should have made it work automatically.
Comment #3
wusel CreditAttribution: wusel commentedI have the same issue.
I add:
If I install the alpha 2 or a alpha2-xxx.dev of the D8-core and select German language during the installation, then I get the german D7-po-file (drupal-7.0.de.po) but nor D8-po-file. A German D8-po-file is available on the translation server of d.o. at http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0-alpha2.de.po
Wusel
Comment #4
penyaskitoThere are different issues happening here, the concrete issue @wusel is experiencing is being tracked at #1914070: Improve version fallback for install language..
Because this issue has some interesting use cases and not sure if everything is covered, I'm against marking this as duplicate.
Comment #5
Gábor HojtsyThat wrong version translations are downloaded and no translations are downloaded in some cases sounds like to me at least a major.
Comment #6
Sutharsan CreditAttribution: Sutharsan commentedIf you add a language to an existing site, a po file is downloaded using the \Drupal::VERSION. Currently this is 8.x-dev for all, and therefore it will not download any translation. No po files are provided for dev's. Edit the 'version' in the info.yml file to test.
Comment #7
Schnitzel CreditAttribution: Schnitzel commentedAs Sutharsan correctly mentioned, there are no po Files for 8.x-dev.
If you change
const VERSION = '8.0-alpha2';
in Drupal.php
and adapt local_project table it works perfectly, see screenshot for some proof.
I think that the Issue of wusel is exactly #1914070: Improve version fallback for install language.
Comment #8
Gábor HojtsySo there are two problems here. #1914070: Improve version fallback for install language. resolves the installer problem. It does not resolve the language addition problem, and that should still work with this. Making it required to change your version is not the way to solve this IMHO. The code for adding language should work with dev releases.
Comment #9
Gábor HojtsyRetitling for the remaining problem. The install problem is handled in #1914070: Improve version fallback for install language.. Needs issue summary update to reduce it only to the remaining problem.
Comment #10
wusel CreditAttribution: wusel commented@#3
I have installed drupal-8.0-alpha4 and drupal-8.x-dev_2013_10_24 on xampp 1.8.2-2 and Win7-64bit.
Thank you all, it now downloads the file "drupal-8.0-alpha2.de.po", if I choose German during install.
Wusel
Comment #11
Gábor HojtsyYeah this bug is about adding another language on that install (or an English install). Is *that* downloading a translation now? :)
Comment #12
wusel CreditAttribution: wusel commentedThere are two bugs:
- like #11: the first language did not download the correct version of the po file:
drupal downloads the file "drupal-8.0-alpha2.de.po" and NOT "drupal-8.0-alpha4.de.po", which we can find at http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0-alpha4.de.po
- if I then add a language like "Hungarian", I read at "admin/config/regional/language" for this installed language Hungarian: "Interface translation = 0/6253 (0%)" and there is no po file at \sites\default\files\translations\
Hint:
If I look at admin/reports/dblog I can find: "Translation file not found: http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0-dev.hu.po."
Wusel
Comment #13
Gábor HojtsyYeah the download of alpha2 is "expected" ATM. About two weeks ago it was downloading Drupal 7 translations. As per the current code it will download translations for the first published version of that series (alpha2 is the first with a tarball). So once betas are out, beta1 translations would be downloaded for all betas. Once rcs are out, rc1 translations, etc. This needs some serious improvements on the server side to make more intelligent. See #2113957: Build server side version fallback system for translations and #2113955: Rely on proper server side version fallback for translations for those.
As for the adding new language is where this bug is at, look at the title and issue summary :)
Comment #14
Sutharsan CreditAttribution: Sutharsan commentedThe version of the translation to be downloaded is based on the
VERSION
constant. When writing the code for the version fallback I assumed this string would reflect the version of the tarball, but it doesn't. I've downloaded the D8-apha4 tarball, and found out it hasconst VERSION = '8.0-dev';
(in core/lib/Drupal.php). I should have been set to '8.0-alpha4' in order to load the corresponding translation file.Comment #15
Gábor HojtsyYeah, this would need the same fallback logic that the installer has *for dev* versions no?
Comment #16
Sutharsan CreditAttribution: Sutharsan commentedHm, I wasn't paying proper attention to the issue subject/title. I was talking about the 'during installation' situation. A solution is that VERSION is set before and reset after creating the release. (unless the below packaging script can handle this).
When enabling a language, the project version is used. This version is based on the date in the .info.yml file. The packaging script is not working correctly yet, as it does not include the right version in the info.yml files. For example system.info.yml contains
version: VERSION
. This should beversion: 8.0-alpha4
. #1963092: Convert .info file rewriting in packaging plugins to deal with D8 .info.yml files is about dealing with info.yml files.Comment #17
Gábor HojtsyYeah, well even if that is made to work it will never work on -dev versions when adding a language, so we need that code applied to when adding a new language. Sounds like it may be just guessing a version if we don't know the core version reusing the same function as in the installer? :)
Comment #18
Sutharsan CreditAttribution: Sutharsan commentedWithout version info, the installer code falls back to hardcoded "8.0-alpha2" version. That can be done. It its simplest form we use "8.0-alpha2" (or 4) if no version info is available. But only for core. I we want to fallback to multiple version (8.0 > RC1 > beta1 > alpha2), the impact is bigger. Currently we can not handle multiple 'check if translation version x exists' requests.
Comment #19
Gábor HojtsySo our other option is we suggest people to install git_deploy?
Comment #19.0
YesCT CreditAttribution: YesCT commentedUpdated issue summary.
Comment #20
penyaskitoI checked that the last alpha (8.0-ALPHA7) had const VERSION = '8.0-dev'.
What's next in this issue?
Comment #21
Gábor Hojtsy@penyaskito: the installer has special code to take fallback version numbers in case of dev versions. This was not added to the regular locale update code though (with the expectation that alpha/beta/rc releases would have actual version numbers instead of dev). To support this dev approach, we would need to have similar fallback code in the runtime path as well like in the installer.
Comment #22
robertdbailey CreditAttribution: robertdbailey commentedI'll try to make some progress on this while at DDD in Szeged.
Comment #23
robertdbailey CreditAttribution: robertdbailey commentedThe logic for version fallback in core/includes/install.core.inc:
function install_get_localization_release($version = \Drupal::VERSION)
Its calling function, "install_check_translations", loops through its results, which would be the same logic required when adding a new language. This logic could be added each time a new language is saved, but perhaps the localizations should be downloaded automatically only when setting or changing the default language. If it were added only when the default language is set, then the correct place to add this would be in function "language_system_regional_settings_form_submit" in core/modules/language/language.module.
Does this sound correct, or should localizations be downloaded every time any language is added?
Comment #24
robertdbailey CreditAttribution: robertdbailey commentedThe logic for language fallback during Drupal installation is in code that is for core only. The place where translations are downloaded when languages are added is used more broadly. It looks like it is used for translation downloads for core as well as all other modules and themes. Should the language-fallback code be extended (when languages are added) to include checking and downloading localizations for all modules and themes?
Comment #25
Sutharsan CreditAttribution: Sutharsan commentedThe fallback code was written specifically for installing Drupal and downloading the drupal core translation (and only core, not contrib modules). It has an hardcoded fallback scheme. It would be silly to use a hardcoded scheme beyond the installation process. Once update module is available (after installation), it is used to provide the latest release version of a module (and core).
The root of the problem in this issue is that info.yml files currently do not provide a version. In the file you find "version: VERSION" instead of for example: "version: 8.0-alpha10". This is taken care of by #1963092: Convert .info file rewriting in packaging plugins to deal with D8 .info.yml files. Pending this issue, I mark the current issue as postponed. I expect that the current issue is resolved once the .info(.yml) file issue is fixed.
Comment #26
Gábor Hojtsy@Sutharsan: but what would happen for git checkouts and dev branches? Do we require git_deploy for those to work in the first place?
Comment #27
Gábor HojtsySo I wanted to look into resolving this sooner than later. The idea was that the runtime will have real version numbers to work with thanks in part to git_deploy. Tonight I decided to try to resolve this by porting git_deploy to D8, and so I did: #2305311: Complete, fully working Drupal 8 port of git_deploy It works well for what it does, but does not really solve the problem for core :(
1. git_deploy decidedly skips core. And in my tests it did not even get the right version (it detects 'HEAD' for a 8.x-alpha13 checkout).
2. For contribs detected, it finds full dev version numbers, eg. 8.x-1.0-alpha1.10-dev in my case because I have a checkout aligning with that. The translation updater successfully falls back from that to look at the update XML and downloads the translation well, YAY!
What do we do for core?
Comment #28
Gábor HojtsyAll right, I made git_deploy work for core as well, at least for tags. See #2305311: Complete, fully working Drupal 8 port of git_deploy. Works with update status. The localization update feature still thinks its a 8.0-dev :/ Also the git deploy module version got very confused...
Comment #29
Sutharsan CreditAttribution: Sutharsan commentedThe fallback only worked for contrib modules/themes. This patch adds a version fallback for core.
Comment #30
Gábor HojtsyThis "works" now in my testing, it detects alpha13 properly with git_deploy 8.x. Unfortunately localize.drupal.org did not identify alpha13 as a version to parse for some reason so that is not available. So now it attempts to download the right version but its the fault of l.d.o that it does not work. Are there any tests for this that should be updated or added to?
Comment #31
Gábor HojtsySo I started look into tests for this today. However in the meantime Drupal switched to semantic versioning so your patch will not be correct anymore. I wanted to update the patch but then realized, I don't know how the new semantic versions will be represented in the XML. Will there be a "minor_version" key? I wanted to look at the test coverage for this in update_status but then found it still has 7.x update tests. Ho-hum. Yeah. See #2170443-82: [meta] Create a plan for implementing semantic versioning for core.
Also was not sure how that affects the contrib version numbers and whether we should support old schemes and new as well. That is not exactly in the scope of this issue, but we need to fix that too.
The first alpha based on the new semantic versioning is coming out tomorrow, so we'll maybe see some useful data coming up on http://updates.drupal.org/release-history/drupal/8.x as to how it is structured for us to support that.
Comment #32
Sutharsan CreditAttribution: Sutharsan commentedalpha14 is out now, but no (Dutch) translation is yet available. This patch now updates the version detection to the new semantic version and does not support pre-semantic versions (i.e. alpha13 and before).
Comment #33
Gábor HojtsyWe tried this at the Drupalaton multilingual training and it did not work for some reason.
I thought this is the reason, this will not match 8.0.0-dev, because the extra - added before .* but removing that extra - did not make it work either.
Comment #34
Sutharsan CreditAttribution: Sutharsan commentedTo test the #32 patch (manually) you also need patch in #2317005: Core version not detected due to new semantic versioning for Git Deploy.
Comment #35
Gábor HojtsyIt works with that yup. I don't fully why/how it does with that regex, but it does :)
Comment #36
janoka CreditAttribution: janoka commentedComment #37
Sutharsan CreditAttribution: Sutharsan commentedThe regex currently evaluates "8.0.0-alpha14-dev", and that works. But I realize that I did not look at future version number patterns. Beta and RC will probably be fine but will it be 8.0.0-dev or 8.0.x-dev? Both will fail.
Comment #38
Sutharsan CreditAttribution: Sutharsan commented8.0.0-alpha14-dev: current situation when using Git Deploy
8.0.0-alpha14+42-dev: current situation when downloading the dev package.
8.0.0-dev: after 8.0.0 release when using Git Deploy.
8.0.0+3-dev: 3 commits after 8.0.0 release when downloading the dev package. (see
DrupalorgProjectPackageRelease::computeRebuildVersion()
)Comment #39
Sutharsan CreditAttribution: Sutharsan commented@janoka, I have not seen any activity in 24 hours, could not find you in IRC either. Are you still working on this issue? I have taken the liberty to continue working on it.
Comment #40
Gábor HojtsyWhat git deploy identifies for me is
8.0.0-alpha14.25-dev
, note the dot between the alpha and the number and dev. I think this is due to 25 commits since the alpha? Should it use a + when doing this number format?Comment #41
Sutharsan CreditAttribution: Sutharsan commentedI did not work with the latest git deploy checkout, but now I see. Is should be
8.0.0-alpha14+25-dev
. Look at the .info.yml files in a D8 dev release package or check the code inDrupalorgProjectPackageRelease::computeRebuildVersion()
of the drupalorg project.Comment #42
Gábor HojtsyFixed git_deploy at #2319775-1: Dev delta versions should use + separator not ..
Comment #43
janoka CreditAttribution: janoka commented@Sutharsan I'm working on this issue.
Comment #44
Gábor HojtsySo while it seems like @janoka made no progress in 18 days, he has a testing issue at #2324551: János Kuszing's test issue. Since that is also not moving a lot, let's at least discuss the plan and validate with Sutharsan here:
1. Implement a test module that uses hook_system_info_alter() to alter the version of core to a dev version.
2. Implement a custom update status backend XML provider (or feed the project DB with equivalent data) that provides the fallback items for said altered core version.
3. Test that locale falls back on appropriate version based on that data.
@Sutharsan: does this sound like it makes sense for you or do you have a simpler/more complex suggestion to cover testing for this? So far @janoka's test issue only implements point 1.
Comment #45
Gábor HojtsyUpdated issue summary.
Comment #46
janoka CreditAttribution: janoka commentedI came back from vacation yesterday, so I would like to continue this issue, if it is possible. @Gabor and @Sutharsan: what do you think, what are your opinion?
Comment #47
Sutharsan CreditAttribution: Sutharsan commentedThis test plan makes sense. Some thoughts:
locale_translation_build_projects()
.Comment #48
Sutharsan CreditAttribution: Sutharsan commented@janoka, the sooner the better ;)
Comment #49
janoka CreditAttribution: janoka commented@Sutharsan I totally understand you!
Comment #50
Gábor HojtsyYeah the most logical step seems to be to plant some test data in \Drupal::keyValueExpirable('update_available_releases')->set('drupal', $test_data); directly in the test in a way that avoids all the conditions that would make update_get_available() run a refresh. Instead of fiddling with update XML and other things.
Comment #51
janoka CreditAttribution: janoka commentedInspired by our phone speaking I coded:
I'll check update_get_available(TRUE) and locale_translation_build_projects().
Comment #52
janoka CreditAttribution: janoka commentedI think that is not the final version. I ask you to give me hints in connection with these patches.
Comment #55
Sutharsan CreditAttribution: Sutharsan commentedFrom reading the interdiff, IFAIS this is the right direction. Some details:
Wrong description. Pls modify.
Should be placed in setup().
Use REQUEST_TIME instead of time() is a small performance improvement. Only use time() if using the /actual/ time is critical.
"Not dev" is to me not a very good name for this module. "development release" ('dev' for short) is the best alternative I can think. Either make the description very genric or describe what the module does, not what the related test case is.
Copied from other module? Pls modify.
Comment #56
janoka CreditAttribution: janoka commentedDear @Sutharsan,
Thanks for your comment! I will do it.
Comment #58
Gábor HojtsyThe fails on LocaleUpdateTest do not seem related to the changes introduced here. So I sent #39 for a retest to see how the "fix only" will fair in current D8 with those tests. It passed at the time.
Comment #59
Gábor HojtsyI also ran the fix part of #52 (that is I believe the same as #39) locally with the existing tests in LocaleUpdateTest, and the same fails were produced as in #52, so the fails are not new with #52. They are new due to time spent / alpha release for the pattern looked for being available. So that is down to remote data now being different at this time than it was when #39 was originally written. So I expect the #39 re-run will result in the same 15 fails.
@Sutharsan, @janoka: so looks like we need to fix/update LocaleUpdateTest as well.
Comment #60
janoka CreditAttribution: janoka commentedComment #64
janoka CreditAttribution: janoka commentedComment #65
janoka CreditAttribution: janoka commentedI hope I could help you.
I will not able to undertake correcting of LocaleUpdateTest on a perfect way.
A good solution would be to simulate the response of drupal.org, like in core/modules/update/tests/modules/update_test.
Comment #66
Sutharsan CreditAttribution: Sutharsan commentedTaking a look at LocaleUpdateTest ...
Comment #67
Sutharsan CreditAttribution: Sutharsan commentedThe test errors are caused by a working update system (Update module). Now that the version pattern is corrected the translation update system actually goes out to l.d.o and fetches a translation there. This problem was not discovered before probably because the update system at d.o was not yet working.
I fixed this by overriding the path at which the translation update system is looking for a translation. Therefore it will not go out to l.d.o to check for a translation. Drupal core will always be untranslated. As a bonus, this patch override on the path at which the update module checks for update status data. This prevents update module to go out to d.o to check for available translations. Both changes make the system under test independent from (l.)d.o services.
[edited: for clarity]
Comment #70
Gábor Hojtsy@Sutharsan: great, this should also speed up test runs a great deal. :)
Comment #71
Gábor HojtsyBTW sent for retest because HEAD got broke by a commit of #2148199: Use snapshot to warn users if the configuration has changed since last import, and the fails looked totally due to that.
Comment #72
Gábor HojtsyLooks amazing. Fixes an actual user bug. Speeds up tests. Small patch. More tests. What's not to love? :)
Comment #73
janoka CreditAttribution: janoka commented@Sutharsan, @Gábor sounds good!
Comment #74
webchickThis actually smells critical to me. Great fix.
Just found two small things, fixed them on commit:
that trailing comma looks odd.
missing some whitespace near the if and ==
Committed and pushed to 8.x. Thanks!
Comment #76
Sutharsan CreditAttribution: Sutharsan commentedCreated follow-up cleanup issue #2338011: Cleanup of locale tests
Comment #78
Gábor Hojtsy