Problem/Motivation
Since #1914070: Improve version fallback for install language. at least Drupal 8 will download alpha2 translations properly. Even if Drupal 8 alpha5 is out. (Then once a beta1 is out, that will be the new base for translation downloads and then rc1, but even if rc3 is out).
Proposed resolution
Ideally we'd have a server side version fallback system where if we ask for 8.x-dev downloads, it would *redirect* us to the latest version downloads (so we know the version downloaded from the file name and can track).
Remaining tasks
1. Work out the server side solution (this issue).
2. Remove fallback logic from core as necessary and rely on server side fallback. Rewrite tests with mock server side (#2113955: Rely on proper server side version fallback for translations).
User interface changes
None.
API changes
None.
Related Issues
#1914070: Improve version fallback for install language.
#2113955: Rely on proper server side version fallback for translations
Comment | File | Size | Author |
---|---|---|---|
#39 | interdiff.txt | 1.03 KB | Gábor Hojtsy |
#39 | 2113957-39.patch | 3.73 KB | Gábor Hojtsy |
#38 | 2113957-38.patch | 3.67 KB | Gábor Hojtsy |
#38 | interdiff.txt | 2.9 KB | Gábor Hojtsy |
#36 | l10n_server-purge-on-symlink-2113957-35.patch | 1.35 KB | Sutharsan |
Comments
Comment #1
Gábor HojtsyRemove revisit tag along the lines of #2113955-1: Rely on proper server side version fallback for translations.
Comment #2
Gábor HojtsyAlso added as a blocker to #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1. Depending on the availability and compatibility of git_deploy with translations, this may or may not block a Drupal 8 release.
Comment #3
mgiffordComment #4
YesCT CreditAttribution: YesCT commentedadding "infrastructure blocker for Drupal 8.0.0" tag, since this blocks the drupal 8 core issue: #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1, so that the infrastructure blockers to d8 release (in issue queues outside of the core queue) is accurate. Remove the blocker tag when this issue is fixed, and update 2267715.
Comment #5
webchickTrying to get a handle on release blockers prior to 8.0.0 and what (if anything) the community can do to help things along in preparation of https://events.drupal.org/losangeles2015/sessions/plain-drupal-english-g...
So the issue seems to be that right now translations will only be against the first X release (alpha 1, beta 1, rc 1), not subsequent X releases (alpha 3)?
Is this a place where community members can jump in at LA? If so, could the issue summary be updated with an outline of where to start?
Comment #6
Gábor Hojtsy@webchick: The issue right now is
(1) Drupal core needs to be manually updated to fall back on a certain translation because it does not know its version without git_deploy. Without manual updates like #2285795: Drupal 8 loads alpha2 translations instead of alpha12, core now falls back on beta1 translation downloads.
(2) Drupal core tries multiple HTTP requests to download translations to decide on which one is available which makes the installation slower; if the server would decide for it, it would only need one HTTP request and be directed to the right one with a redirect or symlink or something along those lines and be quicker to install
@catch requested to keep this as a blocker because otherwise we manually need to update fallback versions in core each release if we want new string translations to be downloaded.
Comment #7
webchickHm. Well but we already have to change the VERSION variable manually each release. Couldn't https://www.drupal.org/files/issues/fallback-to-alpha12-2.patch just be added to the release checklist? There's a clear workaround here.
Comment #8
Gábor HojtsyBased on our IRC chat a couple months back @catch preferred not to add it to the release checklist but to keep this issue open (and revisit before release instead).
Comment #9
webchickOK, I'll talk to him.
Then it sounds like what is needed here is:
- A custom module for localize.drupal.org
- That defines a URL at something like /translations/$project/$major-version
- Hitting that URL with "translations/drupal/8.0.x" returns a JSON response or something with the string "8.0.x-beta10" (atm, would change to 8.0.x-beta11 next month)
Am I remotely in the right ballpark here?
Comment #10
opdavies@webchick: https://www.drupal.org/project/localizedrupalorg is now that. :) The 7.x-1.x branch is what I'm using on the l.d.o staging site.
Comment #11
Gábor Hojtsy@webchick: if heavily cached yeah. Update status had this problem before that hitting a PHP equipped server was too expensive so it was switched to static exports and a static only server, which can serve a lot more requests. Currently localize translations are hosted on ftp.drupal.org so Drupal 8 will not hit any dynamic URL but just poll the static files one by one. We can make the static files symlink to prior ones when l.d.o gets to know about a release or make them do URL redirects or switch over the system entirely and do some dynamic stuff like you suggested.
Note that the "JSON" you suggested is available in the update info, so Drupal 8 *could* query that for version info. The reason it does not do that is (a) it would guarantee it needs 2 HTTP requests as opposed to the current where the first polled .po file may already exist (b) the versions available in the update info may not all have a .po file generated yet, so we would still need to go through multiple versions anyway. Ideally ftp.drupal.org would have something at every possible version number for a project, if only a symlink until that .po file is generated and a .po file symlink or redirect on the MAJOR.X version number which is symlinked or redirected to the latest version. I think for server performance this should be solved on ftp.d.o, but better versions of the solution are welcome.
Comment #12
webchickOk, in that case it sounds like this is only something the d.o infra team can do.
Comment #13
Gábor HojtsyWell, the .po files are actually generated by code in l10n_server on l.d.o directly and rsynced to ftp.d.o already. So theoretically it can be solved in l10n_server IF a solution is agreed on and that solution is transparently supported by the current rsync setup. The current setup may need to be tweaked as it only happens once or twice a day. Also the cron for the .po file generation may need to be tweaked. Also core changes are needed to request the MAJOR.X .po file once it actually exists. So its coordinated changes at minimum across l.d.o, infra and finally core.
Comment #14
hestenetComment #15
Gábor Hojtsy@hestenet: this is currently not implemented on Drupal 6, so not sure how is this a D7 port issue?!
Comment #16
hestenet@Gábor Hojtsy - my mistake! Was moving too quickly through the localize issues to see if we'd missed anything for the port.
Comment #17
japerrySo I dived into the D8 code for locale module and it appears that it is not hitting localize.drupal.org at all. Instead its calling ftp.drupal.org
Unless I'm reading this incorrectly, we would need to rewrite the locale module to make a call to localize.drupal.org to grab the correct version.
Psuedo code:
Alternatively, there would need to be a php wrapper between the file call and what is being grabbed.
For example, the url above: http://localize.drupal.org/translations/8.0-dev/image/image-8.0-dev.de.po
would end up going to localize.drupal.org, which would process and cache the current release representation of dev.
The first issue would be undesirable, since two remote calls would be required.
The second issue, while possible, would also be not desirable, since we would be loading po files from drupal, not static ftp. Maybe there is a way to rewrite the ftp folders to symlink to a release?
Objectives
Unless I'm totally misunderstanding the issue.. ;)
Comment #18
Gábor HojtsyYeah the ftp folders are rsynced from l.d.o's files folder. I think the best way here would be to somehow symlink things and ensure the rsync brings those over.
Comment #19
drummThat is no longer true. ftp.drupal.org hits a CDN, which either hits their cache, or forwards the request directly to our servers. Expiration times are relatively long, but we actively clear the CDN's cache when packaging a file happens.
Comment #20
drummPart of the solution might be rewrites in a web server layer. For example: if a request doesn't match an existing file, then serve something like image-8-latest.de.po, which is a symlink to the latest for that branch.
This reduces the number of symlinks we need to maintain, only one per branch we want to separate out.
The CDN cache clearing could need to switch over to a wildcard request, to be sure we are clearing enough.
Comment #21
basic CreditAttribution: basic at Drupal Association commentedComment #22
japerrySo I can pretty much declare this one fixed and/or RTBC at least.
to test new releases, go here (for newly released versions of d8 that don't have packaging yet):
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-beta15....
To test dev, go here:
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-dev.de.po
both should redirect to drupal-8.0.x.de.po which is a symlink to drupal-8.0.0-beta14.de.po
Now onto fixing d8.
Comment #23
Gábor HojtsyBoth of those links result in beta12 downloads for me. Header text:
# German translation of Drupal core (8.0.0-beta12)
Looking at https://localize.drupal.org/translate/languages/de German does have a download for beta14, so this does not seem to be right?
A final important question is how much max-age is put on these files and/or how would a front CDN invalidate them when the symlinks change? Drumm mentions it could be done, but nobody mentioned it was done.
Comment #24
Gábor HojtsyAnother problem. Someone reported that https://ftp.drupal.org/files/translations/7.x/pathauto_persist/pathauto_... does not exist (which is not surprising given https://localize.drupal.org/download?project=pathauto_persist does not list German translation for this module either). However, it redirects to https://ftp.drupal.org/files/translations/7.x/pathauto_persist/pathauto_... (rightfully) which redirects to itself in an infinite loop unfortunately :/ I think the redirect needs to be modified to not redirect if the file did not exist.
Comment #25
basic CreditAttribution: basic at Drupal Association commentedThis /should/ be working now.
This cleans up the regex a bit, preventing it from matching itself and will now send you a 302 -> 404 for things that don't exist.
As noted on IRC, some further cleanup could happen:
Comment #26
webchickBack to needs review.
Comment #27
drummDebugging was never added to the Fastly purge requests. I'll be deploying that.
Comment #28
drummThat is now deployed, and as far as I can tell, the CDN cache is not cleared properly.
l10n_packager_create_latest_symlink()
should invokehook_l10n_packaged()
sol10n_drupal_rest_l10n_packaged()
makes the PURGE request.Comment #29
Gábor Hojtsy@drumm: I think you meant *now* working properly not *not*?
Comment #30
Gábor HojtsyTested the following:
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-beta15.... (now is actual translation file, no redirect, YAY)
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-beta25.... (redirects to .x, YAY)
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-dev.de.po (downloads beta1 translation NOT GOOD :/)
http://ftp.drupal.org/files/translations/7.x/pathauto_persist/pathauto_p... (redirect to .x and then a 404 YAY)
Given the NOT GOOD item, looks like the implementation of this one and #2113955: Rely on proper server side version fallback for translations would get us back where we started, it would still not download the most recent .po for a dev or a nonexistent version, but instead an old one (beta1). The goal of doing it on the server side is/was because the server knows about the release history and can symlink to the most recent one not a much older one.
Comment #31
drummThe debugging is working well as debugging. I didn't do a thorough read of the logs yet. I expect to see a PURGE for the .x files when their equivalent full version is repackaged. I haven't seen that, but only was able to look at one packaging cron run.
We should make sure the redirects themselves are either not cached, have a short-lived cache, or are also PURGEd appropriately.
Comment #32
Sutharsan CreditAttribution: Sutharsan as a volunteer commentedhttp://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-dev.de.po currently downloads drupal-8.0.x.de.po
Am I missing something or was the CDM purged/expired in the mean time?
Comment #33
Sutharsan CreditAttribution: Sutharsan as a volunteer commentedBut the file's content is 8.0.0-beta1. Same observation as @Gabor in #30 :(
Comment #34
drummThere wouldn't have been a purge pushed today. A change would have been a normal expiration, or hitting a different state on a different point of presence.
Comment #35
drummThis looks like a symptom of the problem I described in #28. Notice the second request, I added
?a
to bypass the CDN's cache, and got the correct file.The symnlink in the filesystem does look good:
drupal-8.0.x.de.po -> /var/www/localize.drupal.org/htdocs/sites/default/files/l10n_packager/8.x/drupal/drupal-8.0.0-beta15.de.po
Comment #36
Sutharsan CreditAttribution: Sutharsan as a volunteer commentedI guess we have now proven that the CDN must be purged. This patch should purge the symlink when a new one is created as requested in #28. Patch is not tested.
Comment #37
Gábor HojtsyTo verify this patch I went to look at how this hook is defined. It is not. Looks like it was not documented when introduced. It is used in http://cgit.drupalcode.org/l10n_server/tree/l10n_packager/l10n_packager.... with a pretty full file object. Its hard to know what should the file object contain. The hook was added by @drumm in #2419637: Add CDN purging on packaging. @drumm can you help us document it?
Comment #38
Gábor HojtsySo I did some digging for the source and found that we only use the URI of the file anyway. We can change this API to pass on the file URI string or document the object. First pass at documenting the hook and renaming it for responsible contrib patterns to prefix with module name.
Comment #39
Gábor HojtsyMinor code style updates.
Comment #41
Gábor HojtsyCommitted that. We'll need to deploy and then monitor if things work as expected.
Comment #42
Gábor HojtsyComment #44
Gábor HojtsyFailed because I was happily committing the patch before the test finished. The branch test is green after commit, see https://www.drupal.org/node/148272/qa.
Comment #45
drummLooks good:
Comment #46
drummMy other concern was:
I think this is okay regardless of caching. Any redirect will be consistent until a real file takes its place. When that happens, the new file will come with a PURGE request.
Comment #47
drummI forgot to actually read the URI, that's not right.
/files/translations/var/www/localize.drupal.org/htdocs/sites/default/files/l10n_packager/7.x/oa_core/oa_core-7.x-2.x.hu.po
vs.
/files/translations/7.x/combo-box/combo-box-7.x-1.0-alpha1.ja.po
Comment #49
drummLet's try this commit.
Comment #50
drummCloser, the
/7.x
is missing.PURGE /files/translations/openaid-7.x-2.x.ja.po
vs.
PURGE /files/translations/7.x/openaid/openaid-7.x-2.0-rc3.ja.po
Comment #52
drummThe custom final parameter of
l10n_packager_get_filepath($release, $language, '%project-%branch.%language.po');
is what's not getting the full path. That might be able to be rewritten to be a bit better for the symlink creation. For now, I didn't want to risk breaking it, and returned those lines of code to as originally written.I added a second call to
l10n_packager_get_filepath($release, $language)
to construct theuri
for clearing.Comment #53
drummThe default
%release
does need to be%branch
.Comment #55
drummI think this is finally good now:
vs.
All my concerns have been addressed. We may want a site-wide CDN clear to get us to a clean cache slate.
Comment #56
Gábor Hojtsy@drumm: amazing! If we can get a purge for
/files/translations/*
that would be great! No need to purge the rest of the d.o project downloads.Comment #57
drummRudy went ahead and purged ftp.drupal.org. This type of purge happens for the whole ftp.drupal.org subdomain.
I confirmed the point of presence I was hitting for https://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.x.de.po went from beta1 to beta15. (See comment #35)
I'm going to be bold and call this fixed.
Comment #58
Gábor HojtsyYay, looks good indeed :)