Problem/Motivation

Since #1914070: Improve version fallback for install language. at least Drupal 8 will download alpha2 translations properly. Even if Drupal 8 alpha5 is out. (Then once a beta1 is out, that will be the new base for translation downloads and then rc1, but even if rc3 is out).

Proposed resolution

Ideally we'd have a server side version fallback system where if we ask for 8.x-dev downloads, it would *redirect* us to the latest version downloads (so we know the version downloaded from the file name and can track).

Remaining tasks

1. Work out the server side solution (this issue).
2. Remove fallback logic from core as necessary and rely on server side fallback. Rewrite tests with mock server side (#2113955: Rely on proper server side version fallback for translations).

User interface changes

None.

API changes

None.

#1914070: Improve version fallback for install language.
#2113955: Rely on proper server side version fallback for translations

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Gábor Hojtsy’s picture

Issue summary: View changes
Issue tags: -revisit before beta
Gábor Hojtsy’s picture

Also added as a blocker to #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1. Depending on the availability and compatibility of git_deploy with translations, this may or may not block a Drupal 8 release.

mgifford’s picture

YesCT’s picture

adding "infrastructure blocker for Drupal 8.0.0" tag, since this blocks the drupal 8 core issue: #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1, so that the infrastructure blockers to d8 release (in issue queues outside of the core queue) is accurate. Remove the blocker tag when this issue is fixed, and update 2267715.

webchick’s picture

Trying to get a handle on release blockers prior to 8.0.0 and what (if anything) the community can do to help things along in preparation of https://events.drupal.org/losangeles2015/sessions/plain-drupal-english-g...

So the issue seems to be that right now translations will only be against the first X release (alpha 1, beta 1, rc 1), not subsequent X releases (alpha 3)?

Is this a place where community members can jump in at LA? If so, could the issue summary be updated with an outline of where to start?

Gábor Hojtsy’s picture

Version: » 7.x-1.x-dev

@webchick: The issue right now is

(1) Drupal core needs to be manually updated to fall back on a certain translation because it does not know its version without git_deploy. Without manual updates like #2285795: Drupal 8 loads alpha2 translations instead of alpha12, core now falls back on beta1 translation downloads.

(2) Drupal core tries multiple HTTP requests to download translations to decide on which one is available which makes the installation slower; if the server would decide for it, it would only need one HTTP request and be directed to the right one with a redirect or symlink or something along those lines and be quicker to install

@catch requested to keep this as a blocker because otherwise we manually need to update fallback versions in core each release if we want new string translations to be downloaded.

webchick’s picture

Hm. Well but we already have to change the VERSION variable manually each release. Couldn't https://www.drupal.org/files/issues/fallback-to-alpha12-2.patch just be added to the release checklist? There's a clear workaround here.

Gábor Hojtsy’s picture

Based on our IRC chat a couple months back @catch preferred not to add it to the release checklist but to keep this issue open (and revisit before release instead).

webchick’s picture

OK, I'll talk to him.

Then it sounds like what is needed here is:

- A custom module for localize.drupal.org
- That defines a URL at something like /translations/$project/$major-version
- Hitting that URL with "translations/drupal/8.0.x" returns a JSON response or something with the string "8.0.x-beta10" (atm, would change to 8.0.x-beta11 next month)

Am I remotely in the right ballpark here?

opdavies’s picture

A custom module for localize.drupal.org

@webchick: https://www.drupal.org/project/localizedrupalorg is now that. :) The 7.x-1.x branch is what I'm using on the l.d.o staging site.

Gábor Hojtsy’s picture

@webchick: if heavily cached yeah. Update status had this problem before that hitting a PHP equipped server was too expensive so it was switched to static exports and a static only server, which can serve a lot more requests. Currently localize translations are hosted on ftp.drupal.org so Drupal 8 will not hit any dynamic URL but just poll the static files one by one. We can make the static files symlink to prior ones when l.d.o gets to know about a release or make them do URL redirects or switch over the system entirely and do some dynamic stuff like you suggested.

Note that the "JSON" you suggested is available in the update info, so Drupal 8 *could* query that for version info. The reason it does not do that is (a) it would guarantee it needs 2 HTTP requests as opposed to the current where the first polled .po file may already exist (b) the versions available in the update info may not all have a .po file generated yet, so we would still need to go through multiple versions anyway. Ideally ftp.drupal.org would have something at every possible version number for a project, if only a symlink until that .po file is generated and a .po file symlink or redirect on the MAJOR.X version number which is symlinked or redirected to the latest version. I think for server performance this should be solved on ftp.d.o, but better versions of the solution are welcome.

webchick’s picture

Ok, in that case it sounds like this is only something the d.o infra team can do.

Gábor Hojtsy’s picture

Well, the .po files are actually generated by code in l10n_server on l.d.o directly and rsynced to ftp.d.o already. So theoretically it can be solved in l10n_server IF a solution is agreed on and that solution is transparently supported by the current rsync setup. The current setup may need to be tweaked as it only happens once or twice a day. Also the cron for the .po file generation may need to be tweaked. Also core changes are needed to request the MAJOR.X .po file once it actually exists. So its coordinated changes at minimum across l.d.o, infra and finally core.

hestenet’s picture

Issue tags: +Localize D7 port
Gábor Hojtsy’s picture

@hestenet: this is currently not implemented on Drupal 6, so not sure how is this a D7 port issue?!

hestenet’s picture

Issue tags: -Localize D7 port

@Gábor Hojtsy - my mistake! Was moving too quickly through the localize issues to see if we'd missed anything for the port.

japerry’s picture

So I dived into the D8 code for locale module and it appears that it is not hitting localize.drupal.org at all. Instead its calling ftp.drupal.org

 default_server_pattern: 'http://ftp.drupal.org/files/translations/%core/%project/%project-%version.%language.po'

Unless I'm reading this incorrectly, we would need to rewrite the locale module to make a call to localize.drupal.org to grab the correct version.
Psuedo code:

  $release_version = get_version_from_ldo(_install_get_version_info($version))
  $package = download_package_po(LOCALE_TRANSLATION_DEFAULT_SERVER_PATTERN)

Alternatively, there would need to be a php wrapper between the file call and what is being grabbed.

For example, the url above: http://localize.drupal.org/translations/8.0-dev/image/image-8.0-dev.de.po
would end up going to localize.drupal.org, which would process and cache the current release representation of dev.

The first issue would be undesirable, since two remote calls would be required.
The second issue, while possible, would also be not desirable, since we would be loading po files from drupal, not static ftp. Maybe there is a way to rewrite the ftp folders to symlink to a release?

Objectives

  • Keep one call to remote po files
  • Remote server must cache request
  • Remove complexities for finding last release from dev

Unless I'm totally misunderstanding the issue.. ;)

Gábor Hojtsy’s picture

Yeah the ftp folders are rsynced from l.d.o's files folder. I think the best way here would be to somehow symlink things and ensure the rsync brings those over.

drumm’s picture

The current setup may need to be tweaked as it only happens once or twice a day.

That is no longer true. ftp.drupal.org hits a CDN, which either hits their cache, or forwards the request directly to our servers. Expiration times are relatively long, but we actively clear the CDN's cache when packaging a file happens.

drumm’s picture

Part of the solution might be rewrites in a web server layer. For example: if a request doesn't match an existing file, then serve something like image-8-latest.de.po, which is a symlink to the latest for that branch.

This reduces the number of symlinks we need to maintain, only one per branch we want to separate out.

The CDN cache clearing could need to switch over to a wildcard request, to be sure we are clearing enough.

basic’s picture

japerry’s picture

Status: Active » Reviewed & tested by the community

So I can pretty much declare this one fixed and/or RTBC at least.

to test new releases, go here (for newly released versions of d8 that don't have packaging yet):
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-beta15....

To test dev, go here:
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-dev.de.po

both should redirect to drupal-8.0.x.de.po which is a symlink to drupal-8.0.0-beta14.de.po

Now onto fixing d8.

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Needs work

Both of those links result in beta12 downloads for me. Header text:

# German translation of Drupal core (8.0.0-beta12)

Looking at https://localize.drupal.org/translate/languages/de German does have a download for beta14, so this does not seem to be right?

A final important question is how much max-age is put on these files and/or how would a front CDN invalidate them when the symlinks change? Drumm mentions it could be done, but nobody mentioned it was done.

Gábor Hojtsy’s picture

Another problem. Someone reported that https://ftp.drupal.org/files/translations/7.x/pathauto_persist/pathauto_... does not exist (which is not surprising given https://localize.drupal.org/download?project=pathauto_persist does not list German translation for this module either). However, it redirects to https://ftp.drupal.org/files/translations/7.x/pathauto_persist/pathauto_... (rightfully) which redirects to itself in an infinite loop unfortunately :/ I think the redirect needs to be modified to not redirect if the file did not exist.

basic’s picture

This /should/ be working now.

+    rewrite '^/files/translations/(.+)/(.+)/drupal-([0-9]\.([0-9]\.)?)[0-9].+\.([a-z-]+)\.po$' 
https://ftp.drupal.org/files/translations/$1/$2/drupal-$3x.$5.po;
+    rewrite '^/files/translations/(.+)/(.+)/(.+)-([0-9]\.([0-9]\.)?x)-([0-9]+)\.([0-9]+)\.([a-z-]+)\.po$' 
https://ftp.drupal.org/files/translations/$1/$2/$3-$4-$6.x.$8.po;

This cleans up the regex a bit, preventing it from matching itself and will now send you a 302 -> 404 for things that don't exist.

As noted on IRC, some further cleanup could happen:

<vandemar>	the groups that you don't end up using, perhaps make them (?:...) instead of (...) so the rewrite target numbering can be consecutive?
<basic`>	oh, that's a good idea as well, ?: will do that?
<vandemar>	I'd probably name them if I was doing that, to avoid having to count groupings
<vandemar>	yep, (?:blah) means don't capture it to a variable, and (?<name>) captures it to be used later as $name instead of $1 or whatever the # is
<basic`>	so (?<release>[0-9]\.([0-9]\.)?)
webchick’s picture

Status: Needs work » Needs review

Back to needs review.

drumm’s picture

Issue tags: +needs drupal.org deployment

Debugging was never added to the Fastly purge requests. I'll be deploying that.

drumm’s picture

Project: localize.drupal.org » Localization server
Component: Infrastructure » Code
Status: Needs review » Needs work
Issue tags: -needs drupal.org deployment

That is now deployed, and as far as I can tell, the CDN cache is not cleared properly.

l10n_packager_create_latest_symlink() should invoke hook_l10n_packaged() so l10n_drupal_rest_l10n_packaged() makes the PURGE request.

Gábor Hojtsy’s picture

@drumm: I think you meant *now* working properly not *not*?

Gábor Hojtsy’s picture

Tested the following:

http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-beta15.... (now is actual translation file, no redirect, YAY)
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-beta25.... (redirects to .x, YAY)
http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-dev.de.po (downloads beta1 translation NOT GOOD :/)

http://ftp.drupal.org/files/translations/7.x/pathauto_persist/pathauto_p... (redirect to .x and then a 404 YAY)

Given the NOT GOOD item, looks like the implementation of this one and #2113955: Rely on proper server side version fallback for translations would get us back where we started, it would still not download the most recent .po for a dev or a nonexistent version, but instead an old one (beta1). The goal of doing it on the server side is/was because the server knows about the release history and can symlink to the most recent one not a much older one.

drumm’s picture

I think you meant *now* working properly not *not*?

The debugging is working well as debugging. I didn't do a thorough read of the logs yet. I expect to see a PURGE for the .x files when their equivalent full version is repackaged. I haven't seen that, but only was able to look at one packaging cron run.

We should make sure the redirects themselves are either not cached, have a short-lived cache, or are also PURGEd appropriately.

Sutharsan’s picture

http://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.0-dev.de.po currently downloads drupal-8.0.x.de.po
Am I missing something or was the CDM purged/expired in the mean time?

Sutharsan’s picture

But the file's content is 8.0.0-beta1. Same observation as @Gabor in #30 :(

drumm’s picture

There wouldn't have been a purge pushed today. A change would have been a normal expiration, or hitting a different state on a different point of presence.

drumm’s picture

This looks like a symptom of the problem I described in #28. Notice the second request, I added ?a to bypass the CDN's cache, and got the correct file.

drumm@Neils-MacBook-Pro:~/Downloads$ curl https://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.x.de.po | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# German translation of Drupal core (8.0.0-beta1)
# Copyright (c) 2015 by the German translation team
#
msgid ""
msgstr ""
"Project-Id-Version: Drupal core (8.0.0-beta1)\n"
"POT-Creation-Date: 2015-08-31 09:34+0000\n"
"PO-Revision-Date: YYYY-mm-DD HH:MM+ZZZZ\n"
"Language-Team: German\n"
"MIME-Version: 1.0\n"
curl: (23) Failed writing body (0 != 16384)
drumm@Neils-MacBook-Pro:~/Downloads$ curl https://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.x.de.po?a | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# German translation of Drupal core (8.0.0-beta15)
# Copyright (c) 2015 by the German translation team
#
msgid ""
msgstr ""
"Project-Id-Version: Drupal core (8.0.0-beta15)\n"
"POT-Creation-Date: 2015-09-07 10:13+0000\n"
"PO-Revision-Date: YYYY-mm-DD HH:MM+ZZZZ\n"
"Language-Team: German\n"
"MIME-Version: 1.0\n"
curl: (23) Failed writing body (0 != 11584)

The symnlink in the filesystem does look good:

drupal-8.0.x.de.po -> /var/www/localize.drupal.org/htdocs/sites/default/files/l10n_packager/8.x/drupal/drupal-8.0.0-beta15.de.po

Sutharsan’s picture

Status: Needs work » Needs review
FileSize
1.35 KB

I guess we have now proven that the CDN must be purged. This patch should purge the symlink when a new one is created as requested in #28. Patch is not tested.

Gábor Hojtsy’s picture

Status: Needs review » Needs work

To verify this patch I went to look at how this hook is defined. It is not. Looks like it was not documented when introduced. It is used in http://cgit.drupalcode.org/l10n_server/tree/l10n_packager/l10n_packager.... with a pretty full file object. Its hard to know what should the file object contain. The hook was added by @drumm in #2419637: Add CDN purging on packaging. @drumm can you help us document it?

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
2.9 KB
3.67 KB

So I did some digging for the source and found that we only use the URI of the file anyway. We can change this API to pass on the file URI string or document the object. First pass at documenting the hook and renaming it for responsible contrib patterns to prefix with module name.

Gábor Hojtsy’s picture

FileSize
3.73 KB
1.03 KB

Minor code style updates.

  • Gábor Hojtsy committed 92ea4e8 on 7.x-1.x
    Issue #2113957 by Gábor Hojtsy, Sutharsan, drumm, basic, japerry: Build...
Gábor Hojtsy’s picture

Committed that. We'll need to deploy and then monitor if things work as expected.

Gábor Hojtsy’s picture

Issue tags: +needs drupal.org deployment

Status: Needs review » Needs work

The last submitted patch, 39: 2113957-39.patch, failed testing.

Gábor Hojtsy’s picture

Failed because I was happily committing the patch before the test finished. The branch test is green after commit, see https://www.drupal.org/node/148272/qa.

drumm’s picture

Status: Needs work » Needs review
Issue tags: -needs drupal.org deployment

Looks good:

14:59:00     [request] => PURGE
14:59:00 /files/translations/var/www/localize.drupal.org/htdocs/sites/default/files/l10n_packager/7.x/oa_core/oa_core-7.x-2.x.hu.po
…
14:59:00     [code] => 200
drumm’s picture

Status: Needs review » Reviewed & tested by the community

My other concern was:

We should make sure the redirects themselves are either not cached, have a short-lived cache, or are also PURGEd appropriately.

I think this is okay regardless of caching. Any redirect will be consistent until a real file takes its place. When that happens, the new file will come with a PURGE request.

drumm’s picture

Status: Reviewed & tested by the community » Needs work

I forgot to actually read the URI, that's not right.

/files/translations/var/www/localize.drupal.org/htdocs/sites/default/files/l10n_packager/7.x/oa_core/oa_core-7.x-2.x.hu.po
vs.
/files/translations/7.x/combo-box/combo-box-7.x-1.0-alpha1.ja.po

  • drumm committed 228e1c7 on 7.x-1.x authored by Gábor Hojtsy
    Issue #2113957 by Gábor Hojtsy, Sutharsan: Build server side version...
drumm’s picture

Status: Needs work » Needs review
Issue tags: +needs drupal.org deployment

Let's try this commit.

drumm’s picture

Closer, the /7.x is missing.

PURGE /files/translations/openaid-7.x-2.x.ja.po
vs.
PURGE /files/translations/7.x/openaid/openaid-7.x-2.0-rc3.ja.po

  • drumm committed d68d33f on 7.x-1.x
    Issue #2113957 by Gábor Hojtsy, Sutharsan, drumm: Build server side...
drumm’s picture

The custom final parameter of l10n_packager_get_filepath($release, $language, '%project-%branch.%language.po'); is what's not getting the full path. That might be able to be rewritten to be a bit better for the symlink creation. For now, I didn't want to risk breaking it, and returned those lines of code to as originally written.

I added a second call to l10n_packager_get_filepath($release, $language) to construct the uri for clearing.

drumm’s picture

The default %release does need to be %branch.

  • drumm committed b30d62f on 7.x-1.x
    Issue #2113957 by Gábor Hojtsy, Sutharsan, drumm: Build server side...
drumm’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -needs drupal.org deployment

I think this is finally good now:

18:04:00     [request] => PURGE
18:04:00 /files/translations/7.x/panelizer/panelizer-7.x-3.0.ja.po HTTP/1.0

vs.

18:04:00     [request] => PURGE
18:04:00 /files/translations/6.x/gallerix/gallerix-6.x-1.x.ja.po HTTP/1.0

All my concerns have been addressed. We may want a site-wide CDN clear to get us to a clean cache slate.

Gábor Hojtsy’s picture

@drumm: amazing! If we can get a purge for /files/translations/* that would be great! No need to purge the rest of the d.o project downloads.

drumm’s picture

Status: Reviewed & tested by the community » Fixed

Rudy went ahead and purged ftp.drupal.org. This type of purge happens for the whole ftp.drupal.org subdomain.

I confirmed the point of presence I was hitting for https://ftp.drupal.org/files/translations/8.x/drupal/drupal-8.0.x.de.po went from beta1 to beta15. (See comment #35)

I'm going to be bold and call this fixed.

Gábor Hojtsy’s picture

Yay, looks good indeed :)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.