Problem/Motivation

Following up from #2669964: Migrate Drupal 7 core node translations to Drupal 8, there are lots of related data to nodes that are now united in the migration to the same nodes. Say nodes 6, 7, 8, 9 and 10 all get migrated to node/6 (because 7, 8, 9, 10 are all translations of node 6), this leaves 404s at node/7, node/8, /node/9 and /node/10. Their respective path aliases are migrated to node 6 with #2850984: Fix path alias migration of translated nodes [D7] and #2827644: Fix path alias migration of translated nodes [D6].

However there is likely other related data (entity references, menu items, etc) that is not updated (yet?). This is arguably data integrity for dead links in content etc., so making this major and 'migrate critical' to at least document if not fix.

Proposed resolution

We don't have path_redirect in core, and aliases probably aren't the right fix here (to alias the node/9 path itself to node/6), but for example it would be possible to keep an nid->tnid map somewhere as part of the migration and have a listener that looks for 404s, consults the URL vs. the map and redirects if it finds a match (in an optional module).

Remaining tasks

Discuss. Implement.

User interface changes

None.

API changes

None anticipated.

Data model changes

None anticipated.

CommentFileSizeAuthor
#107 interdiff-2850085-103-107.txt2.48 KBmaxocub
#107 2850085-107.patch17.04 KBmaxocub
#103 2850085-103.patch17.02 KBmaxocub
#103 interdiff-2850085-96-103.txt2.12 KBmaxocub
#96 interdiff-2850085-95-96.txt919 bytesmaxocub
#96 2850085-96.patch17 KBmaxocub
#95 interdiff-2850085-81-95.txt13.34 KBmaxocub
#95 2850085-95.patch17.02 KBmaxocub
#81 interdiff-2850085-79-81.txt3.33 KBmaxocub
#81 2850085-81.patch14.18 KBmaxocub
#79 interdiff-2850085-78-79.txt17.32 KBmaxocub
#79 2850085-79.patch14.19 KBmaxocub
#78 interdiff-2850085-71-78.txt5.15 KBmaxocub
#78 2850085-78.patch17.7 KBmaxocub
#71 interdiff-2850085-62-71.txt4.31 KBmaxocub
#71 2850085-71.patch15.24 KBmaxocub
#62 interdiff_57-62.txt1.35 KBheddn
#62 2850085-62.patch15.61 KBheddn
#61 2850085-61.patch15.63 KBheddn
#57 interdiff-2850085-51-57.txt10.46 KBmaxocub
#57 2850085-57.patch15.64 KBmaxocub
#51 interdiff-45-51.txt10.46 KBmaxocub
#51 2850085-51.patch16.99 KBmaxocub
#45 interdiff-42-45.txt2.49 KBmaxocub
#45 2850085-45.patch15.64 KBmaxocub
#42 interdiff-40-42.txt950 bytesmaxocub
#42 2850085-42.patch15.13 KBmaxocub
#40 interdiff-39-40.txt796 bytesjofitz
#40 2850085-40.patch15.04 KBjofitz
#39 interdiff-37-39.txt889 bytesmaxocub
#39 2850085-39.patch15.05 KBmaxocub
#37 interdiff-34-37.txt7.98 KBmaxocub
#37 2850085-37.patch14.96 KBmaxocub
#34 interdiff-33-34.txt1.27 KBmaxocub
#34 2850085-34.patch15.2 KBmaxocub
#33 interdiff-31-33.txt3.53 KBmaxocub
#33 2850085-33.patch15.17 KBmaxocub
#31 interdiff-20-31.txt1.19 KBmaxocub
#31 2850085-31.patch14.7 KBmaxocub
#20 interdiff-18-20.txt5.04 KBmaxocub
#20 2850085-20.patch14.4 KBmaxocub
#18 2850085-18.patch12.09 KBmaxocub
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

catch created an issue. See original summary.

mikeryan’s picture

Title: Aliases/redirects for translation set migration path » Aliases/redirects for translation set migration path in D7
Status: Active » Postponed
Issue tags: +migrate-d7-d8
Related issues: +#2827644: Fix path alias migration of translated nodes [D6], +#2669964: Migrate Drupal 7 core node translations to Drupal 8

This was addressed for D6 in #2827644: Fix path alias migration of translated nodes [D6] - it looks like all that's needed for D7 is to make the corresponding changes to d7_url_alias.yml and add tests. We can either do this as a follow-up to #2669964: Migrate Drupal 7 core node translations to Drupal 8, or incorporate it into that patch.

Gábor Hojtsy’s picture

Status: Postponed » Active

IMHO do it as a followup, we don't do any other reference updates, eg. where comments were posted to nodes, or where menu items are pointing to the node, etc. All of those are also broken. Aliases are not anywhere more special than the rest of the data referencing the node that is no longer there.

Also we do not have a redirection solution in core that I know, so we could only do the aliasing updates.

Gábor Hojtsy’s picture

Status: Active » Postponed
catch’s picture

Also we do not have a redirection solution in core that I know, so we could only do the aliasing updates.

We don't have a generic redirect solution in core, but we could have an optional module that handles redirects for old translation set node IDs to the merged node. It could listen to 404s, check the request path, look up the map, send a RedirectResponse.

Gábor Hojtsy’s picture

In that case this would reach back to Drupal 6's data too.

catch’s picture

Yes we'd need a k/v collection or similar that keeps the nid -> tnid mapping.

Gábor Hojtsy’s picture

I believe #2827644: Fix path alias migration of translated nodes [D6] is using migrate's ID map tracking, but for this we would need to somehow have a more permanent tracking data store(?)

maxocub’s picture

Gábor Hojtsy’s picture

Title: Aliases/redirects for translation set migration path in D7 » Redirects for translation set migration path in Drupal 6 and 7
Issue summary: View changes

Updated issue summary given #2850984: Fix path alias migration of translated nodes [D7] so we can focus on the redirect problem here.

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

maxocub’s picture

I'd like to work on this. Why is it postponed?

Gábor Hojtsy’s picture

Issue tags: +D8MI, +language-content, +sprint

It was originally postponed on #2669964: Migrate Drupal 7 core node translations to Drupal 8. I think it may still be postponed on #2850984: Fix path alias migration of translated nodes [D7] unless there is no overlap in where they change things.

maxocub’s picture

Assigned: Unassigned » maxocub
Status: Postponed » Active

I don't see any overlap with #2850984: Fix path alias migration of translated nodes [D7], so I put it back to Active.

I have a working proof of concept, but I'm wondering where to put the code, right now it's in a custom module.

The code consists only of two event subscribers, one for the migrate.post_row_save event (to populate a key/value collection) and the other for the kernel.exception event (to redirect 404s based on the key/value collection).

I can think of these three possible locations:

  • As a hidden sub-module of the node module (although I don't see any of those in core)
  • Directly in the node module (If adding two services only used in on migrated multilingual sites is not a problem)
  • In a contrib module

What would be the best approach here?

Gábor Hojtsy’s picture

@maxocub: I think @catch meant a separate module in core that the migration enables but otherwise is hidden.

maxocub’s picture

@Gábor Hojtsy: OK, thanks, I'll upload a patch when my tests are fixed.

maxocub’s picture

Assigned: maxocub » Unassigned
Status: Active » Needs review
Issue tags: +migrate-d6-d8
FileSize
12.09 KB

Here's a first draft.

I have a hard time trying to test the redirection, if anyone have a better idea, please tell me.

Status: Needs review » Needs work

The last submitted patch, 18: 2850085-18.patch, failed testing.

maxocub’s picture

Status: Needs work » Needs review
FileSize
14.4 KB
5.04 KB

Trying to fix the failing tests.

heddn’s picture

Can we just use redirect.module instead? I really don't like the feel of this direction...

Gábor Hojtsy’s picture

@heddn: is that in core? :)

catch’s picture

redirect module, at least in Drupal 6 and 7 has a performance issue on large sites given it does a database lookup on every request to see if there's a redirect. I haven't looked to see how it does things in 8.x yet.

The approach here allows us to do a lookup only if the response would otherwise return a 404, so it's a lot more efficient, and it'll only be enabled on migrated sites anyway.

heddn’s picture

I'm suggesting we add a sub-module to redirect.module that does just what we need. But a lot of the plumbing and test cases are pretty solid in that module and I'd be worried about conflicts between the two modules. And a UI and the UX of a similar but different module in core.

catch’s picture

There'd be no UI for this at all, it wouldn't even be visible on sites without migrate installed.

Gábor Hojtsy’s picture

Assigned: Unassigned » Gábor Hojtsy

Assigning to myself for review.

Gábor Hojtsy’s picture

Assigned: Gábor Hojtsy » Unassigned
  1. +++ b/core/modules/migrate_drupal/migrate_drupal.services.yml
    --- /dev/null
    +++ b/core/modules/migrate_drupal/src/EventSubscriber/NodeTranslationSubscriber.php
    
    +++ b/core/modules/migrate_drupal/src/EventSubscriber/NodeTranslationSubscriber.php
    @@ -0,0 +1,95 @@
    +  public function onPostRowSave(MigratePostRowSaveEvent $event) {
    +    if (substr($event->getMigration()->getPluginId(), 2, 17) == '_node_translation') {
    ...
    +  public function onPostImport(MigrateImportEvent $event) {
    +    if (substr($event->getMigration()->getPluginId(), 2, 17) == '_node_translation') {
    

    Hm, should node module handle this? I don't know how these subscribers are implemented elsewhere.

  2. +++ b/core/modules/node/tests/src/Kernel/Migrate/d6/MigrateNodeTest.php
    @@ -30,6 +30,8 @@ protected function setUp() {
    +      'd6_language_types',
    +      'd6_language_negotiation_settings',
    
    +++ b/core/modules/node/tests/src/Kernel/Migrate/d7/MigrateNodeTest.php
    @@ -46,6 +46,8 @@ protected function setUp() {
    +      'd7_language_types',
    +      'd7_language_negotiation_settings',
    
    +++ b/core/modules/path/tests/src/Kernel/Migrate/d6/MigrateUrlAliasTest.php
    @@ -31,6 +31,8 @@ protected function setUp() {
    +      'd6_language_types',
    +      'd6_language_negotiation_settings',
    

    This seems unrelated?

  3. +++ b/core/modules/node/tests/src/Kernel/Migrate/d7/MigrateNodeTest.php
    --- /dev/null
    +++ b/core/modules/node_translation_redirect/node_translation_redirect.info.yml
    
    +++ b/core/modules/node_translation_redirect/node_translation_redirect.info.yml
    +++ b/core/modules/node_translation_redirect/node_translation_redirect.info.yml
    @@ -0,0 +1,7 @@
    
    @@ -0,0 +1,7 @@
    +name: Node translation redirect
    

    Should this be a sub-module under migrate_drupal?

  4. +++ b/core/modules/node_translation_redirect/src/EventSubscriber/ExceptionSubscriber.php
    @@ -0,0 +1,86 @@
    +/**
    + * TODO
    + */
    +class ExceptionSubscriber implements EventSubscriberInterface {
    

    Should this be same as the module description? Don't think we need to be very creative here :)

  5. +++ b/core/modules/node_translation_redirect/src/EventSubscriber/ExceptionSubscriber.php
    @@ -0,0 +1,86 @@
    +    $path = $event->getRequest()->getPathInfo();
    +    if ($status_code == 404 && preg_match('/^\/node\/(\d+)$/', $path, $matches)) {
    

    This is always the resolved internal URL? I think it would be useful to highlight that we already migrate path aliases too.

  6. +++ b/core/modules/node_translation_redirect/src/EventSubscriber/ExceptionSubscriber.php
    @@ -0,0 +1,86 @@
    +      $collection = $this->keyValue->get('node_translation_redirect');
    

    Hm, how does this work if a site migrates data from different sources? Eg. a Drupal 6 and a Drupal 7 site? That may not be a 80% use case, but sounds like this would break there, given the different source node ids possibly overlapping.

quietone’s picture

Status: Needs review » Needs work

Looks like there are questions to be answered... Setting to NW

mikeryan’s picture

Issue tags: +Baltimore2017
maxocub’s picture

I'm working on this in Baltimore.

maxocub’s picture

Status: Needs work » Needs review
FileSize
14.7 KB
1.19 KB
  1. I don't know, the only other non-migrate module implementing those subscribers is content_translation in ContentTranslationUpdatesManager. I'm really not sure where it's best to put those, maybe @catch could weight in?
  2. This is required for the tests to pass because the language module implements hook_modules_installed(), which is call when we install the node_translation_redirect module after we migrate translated nodes. And than hook updates language types:
    $negotiator = \Drupal::service('language_negotiator');
    $configurable = \Drupal::config('language.types')->get('configurable');
    $negotiator->updateConfiguration($configurable);
    
  3. I don't know, in #15 and #16 we kind of assumed that @catch meant a separate module. Also, there's no sub-module in core.
  4. Done.
  5. Added a comment.
  6. I don't think there's an acceptable way to diferentiate two identical ids comming from to Drupal sources. Should we just document the risks of migrating more than one Multilingual Drupal site? We could also throw an exception if someone try to migrate a second Drupal site and if one ID is already present in the key/value collection.

Status: Needs review » Needs work

The last submitted patch, 31: 2850085-31.patch, failed testing.

maxocub’s picture

Status: Needs work » Needs review
FileSize
15.17 KB
3.53 KB

Fixing coding standards and two tests.

maxocub’s picture

Fixing the remaining failing tests.

The last submitted patch, 33: 2850085-33.patch, failed testing.

heddn’s picture

Status: Needs review » Needs work

#27.1 I think this should be move into node module. We've tried pushing that type of stuff out to the module itself as much as possible.
#27.3 We split out sub-modules in contrib to bundle in the same package for drush, but in core, we don't need that. I don't think we need to switch from that and start creating sub-modules.
#27.6 I don't think this is a possibility that can be addressed. Unless I misunderstand your comment.

Moving back to NW for item #1.

maxocub’s picture

Status: Needs work » Needs review
FileSize
14.96 KB
7.98 KB

Moved the subscriber from migrate_drupal to the node module

Status: Needs review » Needs work

The last submitted patch, 37: 2850085-37.patch, failed testing.

maxocub’s picture

Status: Needs work » Needs review
FileSize
15.05 KB
889 bytes

We need to check if the MigrateEvents class exists before subscribing to migrate events now that the subscriber is in node.
Hope this fixes all the failing tests...

jofitz’s picture

Minor coding standards correction: use short array syntax, [].

mikeryan’s picture

Status: Needs review » Needs work
+++ b/core/modules/node/src/EventSubscriber/NodeTranslationRedirectSubscriber.php
@@ -0,0 +1,99 @@
+    if (substr($event->getMigration()->getPluginId(), 2, 17) == '_node_translation') {
...
+    if (substr($event->getMigration()->getPluginId(), 2, 17) == '_node_translation') {

Haven't done a full code review, but a casual look gives me some concern here - I'm not thrilled with depending on magic ID names. Could we determine this instead from the source plugin having ['translations' => TRUE]?

If not, at the very least s/2, 17/-17/ would enable this to work with migrate_upgrade-generated migrations of the form 'upgrade_d6_node_translation'.

maxocub’s picture

Status: Needs work » Needs review
FileSize
15.13 KB
950 bytes

@mikeryan: You're right, that was not very strong or flexible.
Now I check if we are migrating node entities by looking at the destination plugin and if they are translatable by looking at the translations configuration.
Although, instead of looking at the 'translations' key in the source plugin, as you suggested, I look at the 'translations' key in the destination plugin, since its already loaded. I don't think it matters, but I might be wrong.

Status: Needs review » Needs work

The last submitted patch, 42: 2850085-42.patch, failed testing.

c.nish2k3’s picture

@maxocub - this needs to be done for onPostImport() as well.

maxocub’s picture

Status: Needs work » Needs review
FileSize
15.64 KB
2.49 KB

@c.nish2k3: Oups, forgot about that one. To avoid duplicating the code there, I added a helper method.

Also, finally, I look at the 'translations' key on the source configuration instead of the destination, to avoid running that code if we are migrating sources other than Drupal. Let's see if it makes the 'MigrateExternalTranslatedTest' pass.

vasi’s picture

Assigned: Unassigned » vasi
heddn’s picture

Priority: Major » Critical

Parent meta is critical. Moving that priority down here. And closing out parent as this was the last active task.

Gábor Hojtsy’s picture

Priority: Critical » Major
maxocub’s picture

Status: Needs review » Needs work

It has been discussed during the weekly migrate hangout that we should use a dedicated table to store the redirection map instead of the key/value store, since it could become bloated on a big site. Back to NW.

Gábor Hojtsy’s picture

Not just the bloat. We discussed that you may be *done* with your migrations, but want to keep your site up including the redirection feature, which is useful way past when migrate was needed on the site.

maxocub’s picture

Status: Needs work » Needs review
FileSize
16.99 KB
10.46 KB

Here's a new version using a mapping table instead of the key/value store.

mikeryan’s picture

Assigned: vasi » mikeryan
heddn’s picture

Assigned: mikeryan » heddn

Assigning to myself for review.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

heddn’s picture

Assigned: heddn » Unassigned
Status: Needs review » Reviewed & tested by the community

I cannot see anything wrong in the code. I'm not sure I like the yucky feeling I get with the architecture implemented here, but I don't have any better ideas. So let's move along to RTBC.

catch’s picture

Status: Reviewed & tested by the community » Needs work

Why not make a new key/value collection for this rather than making a custom table? The k/v collection absolutely should be able to persist after migrations are finished, but putting it in a custom database table prevents for example moving the storage to a redis implementation or similar.

maxocub’s picture

Status: Needs work » Needs review
FileSize
15.64 KB
10.46 KB

I kind of prefer the key value collection too.
But since the node_translation_redirect module is essentially a single event subscriber, wouldn't it be better to just move it to the node module instead of adding a new module to Core?

heddn’s picture

Assigned: Unassigned » heddn
heddn’s picture

Issue tags: +Migrate Drupal

Inventing a tag to mark this as pertaining to migrate_drupal and therefore blocking its stable release.

heddn’s picture

heddn’s picture

Status: Needs review » Reviewed & tested by the community
FileSize
15.61 KB
1.35 KB

re #57 why a new module: I think the idea for a new module is that the K/V lookup is somewhat expensive and to turn it on/off, it is easier to do this with a new module. It is only needed for nodes that were migrated from an i18n source.

+++ b/core/modules/node_translation_redirect/tests/src/Kernel/Migrate/d6/NodeTranslationRedirectTest.php
@@ -0,0 +1,68 @@
+    $response = $kernel->handle($request);

+++ b/core/modules/node_translation_redirect/tests/src/Kernel/Migrate/d7/NodeTranslationRedirectTest.php
@@ -0,0 +1,70 @@
+    $response = $kernel->handle($request);

Nit: unused variable.

I've fixed the nit in the attached patch and marked this RTBC. I don't think removing an un-used variable makes me disqualified.

heddn’s picture

Should we be worried about bloat of K/V if someone has a TON of translated nodes on their site?

larowlan’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/core/modules/node/node.services.yml
    @@ -50,3 +50,8 @@ services:
    +  node.node_translation_redirect_subscriber:
    +    class: Drupal\node\EventSubscriber\NodeTranslationRedirectSubscriber
    +    arguments: ["@keyvalue", "@module_handler", "@module_installer"]
    +    tags:
    +      - { name: event_subscriber }
    
    +++ b/core/modules/node/src/EventSubscriber/NodeTranslationRedirectSubscriber.php
    @@ -0,0 +1,116 @@
    +    if (class_exists('\Drupal\migrate\Event\MigrateEvents')) {
    

    I was concerned about having this in the global container/event dispatcher for all sites - and seeing the way the getSubscribedEvents method is implemented confirms my concerns.

    We need to add a NodeServiceProvider class here that implements ServiceModifierInterface and only adds the event subscriber service if migrate module is enabled.

    For more info see https://www.previousnext.com.au/blog/overriding-services-drupal-8-advanc... and for a concrete example of conditionally adding a service if a module is enabled - see https://github.com/drupal-media/file_entity/blob/8.x-2.x/src/FileEntityS...

  2. +++ b/core/modules/node/src/EventSubscriber/NodeTranslationRedirectSubscriber.php
    @@ -0,0 +1,116 @@
    +    return !empty($source_configuration['translations']) && $destination_configuration['plugin'] == 'entity:node';
    

    nit: we can use === here

  3. +++ b/core/modules/node_translation_redirect/node_translation_redirect.info.yml
    @@ -0,0 +1,7 @@
    +name: Node translation redirect
    

    I'm also not convinced that a new module is the answer here. Will dig about with the service provider notion and see if we can conditionally add this in node too.

  4. +++ b/core/modules/node_translation_redirect/src/EventSubscriber/ExceptionSubscriber.php
    @@ -0,0 +1,89 @@
    +    if ($status_code == 404 && preg_match('/^\/node\/(\d+)$/', $path, $matches)) {
    

    We can't match this on path - because paths can be modified. We need to match it on route name - entity.node.canonical

  5. +++ b/core/modules/node_translation_redirect/src/EventSubscriber/ExceptionSubscriber.php
    @@ -0,0 +1,89 @@
    +      $old_nid = isset($matches[1]) ? $matches[1] : NULL;
    

    Similarly, once we use the route match service, we can use the raw parameter here instead of the regex matches

  6. +++ b/core/modules/node_translation_redirect/src/EventSubscriber/ExceptionSubscriber.php
    @@ -0,0 +1,89 @@
    +        $response->send();
    

    I think we should be doing $event->setResponse($response) here. Sending the response immediately prevents further execution.

  7. +++ b/core/modules/node_translation_redirect/tests/src/Kernel/Migrate/d6/NodeTranslationRedirectTest.php
    @@ -0,0 +1,68 @@
    +    $this->expectOutputString('<!DOCTYPE html>
    +<html>
    +    <head>
    +        <meta charset="UTF-8" />
    +        <meta http-equiv="refresh" content="1;url=/fr/node/10" />
    +
    +        <title>Redirecting to /fr/node/10</title>
    +    </head>
    +    <body>
    +        Redirecting to <a href="/fr/node/10">/fr/node/10</a>.
    +    </body>
    +</html>');
    +    $kernel->handle($request);
    
    +++ b/core/modules/node_translation_redirect/tests/src/Kernel/Migrate/d7/NodeTranslationRedirectTest.php
    @@ -0,0 +1,70 @@
    +    $this->expectOutputString('<!DOCTYPE html>
    +<html>
    +    <head>
    +        <meta charset="UTF-8" />
    +        <meta http-equiv="refresh" content="1;url=/is/node/2" />
    +
    +        <title>Redirecting to /is/node/2</title>
    +    </head>
    +    <body>
    +        Redirecting to <a href="/is/node/2">/is/node/2</a>.
    +    </body>
    +</html>');
    +    $kernel->handle($request);
    

    Once we change the code above to not use $response->send() we can then go $response = $kernel->handle() and then assert that the status code is 301 and the location is as expected - instead of asserting the raw output.

larowlan’s picture

Would anyone object to people needing this functionality having to add a container parameter to enable the redirect, instead of the hidden module?

so in sites/default/services.yml they need to add

parameters:
  migrate.node_translation_redirect: 1

And then we use that in the service provider to enable the subscriber?

Gábor Hojtsy’s picture

@larowlan: trick question: can migrate add that container parameter?

larowlan’s picture

If Drupal can write to sites/default/services.yml then yes it can.

I think we do that in tests...will confirm

larowlan’s picture

\Drupal\Core\Test\FunctionalTestSetupTrait::setContainerParameter is what I'm referring to.

But it also relies on having this in settings.php

$settings['container_yamls'][] = $app_root . '/' . $site_path . '/services.yml';

Although I note that is the default in default.settings.php.

Can migrate check those two things first?

i.e. that services.yml exists and is writable and that Settings::get('container_yamls', []) includes the services.yml file?

I will also ask around in case there is a better way

heddn’s picture

64.1: where we not trying to do the redirect in another module outside of migrate and using k/v instead of the migrate mapping takes so we could disable migrate module?

Also, I'd like a confirmation on my concern with k/v bloat

catch’s picture

The k/v table could get huge, if we're worried about that we should consider either changing the k/v database storage to be table-per-collection or subclassing maybe. No-one's going to be doing any joins on that table though, so it's really only inserts/deletes/updates that could cause any issues.

Sites running into problems due to that much content may need to use alternative storage for things anyway though.

maxocub’s picture

For a start, I fixed #64.2, #64.6 & #64.7.

About #64.4 & #64.5, I am not able to retrieve the route name with RouteMatch in the ExceptionSubscriber, it's always returning NULL, anybody have a clue why? This is what I'm doing:

  public function onException(GetResponseForExceptionEvent $event) {
    $route_name = RouteMatch::createFromRequest($event->getRequest())->getRouteName();
    // $route_name is always NULL.
    // ...

If the URL we are trying to reach does not exists, isn't it normal that we don't have a route name for it?
If so, how else can we check that the requested path is a node path, other than $request->getPathInfo();?

maxocub’s picture

About #64. & #64.5:
After a few shots at this, I don't think there's a way to use the route match service to replace the preg_match. If, for example, '/node/2' was a translation on D7 and so does not exists anymore on D8, then there's no way to match this path to a route. We will always end up with a NullRouteMacth. So for now, I don't see any way around a regex, unless someone has a better idea?

One thing though, the current regex is not OK, since it's not catching paths like '/fr/node/2'.

catch’s picture

Priority: Major » Critical

I'm looking at PathValidator, this is the closest to what we want to do:


  protected function getPathAttributes($path, Request $request, $access_check) {
    if (!$access_check || $this->account->hasPermission('link to any page')) {
      $router = $this->accessUnawareRouter;
    }
    else {
      $router = $this->accessAwareRouter;
    }

    $initial_request_context = $router->getContext() ? $router->getContext() : new RequestContext();
    $path = $this->pathProcessor->processInbound('/' . $path, $request);

    try {
      $request_context = new RequestContext();
      $request_context->fromRequest($request);
      $router->setContext($request_context);
      $result = $router->match($path);
    }
    catch (ResourceNotFoundException $e) {
      $result = FALSE;
    }
    catch (ParamNotConvertedException $e) {
      $result = FALSE;
    }
    catch (AccessDeniedHttpException $e) {
      $result = FALSE;
    }
    catch (MethodNotAllowedException $e) {
      $result = FALSE;
    }

In the case of an entity not existing, we should get ParamNotConvertedException back from the route matcher. At the point ParamConverterManager throws that exception, it puts the route path and route name in the exception message.

We could look at something like adding a route name property to ParamNotConvertedException, then it would be possible to take a path, try a route match, and if the path matches a route but the param can't be converted (i.e. entity doesn't exist) still determine what the route would have been.

The other approach would be to add a public version of Drupal\Core\Routing\Router::matchCollection() that gets back the possible routes for the path before processing, then could just check if any of them are entity.node.canonical.

Since this is tagged migrate critical and partially blocks #2746527: [META] Handle data related to Drupal 6 and 7 node translations with different IDs bumping to critical.

heddn’s picture

Assigned: heddn » Unassigned
heddn’s picture

Issue tags: +Vienna2017

Tagging this for some in person brainstorming in Vienna.

heddn’s picture

What would be nice is a list of issues that /need/ to be resolved for migrate drupal stability. Or at least as much as possible, a full list of what is remaining.

catch’s picture

So the migrate critical tag should be that list ideally, but there might be issues in it which aren't actually stable blockers and some not tagged that are - that's fixable though via triage.

maxocub’s picture

Status: Needs work » Needs review
FileSize
17.7 KB
5.15 KB

WIP, am I on a good track?

maxocub’s picture

Still a WIP. Got rid of the new module in favor of two new node services:

  1. NodeTranslationMigrateSubscriber, which is only added to the container if migrate is enabled and is responsible of saving the mapping between old translated nids to new ones during a multilingual migration.
  2. NodeTranslationExceptionSubscriber, which is only added to the container if a container parameter is set (still needs work) and is responsible of redirecting old translated nids to new ones.

I now have to work on setting that container parameter after a multilingual migration.

Status: Needs review » Needs work

The last submitted patch, 79: 2850085-79.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

maxocub’s picture

Status: Needs work » Needs review
FileSize
14.18 KB
3.33 KB

This should fix the tests.

maxocub’s picture

+++ b/core/modules/node/src/NodeServiceProvider.php
@@ -0,0 +1,38 @@
+    // Register the node.node_translation_exception service to the container if
+    // the node.node_translation_redirect container parameter is set
+    if (TRUE) {

This is the part that needs work an discussion.

What was suggested between #65 and #68 is that we add a container parameter to services.yml if it exists, is writable and is included in settings.php.

Those conditions might not be met in the majority of cases, and in those case the user might not be familiar or have no access to the creation of a services.yml file.

Should we look for another way or is that an acceptable solution?

catch’s picture

+++ b/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
@@ -0,0 +1,94 @@
+
+  /**
+   * Redirects not found node translations using the key value collection.
+   *
+   * @param \Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent $event;
+   *   The exception event.
+   */
+  public function onException(GetResponseForExceptionEvent $event) {
+    $exception = $event->getException();
+    $previousException = $exception->getPrevious();
+    if ($exception instanceof NotFoundHttpException && $previousException instanceof ParamNotConvertedException) {
+      $routeName = $previousException->getRouteName();
+      $parameters = $previousException->getParameters();
+      if ($routeName == 'entity.node.canonical' && isset($parameters['node'])) {
+        $oldNid = $parameters['node'];
+        $collection = $this->keyValue->get('node_translation_redirect');
+        if ($oldNid && $collection->has($oldNid)) {
+          list($nid, $langcode) = $collection->get($oldNid);
+          $language = $this->languageManager->getLanguage($langcode);
+          $url = $this->urlGenerator->generateFromRoute('entity.node.canonical', ['node' => $nid], ['language' => $language]);
+          $response = new RedirectResponse($url, 301);
+          $event->setResponse($response);
+        }
+      }
+    }
+  }

This looks a lot better. Even though I suggested extending the exception, architecturally it seems wrong. However that's comes from the entire pattern we've inherited from Symfony of using exceptions for control structure - 404s and 403s aren't actually exceptional but the normal functioning of a website. Given that limitation, it's good to get rid of the regexp.

We could probably get the previous exception only if the instanceof check passes for the first with an early return to save an extra level of nesting?

heddn’s picture

This is tough, the writing to a new or existing services file during the migration. It won't work for example on platform.sh, because the disk will be read-only. Instead, we can enable/disable a module. Which gets us back to where we started.

maxocub’s picture

Would a simple $container->setParameter() in the first if statement of NodeServiceProvider work? That seems too easy and I'm not familiar with container parameters.

If this works, it means we are 'activating' the redirections only if we have the migrate & language modules enabled, but them we can disable the migrate module and still have our redirections.

Also, re #83:
I added the early return if it's not a NotFoundHttpException.
One thing though, I don't understand if you are saying that the added parameters/methods to ParamNotConvertedException are not a good idea and we need to find another solution, or if it's bad but we have to live with it?

maxocub’s picture

Re #85:
No that won't work, because if we uninstall migrate then the container gets rebuilt and the parameter is lost. Bummer.

larowlan’s picture

Can we:

  • unconditionally add the event subscriber service to node
  • add a state flag in migrate that says 'i18n nodes were migrated'
  • in the event subscriber return early if that flag isn't set - possibly in the getSubscribedEvents method
quietone’s picture

I have the read the issue, installed the patch and had a brief look at the code. Everything looks good, so far. I mean that all the comments have been addressed but one.

What remains to do is to get an answer to maxocub question in #82.

Discussed with phenaproxima, and he would like to have input from another framework manager on this.

phenaproxima’s picture

I would like input from another framework manager, but if we can't get it:

in the event subscriber return early if that flag isn't set - possibly in the getSubscribedEvents method

This looks like a great suggestion to me.

phenaproxima’s picture

Status: Needs review » Needs work

I really like this patch a lot. I feel like it's a pretty elegant solution to this persistent problem, and I can't wait to see it land.

  1. +++ b/core/lib/Drupal/Core/ParamConverter/ParamNotConvertedException.php
    @@ -7,4 +7,45 @@
    +  public function __construct(string $message = "", int $code = 0, \Throwable $previous = NULL, $routeName = "", $parameters = []) {
    

    I could be wrong, but I think \Throwable is not a thing in PHP 5, so $previous needs to be \Exception.

    Also, $routeName should be $route_name, $parameters should be type hinted, and if $routeName and $parameters are required, we should make them required constructor parameters :) PHP lets constructors have any signature you want, even if it's incompatible with the parent constructor.

    Additionally, we need a full doc block (not @inheritdoc), since we're defining new parameters here.

  2. +++ b/core/lib/Drupal/Core/ParamConverter/ParamNotConvertedException.php
    @@ -7,4 +7,45 @@
    +  /**
    +   * Get the route name.
    +   *
    +   * @return string
    +   */
    +  public function getRouteName() {
    

    @return needs documentation, even if it's just "The route name" (lame, I know).

  3. +++ b/core/lib/Drupal/Core/ParamConverter/ParamNotConvertedException.php
    @@ -7,4 +7,45 @@
    +  /**
    +   * Get the route parameters.
    +   *
    +   * @return array
    +   */
    +  public function getParameters() {
    

    @return needs documentation. It's probably worth mentioning if these are the raw parameters, or the so-far converted ones.

  4. +++ b/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
    @@ -0,0 +1,98 @@
    +   * @param \Drupal\Core\KeyValueStore\KeyValueFactoryInterface $keyValue
    +   *   The key value factory.
    +   * @param \Drupal\Core\Language\LanguageManagerInterface $languageManager
    +   *   The language manager.
    +   * @param \Drupal\Core\Routing\UrlGeneratorInterface $urlGenerator
    +   *   The URL generator.
    

    Nit: These variable names should be using $snake_case, not $camelCase.

  5. +++ b/core/modules/node/src/EventSubscriber/NodeTranslationMigrateSubscriber.php
    @@ -0,0 +1,62 @@
    +   * @param \Drupal\Core\KeyValueStore\KeyValueFactoryInterface $keyValue
    +   *   The key value factory.
    

    Nit: Should be $key_value.

  6. +++ b/core/modules/node/src/EventSubscriber/NodeTranslationMigrateSubscriber.php
    @@ -0,0 +1,62 @@
    +   * Maps the old nid to the new one in the key value collection.
    

    This is a bit vague, can it be expanded to explain why it's storing this mapping? A paragraph or two would be useful. A change like this is best documented extensively in the code.

  7. +++ b/core/modules/node/src/NodeServiceProvider.php
    @@ -0,0 +1,39 @@
    + * Register the node_translation_redirect service if migrate is enabled.
    

    Nit: This doc comment isn't really accurate, and it repeats the comment in register(), which is more useful because the code is right there. So I think this can just say something generic, like "Registers services in the container."

  8. +++ b/core/modules/node/src/NodeServiceProvider.php
    @@ -0,0 +1,39 @@
    +      $container->register('node.node_translation_migrate', 'Drupal\node\EventSubscriber\NodeTranslationMigrateSubscriber')
    

    Nit: Can this use the ::class syntax instead?

  9. +++ b/core/modules/node/src/NodeServiceProvider.php
    @@ -0,0 +1,39 @@
    +      $container->setParameter('node.node_translation_redirect', TRUE);
    

    In discussion, we agreed to ditch the container parameter in favor of setting a state or config key. Given the circumstances, a container parameter may be too volatile for what we're trying to accomplish.

  10. +++ b/core/modules/node/src/NodeServiceProvider.php
    @@ -0,0 +1,39 @@
    +    if ($container->hasParameter('node.node_translation_redirect') && $container->getParameter('node.node_translation_redirect')) {
    

    As per the last comment, we don't need the if statement anymore.

  11. +++ b/core/modules/node/src/NodeServiceProvider.php
    @@ -0,0 +1,39 @@
    +      $container->register('node.node_translation_exception', 'Drupal\node\EventSubscriber\NodeTranslationExceptionSubscriber')
    

    Can we use ::class here?

  12. +++ b/core/modules/node/tests/src/Kernel/Migrate/d6/NodeTranslationRedirectTest.php
    @@ -0,0 +1,58 @@
    + * @group node_translation_redirect
    

    Not a valid group anymore :) This should probably have:

    @group migrate_drupal
    @group node
    
  13. +++ b/core/modules/node/tests/src/Kernel/Migrate/d7/NodeTranslationRedirectTest.php
    @@ -0,0 +1,60 @@
    + * @group node_translation_redirect
    

    Need to adjust this too. ;)

jofitz’s picture

Assigned: Unassigned » jofitz

I'll take a look at addressing @phenaproxima's review.

maxocub’s picture

Sorry Jo, I'm already on it at the migrate sprint in vienna, I just didn't thought of assigning it to myself

maxocub’s picture

Assigned: jofitz » maxocub
jofitz’s picture

No problem, @maxocub. Go for it!

I just wish I was with y'all in Vienna.

maxocub’s picture

Status: Needs work » Needs review
FileSize
17.02 KB
13.34 KB

@Jo Fitzgerald: Wish you were here to meet you!

Re #90:

  1. You're right, \Throwable is new in PHP7. I don't think we can make routeName and parameters required because we would need to change all the throw new ParamNotConvertedException calls everywhere and I think this will break BC.
  2. Done.
  3. Done.
  4. Done.
  5. Done.
  6. I added the documentation in the class doc block instead.
  7. Done.
  8. Done.
  9. Done.
  10. Done.
  11. Done.
  12. Done.
  13. Done.
maxocub’s picture

Removed "string" & "int" type hints since they won't work on PHP5.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Oh, I like it.

On the assumption that Drupal CI will pass it, I think this is RTBC. Thanks, @maxocub!

maxocub’s picture

Assigned: maxocub » Unassigned

Unassigning.

catch’s picture

Massive crosspost, one option that hasn't been discussed, haven't read comments from #88 onwards. If #87 stops the event subscriber running on every 404 then that seems like a better option than the below anyway:

Drupal\language\LanguageServiceProvider is probably the closest example and uses config - conditionally register the service provider in a compiler pass.

State is probably better than config here but principle should be the same - means it'll get checked on container rebuild as opposed to every 404.

One potential problem with State is that LanguageServiceProvider has to check ConfigBootstrapStorage - there's a @todo about the dependency issues with the compiler pass, not sure what that looks like for state. That's a general problem with compiler passes depending on config/state.

maxocub’s picture

Following discussions with @phenaproxima and @catch, here are some conclusions to questions we had to improve this patch:

Could we merge the two subscriber into one?

  • Based on #64.1, I'd say it is better as it is since we don't have to check if the migrate event classes exists.

What about checking for the state in the getSubscribedEvents()?

  • It doesn't work, even with \Drupal::state(). This is because that the container is not initialized at that point.
catch’s picture

Status: Reviewed & tested by the community » Needs work

This is looking very close. Found a few things though:

  1. +++ b/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
    @@ -0,0 +1,129 @@
    +
    +    // If the node_translation_redirect state is not set, we don't need to check
    +    // for a redirection.
    +    if (!$this->state->get('node_translation_redirect')) {
    +      return;
    +    }
    +
    +    $previousException = $exception->getPrevious();
    +    if ($previousException instanceof ParamNotConvertedException) {
    

    What about moving the state check after the ParamNotConvertedException check? Seems like $exception->getPrevious() should be cheaper than fetching from state. Possibly even after checking the route name.

    Also $routeName should be $route_name and $previousException should be $previous_exception per coding standards.

  2. +++ b/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
    @@ -0,0 +1,129 @@
    +        $collection = $this->keyValue->get('node_translation_redirect');
    +        if ($oldNid && $collection->has($oldNid)) {
    +          list($nid, $langcode) = $collection->get($oldNid);
    

    If we do $collection->has() followed by $collection->get() does this issue two queries or one? Seems like we could probably do $collection->get() all the time and check the result.

  3. +++ b/core/modules/node/src/NodeServiceProvider.php
    @@ -0,0 +1,42 @@
    +
    +    // Register the node.node_translation_exception service in the container if
    +    // the language module is enabled.
    +    if (isset($modules['language'])) {
    +      $container->register('node.node_translation_exception', NodeTranslationExceptionSubscriber::class)
    +        ->addTag('event_subscriber')
    +        ->addArgument(new Reference('keyvalue'))
    +        ->addArgument(new Reference('language_manager'))
    +        ->addArgument(new Reference('url_generator'))
    +        ->addArgument(new Reference('state'));
    +    }
    +  }
    

    Just to double check, is there a reason we can't do the state check here instead and avoid registering the listener altogether?

maxocub’s picture

Assigned: Unassigned » maxocub

Thanks for the review @catch, I'm on it.

maxocub’s picture

Assigned: maxocub » Unassigned
Status: Needs work » Needs review
FileSize
2.12 KB
17.02 KB

1. I moved the fetching from state after the route name check, and changed the variable names.
2. I removed the call to $collection->has(), $collection->get() already returns NULL if it's not set.
3. For the same reasons as why it can't be in the getSubscribedEvents() call, the container is not yet initialized.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

#103 looks lovely to me.

And +1 to @maxocub's reply to #101.3 -- we can't use container services in a service provider, because the container is in an indeterminate state (i.e., still in the middle of being built). LanguageServiceProvider, it seems to me, realized this and had to hack around it.

catch’s picture

One more thing, I've had about five crossposts trying to post this one, thanks for the quick responses:

+++ b/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
@@ -0,0 +1,129 @@
+ * Redirects non-existent migrated node translations.

- maybe this should be "Redirect migrated node translations to the combined node." Or "Redirect node translations that have been consolidated by migration."

This is a difficult concept to describe in one sentence, but the issue isn't that that translations don't exist but that they've been combined, it's the old nodes that don't exist any more.

maxocub’s picture

Assigned: Unassigned » maxocub

On it.

maxocub’s picture

Assigned: maxocub » Unassigned
FileSize
17.04 KB
2.48 KB

Here's it is.

webchick’s picture

Assigned: Unassigned » catch

Looks like @catch is all over this one.

  • catch committed 2df0fc0 on 8.5.x
    Issue #2850085 by maxocub, heddn, Jo Fitzgerald, Gábor Hojtsy, catch,...

  • catch committed 1c9a87d on 8.4.x
    Issue #2850085 by maxocub, heddn, Jo Fitzgerald, Gábor Hojtsy, catch,...
catch’s picture

Status: Reviewed & tested by the community » Fixed

I think this looks good now. I pondered whether we'd use it for anything other than redirections, but migration should handle the other cases per the meta-issue, so being clear what it's for seems better than leaving it open.

Fixed this on commit:

diff --git a/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php b/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
index 22a9014..795670f 100644
--- a/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
+++ b/core/modules/node/src/EventSubscriber/NodeTranslationExceptionSubscriber.php
@@ -80,7 +80,7 @@ public function __construct(KeyValueFactoryInterface $key_value, LanguageManager
   /**
    * Redirects not found node translations using the key value collection.
    *
-   * @param \Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent $event;
+   * @param \Symfony\Component\HttpKernel\Event\GetResponseForExceptionEvent $event
    *   The exception event.
    */

Committed/pushed to 8.5.x and cherry-picked to 8.4.x. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.