[meeting] Migrate Meeting 2020-08-06 [#3162502]

Hello all, it’s time for the weekly migration initiative meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to here: https://www.drupal.org/project/drupal/issues/3162502. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues:

0️⃣ Who is here today? Is there one thing in the Migrate system you would love to see completed by the end of August?

steinmb	From my understanding, all/most documentation goes into code, right? Found #2916356: Add a documentation page for upgrading multilingual sites pending
alison	Hellooooo, Alison here. Nope, but I look forward to seeing what other ppl say.
dww	Derek. I can't believe I'm still up. ;) About to crash. But the 1 thing I'd most like to see fixed is #2780839: Optimize migration of specific source IDs for SQL sources
dinarcon	Hello, Mauricio here :wave:
quietone	Vicki, Multilingual migrations!!! #2208401: [META] Remaining multilingual migration paths

1️⃣ What should we talk about today? Suggest topics here and I will add threads.

dww	Potentially #2780839: Optimize migration of specific source IDs for SQL sources ?
dww	(Although I don't have time to discuss it myself synchronously right now, but I can come back to the thread later if we open it) ;)

2️⃣ Action items. To be added later.

benjifisher	NR: #3006750: Move memory management from MigrateExecutable to an event subscriber, @benjifisher
benjifisher	NW: #3134470: Switch to entity owner in EntityContentBase during validation, volunteer needed
benjifisher	NR: #3047328: Allow logging for non-strings values, @benjifisher
benjifisher	NW: #3063856: Add ability to view migrate_message table data, volunteer needed
benjifisher	NR: #2925899: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations, @benjifisher
benjifisher	NR: #3143717: Add helper methods to the migrate_drupal_ui functional tests, @benjifisher
benjifisher	Documentation: volunteer needed to review the handbook. See which pages can have "incomplete" removed, request updates for pages that actually need them.

3️⃣ Statistics

benjifisher	A bunch of issues were Fixed since the last meeting. Going by last comment (not exact) and ignoring the meeting issue, I count 7 issues.
benjifisher	RTBC: just two issues, tied for the lowest I have seen. Both have Normal priority and have been updated in the last 2 days.
benjifisher	NR: 47, including 8 that have been waiting more than two months and 3 Major.Thanks to @quietone, who got over her cold and moved several issues to NR in the last week.Several of these are my responsibility, but the Migrate maintainers cannot keep up without some community help!

4️⃣ Move memory management from MigrateExecutable to an event subscriber

benjifisher	#3006750: Move memory management from MigrateExecutable to an event subscriber
benjifisher	This is one of the issues that @quietone workd on recently. I plan to review before the next meeting.

5️⃣ Switch to entity owner in EntityContentBase during validation

benjifisher	#3134470: Switch to entity owner in EntityContentBase during validation
benjifisher	No updates since last week. @heddn has worked on this issue, but anyone could volunteer to fix things pointed out in the latest review.
heddn	I'm still swamped. and going on vacation next week, so it might be a few days before I can get to this.

6️⃣ Allow logging for non-strings values

benjifisher	#3047328: Allow logging for non-strings values
benjifisher	No updates since the last meeting. I plan to review soon.

7️⃣ Expose full set of debugging data in migrate_message table (filterable/searchable)

benjifisher	#3063856: Add ability to view migrate_message table data
benjifisher	No updates since the last meeting. @quietone and @Fredy have worked on this issue, but someone else could help fix things from the latest review.
steinmb	Was there not there an issue/idea a few weeks ago about exploring the possibility turning migrate_messages into entities? Enable user to leverage Views to list messages and so on?
benjifisher	I suggested that. There may already be an issue for that idea.
benjifisher	We probably still need this issue since migrate_drupal_ui should not depend on views. But if the messages were entities, then we might rework this issue to use entity queries.

8️⃣ MigrateUpgradeImportBatch does not use source_private_file_path ...

benjifisher	#2925899: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations
benjifisher	Yet another issue that @quietone has updated, and I plan to re-review.

9️⃣ Add helper methods to the migrate_drupal_ui functional tests

benjifisher	#3143717: Add helper methods to the migrate_drupal_ui functional tests
benjifisher	Yet another issue that @quietone has updated, and I plan to re-review.

1️⃣ 0️⃣ Documentation

benjifisher	#2916356: Add a documentation page for upgrading multilingual sites
benjifisher	There is certainly a lot of documentation in code, and that is used to generate api.drupal.org
benjifisher	But we also have a User Guide and also a handbook at https://www.drupal.org/docs/drupal-apis/migrate-api
benjifisher	Documentation in code is largely focused on what individual functions do. How-to guides are more likely to be in the Handbook.
benjifisher	The User Guide (https://www.drupal.org/docs/user_guide/en/index.html) does not have a chapter on migration, and maybe does not need one.
steinmb	Quite a lot of those handbook pages are tagged 'incomplete' though some are really good, example the source and process plugin pages or is it just me?
steinmb	Anyone got a better overview of the state of the handbook part?
steinmb	Regarding the "The User Guide" I personally think it is OK that migrate is not part of it. Perhaps in the future when there might be more functionality in a UI new users could leverage it to pull inn data without a lot of tech. knowhow?
benjifisher	I do not have an overview of the handbook. I updated the page on process plugins a couple of months ago, adding all the process plugins from other core modules. Maybe I should have removed the "incomplete" tag.It is hard to remove that tag. Do you really want to make the claim that documentation is complete?
steinmb	I find it hard to spot what is not there... well it always are :slightly_smiling_face:
dww	That's why so many modules are in alpha/beta forever. No one (myself included) wants to call something "1.0".
dinarcon	One area in which I would like to see more documentation automated testing. https://www.drupal.org/docs/8/api/migrate-api/migration-tests-in-drupal-8 is listed as incomplete. https://www.drupal.org/docs/8/api/migrate-api/generating-database-fixtur... points to https://www.drupal.org/docs/automated-testing/phpunit-in-drupal/running-... which is marked as out of date. If someone is available to give me an overview of their workflow, I could update the documentation pages. (edited)
alison	I'm usually good at writing documentation, or do people tell me. Idk if I can help -- I know I can't help with automated testing or multilingual documentation, but maybe other documentation... So maybe not today's highlighted topics, but maybe in the future.Removing the incomplete indicator: Super difficult, hmm... There isn't documentation about the incomplete indicator, is there? (I'm only 20% kidding.)^^ I guess I don't think there seems to be...https://www.drupal.org/drupalorg/docshttps://www.drupal.org/drupalorg/do... I mean, we could ask #documentation. I'll do that, see if they have advice. (edited)
dww	Hah, so meta!
alison	Related thread over there: https://drupal.slack.com/archives/C220WV2TW/p1595014261063200?thread_ts=... Doesn't seem to be a "rule," but that's not surprising. Place to ask for advice (if it's a concern) might be the the Documentation working group / Documentation issue queue:https://www.drupal.org/governance/doc-working-group
alison	Maybe it's not important,but once I started down the rabbit hole, I couldn't stop.

1️⃣ 1️⃣ Optimize migration of specific source IDs for SQL sources

benjifisher	#2780839: Optimize migration of specific source IDs for SQL sources
benjifisher	:+1: for this issue. It could use an issue summary update, using the template. Especially the "Remaining tasks" section.
dww	I might have funding to work on this in the next couple of weeks. If so, I'll definitely rock it. But if anyone else wants to be involved, please do! I agree that the exact implementation in the current patch referencing 'idlist' which only migrate_extras knows about isn't going to fly as-is.
dww	But yeah, even getting the summary, scope, plan in shape would be great, then when I finally have dedicated time to work on it, I can implement it exactly how we want. :wink:
dww	It'd also be a nice stretch goal if the API we implement could work on non-SQL sources, if the source itself can handle it in some fancier way than raw iteration and matching each ID directly.
benjifisher	There are precedents for providing an API that is only used in contrib. For example,Core provides the node grant system, but does not use it.Core allows multiple parents and aliases for taxonomy terms, but does not provide a way to create those.
dww	Yeah, I'm fine if core doesn't directly use the API (at least not at first), but it shouldn't be relying on specific implementation details of contrib if something cleaner + more obvious is available.
dww	That said, maybe migrate UI wants to expose this feature... could be really helpful for UI migrate users, in the same way drush mim --idlist is (could be) useful for CLI folks developing / debugging specific migrations.
benjifisher	That would be a follow-up issue. :wink:
benjifisher	I can paste in the IS template now.
benjifisher	Done. @mikelutz, if you have something more specific to say in the "Proposed resolution" section, that would be welcome.
mikelutz	I remember this one. IIRC, I objected to doing it in core since core had no means to limit source ids, but confess that for the sake of migrate_plus, the code to manage it really needed to be in SqlBase
mikelutz	As long as it can be done in a way were SqlBase exposes an api to do it, and then we tweak migrate_plus/tools to use the new api, I’m fine with doing it. What I don’t want to see is SqlBase being bent backwards specifically to make migrate_plus/tools be more efficient as they are.
benjifisher	Any specific ideas about what that API would look like? Maybe a public limitRows() method (for the sake of discussion)? The default implementation would be a noop, but any source plugin that can do better, such as SqlBase, could override it.
benjifisher	Or does it have to be more complicated than that?
benjifisher	There is also an issue out there to provide ways to modify the list of rows by adding conditions. Like a generic way to restrict to a particular content type/bundle. If I can find it, then I will add it as a related issue.
mikelutz	I remember that one. I really like that as a more generic solution.
benjifisher	#3069776: SQL source plugins: allow defining conditions and join in migration yml
heddn	I'm pretty sure that the migrate tools that gabe and wim built could also benefit from a solution here. lots of benefit. especially if we make it generic and build on top of the sql conditions
mikelutz	Well, what I always liked about the sql conditions is that you could basically make one sql source plugin that could be configured for almost any query directly in the migration. You would only need a custom plugin if you needed to do something custom in the prepare_row method.
mikelutz	It would move a good chunk of custom migration work from php programmers into yml configurations.
mikelutz	But I’m also not certain if that would necessarily easily translate into something contrib could use to make --idlist more efficient.
mikelutz	because in theory that means taking the id list, and modifying the migration source configuration before it’s ran to add a sql condition on the id list, but that leaves lots of situations where you might get buggy behavior, from caching, mapping, complex queries that don’t join right, etc.
heddn	I'm only going for an 80% solution. for those cases where it is a complex join or buggy, there's still write your own
mikelutz	Sure, but that’s a downgrade (from the contrib side, I mean). Right now --idlist works 100%, if we add sql conditions to core, and then tools starts using them for --idlist, then --idlist suddenly working 80% of the time wouldn’t be acceptable.
mikelutz	And I don’t think it would be trivial for tools to be able to tell when it needs to fall back to iterator filters.
heddn	another flag to mim :slightly_smiling_face:
benjifisher	Let me see if I understand what you are saying. There is no fundamental difference between adding the conditionssourceid1 = 123type = pageto the query. So the difference between the two issues is being able to add conditions like that from the migration plugin (typically YAML config) or adding a condition when the migration is run.
mikelutz	I suppose another flag it is..
mikelutz	Untitled source: plugin: sql conditions: - type: orgroup Click to expand inline (16 lines)
mikelutz	This would be generally useful, say to use the node source plugin but restrict to a certain content type.
mikelutz	With this setup, tools would load the migration, build a condition around the source ids and the --idlist and append it to the source configuration array prior to running the migration.
mikelutz	But we have to make sure that the source coutn cache doesn’t get polluted by this condition, or you will have weird numbers in ms. You would also have to make sure it doesn’t change the hash for --track_changes ( I don’t recall if source config is included in that hash, or just the source data)

1️⃣ 2️⃣ Wrap up

benjifisher	Thanks for participating! I already updated 2️⃣ . Please continue to add comments. In 1-7 days, I will post a transcript for today's meeting.
alison	Thank you, Benji! .....reading after the fact...

Participants:

steinmb, alison, dww, dinarcon, quietone, benjifisher, heddn, mikelutz