[meeting] Migrate Meeting 2020-07-30 [#3160932]

Hello all, it’s time for the weekly migration initiative meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to here: TODO. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues:

0️⃣ Who is here today? Do you have any pets?

webchick	Hi there! webchick from Vancouver, :canada: I have a cat named cooper. :slightly_smiling_face:
benjifisher	Benji from the Boston area. I have a cat named Qwerty.
heddn	Lucas, from Leon Nicaragua. No pets.
quietone	Vicki, Wellington. Two cats, Oreo and Max.
damienmckenna	Damien in NH, USA. Three cats, one of whom lives to take my seat :wink:
benjifisher	Several cats, no dogs. Who knew?
alison	Hi! Alison in Skaneateles, NY.Had a dog named Poz, he died recently :disappointed: :purple_heart:Here he is, helping me work, and just being his pretty self. (edited)
alison	Lol, timing
benjifisher	I am sorry. I lost one of my cats in January. :disappointed:
damienmckenna	I'm sorry for you both. We lost a dog in iirc November and cat in iirc March :disappointed: It sucks y'all. (edited)
damienmckenna	(not trying to turn this into a pity olympics, just loosing pets sucks)
alison	Oof. I'm sorry for y'all, it absolutely sucks!!
alison	No no, you're sharing / relating / expressing empathy :)
alison	I'm so sorry for your losses :hugs:
dinarcon	Mauricio here :wave: sorry for your losses as well :blue_heart:

1️⃣ What should we talk about today? Suggest topics here and I will add threads.

quietone	#2913428: Concat process plugin should filter empty values just want to confirm moving to Migrate Plus
damienmckenna	Would like to have a chat around documentation in the migration/*.yml files: #3161696: Document how and why d7_node_title_label works
quietone	I would really like this simple one to get to RTBC #3116841: Correct latest revision for node 1 in drupal6 test fixture

2️⃣ Action items. To be added later.

benjifisher	#3006750: Move memory management from MigrateExecutable to an event subscriber: @quietone
benjifisher	#3134470: Switch to entity owner in EntityContentBase during validation: @heddn
benjifisher	#3047328: Allow logging for non-strings values: @benjifisher
benjifisher	#3063856: Add ability to view migrate_message table data: @quietone or @Fredy
benjifisher	#2925899: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations: @benjifisher
benjifisher	#2913428: Concat process plugin should filter empty values: Mark as "won't fix" with a comment. @quietone
benjifisher	#3143717: Add helper methods to the migrate_drupal_ui functional tests: Check the re-roll. @benjifisher
benjifisher	That last one is now RTBC.

3️⃣ Move memory management from MigrateExecutable to an event subscriber

benjifisher	#3006750: Move memory management from MigrateExecutable to an event subscriber
benjifisher	Nothing new since last week's meeting. @quietone?
quietone	No action due to passing a cold through the family. Should see action this week.
benjifisher	Do not overwork yourself! (But I think this one is pretty close.)

4️⃣ Switch to entity owner in EntityContentBase during validation

benjifisher	#3134470: Switch to entity owner in EntityContentBase during validation
benjifisher	No updates since last week's meeting. @heddn?

5️⃣ Allow logging for non-strings values

benjifisher	#3047328: Allow logging for non-strings values
benjifisher	@quietone and I made some progresss on this one. It is now NR, which means it is my responsibility.

6️⃣ Expose full set of debugging data in migrate_message table (filterable/searchable)

benjifisher	#3063856: Add ability to view migrate_message table data
benjifisher	I tested and added some screenshots since last week's meeting. We need an updated patch. @quietone or @Fredy?
quietone	Hopefully, Fredy will get to it.

7️⃣ Statistics

benjifisher	Only one issue Fixed since last week.
benjifisher	RTBC: 6, including one Major. I know that @quietone and I care especially about #3143717: Add helper methods to the migrate_drupal_ui functional tests, part of the meta issue to clean up the functional tests. I would like to see that one land before we continue.@webchick, can you nudge the core committers? (edited)
benjifisher	Also, is there some other channel I should be using to talk to the core committers?
heddn	@benjifisher no other channel. alex and lee have reviewed in the past. if its multi lingual in nature, gabor will jump in and Angie has picked up more UX related improvements to the UI.
benjifisher	NR: 42, including 3 Major.
heddn	Just ping them here or in #contribute if you have a specific item you want to see land.
quietone	A few days ago I asked Gábor to look at #3110669: Migrate d7 menu language content settings
webchick	Yeah, #contribute is where the core committers mostly hang out, tho a few of us are in individual initiative channels too
webchick	https://www.drupal.org/project/issues/drupal?text=&status=14&priorities=... looks like this list?
webchick	Any particularly blocking or just generally would be good to see commit activity?
quietone	#2845485: Refactor and document the MenuLinkParent process plugin - This is the last is one in a Meta for process plugins. Would feel good to get that work done.
webchick	I can probably take #3160031: Fix 18 spelling errors for migrate specific terms#new right now... except DEAR LORD how is it 60K?! :scream:
benjifisher	Earlier in this thread I mentioned one that @quietone and I care about, and there is one Maror.
quietone	#3143717: Add helper methods to the migrate_drupal_ui functional tests This one is for my well being.
benjifisher	Yes, that one^.
webchick	Ok!
webchick	"Migrate maintainers are asking for their RTBC queue to get a poke in the weekly #migrate meeting. #3143717: Add helper methods to the migrate_drupal_ui functional tests "part of the meta issue to clean up the functional tests. I would like to see that one land before we continue."Both seem to need framework manager-ish review."Sent. :slightly_smiling_face:
benjifisher	:clap::clap:
quietone	Unfortunately, #3143717: Add helper methods to the migrate_drupal_ui functional tests needed a reroll. It is running tests now.
benjifisher	Good catch!
benjifisher	I should probably check all the RTBC issues as part of the weekly meeting. RTBC issues are re-tested every two days. The testbot changes the status to NW if a test fails, but not if the patch fails to apply.

8️⃣ MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, ...

benjifisher	#2925899: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations
benjifisher	This will be one of my priorities before the next meeting.

9️⃣ Concat process plugin should filter empty values

benjifisher	#2913428: Concat process plugin should filter empty values
quietone	@heddn, shall we just the issue to Migrate Plus?
heddn	@quietone Well, does it need to go there even? I thought we had a working solution and it might not even need to land there.
quietone	And the idea of making an array_filter process plugin? What happens to that?
heddn	callback?
quietone	Ok. I'll won't fix it and point folks to #2882276: Extended callback process plugin to call functions with multiple parameters
benjifisher	I remember being annoyed when the array_pop plugin went in--or maybe it was the null_coalece plugin--because you could get the same result with a little pipeline: callback with array_filter and array_values and array_pop or something like that.
benjifisher	Right, we could use a few more generic plugins like that ^^ and not a plugin for every use case.Once that one lands, we can deprecate the format_date plugin, or whatever it is called.

1️⃣ 0️⃣ Document how and why d7_node_title_label works

benjifisher	#3161696: Document how and why d7_node_title_label works
quietone	@damienmckenna can you say more about the problem. It seems that skipping rows in that migration was unexpected and caused some confusion.
damienmckenna	This speaks to a bigger issue: if you run a migration and run migrate:status you'll have discrepancies between the source record count and how many were migrated, then need to dig into it to work out why.
damienmckenna	My experience is based from starting with migrate_upgrade and reviewing what comes out at the end (which is how most people should do it imho).
benjifisher	Doc blocks in PHP code are extracted and then shown on a.d.o. The only way to see comments in a YAML file (I think) is if you look at the YAML file.
damienmckenna	You run a migration, you have all these records which appear to be "missing", so I started with the yaml files. (edited)
damienmckenna	IMHO every single record that is skipped should have a message indicating why, so as I work through these few site migrations I'll be working on patches for various modules to add notes about it, thanks to the message option on skip_on_empty that @quietone pointed me towards.
benjifisher	I wonder if a new documentation page might be more useful, or maybe hust a couple of extra paragraphs on an existing one.Based on your experience, is the hard part knowing where to look (modules//migrations/.yml) or understanding the skip_on_empty once you are there? (edited)
damienmckenna	The issues I faced were:What is this specific migration doing,Why did it not migrate all of the records.
benjifisher	The first question could be answered by a short description at the top of each migration file. That might be worth the effort. It would certainly be worth the effort if we had something like phpdoc for YAML files, so we could collect all those descriptions.
quietone	@damienmckenna what is your ideal solution?
benjifisher	The second question is partially answered by knowing to look for skip_on_empty with method: row. The could be a paragraph or two of documentation. In most cases, I do not think that the individual skip_on_empty is worth a comment.
benjifisher	For reference, we are talking about this: label: - plugin: static_map source: title_label bypass: true map: Title: 0 - plugin: skip_on_empty method: row
damienmckenna	I opened an issue somewhere to request that after a migration runs the number of records created + messages logged = number of source records, because someone looking at a migration just won't know what happened, and IMHO you shouldn't be required to look in the codebase to see why. (edited)
damienmckenna	^ would be my ideal solution.
damienmckenna	So yes, Benji, there are two issues.
benjifisher	That would mean a policy of treating the message as required fro method: row.It there ever a way to log more than one message for a single skipped row?
damienmckenna	If you're neck-deep in migrations and comfortable with what's going on you can knowingly dismiss differences between source & destination record counts, when you're not (imho most people will not be) it's a serious point of confusion and wasted effort (& project budget) tracking down why the numbers are different.

1️⃣ :one: Correct latest revision for node 1 in drupal6 test fixture

benjifisher	#3116841: Correct latest revision for node 1 in drupal6 test fixture
benjifisher	That is one of the ones that has been waiting over two months. That puts it near the top of my priority list, but I will probably not get to it this week.
quietone	Queued a retest
quietone	Fortunately, it is just a reroll. It was RTBC by mikelutz.
quietone	@mikelutz can you review this?
benjifisher	It was previously RTBC by @mikelutz. I could do a conditional review pretty quickly, just confirming the re-roll. (edited)
benjifisher	oops, cross-posts
benjifisher	This issue is now RTBC, thanks to @quietone and @dww.

1️⃣ 2️⃣ Wrap up

benjifisher	Thanks for participating! I already updated 2️⃣ . Please continue to add comments. In 1-7 days, I will post a transcript for today's meeting.
quietone	@benjifisher thank you, another well run meeting. If you like, I can post the transcript.
benjifisher	Please do. Thanks!

Participants:

webchick, benjifisher, heddn, quietone, damienmckenna, alison, dinarcon