[meeting] Migrate Meeting 2020-10-08 [#3174626]

Hello all, it’s time for the weekly migration initiative meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to here: https://www.drupal.org/project/drupal/issues/3174626. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

Core migration issues:

0️⃣ Who is here today? What is your favorite flavor of ice cream?

damienmckenna	Mango, or mint, or... ah just pass it over, I'm not that picky when it comes to frozen treats :grin:
larowlan	lee, I don’t like ice cream, but pistachio or coconut gelato is :ok_hand:
benjifisher	I have a recipe for ginger ice cream that is a lot of work, but the result is outstanding.
dinarcon	Hello :wave:
alison	Alison here! Chocolate / death by chocolate type flavors... any flavor with coffee and chocolate... Sometimes lemon... Fresh peach sundaes made with vanilla custard...NOTHING WITH NUTS :woman-gesturing-no: :blob_no: :no_entry_sign:
larowlan	actually, I change my order https://www.weis.com.au/products/macadamia-mango-and-ice-cream-bar/ is top-shelf - worth looking up if you like mango damien
irinaz	hi, I like coffee flavor
quietone	Coconut in the summer and pistachio any time
heddn	Some flavor of dark chocolate. Maybe with nuts.

1️⃣ What should we talk about today? Suggest topics here and I will add threads.

larowlan

I wanted to convey how much the migrate APIs rock, been working on a client migration from Grav and it has come out so well. I’ll release a migrate_source_grav contrib module.

2️⃣ Action items. To be added later.

benjifisher	NR (Major): #3006750: Move memory management from MigrateExecutable to an event subscriber. @benjifisher
benjifisher	Documentation:Alphabetize the glossary and generally review, add, update as needed. @dinarcon, @alison Start a page of pipeline snippets/recipes/examples. @benjifisher
benjifisher	FM review of #3167267: MigrateExecutable should catch not only exceptions, but also fatal errors: @larowlan
benjifisher	Continue work on #3150949: Add a migration source plugin for JSON:API: @quietone, @benjifisher

3️⃣ Statistics

benjifisher	Fixed since last week's meeting: 4 (not counting issues for meetings).
benjifisher	RTBC: 6, including one Major. Two have been waiting more than a month.
benjifisher	NR: 20, including 4 Major. One has been waiting more than a month (and is Major). I started reviewing that one before the meeting, and I will finish today.

4️⃣ Documention

benjifisher	As promised, @dinarcon started work on a glossary: https://www.drupal.org/docs/drupal-apis/migrate-api/migrate-api-overview.... #DrupalThanks!
benjifisher	If you nudge me, I might start a page with examples of process pipelines. For example, clean up whitespace in an HTML string. (This is what I came up with this week. There is room for improvement.) - plugin: callback callable: htmlentities - plugin: str_replace search: - ' ' - ' ' replace: ' ' - plugin: str_replace regex: true search: '@\s+@' replace: ' ' - plugin: callback callable: trim
dinarcon	This bring back memories when there were code snippet sections on drupal .org. I guess they still exists. Having like a "cookbook" with migration "recipes" would be useful.
dinarcon	As for the glossary, I was tempted to add information about source/process/destination plugins, but they have their own section in the documentation. Not sure if it is worth repeating in the glossary and pointing to the respective pages. Also, the example migration goes beyond what a glossary is. Feedback on things to add/change/remove are very welcomed.
alison	Very cool! I was just going to ask if you have a list or anything in mind, for stuff that should be added.
alison	source/process/destination plugins -- repeating in the glossary and pointing to the respective pages^^ I like that idea (edited)
alison	I think the Glossary might need to be on its own page -- but that could wait til it's longer / til we're sure it's going to be longer :slightly_smiling_face:
dinarcon	Some time ago I wrote https://understanddrupal.com/articles/configuration-options-migration-ya... to explain which configuration options are available in migration files. Having every possible key in the glossary might be overkill, but things like migration tags, groups, and dependencies could be added. Also, an overview of id and uuid and how they are used in the system. For instance, the id is used to run the migration and create the migrate map/messages tables. (edited)
alison	Oh and, alphabetization, an important attribute of a glossary :slightly_smiling_face: .....but I recognize that it might be complicated with multi-word (or "phrase") entries. But, still worth alphabetizing, IMO.If folks agree, I'd be happy to help with that -- there aren't too many now, so if we have them alphabetized now, it'll (hopefully) convey to anyone adding more entries to do so alphabetically.
benjifisher	Yes, glossaries should be alphabetical. Be bold, and edit it!I think we agreed at the start to move it to a separate page if it gets big enough.
dinarcon	Yes, having them in alphabetical order makes a lot of sense. The only reason I wrote them in the current order was to try to follow the order in which concepts appear in a migration. But again, having an example migration in a glossary is probably out of scope. +1 on having them in alphabetical order.
alison	Cool, I'll do that -- just wanted to make sure there was consensus :slightly_smiling_face:
alison	re: your message about configuration options etc. -- interesting, yeah, I think it makes sense to add the examples you mention?? I can imagine it becoming complicated, deciding what to include / not include... Sometimes it's ok to just wait to worry about that kind of thing, see how things go. I'm not sure if this is one of those times, buuuut.....
benjifisher	Yes, the idea of the "example pipelines" page is to have usable snippets or recipes. I would also link to the issue to make pipeline snippets reusable; for now I do a lot of copy/paste.
alison	example snippets / recipes -- yessssssssssssss
alison	Also, there are examples in source code files, so, sometimes could be enough to link to those examples?
benjifisher	I think embedded examples will be a lot easier to use than links for this (vaporware) page.
alison	Ok! (What vaporware page?)@dinarcon Are you thinking the process pipeline example will be moved to the future snippet-examples/recipes page? (Not right this second.)
dinarcon	@alison yes, a dedicated snippet page/section is created, the current example in the glossary can be moved over. For now, I would say to keep it there.
benjifisher	Yes, at the top of this thread, I suggested a new page for examples of pipelines. That is the vaporware.
alison	Yeah I saw your message earlier -- hmm definitely not familiar with the term vaporware... Sounds fun. (googled) (edited)
benjifisher	As in "The JSON:API source plugin has progressed from half-baked idea to vaporware".
alison	Alphabetization done.
alison	I was relieved that I didn't have to make a decision about how to treat the connecting word "as".....
benjifisher	Thanks!
dinarcon	Thank you @alison !
alison	my pleasure!
alison	I'll think about what else to add.Do you think it's worth adding words without definitions, like "groups: to do" or something, or no benefit to doing that?I think it could encourage ppl to fill stuff in, or just, having more things there encourages adding more things... Plus, having more examples is helpful for conveying the intent of the section. (edited)

5️⃣ MigrateExecutable should catch not only exceptions, but also fatal errors

benjifisher	#3167267: MigrateExecutable should catch not only exceptions, but also fatal errors
benjifisher	This one is tagged for FM review. @larowlan, since you are here today, can you have a look?
larowlan	sure

6️⃣ Move memory management from MigrateExecutable to an event subscriber

benjifisher	#3006750: Move memory management from MigrateExecutable to an event subscriber
benjifisher	This is the issue I am in the middle of reviewing.

7️⃣ Recent successes that we can share.

benjifisher	From 1️⃣:I’ll release a migrate_source_grav contrib module.
larowlan	yeah, we blew the clients’ socks off with a migration, and it would not have been possible without all the hard work that has gone into the migrate apis
larowlan	so thank you everyone who has made the apis what they are :heart:
dinarcon	I have mentioned this before, but many migrations written for D8 can be run on D9 with minimal or no changes. Last week I updated several of the examples in the 31 days of migration series. The only required change to most modules was adding core_version_requirement: ^8 \|\| ^9 to the *.info.yml file. :blue_heart:
larowlan	yeah, this was a d9 project
benjifisher	I have a few process plugins that I plan to contribute to migrate_plus:dom_select: similar to dom_extract (currently NR) to extract strings/attributes from a DOMDocument objectdom_remove: once you have extracted stuff, remove it from the DOMDocument objecttranspose: if you have multiple URLs in one source property and multiple link titles in another, now you can combine them into one array-of-arrays for use with the sub_process plugin to generate links.
alison	oo dom_remove that would've been amazing to have in our project last year (back when I was more involved here)... yeah I love all of those...Transpose makes me think of Python "zip" :zipper_mouth_face:

8️⃣ Add a migration source plugin for JSON:API

benjifisher	#3150949: Add a migration source plugin for JSON:API
benjifisher	@quietone has done a lot of work recently, and @wimleers (he/him) has given the first review. This is high on my todo list.
benjifisher	@wimleers (he/him): If you read the earlier comments on the issue,@quietone started by copying some code from the Migrate Plus module, which explains why some parts look like they are designed for generic JSON.There are some comments about how the JSON:API source should be different from the generic JSON source.I think you answered a few of the earlier questions, such as whether there is anything better than json_decode(). Thanks!

9️⃣ DrupalSpoons vs. d.o for Migrate Plus, Migrate Tools, etc.

benjifisher	We discussed this last week, even though it came up after the usual end of the meeting. @heddn, are you willing to move these back to d.o?
benjifisher	IIRC, issue forks will be available for all projects in about a month. There has already been one core issue fixed with a fork and a merged branch instead of a patch.
heddn	The UX on drupal's issue forks is a still a little rough. I'm not sure the right answer here. I used it on a recent core issue that just landed and was confused about the whole process the entire time. I wish we had a more unified and easier to use toolset available to us. That would make the decisions easier. Much of the tooling in drupal spoons are head and shoulders above what we see in issue forks. It doesn't come without some downsides though :grimacing:
benjifisher	You have worked with it more than I have. I am optimistic that the DX on d.o will improve over time, probably based on the DrupalSpoons experiment.For the record, here is a tweet with the announcement: https://twitter.com/drupal_org/status/1313575017586716672?s=20

1️⃣0️⃣ Config schema definitions for plugins aren't applied to their derivatives - Migration plugins

benjifisher	#2320937: Config schema definitions for plugins aren't applied to their derivatives - Migration plugins
benjifisher	We discussed this issue at last week's meeting, and there was some further discussion on a thread outside the meeting. I think @quietone and @mikelutz (he/him) were involved.Can we just close it as outdated?
benjifisher	I will close it tomorrow, unless someone beats me to it.

1️⃣1️⃣ Wrap up

benjifisher	Thanks for participating! I have already updated 2️⃣. Please continue to add comments in the threads. In 1-7 days, we will post a transcript for today's meeting.
dinarcon	Thanks for facilitating @benjifisher !
alison	Thank you, Benji!!

Participants:

damienmckenna, larowlan, benjifisher, dinarcon, alison, irinaz, quietone, heddn