Performing a migration process may involve steps other than the actual migration of the data. One that keeps popping up is that certain modules can seriously slow down, say, node_save() (calias and pathauto, for example). When testing migration (unless you're testing that particular functionality), you want them disabled. And even on final migration, you want them disabled during migration, then enable and run bulk updates after all your nodes are created.

Also, you may have different migration processes you want to run. At times you'll just be migrating, say, articles and blog entries; at other times, users and subscriptions. Right now you need to go to the dashboard page, click the right set of toggles, and go. It'd be nice to just click one button for whichever process you want.

So, I want to integrate the migrate module with a workflow module (I'm planning on looking at Rules first, but welcome any other suggestions). The primary requirement is to be able to define any number of processes, each of which can consist of a number of steps that include things like clearing or importing a specified content set, enabling or disabling a module, setting a Drupal variable, etc. Ideally it would also support pausing for manual steps with notification - sometimes you really need to take a manual action in the middle of a migration, thus if the workflow can pause itself and notify you of that action, that would be very nice.

Any and all thoughts about this proposal welcome...

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented

Capturing hint from Moshe - check install_task_list() for ideas...

frankcarey’s picture

frankcarey CreditAttribution: frankcarey commented

Just did some rules integration for the context module, so I'm pretty familiar with it now. Rules has 3 types of hooks. Events, Conditions, and Actions. I think the first thing we should do is get the proper Events laid out. Some of the actions you are talking about (enabling / disabling modules) might be so general that they belong in their own non-migrate related module.

So Let's talk What migrate events to expose:
(feel free to change the labels of events, been a while since i've dug into migrate, but I'm starting it again today)

//---these seem very likely

Migrate Process Start
Migrate process End

Content Set Process Start
Content Set Process End

//---do we need these events?

Migrate Comment Start
Migrate Comment End

Migrate Node Start
Migrate Node End

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented
Priority: Normal » Minor

At the moment, I'm inclined to put off doing the full workflow definition feature for a later version - shell scripting drush commands should be an adequate means for the moment. See Migrate module workflow refactoring.

frankcarey’s picture

frankcarey CreditAttribution: frankcarey commented

I commented at that groups page, and I think that's fine. Just know that Rules integration is a pretty straight forward thing once it's time to take this back up again.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented
Title: Workflow integration » Rules integration
Version: 6.x-1.x-dev » 7.x-2.x-dev

Won't be tackled in Migrate 1. With Migrate 2, you can define steps in the migration process that do anything (not necessarily migrate data), so the original motivation is covered. Rules integration may still be nice to have, so that aspect remains open.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented

What would Rules integration do exactly? IMO this is now Won't Fix

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan commented
Status: Active » Postponed

I can imagine scenarios (not far off from my last big project) where importing one object at a time in the context of a workflow may make sense. That being said, this isn't going to be done until there's a concrete need for it, marking postponed.

manischewitzbacon’s picture

manischewitzbacon CreditAttribution: manischewitzbacon commented

I have a use case for this, I built a magazine site and used migrate to import users/digital subscribers from their old site into the new site. Now i need a way for them to periodically import users from their subscription management software. My first idea involved rules and migrate:

1. Create content type for import file (add field for file)
2. create rule that runs when new content of this type is saved
3. rule grabs file from content and runs migration using this file
4. rule shows success message.

is there another way to achieve this? that I'm missing?

pifagor’s picture

pifagor
Location 🇺🇦 Rivne
CreditAttribution: pifagor as a volunteer and at Drupal Ukraine Community for Drupal Ukraine Community commented
Status: Postponed » Closed (outdated)