Currently the installer just gets a list of all available migrations and then does $migration->processImport($options);
However, processImport() returns a value, and if that value is MigrationBase::RESULT_INCOMPLETE, then the migration should be run again.
Our code currently doesn't do that, which means that when the system is low on memory, the migration completes halfway through and the installation is completed without the complete set of content.
We need to rewrite our batch function to rerun the migration in that case. We also need special code for drush (since the batch doesn't help there), similar to the one in migrate.drush.inc.
Comments
Comment #1
bojanz CreditAttribution: bojanz commentedThis can wait until RC2
Comment #2
vasikethere's commit for batch function for kickstart migration to re-run for incomplete migration
https://code.drupalcommerce.org/#/c/463/
based on migrate_ui_batch() function code.
to do : special code for drush
Comment #3
jsacksick CreditAttribution: jsacksick commentedComment #4
joshmillerDid this get committed and not updated? It's the only 2.0 release blocker that wasn't "complete (fixed)" ?
Curious because I'm going through all the drush related issues in the que and this one seemed like it needed and update (10 weeks old).
Comment #5
nedjoGiven this is a processing-heavy operation, there may be a role here for drupal_set_time_limit(). I've opened a related issue on migrate: #1892296: Allocate sufficient time for migration processes.
Comment #6
nedjoRe #5. As Moshe pointed out in #1892296, batch API takes care of setting the time limit.
Comment #7
lsolesen CreditAttribution: lsolesen commented@joshmiller @bojanz Has this been fixed? It is tagged 2.0 release blocker but still needs review :)
Comment #8
bojanz CreditAttribution: bojanz commentedNo, this was never committed :/ Needs to be tested in a low memory situation (get the system to swap) to confirm that it retries / continues where the previous code doesn't.
Related bug report which might be useful in the future: #1851806: Install slow on import content
Comment #9
mglamanNo patch, putting back to "active" for now
Comment #10
GuyPaddock CreditAttribution: GuyPaddock at RedBottle Design, LLC for House at Work commentedTry this one. This is taken from the migration code we use in all of our installation profiles, which in turn was originally based on Commerce Kickstart's migration code, so this has come full circle. :)
Comment #11
GuyPaddock CreditAttribution: GuyPaddock at RedBottle Design, LLC for House at Work commentedHere's one that will pass Drupal conventions for i18n.
(Our original code has watchdog statements in this function as well, and they share the same messages as the ones we pass through t() for the Batch API, so that's why my original patch puts the strings in a variable. As you know, watchdog requires the untranslated version.)
Comment #12
mglamanLooks good to me, tested it out. The code follows the logic that Migrate provides for this use case. I'm all for getting it in pending bojanz's thoughts per #8 (testing in low memory environment)
Comment #13
rszrama CreditAttribution: rszrama at Centarro commentedCommerce Kickstart 2.x is in minimal maintenance mode. Closing out all outdated tickets now to maintain focus on Commerce Kickstart 3.x.