Problem/Motivation

Migration involves a processing-heavy set of operations that may trigger page timeouts. Typically in Drupal core such operations call drupal_set_time_limit() to allocate additional processing time, with 240 being the usual value used.

Proposed resolution

Add one or more calls to drupal_set_time_limit(240) before migrate processing calls.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

moshe weitzman’s picture

Thats the responsibility of the batch API. Also, long running SQL queries do not count against PHP time, AFAIK.

nedjo’s picture

Status: Active » Closed (works as designed)

Makes sense. It's hard to see a use case for calling a migration outside of a batch, and if someone did they could take care of page execution time themselves. And, yes, sound point about most migration processing being outside of what counts as PHP time (except on Windows).

nedjo’s picture

Status: Closed (works as designed) » Needs review
FileSize
472 bytes

Looking closer, while the cron system does allocate processing time, batch API does not (presumably on the assumption that most batches will chunk operations into batches that don't require extra resources).

I opened this issue coming from #1763508: Reimplement Migrate integration in the installer, where bojanz noted "when the system is low on memory, the migration completes halfway through and the installation is completed without the complete set of content."

Yes, batches should include the sort of code that's in migrate_ui_batch(), examining the return value of the processImport() method and responding accordingly, with the result that incomplete imports will still be completed in subsequent rounds. Still, since migration is a resource-intensive and very infrequent operation, it seems like a good case for setting a high time limit to minimize RESULT_INCOMPLETE migration results, which can result from approaching the PHP time limit (see MigrationBase::timeExceeded()).

Two places we could put this. 1., before calling processImport(). 2., in processImport(), before initiating the import. Attached patch takes approach 2.

mikeryan’s picture

Status: Needs review » Needs work

The issue is specific only to running through the UI so it should be done in the batch processing.

Also, I wouldn't hard-code the value - please make it a MigrationBase member that can be set in the constructor so migration implementors can adjust to fit their needs:

$this->batchTimeLimit = 240;
nedjo’s picture

Status: Needs work » Needs review
FileSize
1.49 KB

Here's a revised patch based on the suggestions in #4:

  • Add a batchTimeLimit property and setBatchTimeLimit method to MigrationBase.
  • Call setBatchTimeLimit in migrate_ui_batch().
mikeryan’s picture

Category: task » feature
Status: Needs review » Fixed

Committed (to wizard_api branch), thanks!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

mikeryan’s picture

Issue tags: +Migrate 2.6

Tagging as a 2.6 feature.