Details

Total Records: ~270k
Records imported: ~240k
Migration type: Multiple XML files with multiple records per file with PDF file attachment per record
Error in httpd log: PHP Fatal error: Maximum execution time of 120 seconds exceeded in /path/to/site/sites/all/modules/migrate/plugins/sources/xml.inc on line 794, referer: http://example.com/batch?op=start&id=57
PHP max_execution_time: 120
PHP memory_limit: 256M

It appears that I have reached some sort of hard limit on my migration. My hard drive and lvm have hundreds of additional GB in space. At first, records were importing at a rate of 1000+ per minute. After passing ~220k records, I noticed it had slowed down considerably to ~15 per minute. It continued to slow and then I hit a wall and cannot get another single record to be migrated.

I updated to the migrate module to 2.4-beta1 to see if that might "magically" solve the issue, but it did not. After troubleshooting a couple changes in file-handling and a modification I made to the xml.inc file I am back where I started. Any help is greatly appreciated!

Sean

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Active » Postponed (maintainer needs more info)

Reason 101 why I always recommend running migrations in drush if you can... To run migrations in the browser, they have to be stopped before the max_execution_time is reached, and restarted in a new batch. While you might imagine it works like the pause button on your DVD player, the new batch is a completely new migration process, so it has to start from the beginning and read through the XML file, checking each migratable element to see if it's already been imported, before it reaches the first new item. The farther you get, the longer it takes to find something to import, and the less time you have to get actual work done in the batch, until you don't even have time to find one new item.

If you run it in drush, you'll get it all done in one fell swoop, no trouble.

pyrello’s picture

pyrello commented

@mikeryan - this appears to have taken care of my issue. I'm going to leave this open until the migration finishes. Thank you very much for the quick response!

Sean

pyrello’s picture

pyrello commented
Status: Postponed (maintainer needs more info) » Fixed

I was able to complete my migration using drush. Fortunately, I don't have very many more collections that are as big as that.

Thanks for the help @mikeryan!

Sean

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Anonymous’s picture

(not verified) commented
Issue summary: View changes

Updated issue summary.