I am running migrate with Drush - the fastest throughput I can achieve is about 125/min - I have lots of system resources free though; how do I go about increasing the throughput? Is it possible to run more than one instance of drush migrate in order to go faster?

The reason why speed is of concern to me is I am migrating a huge dataset - over 2 mil rows total - hence even small speed increases will help!

Comments

moshe weitzman’s picture

moshe weitzman commented
Status: Active » Fixed

WIth 99% certainty, I can say that the problem here is not migrate. You have some custom or contrib code that is slowing down saving of nodes or whatever you are saving. Try disabling unneeded modules. In my experience, token and pathauto can be very expensive.

newmediaist’s picture

newmediaist commented

That's good to know - I will try disabling them now.

A neat future feature would be the ability to specify start and end rows for the migration - that way I could start up a number of drush processes in tandem

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

Note that you can use -d (debug) on the drush command or enable "Display timers when processing" on the migrate settings page to see timing information that should help you narrow down where your bottleneck is - almost certainly it'll show node_save taking most of the time. Then, throw some timer_start/timer_stop calls into node_invoke, node_invoke_nodeapi, and module_invoke_all to figure out exactly which hook(s) form the bottleneck. E.g., alter module_invoke_all() thusly:

    timer_start($function);
    $result = call_user_func_array($function, $args);
    timer_stop($function);
newmediaist’s picture

newmediaist commented

Interesting, thanks.
I disabled a whack of modules and speed improved drastically.

I've also noticed the longer migrate runs through drush, the slower it gets now - I.e. in first hour I was getting 1022/min, 2nd hr 800/min, 3rd hr 600/min, 4th hour 400/min. Any idea why that might be happening?

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

With any database-based system, the more existing data you have, the slower it is to insert more - but the slowdown shouldn't be linear. This is generally a sign that there's a query out there doing a table scan (i.e., not using an index). You can use the devel module query log to help find that query, and/or instrument the invoke functions to figure out what hook is slowing you down.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

dgsiegel’s picture

dgsiegel commented

it seems like pathauto is a huge performance killer. have a look at those two examples:

$this->addFieldMapping('pathauto_perform_alias')->defaultValue('1');
Imported 2 (0 failed) in 15 sec (8/min) - done with 'ExampleMigration'

$this->addFieldMapping('pathauto_perform_alias')->defaultValue('0');
Imported 2 (0 failed) in 0.5 sec (240/min) - done with 'ExampleMigration'

i guess, you can disable pathauto for the migration and then do the "bulk pathauto creation" afterwards. another possibility would be to call node_pathauto_bulkupdate() in postImport(), but that will take the same time anyway.