Closed (fixed)
Project:
Migrate
Version:
7.x-2.x-dev
Component:
Code
Priority:
Normal
Category:
Bug report
Assigned:
Reporter:
Created:
18 Apr 2012 at 11:26 UTC
Updated:
12 May 2012 at 15:30 UTC
Jump to comment: Most recent file
Here's a simplified scenario of my case:
"Peter", "Pan", 1330855930The source column definitions:
array(
0 => array('firstname', 'First name'),
1 => array('lastname', 'Last name'),
2 => array('updated', 'Last updated'),
);
Added this hightwater mark:
$this->highwaterField = array(
'name' => 'updated',
'type' => 'int',
);
Now... I'm expecting that, on the second import, Migrate will bypass all records that were previously imported and not changed. I didn't touch the CSV file between the imports. I see that {migrate_status}.highwater is filled with the greatest "updated" column from source.
I used both:
$ drush migrate-import MyMigrate
$ drush migrate-import MyMigrate --update
Both are importing again, everything. My intention is to keep my Drupal in-sync with CSV so it's not a one-time migration. So, performance is critical... I need only fresh data (new and updated records) to come in.
| Comment | File | Size | Author |
|---|---|---|---|
| #4 | migrate-highwater_prepare-1538046-4.patch | 2.56 KB | mikeryan |
| #2 | migrate-unprepared-highwater-field-1538046-2.patch | 1.71 KB | claudiu.cristea |
Comments
Comment #1
claudiu.cristeaIt seems that Migrate is using the un-prepared value of timestamp when deciding to parse the row. In my case I implemented a
prepareRow()method to transform CSV milliseconds timestamps to regular UNIX timestamp.In includes/source.inc we have:
So...
$row->{$this->highwaterField['name']}will be in milliseconds (unprepared)$this->activeMigration->getHighwater()will be in seconds, it was prepared before stored.I think the above
elseifstatement must take the prepared$row->{$this->highwaterField['name']}Comment #2
claudiu.cristeaHere's a patch. I admit that is not so nice cloning the
$rowobject but I didn't wanted to messup all the logic there by preparing the row in an early stage. This works for me.Comment #3
mikeryanNo, there's got to be a better way. Doubling the calls to prepareRow() for the sake of an edge case is not acceptable. Better would be to find a way to insert a call to prepareRow() just before testing the highwater mark, and make sure it doesn't get called again below if it was called here...
Comment #4
mikeryanThe attached patch is what I was talking about - does it address the issue for you?
Thanks.
Comment #5
claudiu.cristeaWorks as expected with #4. I had to apply it manually.. the branch is ahead.
Thank you!
Comment #6
mikeryanCommitted for D6 and D7, thanks!
Comment #7
claudiu.cristeaChecked twice... Now existing, untouched records are updated every import :(
Tested and discovered that every time this condition is satisfied:
This mean that the existing rows doesn't have the
migrate_map_sourceid1property set at this moment. Not sure that this is coming from your patch.Comment #8
mikeryanThe line comes from #1529362: Migrate not respecting existing map statuses. I don't see why a previously-imported record would not have migrate_map_sourceid1 set...