Highwater marks

By default, each time a migration is run, any previously unimported source items are imported (along with any previously-imported items marked for update). If the source data contains a timestamp that is set to the creation time of each new item, and changed to the update time every time the item is updated, then you can have those updated items automatically reimported by setting the field as your highwater field.

To take advantage of highwater marks, define $this->highwaterField in your Migration constructor to indicate what field returned by your source query will reflect updates to your data.

So, if your query looks like

    $query = db_select('migrate_example_wine', 'w')
             ->fields('w', array('wineid', 'name', 'body', 'excerpt', 'accountid',
              'posted', 'last_changed', 'variety', 'region', 'rating'));

and the last_changed field is a (UNIX integer) timestamp that is set to the created date/time when the source object is created, and updated whenever it is changed, in your constructor include:

    $this->highwaterField = array(
      'name' => 'last_changed',
      'alias' => 'w',  
      'type' => 'int', 
    );

Because the last_changed field is a UNIX timestamp, 'type' => 'int" is required here. If you have date/time fields that are lexicographically sortable (e.g., '2011-05-19 17:53:12'), you can omit the 'type' entry.

If your source is an SQL database, sort your query by the highwater field:

    $query->orderBy('last_changed');

What precisely happens when your migration is setup to use highwater marks?

1. The first time you import, Migrate saves the highwater field value as the "highwater mark".
2. Over time, new content is added (with last_updated values greater than the highwater mark saved by Migrate), and old content is updated (changing the last_updated value to be greater than the highwater mark).
3. The next time you run import, Migrate will automatically alter your source query to pull all content where the highwater field is greater than its saved highwater mark. That means it will both import any content added since the last time it ran, and also re-import any content changed since that time.

Thus, if you want to schedule (e.g., via cron) regular updates of your destination site taking into account both inserts and updates of your source data, highwater marks are very useful.