While using the track_changes option, we can repeatedly use the same class for continuous migrations, but we lose logged entries in message tables from each previous migration. The delete() method in MigrateSQLMap is deleting old messages, prior to saving of new ones. Could that become optional in the MigrateMap constructor?

Comments

bdone’s picture

Status: Needs work » Active
bdone’s picture

Status: Active » Needs review
StatusFileSize
new1.83 KB

here's a first pass at something that achieves this

bdone’s picture

skipping MigrateSQLMap's delete() method is logging lots of undefined hashes in messages:

Undefined property: stdClass::$migrate_map_hash
File .../migrate/includes/source.inc, line 420(file: .../migrate/includes/source.inc, line 420)
bdone’s picture

still not sure about the implementation, but a usage example with MigrateSQLMap:

    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'YOURKEY' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
        ),
      ),
      MigrateDestinationNode::getKeySchema(),
      'default',
      array(
        'track_last_imported' => FALSE,
        'append_messages' => TRUE,
      )
    );
bdone’s picture

any other plans, or suggestions, for message logging in continuous migrations?

mikeryan’s picture

Status: Needs review » Postponed (maintainer needs more info)

Can you explain the motivation here? What's the scenario where subsequent re-migrations of the same item generate different messages - i.e., why do you want to keep a history of the messages? It seems like an awfully narrow use case to include as a general feature.

Thanks.

bdone’s picture

Status: Postponed (maintainer needs more info) » Needs review

@mike: I’ll try to explain the motivation best i can. I'm using a custom node migration, with system of record of source, using track_changes, and no high water mark. The same migration is used repeatedly (weekly, soon to be daily). Things work fine as they are, except for retaining message logging between migrations.

The imported/unimported counts are getting out of sync, which makes sense per #1732680: removing records from source causes unimported count to be negative. But, more importantly, between re-migrations, Migration class is calling delete() which removes messages prior to each import. In this issue, I was trying to append messages for logging purposes.

// From migration.inc...
// Wipe old messages, and save any new messages.
$this->map->delete($this->currentSourceKey(), TRUE);

Have you ever retained messages, before truncating the table? I've hacked around a bit trying an INSERT INTO to replicate the message and mappings tables, and even tried to register those migrations dynamically with an appended timestamp, like MyMigration_REQUEST_TIME. There were some issues there that i can’t recall specifically, but i beleive having the same sourceid in other mapping tables was creating duplicate nodes from the source.

I talked with moshe in portland and thru conversation, he thought migrate was not really setup for this. But, maybe there's another approach you've already come up with?

mikeryan’s picture

I've never seen a reason to retain messages. Usually if there's a message and an item is reimported, it will either generate the exact same message again, or if whatever condition caused the condition is fixed, the message disappears (and good riddance!). I'm curious what your use case is where a history of messages would be useful (especially considering there's no information in the message table to help tell you which run of the import resulted in a given message).

13rac1’s picture

Issue summary: View changes
Status: Needs review » Closed (won't fix)

I agree with mikeryan. This is use-case is too narrow, plus no one else has commented on this functionality since it was posted. Closing. Re-open to provide a wider use-case.

rhabbachi’s picture

Rerolled the above patch. We are trying to add some user feedback to our, migrate based, solution for harvesting data from remote/local sorces.dkan_harvest and this patch might come in handy.