META: Refactoring of file destination/field handlers

I did commit a couple of tweaks that were helpful - however, as I dig deeper (e.g., to try to support media_youtube), I'm starting to think we may need to do some significant refactoring. For one thing, adding hard-coded file_functions complicates the code while limiting flexibility - the file_functions should be pluggable, so derived migration integrations for media, media_youtube, etc. can add their own. That's a bit much to do right away - my thinking at the moment is to wrap up a 2.2 release with a little further cleanup of the file support, and address refactoring for 2.3.

See also #1340404: Renaming destination file during migration - the field handlers should be flexible enough to permit renaming during migration.

Rewrote the original issue, pointing at the specific issues we want to address. I want to get Migrate 2.3 out the door - I'll release 2.3-rc1 today, hopefully have a final 2.3 release within a week or two, then this will be the first priority after that.

Make into a meta-issue

I fixed it back - as it says, this work is to be done after the 2.3 release - i.e., for 2.4.

Fix version

Issue wasn't really about files specifically

Just a note that I am actively working on a complete rewrite of the file handlers. I believe at this point, rather than break code using the current file handler implementations, or seriously complicate the rewrite trying to provide backwards compatibility, it's best to implement brand-new handlers and deprecate the old ones.

My approach is to offload most of the real work to a class derived from the new MigrateFileSource, specifiable in the field mappings, e.g.:

    $this->addFieldMapping('field_image', 'image');
    $this->addFieldMapping('field_image:source_class')
         ->defaultValue('MigrateFileBlogSource');

You'll thus be able to implement your own source classes for special cases. Also, this will enable sharing most of the logic between the destination and field handlers, which right now are entirely separate.

Thoughts?

At long last, a first stab. My next step will be working on #1535786: Update media support for Migrate 2.4, which I'm sure will suggest some tweaking of the approach, but the curious may peruse where this is going. The key is the file class (implementation of MigrateFileInterface) - there are three such classes to choose from, FileMigrateUri (the default - migrating from a local file or a URI), FileMigrateBlob (migrating from blob data to a real file), and FileMigrateFid (attached a file field to a previously-migrated file entity).

Some examples of how to use this version of the file handling (from migrate_example):

    // Migrating images in a local directory to an image field, with associated alt and title text
    $this->addFieldMapping('field_migrate_example_image', 'image');
    // This mapping is unnecessary, MigrateFileUri is the default
    $this->addFieldMapping('field_migrate_example_image:file_class')
         ->defaultValue('MigrateFileUri');
    $this->addFieldMapping('field_migrate_example_image:source_dir')
         ->defaultValue(drupal_get_path('module', 'migrate_example'));
    $this->addFieldMapping('field_migrate_example_image:alt', 'image_alt');
    $this->addFieldMapping('field_migrate_example_image:title', 'image_title');

    // Similarly from remote URLs (note no source_dir in this case)
    $this->addFieldMapping('field_migrate_example_image', 'url');
    $this->addFieldMapping('field_migrate_example_image:file_replace')
         ->defaultValue(FILE_EXISTS_REPLACE);
    $this->addFieldMapping('field_migrate_example_image:alt', 'image_alt');
    $this->addFieldMapping('field_migrate_example_image:title', 'image_title');

    // Files as destinations - just copy from a URI to a file entity
    $this->destination = new MigrateDestinationFile2();
    $this->addFieldMapping('value', 'url');

    // Files as destinations, from a database blob with associated filename
    $this->destination = new MigrateDestinationFile2('file', 'MigrateFileBlob');
    $this->addFieldMapping('destination_file', 'filename');
    $this->addFieldMapping('value', 'imageblob');

Now, the addition of this shouldn't affect existing migrations using the old classes. It does need to be noted, though, that on admin/content/migrate/configure one should have only one of MigrateFileFieldHandler and MigrateFileField2Handler enabled. Running update.php/dbupdate will disable the new field handler - you'll need to go to the configure screen to switch to it when you're ready to start using it.

OK, I think this is ready now. At this point, my inclination is to bite the bullet and replace the former file handlers entirely with the new ones - there will be some upgrade pain for 2.4, but I think it's worth it.

The beer.inc example of using the image field handler is now:

    // Copy an image file, write DB record to files table, and save in Field storage.
    // We map the filename (relative to the source_dir below) to the field itself.
    $this->addFieldMapping('field_migrate_example_image', 'image');
    // Here we specify the directory containing the source files.
    $this->addFieldMapping('field_migrate_example_image:source_dir')
         ->defaultValue(drupal_get_path('module', 'migrate_example'));
    // And we map the alt and title values in the database to those on the image.
    $this->addFieldMapping('field_migrate_example_image:alt', 'image_alt');
    $this->addFieldMapping('field_migrate_example_image:title', 'image_title');

and the wine.inc example of using a file destination to migrate a blog looks like:

    $this->destination = new MigrateDestinationFile('file', 'MigrateFileBlob');

    $this->addFieldMapping('destination_file')
         ->defaultValue('druplicon.png');
    $this->addFieldMapping('value', 'imageblob')
         ->description('An image blob in the DB');

There's no way I can test every possible scenario myself - my inclination is to be aggressive, commit this soon and get a beta release with it out as soon as possible.

Updates

Documentation at http://drupal.org/node/1540106.

Committed. I'm sure it's not perfect, but I think it will be best to get people banging on this and opening new issues for individual issues that are identified.

See http://drupal.org/node/1540106 for details on how to implement file migrations in the new world.