Hello!

I am working on a D6->D7 migration involving Ubercart->Commerce, and Ubercart UC File S3 -> Commerce File S3 (as our downloadable content is purchasable through the site and is hosted on Amazon S3).

My first migration failed to be "finished" because orders I made of products that had been migrated in to D7 did not have any File Entity data that could be loaded, and so Rules that allows "File Licenses" to be created for those orders would fail. So I'm trying again; here is my UCFileS3ToFilesMigration class:

class CommerceMigrateUCFileS3ToFilesMigration extends Migration {
  public function __construct() {
    parent::__construct();
    $this->description = t('Bring in files from uc_file_s3');
    $this->map = new MigrateSQLMap(
      $this->machineName,
      array(
        'fid' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'description' => 'uc_files_s3 key',
        ),
      ),
      MigrateDestinationFile::getKeySchema()
    );
    $connection = commerce_migrate_ubercart_get_source_connection();
    $query = $connection->select('uc_files_s3', 'ufs')
      ->fields('ufs', array('fid', 'filename'));
    $query->condition('filename', 'logs/%', 'NOT LIKE')
      ->condition('filename', 'log/%', 'NOT LIKE')
      ->condition('filename', 'cflog/%', 'NOT LIKE')
      ->condition('filename', 'stats/%', 'NOT LIKE');
    $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE));

    $this->destination = new MigrateDestinationFile();
    $this->addFieldMapping('copy_files')->defaultValue(FALSE);
    $this->addFieldMapping('file_replace')->defaultValue(MigrateFile::FILE_EXISTS_REUSE);
    $this->addFieldMapping('preserve_files')->defaultValue(TRUE);
    $this->addFieldMapping('destination_dir')->defaultValue('s3://');
    $this->addFieldMapping('destination_file', 'filename')->defaultValue(MigrateFile::FILE_EXISTS_REUSE);
    $this->addFieldMapping('uid')->defaultValue(1);
    $this->addFieldMapping('uri', 'uri')->defaultValue(MigrateFile::FILE_EXISTS_REUSE);
    $this->addUnmigratedDestinations(array('fid','filename', 'status',
      'filemime', 'timestamp'));
  }

  public function prepareRow($row) {
    $row->uri = 's3://' . $row->filename;
  }
}

And here is some helpful debug output from Drush:

MigrateFileUri Object
(
    [sourceDir:protected] =>
    [sourcePath:protected] =>
    [destinationDir:protected] => s3://
    [destinationFile:protected] => StarWarsII_RiffTrax.zip
    [fileReplace:protected] => 1
    [preserveFiles:protected] => 1
    [defaultFile:protected] => stdClass Object
        (
            [copy_files] =>
            [file_replace] => -1
            [preserve_files] => 1
            [destination_dir] => s3://
            [destination_file] => StarWarsII_RiffTrax.zip
            [uid] => 1
            [uri] => s3://StarWarsII_RiffTrax.zip
            [migrate] => Array
                (
                    [machineName] => CommerceMigrateUCFileS3ToFiles
                )

        )

)

s3://StarWarsII_RiffTrax.zip
The specified file <em class="placeholder"></em> could not be copied to <em class="placeholder">s3://StarWarsII_RiffTrax.zip</em>.

I also read through #1665334: file_link in migrate to see if there were any helpful comments there, and I'm fairly certain I've followed the available tips there to the letter, but as such, my Migration is still not respecting the fact that I don't need to actually copy these files; I simply need to add their data to {file_managed} and create new File Entities in the process. Without that, I will have to go through and re-save all of our migrated products, which we have lots of.

Any ideas what's happening? I could hack the migration fileCopy handler as a workaround for now, but I'd like to resolve it in a more meaningful way that's helpful to others. Any insight you could provide would be greatly appreciated. Thanks!

CommentFileSizeAuthor
#1 copy_files-1809234-1.patch3.92 KBmikeryan

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Category: support » feature
StatusFileSize
new copy_files-1809234-1.patch3.92 KB

Took a little effort, but tracked down the "Commerce File S3" module you referenced at http://drupal.org/sandbox/rfay/1233708... The main thing is, your destination uses the s3:// stream wrapper, so almost certainly you need to define your own file class to handle it. It's not necessarily that hard - check out the example at http://drupal.org/node/1540106 for handling Youtube.

A couple other notes:

  • There's no such option as "copy_files" in Migrate 2.4 or later.
  • ->defaultValue(MigrateFile::FILE_EXISTS_REUSE) is being appended all over the place - it's only relevant to preserve_files.

Now, an interesting question is whether a "copy_files" option would make MigrateFileUri work as-is for you. It may be useful anyway... So, give the attached patch a try and see if it's helpful.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Active » Needs review
torgospizza’s picture

torgospizza
he/him
Primary language English
Location Portland, OR
commented
Status: Needs review » Needs work

Man, you are fast. I can't thank you enough for the prompt reply. Thanks for the other notes, I will keep them in mind going forward.

With regard to your patch, There was a minor typo, ~line 25:

+    if (isset($arguments['copy_files'])) {
+      $this->preserveFiles = $arguments['copy_files'];
+    }

It should be $this->copyFiles = $arguments['copy_files']; ... looks to me like a copypasta mistake :)

However, I think writing my own File class is probably the better way to go, since it's still attempting to stat() a file that doesn't exist. (In other words it doesn't know anything about the S3 wrapper, so it's still attempting to run functions on what it perceives as a local filepath.) It now errors with:

filesize(): stat failed for s3://FantasticFour_RiffTrax.zip 
File /mnt/www/html/rifftraxdev/docroot/includes/file.inc, line 570

So! Back to square one, but I will follow the YouTube example and write an S3 File Handler for it. Thanks again for the quick response. I'll set this back to "Needs work" for now, but I think you could close it in lieu of the fact that this is really "by design", since copy_files is not currently a valid argument. That being said I would love to see something like that return, so we don't have to rely on always writing a new File class when we switch to a new storage method (like moving from S3 to Rackspace CloudFiles or something). In our case this would benefit us because we would migrate all our files beforehand, since they can reach 7GB.

Anyway thanks again, I'll be sure to ping you if I run into any issues writing a new File class.

torgospizza’s picture

torgospizza
he/him
Primary language English
Location Portland, OR
commented

Okay, I've been trying to write the handler, but can't seem to get past this one hurdle: where does $value in processFile come from? It seems that my class is able to connect to the fieldMapping in the main Migration class, but there seems to be a break in communication between the prepared $row from Migrate and the actual file representation. (I'm still trying to catch up with OOP; that's really my main learning curve here.) So yeah, right now, $value is coming up as not having a set value, and my inexperience is making me unable to wrap my brain around it.

Here's the class I wrote. Any pointers is gladly appreciated!

/**
 * Class for creating commerce_file_s3 file entities.
 */
class MigrateAmazonS3Files implements MigrateFileInterface {

  public function __construct() {
    libraries_load('awssdk');
    $bucket = variable_get('amazons3_bucket', '');
    $handler = new AmazonS3StreamWrapper();
  }

  /**
   * Implementation of MigrateFileInterface::fields().
   *
   * @return array
   */
  static public function fields() {
    return array('uri', 'filename', 'size');
  }

  /**
   * Implementation of MigrateFileInterface::processFiles().
   *
   * @param $value
   *  The URI or local filespec of a file to be imported.
   * @param $owner
   *  User ID (uid) to be the owner of the file.
   * @return object
   *  The file entity being created or referenced.
   */
  public function processFile($value, $owner) {
    $metadata = $handler->_amazons3_get_object($value);
    $destination = file_stream_wrapper_uri_normalize($metadata->uri);

    $file = file_uri_to_object($value, TRUE);
    $file = file_save($file);

    if (is_object($file)) {
      return $file;
    }
    else {
      return FALSE;
    }
  }
}

And I adjusted the field mapping in the Migration to look like this:

// I also tried 'file' and 'commerce_file' for the bundle, just in case.
$this->destination = new MigrateDestinationFile('commerce_file_s3', 'MigrateAmazonS3Files');

Thanks!

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

See the "File destinations" section of http://drupal.org/node/1540106 - your source URI/filespec should be mapped to 'value' in your migration constructor.

As for your implementation, it's important to note that object data in your class must be referenced through $this. I.e., when you set $handler in your constructor, that's a local variable not visible outside the constructor. You need to set $this->handler, then reference it the same way where you need it.

Also note that in fields(), 'uri' and so on should be the array keys, not the values - the values should be descriptive text. Look at the fields() implementations in migrate's file.inc for examples.

pifagor’s picture

pifagor commented
Status: Needs work » Closed (outdated)