So I've been creating a migration module based closely on the provided Beer and Wine examples... I'm working on migrating a bunch of nodes over (from a Drupal 6 site to a new Drupal 7 site), as well as a cck image upload field and the corresponding files... everything seems to be working okay, but I can't figure out how to adjust the destination for the copied file in the directory structure... right now all the files are just ending up in my sites/default/files directory, when I'd like to sort them into folders within that directory... actually, I'd like to just keep the original file paths (so a file that was at sites/default/files/1/2/3/file ends up at sites/default/files/1/2/3/file.

Sorry if this is a stupid question, but I've been looking around all day without figuring this out...

Here's my relevant query code:

    // Get image file info
    $query->leftJoin('content_field_image_upload', 'cfiu', 'n.nid = cfiu.nid');
    $query->leftJoin('files', 'f', 'cfiu.field_image_upload_fid = f.fid');
    $query->addField('f', 'filepath', 'image');

(note that 'image' here includes the full relative filepath)

And field mapping code:

    $image_arguments = MigrateFileFieldHandler::arguments(NULL, 'file_copy', FILE_EXISTS_REPLACE);
    $this->addFieldMapping('field_image', 'image')
         ->arguments($image_arguments);

I assume there's an argument for the destination as part of MigrateFileFieldHandler, or something... but I'm not sure where to find the documentation...

Thanks for any help!

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Component: Documentation » Code
Status: Active » Postponed (maintainer needs more info)

Where is the file being copied from? Or, more precisely, how are you telling the Migration module to find the file? If you're scraping it from the D6 website, your first argument to MigrateFileFieldHandler::arguments() should be 'http://www.example.com/'.

Now, given that the source file spec is something like sites/default/files/1/2/3/file, on the D7 side it should end up as sites/default/files/sites/default/files/1/2/3/file - it will by default preserve the full incoming filespec within the local files directory - to get the original full filespec you would want to strip the sites/default/files/ prefix, probably in a callback:

    $image_arguments = MigrateFileFieldHandler::arguments('http://www.example.com/', 'file_copy', FILE_EXISTS_REPLACE);
    $this->addFieldMapping('field_image', 'image')
      ->arguments($image_arguments)
      ->callback(array($this, 'filterImage'));
...
  protected function filterImage($image) {
    return substr($image, strlen('sites/default/files/'));
  }

Is this helpful?

jordanmagnuson’s picture

jordanmagnuson commented
Status: Postponed (maintainer needs more info) » Active

Thanks for the reply Mike--much appreciated! Unfortunately, I'm still a bit confused.

In terms of where I'm getting the files from, I have them in a /legacy folder within my sites/default/files directory of my new D7 site. Forgot to mention that. In any case, that's why I left the source_path null in the MigrateFileFieldHandler... I updated it in my prepareRow function like so:

<?php
$current_row->image = str_replace('sites/default/files', 'sites/default/files/legacy', $current_row->image);
?>

So the images are being found in the legacy folder, but they are not being copied to sites/default/files/sites/default/files/legacy/1/2/3/file.ext as I would expect from your comment... rather, they are just ending up at sites/default/files/file.ext... ie in the root files folder, without their original path.

I've experimented with the callback function, as per your example, but it seems to be acting on source path before the file is found, rather than on the destination path. For example, I can get rid of my prepareRow function, and use this callback instead:

<?php
    $image_arguments = MigrateFileFieldHandler::arguments(NULL, 'file_copy', FILE_EXISTS_REPLACE);
    $this->addFieldMapping('field_image', 'image')
         ->arguments($image_arguments)
         ->callbacks(array($this, 'filterImage'));
?>
<?php
  protected function filterImage($image) {
    return str_replace('sites/default/files', 'sites/default/files/legacy', $image);
  }   
?>

If I put anything else in the callback though, the file is simply not found, and I remain unable to change where the file is actually being copied to in the destination...

The long and the short of it, in other words, is that I am able to modify the source path, where the file is being looked for, but remain unable to alter the destination path... every file just gets plopped directly into my default file directory: sites/default/files/file.ext.

aendra’s picture

aendra commented

I'm having the same problem. It doesn't seem that Migrate is preserving the source path structure after copying from the http:// address. So instead of the precise directory structure all my source images are arranged in, I'm getting them all dumped into the same directory. Worse yet, I'm getting the following error, I'm guessing because FILE_EXISTS_RENAME isn't making new filenames unique enough:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'public://Cover_8.gif' for key 2: INSERT INTO {file_managed} (uid, filename, uri, filemime, filesize, timestamp, type) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => 0 [:db_insert_placeholder_1] => Cover.gif [:db_insert_placeholder_2] => public://Cover_8.gif [:db_insert_placeholder_3] => image/gif [:db_insert_placeholder_4] => 185405 [:db_insert_placeholder_5] => 1328464279 [:db_insert_placeholder_6] => image ) (/home/blurg/d7/includes/common.inc:6975)

Any help? This is an active issue and hasn't had a response since October...

Thanks!

StuartDH’s picture

StuartDH commented

aendrew, for the duplicate problem you can dedupe your fieldmappings. To make it work for files you'll need to change line 1002 in migration.inc from

      $candidate = $original . '_' . $i;

to

      $candidate = $i . '_' . $original;
aendra’s picture

aendra commented

@StuartDH -- That's helpful and I'll definitely try that. Thanks!

That said, I'd still *really* like a way to preserve the existing directory structure. Any help, mikeryan? Pretty please? :)

StuartDH’s picture

StuartDH commented

aendrew, I think this might be the same sort of issue as this http://drupal.org/node/1436838

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Title: How to assign destination path during file migration? » Support specifying the destination file name/path
Category: support » feature
Status: Active » Postponed
Issue tags: +Migrate 2.4

See also #1434508: When to rename file during import. This falls under the meta-issue #1240928: META: Refactoring of file destination/field handlers, and is targeted for Migrate 2.4.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Postponed » Active
mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Active » Closed (duplicate)

This is being dealt with in #1240928: META: Refactoring of file destination/field handlers - you will be able to specify destination_dir and destination_path to get the hierarchy you want.