The simplest and most efficient way to get your files imported is to use the distinct file migration class, rather than try to map them directly in the referencing classes. These classes basically map the source files (or file_managed) table to the destination file_managed table, copying referenced files as necessary. The supported arguments:

  • user_migration: The machine name of your user migration, used to properly assign ownership of the files.
  • default_uid: A default destination uid to use as the default owner of files where a legacy owner isn't identified (for example, files owned by the account with uid 1 in the legacy system, since that account is not migrated).
  • source_dir: The source destination from which to copy migrated files. See here for more info.
  • destination_dir: The destination to which to copy migrated files. This defaults to public://.
  • file_class: An override for the default MigrateFileUri class - you would use this if you had extended that class with application-specific behavior.
$api['migrations']['ExampleFile'] = $common_arguments + array(
  'class_name' => 'DrupalFile6Migration',
  'description' => t('Import Drupal 6 files'),
  'user_migration' => 'ExampleUser',
  'default_uid' => 1,
  'source_dir' => 'http://example.com',
  'destination_dir' => 'public://legacy_files',
);

Using this strategy, files are migrated before being used in a subsequent migration, such as for a node containing a referenced image field. In the (custom) node migration, the referenced file should be migrated with a field mapping specifying a file_class of MigrateFileFid. This migrates only the reference to the file, not the file itself. Additionally, preserve_files should be set to TRUE to avoid deleting the actual file, should the node migration be rolled back (requires Migrate 7.x-2.6 or later). The mapping in the constructor for the node migration might contain:

    $this->addFieldMapping('field_my_image', 'field_my_source_image')
         ->sourceMigration('MyFileMigrationMachineName');
    $this->addFieldMapping('field_my_image:file_class')
         ->defaultValue('MigrateFileFid');
    $this->addFieldMapping('field_my_image:preserve_files')
         ->defaultValue(TRUE);
    $this->addFieldMapping('field_my_image:title', 'image_copyright')
         ->defaultValue(t('(c) 2012 My Site'));
    $this->addFieldMapping('field_my_image:alt', 'image_description')
         ->defaultValue('');
    $this->addFieldMapping('field_my_image:language')
         ->defaultValue(LANGUAGE_NONE);

In the UI

In Migrate 7.x-2.6 or later, one will need to edit the classes after running the UI migration wizard. Each content type or entity will need to be manually set. The destination file field needs to be mapped from the source file field, with the source migration set to the file migration that was automatically created (after agreeing to it in the wizard). And, the file_class subfield "Implementation of MigrateFile to use" needs to have a default value of MigrateFileFid (no source field mapped). It may also be useful to give a default value of TRUE to the field "Option: Boolean indicating whether files should be preserved or deleted on rollback".

Comments

tmetzger’s picture

Hi there,

I'm really struggling with getting this working in a pretty simple case. I'm attempting to move all files from a D6 installation to a D7 installation. I notice that the comments in migrate_d2d/file.inc suggest that the "source_dir" argument is mandatory, but it's not in your example, nor can I see anywhere in the code that it is set to a default value.

Having said that, I've tried setting the source_dir to pretty much every reasonable value I can think of, and I always get an error like this for each file:

The specified file /home/groupanizer/domains/misc/staging/sites/mustsingit.misc.staging.groupanizer.com/files/previews/donya.metzger/45/I Don't Know Enough About You_0.mp3 could not be copied to public://legacy_files/previews/donya.metzger/45/I Don't Know Enough About You_0.mp3

In the above case I was trying to set the path to the exact file system path, but I also tried relative paths and URL's.

What am I doing wrong?

Thanks,
Tom

alexweber’s picture

I updated the page with info on the source_dir param.

Anonymous’s picture

It assumes that, so make it the path to your drupal installation.

xlsg’s picture

The first part works fine (changing the migration class to DrupalFile5Migration of course). But in my Drupal 5 set up the files are not attached to a node by a field (there is the files table with nid and fid, and also file_revisions table which contains a description, fid and vid), so I don't think we can use that 'field_my_source_image' code... Does anybody have a working example of how to do migrate files from Drupal 5 to 7?

rprager’s picture

I have a similar setup with files on my D5 site. I'm currently working on migrating to D7, but I'm not having any luck with the examples found on here and across the web. Anybody have a working example for migrating file attachments (Upload module) from D5 to a file field in D7?

rprager’s picture

After playing around for a while, I figured out how to properly migrate files from D5 node attachements to a D7 file field.

This missing piece for me was that the file field in D7 needs to be mapped to the fid from the D5 file. In order to do that, the fid needs to be included as a source field. I copied the query() function from migrate_d2d/d5/node.inc and modified it in my custom node migration (node.inc in my custom module directory) by adding two lines (see my code comments) to include the fid.

protected function query() {
  $query = Database::getConnection('default', $this->sourceConnection)
  ->select('node', 'n')
  ->fields('n', array('nid', 'vid', 'title',
    'uid', 'status', 'created', 'changed', 'comment', 'promote',
    'moderate', 'sticky'))
    ->condition('n.type', $this->sourceType)
    ->orderBy('n.changed');
  $query->innerJoin('node_revisions', 'nr', 'n.vid=nr.vid');
  $query->innerJoin('files', 'fi', 'n.nid=fi.nid'); //***ADDED LINE***
  $query->fields('nr', array('body', 'teaser', 'format'));
  $query->fields('fi', array('fid')); //***ADDED LINE***
  // Pick up simple CCK fields
  $cck_table = 'content_type_' . $this->sourceType;
  if (Database::getConnection('default', $this->sourceConnection)
      ->schema()->tableExists($cck_table)) {
    $query->leftJoin($cck_table, 'f', 'n.vid=f.vid');
    // The main column for the field should be rendered with
    // the field name, not the column name (e.g., field_foo rather
    // than field_foo_value).
    $field_info = $this->version->getSourceFieldInfo();
    foreach ($field_info as $field_name => $info) {
      if (isset($info['columns']) && !$info['multiple'] && $info['db_storage']) {
        $i = 0;
  	foreach ($info['columns'] as $display_name => $column_name) {
  	  if ($i++ == 0) {
  	    $query->addField('f', $column_name, $field_name);
          }
  	  else {
  	    // The database API won't allow colons in column aliases, so we
  	    // will accept the default alias, and fix up the field names later.
  	    // Remember how to translate the field names.
  	    $clean_name = str_replace(':', '_', $display_name);
  	    $this->fixFieldNames[$clean_name] = $display_name;
  	    $query->addField('f', $column_name);
  	  }
  	}
      }
    }
  }
  return $query;
}

I then added the following dependency and field mappings in the constructor of my custom node migration.

$this->dependencies[] = 'File';
$this->addFieldMapping('field_file_field', 'fid')
  ->sourceMigration('File');
$this->addFieldMapping('field_file_field:file_class')
  ->defaultValue('MigrateFileFid');

I tested the migration and it worked. Records were properly created in the field_data_field_file_field and field_revision_field_file_field tables and properly attached to the nodes. I hope this helps you and anyone else that might be struggling with migrating files from D5.

pearl.liang’s picture

Hi rprager

Thanks for sharing your experience. It helps me. I would also like to share my experience that require fewer codes.

$this->addFieldMapping('field_destination_image', 'field_source_image:fid')
->sourceMigration('file_mirate_machine_name');
$this->addFieldMapping('field_destination_image:file_class')
->defaultValue('MigrateFileFid');

In this way, there is no need to modify query to join fid column. This is for migrating D5 image field to D7.

dafeder’s picture

This only works, as far as I can tell, if your source has a file field. If you're migrating from the core upload files, they don't seem to get mapped without extra code.

dafeder’s picture

The way dynamic queries and method inheritance work, you should be able to add just the extra lines without having to repeat everything. For instance, here's my query() method:

  protected function query() {
    $query = parent::query();
    $query->addExpression("GROUP_CONCAT(fr.fid SEPARATOR ',')", 'fid');
    $query->addExpression("GROUP_CONCAT(fr.list SEPARATOR ',')", 'list');
    $query->leftJoin('file_revisions', 'fr', 'n.vid = fr.vid');
    $query->groupBy('n.nid');
    return $query;
  }

The GROUP_CONCAT expressions let me migrate multiple attachments to the same node into a multi value file field (this only works on mysql). The mappings look like this:

    $this->addFieldMapping('field_attachments', 'fid')
      ->sourceMigration('File')
      ->separator(',');
    $this->addFieldMapping('field_attachments:display', 'list')
      ->separator(',');
    $this->addFieldMapping('field_attachments:file_class')
      ->defaultValue('MigrateFileFid');
mano_gr’s picture

I have a problem with file migrate. I am trying to migrate files as attachments fields to my node.
I have mapping node's field with the file name and source path. If the file has an english name, then the migrate is correct. From the other hand if i have a file with greek letters at title( "Καταγραφές.pdf"), the files does not migrate. When the migration proccess complete i have a message like :
The specified file C:/Users/Administrator/Documents/old__iso/Attachments/5d/Καταγραφές.pdf could not be copied to public://Καταγραφές.pdf.
for every greek file-title.

Any idea? I tryied to set

	$this->addFieldMapping('field_attach:language')
			->defaultValue('gr');

but i have the same problem.

micahw156’s picture

Files attached to nodes using Drupal 6 core upload module will need to be moved into fields. The following field mapping should work.

    $this->addFieldMapping('field_name', 'upload')
         ->sourceMigration('MyFileMigrationMachineName');
    $this->addFieldMapping('field_name:file_class')
         ->defaultValue('MigrateFileFid');
    $this->addFieldMapping('field_name:preserve_files')
         ->defaultValue(TRUE);
    $this->addFieldMapping('field_name:description', 'upload:description');
    $this->addFieldMapping('field_name:display', 'upload:list');
    $this->addFieldMapping('field_my_image:language')
         ->defaultValue(LANGUAGE_NONE);
welly’s picture

I've tried this same thing, however migrate reports:

"upload" was used as source field in the "field_files" mapping but is not in list of source fields
"upload:description" was used as source field in the "field_files:description" mapping but is not in list of source fields
"upload:list" was used as source field in the "field_files:display" mapping but is not in list of source fields

I understand from that that I need to expose the upload source some how but I'm unsure how. Have you got any advice?

Thanks!

Sunflower’s picture

I want to migrate only images & files associated with nodes. Not all files & images, so I am thinking the "Source Migration" way may not work. How do I achieve this?

Jaypan’s picture

To expand on micahw156's comment, which helped me a lot, everywhere he put 'field_name', you need to replace it with the field name on the destination content type. Example: field_article_file.

Jaypan’s picture

I was having a heck of a time with importing files for the Video module. Whenever I rolled back, I would lose all my files, even though I had preserveFiles set to TRUE.

It turns out that a special handler is needed for the video module. I used this: https://www.drupal.org/node/1835478#comment-7481906