Handling files in Drupal 7

Last updated April 24, 2013. Created by mikeryan on April 19, 2012.
Edited by Ryan Weal, DanChadwick. Log in to edit this page.

NOTE: This applies to Migrate 2.4, for earlier versions of Migrate see http://drupal.org/node/1224042 and http://drupal.org/node/1349726. Also note that this requires Drupal 7.12 or later.

With Migrate 2.4, we completely redesigned the support for importing files into Drupal 7 file entities. This scheme makes the file destination and the file field handlers more consistent and more flexible, and coupled with improved handling of options and subfields, easier to implement. It is, however, incompatible with previous versions, so when upgrading you will need to change your migrations to use these techniques.

The redesigned support changes the syntax from mapping a field with a single entry such as $this->addFieldMapping('dest', 'src')->arguments($arguments); to using multiple entries. The first mapping you do will work like any standard field, taking the 'dest' and 'src' options, default values, etc. Meanwhile, the second and subsequent mappings will follow the form $this->addFieldMapping('dest:param')->defaultValue(); or $this->addFieldMapping('dest:param', 'src2');. While adding these fields it is helpful to review the source and destination fields in the Migrate UI to see what subfields await mapping.

File classes

File classes, implementing the PHP interface MigrateFileInterface, are the key to the new file import approach. A file class represents the logic for taking a particular representation of a file (such as a URL or a database blob) and tying it to a Drupal 7 file entity in the appropriate way. Each file field mapping, or migration to a file destination, will have a file class associated with it, and that file class determines what options and fields are available when mapping it.

There are three file classes provided by the Migrate module itself:

MigrateFileUri

This is the default file class - the one that will be used if you don't specify a file class. When using this class, the primary value (the value you map to a file or image field, or to 'value' in MigrateDestinationFile) is taken to be a URI or a local filespec, and the assumption is that the resource referenced will be copied into a Drupal file scheme.

It takes the following subfields and options:

source_dir - This is, logically, the source directory relative which to interpret the primary value. As a practical matter, it's a prefix applied to all primary values. That is, the migration process will look for the source file at $source_dir/$value. If omitted, the primary value will be treated as an absolute path.

destination_dir - This represents the parent path within Drupal to store the file. It should use a Drupal-defined stream wrapper such as public://. If omitted, it defaults to public:// for file destinations, and the configured file directory for file and image fields.

destination_file - This is the filename relative to destination_dir at which to store the file. If omitted, the filename portion of the primary value will be used. If you have a hierarchical structure you wish to preserve, pass the same value here as you do for the primary value. For example, suppose you implement a migration given physical files at:

/mnt/files/images/foobar.jpg
/mnt/files/images/2012/newfile.png
/mnt/files/misplaced.gif

and a database table with these filename values:

images/foobar.jpg
images/2012/newfile.png
misplaced.gif

with these mappings:

You will end up with files in Drupal at public://foobar.jpg, public://newfile.png, and public://misplaced.gif. However, if you add

you will get public://images/foobar.jpg, public://images/2012/newfile.png, and public://misplaced.gif.

file_replace - This determines the behavior when attempting to import a file and discovering there is already a file at the destination location. It takes three possible values, two of them core constants and one a Migrate extension:

MigrateFile::FILE_EXISTS_REUSE - If the destination file already exists, do not attempt to copy in the source file - just use the existing file (creating a file entity for it if one doesn't already exist).

FILE_EXISTS_REPLACE - Copy the new source file over the pre-existing one, rewriting the pre-existing file entity if it exists.

FILE_EXISTS_RENAME - Create a new, unique name for the incoming file (by appending _1, _2, etc.) and make a file entity pointing to that. This is the default behavior.

preserve_files - Normally, importing and rolling back a migration involving file imports will delete the files on rollback. Sometimes this is not desirable - for example, one approach to file migration may be to simply copy all the files into their desired location in the Drupal installation and use FILE_EXISTS_REUSE to just link to them, or you may have a symbolic link to a readonly mounted filesystem containing the files. Setting preserve_files to TRUE will cause Migrate to add a file_usage entry so that the deletion of referencing entities or fields will not delete the actual file. Note that preserve_files requires Migrate 7.x-2.6 (or latest dev) for use with a file_class of MigrateFileFid.

urlencode (new in Migrate 2.6) - When fetching files from a remote URL, if any of the path components contain special characters like % they need to be encoded - MigrateFileUri will do this encoding by default. However, sometimes the characters are already encoded in your source data - in this case, you can disable the encoding:

MigrateFileBlob

This file class will interpret the primary value (the value you map to a file or image field, or to 'value' in MigrateDestinationFile) as file data (presumably from a database blob field) to be saved as a true file in Drupal.

MigrateFileBlob has the same subfields and options as MigrateFileUri, except for source_dir, which does not apply. The one difference in interpretation is that destination_file is required.

MigrateFileFid

This file class interprets the primary value as an existing file entity ID (fid). It has no subfields or options, and is only relevant with fields, not with file destinations. This is used to tie files migrated to MigrateDestinationFile to a file field:

File destinations

To migrate files directly to file entities, define a migration with a destination of MigrateDestinationFile. The constructor takes three arguments. First is the entity bundle, which defaults to 'file' - this is what you'll usually use with core file entities. Second is the file class, described above, which defaults to MigrateFileUri. Finally is an array of destination options, passed through to the parent destination class - MigrateDestinationFile defines none of its own.

The critical field to map in a file migration is 'value' - this is the primary value, dependent (per above) on the file class. Besides the file class fields described above, the key fields available for mapping are uid (owner of the file) and timestamp (upload date of the file). A simple blob migration would would like this:

File and image fields

Using path as the source field

To migrate files directly into a file or image field, map the subfields directly. Note that you generally don't need to specify destination_dir, which is defaulted in the field definition, and that you should not specify destination_file if migrating multiple values per field (i.e., with an array for image_filename).

Using FID as the source field

If you are working on an offline copy of your site without a full copy of the files folder migrating from a previous version of Drupal you may want to take a shortcut by using the FID of the files as the source rather than the file path. Using this method it is not necessary to setup a separate migration class so you can use it within your existing node migraton. Furthermore, Drupal will not bother to look at the local file system using this method.

First, be sure that your base query includes the desired file field and use the resulting FID to join to the files table so you will have all of the needed fields available to you. When this is ready, use your FID field as the SRC in your initial mapping, then add secondary mapping to specify what class you wish to use.

Here is an example of how you would migrate using FIDs from the D6 upload table to the same field on an upgraded D7 site. The two additional subfields prevent Drupal from consulting the disk to see if the files actually exist or not:

Implementing your own file class

To make your own file class, you simply need to implement MigrateFileInterface, with concrete implementations of fields() (documenting any options or arguments your implementation supports) and processFile (which will take whatever the incoming file representation is and return a file entity). This complete example, from migrate_extras, implements support for migrating Youtube URLs into the media_youtube module: