Option to flatten or preserve file hierarchies on import

The intent of the file handler is to migrate into a file Field. Each file field has a configured destination already. I'm not clear why you would want to migrate into some other destination. If you really want that, just copy it yourself in a prepare hook.

The most native Drupal way is to enable user pictures and then import right into that. An alternative is to put an Image field on the user entity or on a new profile entity (see Profile2 module).

You can go with a custom picture implementation like you describe but then you have to build own UI and code for add/edit/delete

OK, so what you would like is an option to flatten the file hierarchy on import, correct?

My current solution is to create a sub-class of MigrateDestinationFile and and override the import() method.

myname.com, can you share what you've done, please?

I'm having a similar problem with MigrateFileFieldHandler when my Migrate class downloads files from FTP to /tmp/mypics/filename.jpg (in prepareRow) and then map them to an image field. Since the files are no longer remote I use the 'file_move' operation to let the field handler move things into public://myphotos (set as the image field's destination).

Since I know no filenames will collide, I would have expected all files to end up as www.example.com/sites/default/files/myphotos/filename.jpg. (public:// is pointing to sites/default/files)
But, all filenames are modified so the path ends up like this: www.example.com/sites/default/files/myphotos/-tmp-mypics-filename.jpg.
This is a bit of a problem since I want the filenames to stay intact.

My field mapping looks like this.

    // prepareRow() fills $row->photos with a pipe-delimited list of downloaded files: /tmp/mypics/filename1.jpg|/tmp/mypics/filename2.jpg
    $arguments = MigrateFileFieldHandler::arguments(NULL, 'file_move', FILE_EXISTS_REPLACE);
    $this->addFieldMapping('field_photos', 'photos')
      ->arguments($arguments)
      ->separator('|');

I traced it back to fields.inc (around line 320 in 7.x-2.1, which is what I'm using):

    $destination_dir = file_field_widget_uri($field_info, $instance);
    file_prepare_directory($destination_dir, FILE_CREATE_DIRECTORY);
    if ($remote) {
      $destination_file = $destination_dir . "/" . basename($full_path);
    }
    else {
      $destination_file = file_stream_wrapper_uri_normalize($destination_dir . "/" . str_replace('/', '-', $full_path));
    }

    $real_destination_file = drupal_realpath($destination_file);

Why is basename() never used for local files?

Could we add an argument to tell the file handler to apply basename() even if the file is local?
I put together a simple patch for 7.x-2.x which does what I need, but it hasn't had much testing with alternate setups yet.

I'm in almost exactly the same boat as myname.com

I'm importing an old forum, with user avatars. I've chosen to import from a http:// uri and have supplied the path from web root to the location of the files.

In the SQL source, I have a file name. In order for migrate to find the file, I'm prepending the full URL to the file name giving me something like http://example.com/images/avatars/upload/filename.ext

I've set the migrate destination like this:

    $this->destination = new MigrateDestinationFile(array(
      'copy_file' => TRUE,
      'copy_destination' => file_build_uri(variable_get('user_picture_path', 'pictures')) . '/'
    ));

Which gets the set path for user pictures (public://pictures/)

The result is, I get files in:
public://pictures/images/avatars/upload/filename.ext
Where as, I would like them to be:
public://pictures/filename.ext

This would give me consistency with any new avatars uploaded after the migration.

It seems to me a option to only use the basename of the migrated file would be exactly what I need.

as a workaround I just downloaded the files on my local drive and then imported from there. The advantage is that the import is now faster, I can remove the extra folder structure, can rename the filenames and keep a backup copy, just in case ;-)

I've got ~5000 images to import from FTP in one go, then there's maybe up to 40 new images each day, separated across hundreds of directories and subdirectories...

I just write PHP scripts to curl the files. I'm downloading 100,000+ images and videos. I'm glad TB harddrives are cheap and comcast hasn't throttled my Internet connection (yet) ;-)

I just noticed that I migrated some node images (from remote url) and they got saved in the "sites/default/files" directory even though the remote url has 4 or 5 folders.

I used this code:

    $arguments = MigrateFileFieldHandler::arguments(NULL,'file_copy', FILE_EXISTS_RENAME, NULL);
    $this->addFieldMapping('field_logo', 'pathtophoto')
         ->arguments($arguments);

I don't know how many files each imported item has beforehand, I just know the naming scheme - based on item id - and in which folder they should exist (sorted into subdirs by item group and then by image number). Using the same folder hierarchy on the site's server makes no sense, which is why this patch is important to me.

@mototribe, I'll probably approach those numbers when my client switches data provider, but I've got a 30s execution time limit to battle too, so that's not likely an option for me.

I'm finding that MigrateDestinationFile maintains folder structures, which is what I need for my project

However, when I then use the MigrateFileFieldHandler the folders are flattened...which is the opposite of what I need

See this issue http://drupal.org/node/1436838

See also #1436838: copy folder structure bug. Both fall under #1240928: META: Refactoring of file destination/field handlers, which we plan on addressing for Migrate 2.4.

This is dealt with in #1240928: META: Refactoring of file destination/field handlers - by default, the hierarchy will be flattened, but by setting destination_file to the hierarchy you want you can preserve it.