I used various examples I found in the issue queues to come up with the following for importing users with pictures. Due to the issues with the http-based file copy, I manually copied the avatars to the pictures directory and have copy_file set to false. My problem is that the avatars are named (avatar_{UID}.png) based on the UID, which always exists, but not every user has an avatar. The import is working for users that have avatars, but not for those who don't with error: filesize() [function.filesize]: stat failed for ...

Is it possible to detect the missing avatar and default the picture value to 0 for those users that don't have one so that they may still be imported?

Here's my code:


class UsersMigration extends TUMigration {
  public function __construct() {
    parent::__construct();

    $this->description = t('Migrate TU Users');

    $this->dependencies = array('Avatars');

    $source_fields = array(
      'id_member' => t('User ID'),
      'roles' => t('The set of roles assigned to a user.'),
    );

    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'id_member' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'description' => 'SMF Unique User ID',
          'alias' => 'u',
        )
      ),
      MigrateDestinationUser::getKeySchema()
    );

    $query = Database::getConnection('default', 'tuprod')
      ->select('base_members', 'u')
      ->fields('u', array('ID_MEMBER', 'memberName', 'dateRegistered', 'passwd', 'emailAddress'))
      ->condition('u.is_activated', 1, '=')
      ->condition('u.ID_MEMBER', 1, '>');

    $this->source = new MigrateSourceSQL($query, $source_fields, NULL, array('map_joinable' => FALSE));

    $this->destination = new MigrateDestinationUser();

    $this->addFieldMapping('uid', 'id_member');
    $this->addFieldMapping('name', 'membername');
    $this->addFieldMapping('pass')->defaultValue('');
    $this->addFieldMapping('mail', 'emailaddress');
    $this->addFieldMapping('theme')->defaultValue('');
    $this->addFieldMapping('signature')->defaultValue('');
    $this->addFieldMapping('created', 'dateregistered');
    $this->addFieldMapping('status')->defaultValue(1);
    $this->addFieldMapping('language')->defaultValue('');
    $this->addFieldMapping('init', 'emailaddress');
    $this->addFieldMapping('picture', 'user_avatar')->sourceMigration('Avatars');
    $this->addFieldMapping('is_new')->defaultValue(TRUE);

  }
}

class AvatarsMigration extends Migration {
  private $base_path;
  public function __construct() {
    parent::__construct();

    $this->base_path = 'public://' . variable_get('user_picture_path', 'pictures') . '/';

    $this->description = t('User avatars images');

    $source_fields = array(
      'id_member' => t('User ID'),
      'roles' => t('The set of roles assigned to a user.'),
    );

    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'id_member' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'description' => 'SMF Unique User ID',
          'alias' => 'u',
        )
      ),
      MigrateDestinationFile::getKeySchema()
    );

    $query = Database::getConnection('default', 'tuprod')
      ->select('base_members', 'u')
      ->fields('u', array('ID_MEMBER'))
      ->condition('u.is_activated', 1, '=')
      ->condition('u.ID_MEMBER', 1, '>');

    $query->distinct();

    $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE));

    $this->destination = new MigrateDestinationFile(array(
      'copy_file' => false,
      'preserve_files' => true,
    ));

    $this->addFieldMapping('uri', 'id_member')->callbacks(array($this, 'add_path'));
    $this->addUnmigratedDestinations(array('fid', 'uid', 'status', 'filemime', 'timestamp'));

  }

  public function add_path($uri) {
    return $this->base_path . 'avatar_' . $uri . '.png';
  }
}

Comments

StuartDH’s picture

Have you tried a null default value for the avatar? Or maybe use a default image to give all users a blank face if they haven't uploaded an avatar.

    $this->addFieldMapping('picture', 'user_avatar')->sourceMigration('Avatars')
            ->defaultValue(null);
cameron prince’s picture

Changing the addFieldMapping to $this->addFieldMapping('picture', 'user_avatar')->sourceMigration('Avatars')->defaultValue(0); made no difference. I ended up removing $this->dependencies = array('Avatars'); and this allows the users to import, but the migration is still full of errors due to missing files. Is there a way to define specific methods as soft fail?

cameron prince’s picture

I learned about $this->softDependencies and associated Avatars to Users with it. This allows the users to be inserted even if they don't have an avatar but I still get the filesize() [function.filesize]: stat failed for public://pictures/....

When I did a test with all 2000+ users, at about the 800 mark the import died from the UI with an insanely long AJAX error. I wonder if the number of errors just overloaded the buffer. I haven't tried from Drush yet.

As you can see in this code, I'm using a callback to create the file names. I tried returning NULL here to skip the files but this produces an SQL duplicate key error due to the unique index on the uri field in the file_managed table because 0 is being returned as the URI for each missing image.


...

    $this->addFieldMapping('uri', 'id_member')->callbacks(array($this, 'add_path'));
    $this->addUnmigratedDestinations(array('fid', 'uid', 'status', 'filemime', 'timestamp'));

...

  public function add_path($uri) {
    $path = '/home/dev/htdocs/sites/default/files/pictures/';
    $file = 'avatar_' . $uri;
    foreach (array('.png', '.jpg', '.gif') as $ext) {
      if (file_exists($path . $file . $ext)) {
        $file .= $ext;
      }
    }
    return $this->base_path . $file;
  }

I'm now wondering if I could use a row function to check for file existence and somehow prevent the FieldMapping. Any ideas?

Thanks,
Cameron

mikeryan’s picture

Status: Active » Postponed (maintainer needs more info)

I have just the same setup, an avatar migration for avatars which may or may not exist, followed by a user migration. The cleanest way to handle this is to check for existence in prepareRow() and return FALSE, telling Migrate to ignore that record:

public function prepareRow($row) {
  $row->id_member = $this->add_path($row->id_member);
  if (!file_exists($row->id_member)) {
    return FALSE;
  }
}

Is this helpful?

cameron prince’s picture

Hi Mike,

Yes, absolutely... That was next on my list to try. Thank you for confirming.

Cameron

mikeryan’s picture

Status: Postponed (maintainer needs more info) » Fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

jaxxed’s picture

Title: Don't fail import when files (pictures/avatars) are missing from local path » Don't fail import when files (pictures/avatars) are missing from local path [patch]
StatusFileSize
new425 bytes

This is an older issue,

I have a similar problem, but I am using remote files, and I don't want my row import to fail, so failing on the prepareRow() is not my solution.

I traced the problem down to the destination plugin. The plugin throws an exception if the file copy fails.

  protected function copyFile($destination) {
    // Perform the copy operation.
    if (!@copy($this->sourcePath, $destination)) {
       throw new MigrateException(t('The specified file %file could not be copied to ' .
         '%destination.',
         array('%file' => $this->sourcePath, '%destination' => $destination)));
    }
    else {
      return TRUE;
    }
  }

This exception carries all the way up to the base class, and import, where it fails the row import.

As a quick fix, you can simply rem out the exception using the patch I've attached. I don't like this patch, but it works. If someone could recommend a better place to catch the exception before it makes it all the way up the chain, preferably in the custom migrate class, then we could suggest a custom catch to fix things up properly.

jaxxed’s picture

I am adding the same patch in a manner that can be used in a make file.

jaxxed’s picture

And one more try here. Ignore the previous files if you are using .make

jaxxed’s picture

kill me, or at least delete my previous files