This Cookbook shows, how you import one CSV file into a Drupal 7 site, creating new users, each with Profile2 fields.

The trick is to create two separate migrations from one source. The first creates the users, the second creates the profiles.

The second migration connects up the profiles it creates with the users that now exist by mapping the source unique key, MID, to the user uid. This is achieved by applying the line
$this->addFieldMapping('uid', 'MID')
to the code.

Remark
The intention of this cookbook is, with detailed notes on this page and comments in the code to explain the function of each component of this migration as a working demo. For your first try use a fresh installed 'drupal 7 with default profile' and then start with step 1.

 

Cookbook:

 

Step 1: Before we start

  1. If not enabled, (download, install to your modules-path like sites/all/modules and) enable these modules:
  2. Add one 'profile type' in 'http://example.com/admin/structure/profiles':
    Label ='Memberlist'
  3. Add some fields in 'http://example.com/admin/structure/profiles/manage/memberlist/fields':
    • 'Text'-field: Label ='Member-ID', Field name = 'field_mnr', length = 6
    • 'Text'-field: Label ='Username', Field name = 'field_username'
    • 'Text'-field: Label ='Complete Name', Field name = 'field_name'
    • 'Date'-field[5]: Label ='Birthday', Field name = 'field_birthday', Date attributes to collect: Year = yes, Month = yes, Day = yes, Hour = no, Minute = no, Second = no
    • 'Text'-field: Label ='Tel. 1', Field name = 'field_tel_1'
  4. Check the permissions for 'Memberlist' on 'http://example.com/admin/people/permissions#module-profile2'

 

Step 2: Build the import-module

Containing directory

Create a containing directory for your module. For example, all later steps in this section assume that you're in the directory sites/all/modules/a_wusel_migration.

.info file

In your module directory, create the file a_wusel_migration.info containing the following:

name = "A Wusel Migration"
description = "Importing a CSV file as new users, each with Profile2 fields"
package = Migration

dependencies[] = migrate (>=2.8)
dependencies[] = migrate_extras
dependencies[] = profile2

; only for the 'Date'-field 'Birthday':
dependencies[] = date (>=2.8)
dependencies[] = date_api

; include the Classes file 
files[] = a_wusel_migration.migrate.inc

; TIP: to show the migrate user interface delete the ";" at the beginning of the next line (or use drush)
;dependencies[] = migrate_ui

version = "7.x-1.2"
core = 7.x

Hints:
The line "dependencies[] = migrate (>=2.8)" enforces to have a module-version of the migrate module like 7.x-2.8 or newer (e.g. like 7.x-2.8+xx-dev).
Currently being tested with "7.x-2.8".

The line "dependencies[] = date (>=2.8)" enforces to have a module-version of the date module like 7.x-2.8 or newer.
Currently being tested with "7.x-2.8".

.module file

In your module directory, create the file a_wusel_migration.module[1].


/**
 * @file
 * THIS FILE INTENTIONALLY LEFT BLANK.
 *
 * Yes, there is no code in the .module file. Migrate operates almost entirely
 * through classes, and by adding any files containing class definitions to the
 * .info file, those files are automatically included only when the classes they
 * contain are referenced. The one non-class piece you need to implement is
 * hook_migrate_api(), but because .migrate.inc is registered using
 * hook_hook_info, by defining your implementation of that hook in
 * a_wusel_migration.migrate.inc, it is automatically invoked only when needed.
 */

This is all you require in your .module file. And you need this file!

Classes file

In your module directory, create the file a_wusel_migration.migrate.inc, starting with the following code[1]:

/**
 * @file
 * Our own hook implementation.
 */

/**
 * Implements hook_migrate_api()
 *
 * Returns 'api' => 2 for the 7.x-2.x branch of Migrate.
 * Registers the migration classes for the 7.x-2.6 branch of Migrate (including
 * 7.x-2.6+xx-dev).
 */
function a_wusel_migration_migrate_api() {
  $api = array(
    'api' => 2,

    // Migrations can be organized into groups. The key used here will be the
    // machine name of the group, which can be used in Drush:
    //  drush migrate-import --group=WuselMigrate
    // The title is a required argument which is displayed for the group in the
    // UI. You may also have additional arguments for any other data which is
    // common to all migrations in the group.
    'groups' => array(
      'WuselMigrate' => array(
        'title' => t('WuselMigrate Imports'),
      ),
    ),

    // Here we register the individual migrations. The keys (Wusel_Step1_User,
    // etc.) are the machine names of the migrations, and the class_name
    // argument is required. The group_name is optional (defaulting to 'default')
    // but specifying it is a best practice.
    'migrations' => array(
      'Wusel_Step1_User' => array(
        'class_name' => 'Wusel_Step1_UserMigration',
        'group_name' => 'WuselMigrate',
      ),
      'Wusel_Step2_Memberlist' => array(
        'class_name' => 'Wusel_Step2_MemberlistMigration',
        'group_name' => 'WuselMigrate',
      ),
    ),
  );
  return $api;
}

Continuing with this same file, we then add an abstract class[1]:

/**
 * Migration classes for migrating users and profiles
 *
 * based on: drupal.org/node/1269066#comment-4988994
 * and:      drupal.org/node/1190958#comment-4616032
 */

/**
 * Abstract class as a base for all our migration classes
 */
abstract class Wusel_Basic_Migration extends Migration {
  public function __construct($arguments) {
    // Always call the parent constructor first for basic setup
    parent::__construct($arguments);
    // Avoid known line ending issue: "Invalid data value" at drupal.org/node/1152158#InvalidDataValue
    ini_set('auto_detect_line_endings', TRUE);
  }
}

You might find you don't need this abstract class, but it helps avoid a problem with line ending encodings that some people have reported. Basically, rather than using Migration as our base class below, we always use Wusel_Basic_Migration.

Continuing with this same file, we then add a class to migrate our users[1]:

/**
 * User-only migration - not profile fields
 *
 * The data file is assumed to be in
 *   sites/all/modules/a_wusel_migration/data_sources/
 */
class Wusel_Step1_UserMigration extends Wusel_Basic_Migration {
  public function __construct($arguments) {
    parent::__construct($arguments);
    $this->description = t('Import an CSV-file (only "Account"-fields)');
    $columns = array(
      // "Source": ('Fieldname', 'Description')
      0 => array('MID', t('Member-ID (must be unique)')),
      1 => array('mail', t('Email (Account)')),
      2 => array('name', t('Name (Account)')),
      3 => array('password', t('Password (Account)'))
    );
    // TIP: delete ", array('header_rows' => 1)" in the next line, if the CSV-file has NO header-line
    $this->source = new MigrateSourceCSV(DRUPAL_ROOT . '/' . drupal_get_path('module', 'a_wusel_migration') . '/data_sources/drupaluser_import.csv', $columns, array('header_rows' => 1));
    $this->destination = new MigrateDestinationUser();
    $this->map = new MigrateSQLMap($this->machineName,
        array('MID' => array( // this field is used to connect user und profile2
                'type' => 'varchar',
                'length' => 6,
                'not null' => TRUE,
                'description' => t('User\'s Member-ID') // description never used
              )
             ),
        MigrateDestinationUser::getKeySchema()
    );

    // Mapped fields
    $this->addSimpleMappings(array('name'));
    $this->addFieldMapping('mail', 'mail') 
      ->defaultValue('')
      ->description(t('Email address'));
    $this->addFieldMapping('init')
      ->defaultValue('')
      ->description(t('Email address used for initial account creation'));
    $this->addFieldMapping('pass', 'password')
      ->defaultValue('asdf')
      ->description(t("User's password (plain text)"));
    $this->addFieldMapping('is_new')
      ->defaultValue(TRUE)
      ->description(t('Build the new user (0|1)'));
    $this->addFieldMapping('roles')
      ->defaultValue(DRUPAL_AUTHENTICATED_RID) 
      ->description(DRUPAL_AUTHENTICATED_RID . t(' = "authenticated user"'));
    $this->addFieldMapping('theme')
      ->defaultValue('')
      ->description(t("User's default theme"));
    $this->addFieldMapping('signature')
      ->defaultValue('')
      ->description(t("User's signature"));
    $this->addFieldMapping('signature_format')
      ->defaultValue('filtered_html')
      ->description(t('Which filter applies to this signature'));
    $this->addFieldMapping('created')
      ->defaultValue(time())
      ->description(t('UNIX timestamp of user creation date'));
    $this->addFieldMapping('access')
      ->defaultValue(0)
      ->description(t('UNIX timestamp for previous time user accessed the site'));
    $this->addFieldMapping('login')
      ->defaultValue(0)
      ->description(t('UNIX timestamp for user\'s last login'));
    $this->addFieldMapping('status')
      ->defaultValue(1)
      ->description(t('Whether the user is active(1) or blocked(0)'));
    $this->addFieldMapping('timezone')
      ->defaultValue(t('Europe/London')) // 'America/Los_Angeles', 'Europe/Berlin', 'UTC', ... from drupal.org/node/714214
      ->description(t("User's time zone"));
    $this->addFieldMapping('language')
      ->defaultValue(t('en')) // e.g.: 'en', 'fr', 'de', ...
      ->description(t("User's default language"));
    $this->addFieldMapping('picture')
      ->defaultValue(0)  
      ->description(t('Avatar of the user'));
  
    // Other handlers
    if (module_exists('path')) {
      $this->addFieldMapping('path')
           ->defaultValue(NULL)
           ->description(t('Path alias'));
    }
    if (module_exists('pathauto')) {
      $this->addFieldMapping('pathauto')
        ->defaultValue(1) 
        ->description(t('Perform aliasing (set to 0 to prevent alias generation during migration)'));
    }
    // Unmapped destination fields
    $this->addUnmigratedDestinations(array('role_names', 'data'));
  }
}

Notice the use of addSimpleMappings: $this->addSimpleMappings(array('name'));
addSimpleMappings may be used when the source field and the destination field are assigned the same name/identifier. In cases where they have different names/identifiers you need to use addFieldMapping to map the source to the destination.

In the same file we also write a second class to migrate the Profile2 fields[1]:

/**
 * Profile2 field migration
 *
 * The data file is assumed to be in
 *   sites/all/modules/a_wusel_migration/data_sources/
 */
class Wusel_Step2_MemberlistMigration extends Wusel_Basic_Migration {
  public function __construct($arguments) {
    parent::__construct($arguments);
    global $user;
    $this->description = t('Import an CSV-file with Profile2-fields ("memberlist"-fields)');
    $columns = array(
      // "Source": ('Fieldname', 'Description')
      0 => array('MID', t('Member-ID (must be unique)')),
      1 => array('mail', t('Email (Account)')),
      2 => array('name', t('Name (Account)')),
      3 => array('password', t('Password (Account)')),
      4 => array('complete_name', t('Complete Name (for Memberlist)')),
      5 => array('birthday', t('Birthday (for Memberlist)')),
      6 => array('tel_1', t('Tel.#1 (for Memberlist)'))
    );
    // TIP: delete ", array('header_rows' => 1)" in the next line, if the CSV-file has NO header-line
    $this->source = new MigrateSourceCSV(DRUPAL_ROOT . '/' . drupal_get_path('module', 'a_wusel_migration') . '/data_sources/drupaluser_import.csv', $columns, array('header_rows' => 1));
    // Declare migration 'Wusel_Step1_User' a dependency in migration 'Wusel_Step2_Memberlist' to have them run in the right order, if needed:
    $this->dependencies = array('Wusel_Step1_User');
    $this->destination = new MigrateDestinationProfile2('memberlist');  // 'memberlist' is the "Machine name" of the profile2-"Profile type"
    $this->map = new MigrateSQLMap($this->machineName,
        array('MID' => array( // this field is used to connect user und profile2
                'type' => 'varchar',
                'length' => 6,
                'not null' => TRUE,
                'description' => t('User\'s Member-ID') // description never used
              )
             ),
        MigrateDestinationProfile2::getKeySchema()
    );

    $this->addFieldMapping('uid', 'MID') // Connecting the profile2 to the user using 'MID' - this row is "the trick"
         ->sourceMigration('Wusel_Step1_User')  // If your user migration class was named 'MyUserMigration', the string is 'MyUser'
         ->description(t('The assignment of profile2-items to the respective user'));

    // Mapped fields
    $this->addFieldMapping('field_mnr', 'MID')
      ->defaultValue(0)
      ->description(t('The Member-ID (must be unique)'));
    /* Delete this line, if you need the following:
    $this->addFieldMapping('field_mnr:format')
      ->defaultValue('plain_text')
      ->description(t('The Text-Format of the Member-ID'));
    /* */
    $this->addFieldMapping('field_mnr:language')
      ->defaultValue('und')
      ->description(t('The language of the Member-ID<br />("und" = no language)'));
    $this->addFieldMapping('field_username', 'name')
      ->defaultValue('')
      ->description(t('The login name'));
    /* Delete this line, if you need the following:
    $this->addFieldMapping('field_username:format')
      ->defaultValue('plain_text')
      ->description(t('The Text-Format of the login name'));
    /* */
    $this->addFieldMapping('field_username:language')
      ->defaultValue(t('en'))
      ->description(t('The language of the login name'));
    $this->addFieldMapping('field_name', 'complete_name')
      ->defaultValue('')
      ->description(t('The complete name (for Memberlist)'));
    /* Delete this line, if you need the following:
    $this->addFieldMapping('field_name:format')
      ->defaultValue('plain_text')
      ->description(t('The Text-Format of the complete name'));
    /* */
    $this->addFieldMapping('field_name:language')
      ->defaultValue(t('en'))
      ->description(t('The language of the complete name'));
    $this->addFieldMapping('field_birthday', 'birthday')
      ->defaultValue('') // empty = unknown
      ->callbacks(array($this, 'fixTimestamp'))
      ->description(t('The birthday (for Memberlist)') . '.<br />' . t('Format') . ': "YYYY-MM-DD" <br />' . t('or') . ' "MM/DD/YYYY" <br />' . t('or') . ' "DD.MM.YYYY"');
    /* Delete this line, if you use version 7.x-2.6+24-dev of the module Date or newer
    $this->addFieldMapping('field_birthday:timezone')
      ->defaultValue('UTC') // !!! NO time conversion !!!
      ->description(t('The timezone of the birthday field'));
    $this->addFieldMapping('field_birthday:rrule')
      ->defaultValue(NULL)
      ->description(t('Rule string for a repeating date field [this field is not present]'));
    $this->addFieldMapping('field_birthday:to')
      ->defaultValue(NULL) 
      ->description(t('End date date'));
    /* */
    $this->addFieldMapping('field_tel_1', 'tel_1')
      ->defaultValue('')
      ->description(t('The main telephone-number (for Memberlist)'));
    /* Delete this line, if you need the following:
    $this->addFieldMapping('field_tel_1:format')
      ->defaultValue('plain_text')
      ->description(t('The Text-Format of the main telephone-number'));
    /* */
    $this->addFieldMapping('field_tel_1:language')
      ->defaultValue('und')
      ->description(t('The language of the main telephone-number<br />("und" = no language)'));
      
    // Other handlers
    /* Delete this line, if you need the following:
    if (module_exists('path')) {
      $this->addFieldMapping('path')
           ->defaultValue(NULL)
           ->description(t('Path alias'));
    }
    /* */

    // some internal fields
    $this->addFieldMapping('revision_uid')
      ->defaultValue($user->uid)
      ->description(t('The user ID of the user, who started the migration'));
    $this->addFieldMapping('language')
      ->defaultValue(t('en'))
      ->description(t("The default language of the user (e.g. 'en', 'fr', 'de')"));

    // Unmapped fields (this fields are in core and not needed as profile2-fields)
    $this->addUnmigratedSources(array('mail', 'password'));
  }

  public function fixTimestamp($date) {
    // enable empty (= unknown) birthday-string:
    if (strlen($date) > 0) {
      $date = substr($date, 0, 10) . 'T12:00:00'; // we add 'twelve o'clock in the daytime' for automatic compensation of a website time zone difference to UTC
    }
    return $date;
  }  
}


.install file

It is necessary to add a hook_disable() function to your module to deregister the Migration Classes when you uninstall your custom migration module using two Migration::deregisterMigration lines.

So along with your other module files, create a_wusel_migration.install, containing the following[1]:

/**
* @file
* Implements hook_disable().
*
* the migration module should deregister its migrations
*/
function a_wusel_migration_disable() {
  // based on: drupal.org/node/1418350#comment-5557772
  Migration::deregisterMigration('Wusel_Step1_User');
  Migration::deregisterMigration('Wusel_Step2_Memberlist');
}


Create the directory for the data file

Create the directory "sites/all/modules/a_wusel_migration/data_sources/"[4].

 

Step 3: Enable your module and the Migrate modules

If not present, download and install to your modules-path like sites/all/modules these modules:

Then enable your own module "A Wusel Migration"! Because of the dependencies[] in the .info file,
it should also automatically enable:

  • Migrate
  • Migrate UI (only, if you have activated the line 'dependencies[] = migrate_ui' in the .info file)
  • Migrate Extras
  • Date Migration (for the Date field "Birthday")

Your next steps are:

  1. Clear your cache!
  2. Perform the registration
    1. It is recommended that you use Drush to perform your Migration functions. There is a list of Drush commands for Migrate in the Community Documentation, or you can get the latest list with drush help --filter=migrate. After enabling the module, run drush mreg to register your Classes. Then check that they are displayed with drush ms
    2. If you must use the web admin, visit 'http://example.com/admin/content/migrate' and click the "Configure" option then the "Register statically-defined classes" button.
  3. If you want to import big CSV files, visit 'Background operations' on 'http://example.com/admin/content/migrate/configure'. To enable running operations in the background with drush, (which is recommended), some configuration must be done on the server. See the documentation "Running imports and rollbacks from the UI via drush" on drupal.org.

 

Step 4: Prepare the CSV file

Prepare the CSV file[2] and name it drupaluser_import.csv, e.g.:

"member_nr","email","username","password","complete_name","birthday","tel_1"
"1001","new.tester@example.com","new tester","test","New Tester","07/13/1999","00000 12345-213"
"1002","old.tester@example.com","old tester","test","Old Tester","12.07.1945","00000/54321-0"
"1003","another.tester@example.com","another tester","test","Another Tester","1985-07-11","00000 678901"
"1004","","incognito user","test","Incognito User","",""

This file shows the possible forms of the birthday field (ten or none signs!) for testing the import.
The allowed date formats in the CSV-file are:
"YYYY-MM-DD" or "MM/DD/YYYY" or "DD.MM.YYYY".
The delimiters are different for each format and have to be used properly!
This is only for the import!
Within e.g. a view, you can chose the format of the output.

The columns of this CSV file have to follow the order of both

$columns = array(
...
);

-code-parts in the file 'a_wusel_migration.migrate.inc'!

Ideally you may use the same names for the source fields in the class arrays as the column labels in the csv. The labels you use for the columns in the csv source file are ignored during the import via the array('header_rows' => 1) attribute, they are purely for visual inspection. The classes assign names/identifiers to each field in the arrays - so in your arrays you can assign the source fields any arbitrary name, as long as your array follows the order of the columns in the csv from left to right.

 

Step 5: Import

Store the CSV-file drupaluser_import.csv from Step 4 in the place as specified (sites/all/modules/a_wusel_migration/data_sources/drupaluser_import.csv).

Clear your cache!


There are two alternatives for the rest of this step:

a) Import with Drush (recommended)

step 1 to create the user accounts:
drush mi Wusel_Step1_User
step 2 to create the profiles:
drush mi Wusel_Step2_Memberlist

hint: get a list of the registered Class Names with drush ms
If you named the Migration Class "Wusel_Step1_UserMigration" the class name will be "Wusel_Step1_User" (without "Migration").

Or:

b) Import with the Migrate UI via the web admin

If you have enabled the module migrate_ui:
Go to Administration > Content > Migrate and click on "WuselMigrate Imports" or go to http://example.com/admin/content/migrate/groups/WuselMigrate.
Check the box for the first migration Wusel_Step1_User, select "Import" or "Import immediately" and click on "Execute".
Check the box for the second migration Wusel_Step2_Memberlist, select "Import" or "Import immediately" and click on "Execute".

 

Step 6: Show the import

Visit 'http://example.com/admin/people'.

 


 

Notes:

[1]: Some sub-notes:

  1. Include the line "<?php" only once in the very first line of this file!
  2. Don't include the line "?>" in this file!
  3. Here they are several times visible to format this page.

[2]: Some sub-notes:

  1. The CSV-file must have the ","-separator. The ";"-separator is not allowed!
  2. The CSV-file must be in "UTF8 with BOM"-format, then the import of special characters (letters like €, £, ß, ö ,ä, ü, Ö, Ä and Ü or other non-ASCII-signs) is without problems.
  3. If you don't include the line
    ini_set('auto_detect_line_endings', TRUE);
    in the classes:
    The needed/correct line-ending-char(s) of a Feeds-imported CSV-file depends on the type of the operating system of the www-server:
    If you are using a Linux-Server, use only LF at the line-end of the CSV-file.
    If you are using a Windows-Server, use CR+LF at the line-end of the CSV-file.
    If you are using a Mac-Server, use only CR at the line-end of the CSV-file.
    The changing of the line-end of the CSV-file before importing is important, if the source of the CSV-file (e.g. your computer or the database of the CSV-file) has a different operating system!
    For the meaning of LF and CR look at http://en.wikipedia.org/wiki/Newline#Representations
  4. You can use a good editor like 'notepad++' on windows or 'LibreOffice Calc', both when indicated: '... Portable', (or use 'MS Excel') to change this.
    Tip:
    Use "Save as" and change the needed properties before and/or during saving the file ("before / during" is depending on the program used).

[3]: You can translate this enabled module using the Locale module (in core). Or use 'Add a translation' (http://drupal.org/node/1524254).

[4]: It's convenient to have the sample data files (drupaluser_import.csv) in the module directory for the example module, but bad practice with real data. Don't try this at your real live server! Your source data should be outside of the webroot, and should not be anywhere where it may get committed into a revision control system.

[5]: To be able to store dates before "1901-12-14" (12/14/1901), never use the field type "Date (Unix timestamp)"!

 

Good luck!

 

Comments

mortona2k’s picture

I have been struggling with this for a while now, and finally got it working this morning, just before this page went up. Thanks anyway!

edit: Oh, this is based on my forum post! *facepalm*

The other Andrew Morton

ambientdrup’s picture

Trying out this module approach but getting the following errors when I try to run the Wusel_Step_1 import:

Notice: Undefined variable: user in Wusel_Step2_MemberlistMigration->__construct() (line 129 of /Applications/MAMP/htdocs/www/sites/all/modules/custom/a_wusel_migration/a_wusel_migration.migrate.inc).
Notice: Trying to get property of non-object in Wusel_Step2_MemberlistMigration->__construct() (line 129 of /Applications/MAMP/htdocs/www/sites/all/modules/custom/a_wusel_migration/a_wusel_migration.migrate.inc).

Suggestions?

-Trevor

ambientdrup’s picture

I'm also getting this message when running the first import:

Processed 0 (0 created, 0 updated, 0 failed, 0 ignored) in 0 sec (0/min) - done with 'Wusel_Step1_User'

-Trevor

wusel’s picture

@ambientdrup:

Your posting on December 1, 2011 at 10:53pm:
It was my error.
I add a new line 3 of "class Wusel_Step2_MemberlistMigration" as "global $user;", please look at the code above. Please add it in your code, too.

Your other posting on December 1, 2011 at 11:06pm:
Is your CSV-file at the correct path? On the tab/page "Source" of "http://example.com/admin/content/migrate/Wusel_Step1_User", you can find your path to the CSV-file.
And always clear your cache after putting this (e.g. new) CSV-file in the correct path.
If you are using a Linux-Server, please use only LF at the line-end of the CSV-file.
If you are using a Windows-Server, please use CR+LF at the line-end of the CSV-file.
The CSV-file must have the ","-separator. The ";"-separator is not allowed!

Wusel

kannanraj’s picture

In case our csv file content have duplicate user (eg: by mistake to add same user details in mutiple times) how can i check to using conditional statement and where do write the code..

darrylri’s picture

You can do it in a prepareRow() public method:

public function prepareRow($row) {
  if (this is a bad row) {
    return FALSE;  // this row will be skipped in the migration
  }
  return TRUE;
}

Look to the general migrate documentation to find out more about this method.

kannanraj’s picture

when i add user name and other details it shows 'it already exist' and that user name does not saved in database but other field values stored. i wish when user name is same other field values do not have to store in database and also that user account do not have to create. how to write code for this condition?

bjlewis2’s picture

Thank you soooooo much!

Any chance you could put together an "Update Users and Profile2 Info" cookbook?

Anonymous’s picture

This has been a LIFE SAVER for a client job - thank you!!

tranquilrealm’s picture

I tried this example and it worked. At least I can see the Import worked like it should and imported 3 things as in the csv. But where are the results? How do I display those results after the import?

TimelessDomain’s picture

This code looks like it only works for 1 profile type, can you please mod it so that it works with 2 or more? Thanks

freedom isn't free

wusel’s picture

1.) Copy the whole "class Wusel_Step2_MemberlistMigration" as "class Wusel_Step3_SecondProfile2Migration" to the bottom of the file "sites/all/modules/a_wusel_migration/a_wusel_migration.migrate.inc".
2.) Edit this new class to fit your second Profile2.
Here you have to change "memberlist" in the line "MigrateDestinationProfile2('memberlist')" to your second Profile2-Name.
The "$columns" and the "AddFieldMapping"s depend on the fields of your second Profile2.
3.) Add the new fields to your CSV-file.

Good luck!

Wusel

Tom Ash’s picture

Also:

4) Add the new class to 'migrations' => array() in function a_wusel_migration_migrate_api()

Unfortunately the extra column I import in my 3rd class gets imported as blank. Can anyone link to how to debug the migrate module? Alas print_r($this->source) within the functions in the original tutorial doesn't show the CSV contents even within the 2nd class, which imports succesfully.

TJM’s picture

Thanks for the tutorial and code.
For some reason my http://mysite.com/admin/content/migrate does not display the Wusel_Step1_User and Wusel_Step2_Memberlist options.
Only the MigrateExampleProfile2 and Date migration options are displayed.
I have the a_wusel_migration directory/file structure as follows:
modules (folder)
--a_wusel_migration (folder)
----a_wusel_migration.inc
----a_wusel_migration.info
----a_wusel_migration.module
----data_source (folder)
------drupaluser_import.csv
I removed the end ?> from .inc and .module files
Modules installed: migrate, migrate_extras, migrate ui, migrate extra profile2 example, a_wusel_migration, profiles2, entity api, date, date migration, date api, etc...
Added profiles to the user data. The profile fields mirror the ones on the Drupal 6 site.
Clear cache several times.
Basically, I am trying to convert a Drupal 6 to 7 website that has a membership list with profiles. The Drupal 7 site is fresh and I want to do the membership migration before adding content to the site. This new site is being developed on a local wampserver.
Is there a reason that the Wusel_Step1_User and Wusel_Step2_Memberlist options do not display?

wusel’s picture

@TJM:
First implement this Cookbook without any changes on a test-drupal-system (Drupal core 7 without any other module and without any other theme).

Only when there are no errors, then change part for part, but only one part in every step. "Part" may be a value of the description above or an additional module or another theme. If this runs you may change the next part.

Good luck!

Wusel

TJM’s picture

Hi Wusel,
Thanks for your continued support.
I did a fresh install of Drupal 7 and when through the files again. Guess what? The problem was a file name. As seen above in my post, the module file name is not correct.
Wrong: ----a_wusel_migration.module
Correct: ----a_wusel_migration.migrate.module
After the correction, Step 1 and 2 show up correctly. Now I can proceed with your module.
Sorry about my mistake,
Ted

andjules’s picture

I'm getting this error:
SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'field_company_value' cannot be null: etc, etc, etc
I've triple checked everything and looked at migrate issues (where this error only seems to come up for some filefield... in my case it's a plain text field).
- Wondering if anyone has seen this error in relation to this cookbook and has any advice
- And Wusel, I'd be happy to talk about compensation for troubleshooting...

andjules’s picture

Whoops, I realize the field throwing the error is in fact a computed field.
So now I have to figure how to migrate to a destination computed field... any advice, anyone?

TJM’s picture

My users are already in my database by upgrading from Drupal 6 to Drupal 7. However, this upgrading did not transfer the Profiles to the new Profile2. So, currently, I have a database with all the users and their passwords in my database. How should your code be modified to add the user's profiles?
Thanks,
Ted

shamio’s picture

I think your problem is explained on this section and this section provides information about importing user profiles from a CVS file to profile2 to use them in Drupal 7. Please read this section again and if it didn't work do you well, ask your question again with more details that members can help you better. However i am not sure if it helps you, you can enable profile module in Drupal 7 too by going to yoursite.com/admin/modules page.

wusel’s picture

The trick of this import is to create two separate migrations from one source. The first creates the users, the second creates the profiles.
I think you have to do both steps of this migration and not only step two.
Or you have to use another field to connect ("map") user und profile2. But I don't know the trick for you.

Please feel free to create an issue at http://drupal.org/node/add/project-issue/migrate or (look) at http://drupal.org/forum/37
This may help you!

Please report your solution with a new cookbook-page! Thanks from the other users, who have similar problems.

Wusel

TJM’s picture

I have moved on to other projects. I might let the users enter their own profiles again. Possibly, someone will come up with a simple solution for just transferring the profiles of Drupal 6 to profiles2 of Drupal 7.
I appreciate all of your contributions and assistance.
Thanks
Ted

rclemings’s picture

I was able to import profile data for existing users by modifying these files only slightly:

  1. From the .module file, delete the line: Wusel_Step1_User' => array('class_name' => 'Wusel_Step1_UserMigration')
  2. From the .inc file:
  • Delete the entire class: class Wusel_Step1_UserMigration extends Wusel_Basic_Migration
  • Delete this line: $this->dependencies = array('Wusel_Step1_User')
  • Delete this line: ->sourceMigration('Wusel_Step1_User')
  • Keep this line; it tells migrate which user record your data goes with: $this->addFieldMapping('uid', 'MID'), but change the field names to match what's in your destination and source fields.
  • Create new field mappings $this->addFieldMapping to link the columns in your source table to the appropriate profile2 fields, and update the $this->addUnmigratedSources array to include all of the source columns you're not migrating.

To migrate data from the same CSV into multiple profile types, as mentioned elsewhere in the comments, just create new classes like Wusel_Step2_MemberlistMigration and add them to the .inc file, and to the migrations array in the .module file. Since Wusel_Step1_User and Wusel_Step1_UserMigration no longer exist, you might also want to renumber the remaining step(s), although it's probably not required.

Tom Ash’s picture

I don't understand the following, can you give an example of what field names to give and what the import CSV should have for them?:

Keep this line; it tells migrate which user record your data goes with: $this->addFieldMapping('uid', 'MID'), but change the field names to match what's in your destination and source fields.

cobenash’s picture

Thx a lot!

i have a problem about importing taxonomy term.

CSV:

name,sex,phone
cobenash,F,0000

My problem is the data of sex in CSV is "F". However,the taxonomy tid in my drupal site is 7.

How can i get the value of the CSV and then use if~else if to put the correct tid in the field?

nitvirus’s picture

Thanks for the cookbook, its great
Can you help me with giving a role to the migrated user.

thanks
nitish

wusel’s picture

In the file a_wusel_migration.module change the line

      ->defaultValue(DRUPAL_AUTHENTICATED_RID)

to

      ->defaultValue(array(DRUPAL_AUTHENTICATED_RID,4))

if you want e.g. to add the role with RID = 4 to all users. This RID must exist!

You also can import the roles from the CSV file. Add a new column and assign it to the field 'roles'.

Note:
"DRUPAL_AUTHENTICATED_RID" has the RID = 2.
You can view this at 'http://example.com/admin/content/migrate/Wusel_Step1_User', click at the tab 'Mapping: Done'.

Good luck!

Wusel

nitvirus’s picture

You really helped,
Now i have got into another problem.
I have added the birthday field in the registration form only. I have added your timefix function there also, But i am getting:
DateTime::__construct(): Failed to parse time string (1T12:00:00) at position 0 (1): Unexpected character (/var/www/fodler1/sites/all/modules/migrate/includes/base.inc:1019)
error could you help with this.

nitvirus’s picture

I solved it by keeping the mapped field in the same order in CSV as in code.

jrochate’s picture

Thanks, this is great :)

I was wondering if I can map Profile2 fields that are "Field Collection" type.

jrochate’s picture

Guess what? I've just added a Step3 with successfully field collection migration.

This is what i've done:

1. Add this patch to field_collection module
http://drupal.org/node/1175082#comment-6763310

2. Add new class to the .migrate.inc developed on the current post

3. Import one or more childs to profile2. Cool or what? :)

thanks Wusel!

class Wusel__Step3_ChildMigration extends Wusel_Basic_Migration {
  public function __construct() {
    parent::__construct();
    global $user;
    $this->description = t('Import an CSV-file with Field Collection-fields ("child"-fields)');
    $columns = array(
      // "Source": ('Fieldname', 'Description')
      0 => array('MID', t('Member-ID (must be unique)')),
      1 => array('mail', t('Email (Account)')),
      2 => array('name', t('Name (Account)')),
      3 => array('password', t('Password (Account)')),
      4 => array('child_name', t('Name (for Child)')),
      5 => array('child_birth', t('Birthday (for Child)'))
    );
    // TIP: delete ", array('header_rows' => 1)" in the next line, if the CSV-file has NO header-line
    $this->source = new MigrateSourceCSV(DRUPAL_ROOT . '/' . drupal_get_path('module', 'a_wusel_migration') . '/data_sources/drupaluser_import.csv', $columns, array('header_rows' => 1));
    $this->dependencies = array('Wusel_Step2_Memberlist');
    $this->destination = new MigrateDestinationFieldCollection('field_mychildcollection',array('host_entity_type' => 'profile2'));  
    $this->map = new MigrateSQLMap($this->machineName,
        array('MID' => array( // this field is used to connect child und profile2
                'type' => 'varchar',
                'length' => 6,
                'not null' => TRUE,
                'description' => t('User\'s Member-ID') // description never used
              )
             ),
        MigrateDestinationFieldCollection::getKeySchema()
    );

    // Connecting the field collection to the profile2:
    $this->addFieldMapping('host_entity_id', 'MID')
         ->sourceMigration('Wusel_Step2_Memberlist')  // If your user migration class was named 'MyUserMigration', the string is 'MyUser'
         ->description(t('The assignment of collection-items to the respective profile'));

    $this->addFieldMapping('field_child_first_name', 'child_name')
      ->defaultValue('')
      ->description(t('The first name (for Child)'));
    /* Delete this line, if you need the following:
    $this->addFieldMapping('field_name:format')
      ->defaultValue('plain_text')
      ->description(t('The Text-Format of the complete name'));
    /* */
    $this->addFieldMapping('field_child_first_name:language')
      ->defaultValue(t('pt-pt'))
      ->description(t('The language of the complete name'));
    $this->addFieldMapping('field_child_birthday', 'child_birth')
      ->defaultValue('') // empty = unknown
      ->callbacks(array($this, 'fixTimestamp'))
      ->description(t('The birthday (for Child)') . '.<br />' . t('Format') . ': "YYYY-MM-DD" <br />' . t('or') . ' "MM/DD/YYYY" <br />' . t('or') . ' "DD.MM.YYYY"');
  }

  public function fixTimestamp($date) {
    // enable empty (= unknown) birthday-string:
    if (strlen($date) > 0) {
      $date = substr($date, 0, 10) . 'T12:00:00'; // we add 'twelve o'clock in the daytime' for automatic compensation of a website time zone difference to UTC
    }
    return $date;
  }
}
kannanraj’s picture

I did all the things said above. Data in csv file doesn't store in database but table is created, and I can't view the data in my site. How can i show data stored in csv file on drupal site?

kannanraj’s picture

I done this. I forget to execute the import.
Thanks for your post.

kannanraj’s picture

i did to create user profile from import data from csv file. The date format is storied in MM.DD.YYYY but i want to store this format DD.MM.YYYY. how can i write the condition code in this format

wusel’s picture

The allowed date formats in the CSV-file are:
"YYYY-MM-DD" or "MM/DD/YYYY" or "DD.MM.YYYY".
The delimiters are different for each format and have to be used properly!

You have to change "MM.DD.YYYY" to "MM/DD/YYYY".

This is only for the import!
Within e.g. a view, you can chose the format of the output.

You can read this in the "class Wusel_Step2_MemberlistMigration" in the third part of the file "a_wusel_migration.migrate.inc" for the field 'field_birthday'
or after enabling the module "A Wusel Migration" in the UI.

Good luck!

Wusel

kannanraj’s picture

thank your for your replay

albie001’s picture

I keep getting an error returned when I add a new class for field collections as per this post:

"Missing bundle property on entity of type profile2. (../includes/common.inc:7693)"

This is the relevant line in common.inc

  if (!empty($info['entity keys']['bundle'])) {
    // Explicitly fail for malformed entities missing the bundle property.
    if (!isset($entity->{$info['entity keys']['bundle']}) || $entity->{$info['entity keys']['bundle']} === '') {
      throw new EntityMalformedException(t('Missing bundle property on entity of type @entity_type.', array('@entity_type' => $entity_type)));

I can only assume this is related to the function 'MigrateDestinationFieldCollection' as the error is not returned if I remove this section of code.

Any suggestions welcome...?

Thanks

albie001’s picture

Managed to resolve the issue.

I was calling the wrong class for the sourceMigration.

robearls’s picture

I seem to be getting an error when clicking on the "Execute" button. I've checked every line, twice! I've even stripped out my code to just have Step 1, with no solution in sight.

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=572&op=do StatusText: Service unavailable (with message) ResponseText:

I just cannot see what is wrong. No errors are in the error logs or on screen. I'm at the end of my tether, having spent all day, and would really appreciate a pointer as to where to start!

Using Migrate 7.x-2.x-dev - But the same on 7.x-2.5

kannanraj’s picture

i imported user details from csv file using migrate module. in my site i'm using address field module, and i wish to import postal address for each user from cvs file to address field. how can i do this?

wusel’s picture

Please create an issue in the address field module issue queue.

Good luck.

Wusel

kannanraj’s picture

i do like this format to importing address field and its work. This is the csv source to mapping used in array

$columns = array(
0=> array('city', t('city (for pro_contact_information)')),
1=> array('state', t('state(for pro_contact_information)')),
2=> array('zip', t('zip(for pro_contact_information)')),
3=> array('country', t('country(for pro_contact_information)')),
);

i used to following this code to map csv file to postal address field using addFieldMapping

$argument = array( 'thoroughfare' => array('source_field' => 'address'),
'locality' => array('source_field' => 'city'),
'administrative_area' => array('source_field' => 'state'),
'postal_code' => array('source_field' => 'zip'),
);    // The country ISO code should be passed in as the primary value.
    $this->addFieldMapping('field_ftf_address','country')
          ->arguments($argument);
kannanraj’s picture

i'm using migrate module. when i import a cvs file user account created successfully. i want to generate password randomly for each user automatically when import cvs file. how can i do this?

jrochate’s picture

Hi.

The solution is to use the prepareRow function.

Let's assume that you have an CSV column with the title 'password', where you can have pre-generated passwords OR you can leave empty cells and let you migration code create random ones.

In you mapping fields, you have something like this inside __construct():

    $this->addFieldMapping('pass', 'password')
      ->defaultValue('password')
      ->description(t("User's password (plain text)"));

After you close __construct(), and still inside the class, you can add something like this:

  public function prepareRow($current_row) {
	  $pass=$current_row->password;

	  // Validate the password
	  if (strlen($pass)>0) {
		  $current_row->password = $pass;
	  } else {
		  $current_row->password = implode('',array_rand(array_flip(array_merge(range('a','z'),range('A','Z'),range('0','9'))),8));
	  }
   }

This function will analyze every row.
It's very handy! :)

kannanraj’s picture

i'm using migrate module. when i import a cvs file profile2 fields created successfully. In my csv file have some filed to import to create profile field. if any one column is NULL the database as store blank space and its display on only field name. In this situation i want to omit that data storing in database and also do not display on my profile view page how can i do this

albie001’s picture

One question however.

I have gotten this to work as expected with text fields. Having a slight issue getting the date field to work but am more interested in how to go about incorporating 'List (text)' fields?

Any help appreciated

wusel’s picture

For your 'date' field look at https://drupal.org/node/1477602 Step 1 No. 3 and 4 ("Time zone handling" to "No time zone conversion") and then at https://drupal.org/node/1477602#note1 and my hints at "Step 4: Prepare the CSV file" on this page.
If you want to store "date and time" information e.g. in the field 'Birthday', delete the line
->callbacks(array($this, 'fixTimestamp'))
and delete the line
/* Delete this line, if you use version 7.x-2.6+24-dev of the module Date or newer
and use the newest dev-version of the date module. Now you should be able (not tested by me!) to import the timezone of that field using an extra field of your CSV-file (you have to add some code!).

While building the 'List (text)' field, you have typed in the "Allowed values list", one value per line, in the format "key|label". Within the migration you have to import only the "key"-value from the CSV-file. It may be that the keys must exist before the import.

Good luck!

Wusel

albie001’s picture

Thanks for the response. After much testing it turned out it was simply the date format on the .csv file that was causing a problem.

Managed to get the List(text) fields working though!

Thanks again.

albie001’s picture

Is it possible to import these?

Thanks

albie001’s picture

Please can someone possible help?

I am trying to import data into a field that can contains more than one value.

i.e. A List(txt) field with a number of results in the drop down. Users can choose more than one item from this list.

A/ How do I write this into a class to map the fields

B/ What is the formatting of the .csv file to allow for multiple values?

Hopefully this makes some sort of sense?

Thanks

SKAUGHT’s picture

i found the basic routines worked with this change to them.

class Wusel_Step1_UserMigration extends Wusel_Basic_Migration {
   public function __construct($arguments) {
      $this->arguments = $arguments;
      parent::__construct();
      //.... rest of code..//
   }

}

..and thanks for the example. a good base.

albie001’s picture

Hi,

Once again, this cookbook has proved invaluable for user migration including profile2 fields, so big thank you.

However, I would like to know how I can import profile 2 data to existing users via this method?

I'm guessingI need to remove the dependancy on the 'MID' but my code-fu is not good enough to quite suss this out.

Any advice please?

joelstein’s picture

FYI, in case you want to use Feeds to import your data, there's now a module called Feeds Profile2 which extends the built-in User processor so that Profile2 profiles can be created and updated as part of a user importer.

schuffr’s picture

Hi all,

I am having a problem migrating address fields. Everything else in my migration works, but address field simply is blank. I have searched for many hours and havent found a definitive answer that works. I am using Drupal 7.37, Migrate 7.x-2.7, Address Field 7.x-1.1. I have set the default value for address to US and I am using the : subfield notation. Any ideas why this wouldn't work?

many thanks!

My migration class is as follows:

    <?php
    class DOCInfoMigration extends Migration {
      public function __construct($arguments) {
    
        parent::__construct($arguments);
    
        $this->description = t('Loads early profiles data to DOCInfo profile');
            
       	/********* Source *********/
       	// MySQL database as source
        $query = Database::getConnection('default', 'default')
                 ->select('DOCInfo', 'u')
                 ->fields('u', array('uid',
                                     'first',
                                     'last',
                                     'phone',
                                     'phonetype',
                                     'dob',
                                     'gender',
                                     'membertype',
                                     'year',
                                     'make',
                                     'model',
                                     'new_used',
                                     'tshirt_it',
                                     'street',
                                     'apt',
                                     'city',
                                     'state',
                                     'zipcode'));
                 
        $this->source = new MigrateSourceSQL($query);
    
    		$this->destination = new MigrateDestinationProfile2('doc_info'); // use machine name of profile
    
    		/*********** Map **********/
    		// Create a "map" which is used to translate primary keys*/
        $this->map = new MigrateSQLMap($this->machineName,
          array(
            'uid' => array(
              'type' => 'int',
              'alias'=> 'u'
            ),
            ),
          MigrateDestinationProfile2::getKeySchema()      
        );
    
        /*********** Connect DOCInfo to user **********/
        $this->addFieldMapping('uid', 'uid');
    #         ->sourceMigration('DOCInfo')  // If user migration class was named 'MyUserMigration', the string is 'MyUser'
    #         ->description(t('The assignment of DOCInfo source data to the respective DOCInfo fields'));
    
        /******* Field mappings ******/
        $this->addFieldMapping('language')->defaultValue('en');
        $this->addFieldMapping('field_fname','first');
        $this->addFieldMapping('field_fname:language')->defaultValue('en');
        
        $this->addFieldMapping('field_lname','last');
        $this->addFieldMapping('field_lname:language')->defaultValue('en');
    
        $this->addFieldMapping('field_home_phone','phone');
        $this->addFieldMapping('field_home_phone:language')->defaultValue('en');
        $this->addFieldMapping('field_phone_type','phonetype');
    
        $this->addFieldMapping('field_dob','dob');
        $this->addFieldMapping('field_doc_gender','gender');
    
        $this->addFieldMapping('field_doctype','membertype');
        $this->addFieldMapping('field_docbikeyear','year');
        $this->addFieldMapping('field_docmake','make');
        $this->addFieldMapping('field_docmodel','model');
        $this->addFieldMapping('field_docmodel:language')->defaultValue('en');
        $this->addFieldMapping('field_docnewused','new_used');
        $this->addFieldMapping('field_italian_t_shirt','tshirt_it');
    
    
        $this->addFieldMapping('field_address')->defaultValue('US');
        $this->addFieldMapping('field_address:thoroughfare','street');
        $this->addFieldMapping('field_address:premise','apt');
        $this->addFieldMapping('field_address:locality','city');
        $this->addFieldMapping('field_address:administrative_area','state');
        $this->addFieldMapping('field_address:postal_code','zipcode');
    
    
        /*** Unmapped destination fields ***/
        $this->addUnmigratedDestinations(array('revision_uid',
    #                                           'field_address',
                                               'field_address:sub_administrative_area',
                                               'field_address:dependent_locality',
                                               'field_address:sub_premise',
                                               'field_address:organisation_name',
                                               'field_address:name_line',
                                               'field_address:first_name',
                                               'field_address:last_name',
                                               'field_address:data',
                                               'field_dob:timezone',
                                               'field_dob:rrule',
                                               'field_dob:to',));
      }
    } 
wusel’s picture

this cookbook is about "Import new users and their Profile2 fields from one CSV file".

Your question is about importing from a database and using address field.

This a an entirely different issue that I've never used. Please repeat this post in another forum so that this problem is solved.
Here it does not belong.

Good luck!

Added:
Add a link here to your new post!

Wusel

schuffr’s picture

ok. thank you. Not sure where it belongs... address field or migrate.