Terms starting with spaces cause duplicates [#1854382]

Hi,

I import nodes with a free taging field using

$this->addFieldMapping('field_tags', 'Tags')
        ->separator(',')
        ->arguments(array('create_term' => TRUE));

Recently I discovered, to my horror, that my Tags vocabulary contains loads of duplicate tag names - remnants of various rolled back migrations!

Started to experiment and found out that

tags are NOT deleted during rollback
but especially: at another import attempt (same migration) new tags are created instead of re-using the existing ones!!!

Is there any way to force addFieldMapping to re-use an existing term if there is any?

(Note: tags are NOT duplicated if there are several identical ones in one migration.)

Mike or anybody out there -- have you had a problem like this and is there a solution?

Comment	File	Size	Author
#14	avoid_duplicate_terms-1854382-14.patch	952 bytes	Luxian
#8	avoid_duplicate_terms-1854382-8.patch	620 bytes	Luxian

Comments

Comment #1

vacilando commented 29 November 2012 at 21:09

Title:

Terms not rolled back and duplicated in repeated imports

» Terms not rolled back and multiply with repeated imports

Comment #2

vacilando commented 29 November 2012 at 23:27

Title:	Terms not rolled back and multiply with repeated imports	» Terms with spaces multiply with repeated imports
Category:	support	» bug

OK, so I've found out that terms multiply like this only when they contain space (more than one word).

Currently I am not treating multi-word terms in any way, they get imported properly. If I import them in quotes, then the quotes are part of the term, which is also undesirable.

The problem occurs somewhere in the create_term logic -- it does not find existing terms with spaces and ends up creating new ones.

Any idea?

Comment #3

vacilando commented 30 November 2012 at 00:11

In class MigrateTaxonomyTermReferenceFieldHandler, there's this bit:

$existing_terms = db_select('taxonomy_term_data', 'td')
          ->fields('td', array('tid', 'name'))
          ->condition('td.name', $values, 'IN')
          ->condition('td.vid', $vid)
          ->execute()
          ->fetchAllKeyed(1, 0);

I've verified that the select here finds no tid for terms that contain spaces.

Any idea how to fix the query or the array $values to get all the tids?

Comment #4

mikeryan

he/him

English

Pittsfield, MA, USA

commented 4 December 2012 at 01:07

Status:

Active

» Postponed (maintainer needs more info)

A few things:

1. You should be using the subfield notation now for options (arguments() should still work, but is deprecated):

$this->addFieldMapping('field_tags', 'Tags')
        ->separator(',');
$this->addFieldMapping('field_tags:create_term)
        ->defaultValue(TRUE);

2. Terms created as a side-effect of a node migration (when you set create_term on the field) will not be rolled back when the nodes are rolled back, because there's no tracking for them - to be able to rollback terms, they need to have their own migration (and thus map table tracking them).

3. I cannot reproduce the multiple term creation. In beer.inc of migrate_example, I added

    $this->addFieldMapping('migrate_example_beer_styles:create_term')
         ->defaultValue(TRUE);

and removed migrate_example_beer_styles:create_term from addUnmigratedDestinations(). I ran the term and user migrations, then added the row 99999999,'india pale ale' to the migrate_example_beer_topic_node table (so it would be added via create_term rather than the BeerTerm migration. I ran the BeerNode migration and 'india pale ale' was added to the style vocabulary. No matter how many times I rollback and re-import BeerNode, I always get a single instance of the IPA. There must be something else going on in your environment.

Comment #5

vacilando commented 4 January 2013 at 22:09

Status:

Fixed

» Postponed (maintainer needs more info)

Thanks, Mike!

1&2: OK, thanks!

3: I re-tested everything but was still getting duplicates. Eventually found out it's not duplicating just those terms that contain spaces (multi-word) -- my mistake. In fact terms were duplicated when they contained a space at the beginning! The space is not allowed in a term, so the duplicates were identical, but they were being created as a consequence of the initial space in migrated values. I then solved it with trimming each term in prepareRow().
I therefore propose you trim() all values in array $values in MigrateTaxonomyTermReferenceFieldHandler to prevent this from happening to anyone else in the future.

Comment #6

mikeryan

he/him

English

Pittsfield, MA, USA

commented 31 December 2012 at 17:16

Title:	Terms with spaces multiply with repeated imports	» Terms starting with spaces cause duplicates
Category:	bug	» task
Status:	Postponed (maintainer needs more info)	» Fixed

Fix committed, thanks.

Comment #7

18 January 2013 at 22:10

Status:

Postponed (maintainer needs more info)

» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Comment #8

Luxian

he/him

Romanian

Zurich Area

commented 25 June 2013 at 16:12

Status	File	Size
new	avoid_duplicate_terms-1854382-8.patch	620 bytes

Sorry for digging this old issue, but I had the same problem few days ago.

Basically I have some tags to import, with terms separated by comma like this "tag1, tag2, tag3". I used separator(',') and got multiple terms created, but the extra space is causing a lot of duplicate terms. I cannot use trim as a callback for that field (argument will be an array and the function expects a string) and I cannot add it to separator (is not always there).

The problem is that MigrateTaxonomyTermReferenceFieldHandler is matching term name with spaces in the SQL query while the first line in taxonomy_term_save() is removing them. So taxonomy terms created by handler will never be matched by the original value again.