Feeds Randomly Reimports / Duplicates Already Imported Nodes

In the feeds_item table, there are fields for entity_type, entity_id, and guid. For a given node id/entity id do you see multiple records. If you set node id to the guid in the mapper does it work? Do you have multiple importers applied to the same content type?

Can you export your importer settings and attach them?

We are experienced a similar issue:

In total have 7,077 Commerce products. Made use of $conf['feeds_process_limit'] = XXX; to speed up the job.

We experienced an issue where it would automatically start to delete the products which was solved by following #1423912: Delete process starting without asking!

The Sku is mapped to (2) field both the GUID and Commerce SKU. -- Checking the DB both fileds have the data correctly. :-)

This is the only feed assigned to this entity type.

After the the import was deleted the file was manually deleted from the server as it is in the public files.

However if we keep the file or don't it keeps trying to reimport.

Attached please a screen grab of the feeds log along with an export of the feeds configuration. If need anything more please let me know.

Thanks!

$feeds_importer = new stdClass;
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'cruise_ship_import';
$feeds_importer->config = array(
  'name' => 'Cruise Ship Commerce Importer',
  'description' => 'Feed importer for cruise ship dates and prices. Please ask before using.',
  'fetcher' => array(
    'plugin_key' => 'FeedsFileFetcher',
    'config' => array(
      'allowed_extensions' => 'txt csv tsv xml opml',
      'direct' => FALSE,
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsCSVParser',
    'config' => array(
      'delimiter' => ',',
      'no_headers' => 0,
    ),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsCommerceProductProcessor',
    'config' => array(
      'product_type' => 'cruise_dateprice',
      'author' => '3',
      'tax_rate' => '',
      'mappings' => array(
        0 => array(
          'source' => 'SKU',
          'target' => 'sku',
          'unique' => 1,
        ),
        1 => array(
          'source' => 'Ships-Name',
          'target' => 'title',
          'unique' => FALSE,
        ),
        2 => array(
          'source' => 'NID',
          'target' => 'field_tour:nid',
          'unique' => FALSE,
        ),
        3 => array(
          'source' => 'Departure-Time',
          'target' => 'field_grp_tour_trp_date:start',
          'unique' => FALSE,
        ),
        4 => array(
          'source' => 'Arrival-Time',
          'target' => 'field_grp_tour_trp_date:end',
          'unique' => FALSE,
        ),
        5 => array(
          'source' => 'Single-Room-Supplement-Percentage',
          'target' => 'field_singe_room_sup_percentage',
          'unique' => FALSE,
        ),
        6 => array(
          'source' => 'Americas',
          'target' => 'field_cruise_price_amer:amount',
          'unique' => FALSE,
        ),
        7 => array(
          'source' => 'European',
          'target' => 'field_cruise_price_eu:amount',
          'unique' => FALSE,
        ),
        8 => array(
          'source' => 'Asian',
          'target' => 'field_cruise_price_asia:amount',
          'unique' => FALSE,
        ),
        9 => array(
          'source' => 'QTY',
          'target' => 'commerce_stock',
          'unique' => FALSE,
        ),
        10 => array(
          'source' => 'Departure-Port',
          'target' => 'field_cruise_port_dep',
          'unique' => FALSE,
        ),
        11 => array(
          'source' => 'Arrival-Port',
          'target' => 'field_cruise_port_arr',
          'unique' => FALSE,
        ),
        12 => array(
          'source' => 'Cabin-Name',
          'target' => 'field_cruise_cabin_names',
          'unique' => FALSE,
        ),
        13 => array(
          'source' => 'SKU',
          'target' => 'guid',
          'unique' => FALSE,
        ),
      ),
      'update_existing' => '2',
      'input_format' => 'php_code',
    ),
  ),
  'content_type' => '',
  'update' => 0,
  'import_period' => '0',
  'expire_period' => 3600,
  'import_on_create' => 1,
  'process_in_background' => 1,
);

I've been struggling with this issue and I think I've figured out why it's happening.

In trying to import a very large file, I've fiddled with $conf['feeds_process_limit'], cron times and php time outs and I've finally got it working without creating duplicates or random deletes.

The issue seems to be caused by cron getting called while a separate cron process is already running. I thought that cron couldn't execute in parallel like this, but apparently it does. The fix is to choose a balance between the feeds_process_limit, cron scheduling and php time outs so that cron will never get called when it's running already.

I've changed this to a major issue because depending on server resources and potentially other unrelated processes being run, this solution may not always work and it can severely affect import performance if not tuned properly. In addition, the logic that's executed during multiple cron runs doesn't seem to respect the GUID import settings either. At a minimum the feeds background import should create a lock to prevent cron from running the import if it's already running.

In my scenario, I am running cron every minute because I'm in development and I just need to import lots of data quickly. I have a feeds_process_limit of 500. I'm now installing Elysia Cron to see if I can improve my configuration and make it more robust.

fix

If you run into duplicate records, check if you've configured set a target as unique and if Feeds does receive a value for this target. You can check that easily with the Feeds Import Preview module.