Limit number of feed items to import

I managed to hack core for no, File feeds/plugin/FeedsSyndicationParser.inc

class FeedsSyndicationParser extends FeedsParser {

  /**
   * Implements FeedsParser::parse().
   */
  public function parse(FeedsSource $source, FeedsFetcherResult $fetcher_result) {
    $result->items = array_slice($result->items, 0, variable_get('feeds_import_count', 50));
    return $result;

feeds_import_count is my custom value, that is reset after my processing, so the default import is not affected.

This is a handy trick, thanks @selvamkf.

It's particularly useful in development, if you want to keep importing just a few records at a time until you get your own code correct.

It looks like at one point the 'feeds_process_limit' variable was added for this purpose, but it doesn't seem to be working anymore. As far as I can tell the only place it is referenced is in the clear() and expire() methods, not during the import where it is most needed.

Also changing the category to a bug report.

'feeds_process_limit' only limits the batch size, not the total number of items imported.

'feeds_process_limit' is used in FeedsImporter::getLimit() which parsers should be using to limit the number of items they parse. This doesn't always happen though.

It seems like there are 2 different things going on here.

1. Restrict the total number of items imported.
2. Restrict the number of items imported per batch.

Hello,

By mistake I created similiar "support request" I'm also stuck while trying to import 10 000 feed items, it imports them at 2000 at a time and then it gives nginx error.

I'm importing items from here: http://api.steampowered.com/ISteamApps/GetAppList/v2 tried to use both available json parsers none was able to limit them.

So is there a question maybe to somehow convert JSON into CSV and then import the csv ? I think its the only possible workaround...

I keep coming back to this issue because of what @jenlampton said

It's particularly useful in development, if you want to keep importing just a few records at a time until you get your own code correct.

Thanks @selvamkf!

Hi Guys,

Can you please provide the detailed replication steps? I just want to replicate this issue on my local.

Here's a patch to limit the number of feed-item imported.

EDIT: Sorry, but FeedsEntityProcessor.inc has been removed from feed so the patch isn't working anymore. Have to do a new one...

As mentioned by Twistor #5 at https://www.drupal.org/node/1909974,

Is there a solution to "limit the total number of items imported per feed" ? Currently, I use manual file upload/import instead of batch
and would like to put a hard limit on number of items imported each time I manually run the feed import process.

Thus far I only see a solution to "restrict import per batch" by setting $conf['feeds_process_limit'] in settings.php file or
reviewing the patches above, "feeds_process_limit" is set in the patches ...

Just to give more clarity to the problem:

Currently:

1) I use manual feed import instead of batch (which means feeds_process_limit setting will not work for me to limit the imported items as I am not looking to only limit the batch import limit
2) At the end of feed import process, I use "rules module" to eliminate feed items that are not qualified to keep in the imported node.
The rule structure is as follows:

Event: before saving an item imported with xyz feed
Condition: if it meets this condition
Action: skip import of feeds item

I am looking for, say, total of importing 100 items after rules applied.
I "am not" looking for initially importing 100 items, then, eliminating, say 30 % with rules, and end up 70 items.

Hope I am making my use case clear. This is not batch import limit problem. Would like to understand how we can achieve limiting feed items including rules.

Thank you!

About the "feeds_process_limit" setting

The "feeds_process_limit" setting was never intended to serve as a solution for this issue: limit the number of items to import. So there is no bug here. It was indended to serve as a workaround for PHP's max execution time. A feed can take a lot time to import and as Feeds isn't smart enough to predict how many items it will be able to parse and process during one request, the "feeds_process_limit" was introduced to allow users to define this for themselves. They can increase the value if the import sources are small and should decrease it when Feeds isn't able to finish one batch in time.
Unfortunately, it depends on the implementation of the parser for this setting to have effect. Only a few parsers respect this setting (for example the CSV parser and the parsers from the Feeds extensible parsers module.

In Feeds 7.x-2.0-alpha8 and before only one batch was run per cron run and as a side effect the "feeds_process_limit" setting could be used to limit the number of items to import per cron run. This behaviour was changed in #1231332: periodic import imports only one file per cron, which meant Feeds can now run multiple batches per cron run depending on how much time there is left before reaching PHP's max execution time (if I understanded that other issue correctly).

Debugging Feeds configuration: Feeds import preview

If you looking for an answer to this solution for debug purposes (like jenlampton and joelpittet do, see #2 and #7), you can also use my Feeds import preview module, which allows you to see what your source looks like after it has been parsed (before doing a real import).

How to limit the number of items to import using Feeds

Now let's take a look at what you can do to limit the number of items to import. As twistor pointed out in #5 there are two questions asked here:

1. How to restrict the total number of items to import (ignore any other items in the source).
2. How to restrict the number of items to import per cron run (but import everything in the end).

Restrict the total number of items to import

There are several phases during an import where the total number of items to import can be limited:

• After parsing, before processing: hook_feeds_after_parse()
• Before saving a feed item: hook_feeds_presave() or the Rules event "Before saving an item imported via [importer name]".

Restrict the number of items to import after parsing (before processing)

The easiest (and fastest) way to limit the items to import is by implementing hook_feeds_after_parse(). Via that hook all items to import are passed, so you can simply loop through them and unset the items you don't want to import. The con about using this method is that there is less context available. You only have the source to work with, there are no entities loaded yet.

/**
 * Implements hook_feeds_after_parse().
 */
function mymodule_feeds_after_parse($source, $result) {
 $count = 0;
 foreach ($result->items as $item_key => $item) {
   $count++;
   if ($count > 10) {
     unset($result->items[$item_key]);
   }
 }
}

Note: Instead of looping through all the items, array_slice() can also be used if you just want the first 10 items for example.

$result->items = array_slice($result->items, 0, 10);

Restrict the number of items to import during processing (before saving)

The above method restricts the number of items being imported during parsing. However, during processing you could also prevent items from being saved by implementing hook_feeds_presave() and in there, set $entity->feeds_item->skip to TRUE. If you cut down the number of items just before processing (in hook_feeds_after_parse()) and you are also skipping items during processing, you will get a lower number of items imported then you may had intended.

Luckily, the FeedsSource object (which is passed with the hook_feeds_presave() hook) contains information about the number of items that are currently processed. This information can be found inside a FeedsState object. During processing, this FeedsState object can be accessed this way:

$state = $source->state(FEEDS_PROCESS);

This FeedsState object tells you (among more) how many items were created and updated and how many failed to process.

So, if you want to limit the number of items during the process phase, use the sum of $state->created and $state->updated. If that exceeds the desired maximum number of items to import, set $entity->feeds_item->skip to TRUE.

Example:

/**
 * Implements hook_feeds_presave().
 */
function mymodule_feeds_presave(FeedsSource $source, $entity, $item, $entity_id) {
  $state = $source->state(FEEDS_PROCESS);
  if (($state->created + $state->updated) >= 10) {
    // Do not import more than 10 items.
    $entity->feeds_item->skip = TRUE;
  }
}

Beware that if multiple implementations are skipping items, the total number of imported items may also get lower. This depends on which implementation is executed last. If the above implementation is the last one, you're good. Note that the Rules event "Before saving an item imported via [importer name]" (which can also be used to skip items) is always invoked later than the hook_feeds_presave() hook.

Using Rules

With Rules you can also skip items during processing by using the Rules event "Before saving an item imported via [importer name]" and the Rules action "Skip import of feeds item". Unfortunately, in Rules the information about the number of processed items is not available, so trying to limit the number of items being imported using Rules is quite hard. You could work around this by using a "global" variable in which you keep track yourself of the number of items that are processed:

1. At the beginning of the import process you should initiate this variable with zero, to make sure that it doesn't contain a value that is leftover from the previous import. For this, you could use the Rules event "Before importing feed" and the Rules action "Execute custom PHP code" (I don't think there is a Rules action for setting a global variable yet?):

         variable_set('feeds_import_' . $source->id . '_' . $source->feed_nid . '_count', 0);
       

   Note that $source->id contains the machine name of the importer being used and $source->feed_nid the id of the feed node (if the importer is attached to a content type). $source->feed_nid is always zero when using a standalone importer.

2. In the Rules event "Before saving an item imported via [importer name]", you start with getting the global variable. First, add an integer variable within Rules using the "Add a variable" action. Then, using the Rules action "Execute custom PHP code" set this variable to the value of the "global" variable:
   

         $variable_added = variable_get('feeds_import_my_importer_0_count', 0);
       

   You have to specify the machine name of the feeds importer and the feed node id manually, since this context is not passed to the rules event.

3. Check using the Rules Conditional module if the added Rules variable exceeds the desired maximum number of items to import. Within the conditional, add a "Skip import of feeds item" action. Configure eventually other logic in which an item must be skipped inside this Rule.
4. Finally, make sure that you configure the rule in a way that if the action "Skip import of feeds item" does not happen, that you increment the Rules variable with 1 and then write this value back to the "global" variable. For the increment you can use the Rules actions "Calculate a value" and "Set a data value". Setting the "global" variable can be done with the "Execute custom PHP code" action:
   

         variable_set('feeds_import_my_importer_0_count', $variable_added);
       

When using Rules you should also be aware that if you are skipping items to import in multiple rules using the same event, the total number of imported items may get lower.

Restrict the number of items imported per cron run

I have no answer yet to the question of how to limit the number of items to import during each cron run. During a cron run, if an import is not completed, it is put immediately back on the "DrupalQueue" for further processing. See FeedsSource::scheduleImport(). As pointed out earlier this behaviour was introduced in #1231332: periodic import imports only one file per cron.

I suppose you could use the methods from above to achieve this behaviour, but then you will have to keep track yourself of where the import exactly halted (to know with which item to start during the next import). And you would also need to store the source as the source can possibly change between two imports. These are things Feeds does already take care of when using it without manipulation: for example, the HTTP fetcher caches the source in the database. So when a Feed import is not completed, the source isn't refetched, but taken from the cache. If you manipulate the number of items to import using hook_feeds_after_parse() or hook_feeds_presave(), Feeds will consider an import to be completed and then it will refetch the source next time.

Summary

• The "feeds_process_limit" setting is not intended to use for limiting the number of items to import, only for not running out of time during an import batch because of PHP's max execution time.
• Limiting the total number of items to import can be done by unsetting items in hook_feeds_after_parse() or skipping items in hook_feeds_presave().
• Unanswered: limiting the number of items to import per cron run.

Since there isn't a real answer yet for how to limit the number of items to import per cron run, I leave this issue open. Phew, I should have a blog to write about stuff like this.

Thanks MegaChriz ! for such a comprehensive response. Method 1 "Restrict the number of items to import after parsing (before processing)" works out perfect and simple. Will explore rules option if needed.

Drupal 7 reached end of life and the D7 version of Feeds is no longer being developed. To keep the issue queue focused on supported versions, we’re closing older D7 issues.

If you still have questions about using Feeds on Drupal 7, feel free to ask. While we won’t fix D7 bugs anymore, we’re happy to offer guidance to help you move forward. You can do so by opening (or reopening) a D7 issue, or by reaching out in the #feeds channel on Drupal Slack.

If this issue is still relevant for Drupal 10+, please open a follow-up issue or merge request with proposed changes. Contributions are always welcome!