Use Batch API to improve refresh batching

Really amazing!
My ideas here:

• About hook_feedapi_after_refresh() : what about appending this to the last item of the batch queue? This needs an additional function.
  To do this, where do you need contextual information? I see that it needs to be executed after all of the previous tasks in the queue was finished and the $feed variable should be available.

  $func = $module .'_feedapi_after_refresh';
  $func($feed);
  

• About the stats: yeah, if we're doing the updates in this way, a permanent location is needed to store these data. Stats are really important.

All in all, this seems to be a great idea!

As discussed on IRC, the show stopper for using batch API for feed processing is that it assumes a browser with either JS or meta refresh support on the other end. This would kill using drush for refreshing feeds.

Further, I think it does not attack the problem at its core where FeedAPI starts a $half_done feed from scratch because it cannot keep a pointer of where it left off.

Clarify title.

dixon_: I thought more about this problem.

While you said that importing chokes on the processing level and not on the parsing level, I think the parsing level, and more specifically, the fetch (read) level is where batching needs to happen.

Ultimately, this is the very front of the import process and thus the best place to batch. If we add batching somewhere in the middle of the import process (i. e. the processing stage) we don't attack the problem at its core. We might at some later point find out that really, really huge files need to be imported and then we wind up with a scenario where we have batching on more than one level.

I've tried to solve such cascaded batching with some pseudo code yesterday and it inevitably leads to a mess.

Further, we would waste resources by importing a huge but parseable file, even loading it from cache, to then let the processors nibble from it in a batch cycle.

Thus, I think batching should happen in parsers.

We will still have to download full HTTP documents, we will even have to parse them completely, but the parser should only pass on as many items as the rest of the stack can take. This will need to be some sort of setting with a good default.

It's getting much more interesing when we're talking about files. CSV files can be read line by line and fetching and parsing can be batched easily. This would allow us to process extremely big files, gigabytes of data if need be.

In regards to parsing extremely big files, have a look at killes' patch over here: #548312: Allow feeds to be collected from files

#13: I completely see your point.

in the scenario I'm describing in #12, there would be a default setting for how many feed items can be parsed at once - a 'chunk size setting'. It would be pretty low, something like '20'.

With such a default setting, we could guarantee that pretty much any processor set up would be able to complete within PHP execution time (this is _always_ a guess btw). While we wouldn't break most feeds (news feeds with less than 20 items) into chunks.

Further, this setting would only limit how big a chunk FeedAPI can bite off at once, not how many.

If users need more performance or if the processor set up is particulary heavy, this setting can be tuned.

Obviously, the disadvantage of this approach is that there is a setting that needs to be tuned in special cases - but I would argue those are very few cases. On the upside, it keeps FeedAPIs import procedure simple and flexible by moving the slicing of feeds all the way to the front of the process.

In a first iteration, chunk size setting could even be a parser_csv internal setting. You could read N lines from the CSV file, parse it, pass it on to FeedAPI and cache the position of the file pointer. Next time FeedAPI calls you, you keep reading where you've left off until you hit the end of the file where you reset the pointer to the beginning of the file.

subscribing