A transporter interface to allow for batching

So, here is a first take at the new FeedsTransporter interface. The patch comes with two implementations of the interface:

• FeedsDirectTransporter - transports parsed items directly to the processor.
• FeedsBatchTransporter - uses Drupal's built in Batch API to pass items to the processor.

FeedsDirectTransporter is attached to all importers by default as this is the current behavior of Feeds. So to test FeedsBatchTransporter you have to programmatically attach it to an importer.

The patch still needs some work. I haven't implemented any configurable options for the transporters yet etc so some values like the batch size is hard codes at the moment. Also, transporters should do the work in FeedsImporter::clear() instead of processors handling that by them selfs.

I think it makes sense to ship Feeds with FeedsQueueTransporter also, as Feeds have built in support for queuing whole feeds refreshments with drupal_queue. But I haven't written FeedsQueueTransporter yet.

Hey, glad you're attacking batching.

I'm trying to understand the underlying assumptions better. Can you explain why you decided to add a separate transporter plugin type instead of adding batch support into FeedsProcessor class?

I'm struggling to see the inherent advantage of making batching a seperate import stage on the same level as fetching, parsing, processing, I do see a couple of problems though:

1. The batching is a cross section problem, a problem of import execution and not a problem of a particular import stage. While I agree that we should focus our batching work on the processing stage for now, a stage-based like FeedsTransporter approach is painting ourselves into a corner.
2. The patch introduces a novel pattern: a plugin (transporter) executes another plugin for import. FeedsImporter is the only class involved in the execution of an import so far.
3. Moreover, using the plugin concept for batching will require switching the specific class in use depending on the context of the import: when importing from cron you'll have to use a different class than when you're importing from through the browser.
4. A new stage does introduce complexity on the architectural level.

I propose implementing the main batching functionality on the FeedsImporter level while keeping FeedsImporter and plugins agnostic to the batching implementation (Batch API or cron).

1. Introduce an additional parameter: FeedsProcessor::process($parserResult, $source, $batch = FALSE);
2. Introduce an additional parameter: FeedsImporter::import($source, $batch = FALSE);
3. Introduce a return value in FeedsProcessor::process() that indicates whether process() is done: FeedsProcessorResult extends FeedsResult (alternatively, the return value may just be a string that indicates the status).
4. Have FeedsImporter::import() return the FeedsProcessorResult.
5. Modify import() so that it does not invoke fetchers/parsers if a batch is active for a given source. This functionality needs to be concurrency safe: Batching must not break if two processes start a batch at the same time or a batch is running while cron updates the same source. Concretely, this means that a) a process is importing and b) the progress of the batch will need to be stored with the source (in feeds_source table).

This approach is designed to fit snugly with the existing architecture of Feeds, it allows us to implement batching with BatchAPI and (with modifications of FeedsScheduler) on cron. It further doesn't limit us to batching only on the processing level (while we're only focussing on processing for now). It uses the existing infrastructure for modifying FeedsImporter behavior by swapping the FeedsImporter class defined in the Drupal variable 'feeds_importer_class'.

Some further considerations:

• The return value from import() will neatly take care of batching via batch API. Batching via Drupal Queue should not be hard: we just need to add a feed to the queue again if it didn't finish processing. Batching on cron without DrupalQueue will require more refactoring, as FeedsScheduler right now adds a source for next refresh at the time it starts processing it, not at the time it is done processing. This last problem could pose some inefficiencies when using Drupal Queue, too: when a source is scheduled for 'refresh on cron as often as possible' and it is further very large and heavily batched, we will wind up with multiple queue entries for that source.
• We should time batching (15 to 20 seconds a batch) and not set it to a number limit.
• Instead of adding an additional parameter to process() and import() we could think about a class FeedsImportContext that carries a source and a batch flag. But I think at this stage, this is overdoing it.

Thoughts?