Add standard mechanism for "row parsers"

I've talked to Alex about this before and he asked me to post it to the issue queue so that I could explain it better. Here goes. :-)

I would propose that Feeds needs a standard mechanism for row parsers and possibly row processors.

Currently, as I understand how the system works, an incoming feed is parsed en masse to an intermediary form in PHP, usually an array. That intermediary form is then passed en masse to a processor, which takes that data and generates nodes, users, table records, or whatever the desired end result is.

That's all well and good when the incoming format is a unified format, like RSS or CSV. However, the vast majority of feeds one is going to process will follow a very regular format: They will essentially be an array of "things to import", represented in whatever the incoming format is. The format of each item in that array will be largely identical, and processed in a loop. Similarly, what we're saving to will be largely identical.

However, not all incoming feeds will be in a single unified format. Many formats support wrapping an arbitrary other format internally. The example that comes to mind for me is Atom. Atom itself supports a number of different payload formats, including raw text, HTML, arbitrary XML, etc. However, there is no Feeds-core supported way to handle the Atom separately from the payload (that I'm aware of). The feed is treated as a single entity. We ran into this issue when developing the Feeds Atom module. We had a rich, complex payload (a custom RDF-ish XML format for representing Entities and Fields) wrapped in Atom, but the parser and processor only get them in combined form. That means that the Atom parser we wrote cannot be used without that custom XML payload, and if we wanted to wrap that custom XML payload in a different feed format we couldn't.

Instead, I propose that the "feed parser" and the "row parser" be separated. The responsibility of the feed parser is to extract each of the entries in the feed, attach any relevant metadata, and then pass the entry and its metadata along to a configured row parser. The row parser is then responsible for Extracting the relevant data from just that one record into a standardized PHP format. That single-entry record is then passed off to a processor that turns that internal format into a node, user, table row, etc.

This is a direct mirror of row style plugins in Views; Each record in the result set is rendered separately, and then aggregated into a single output by the overall style plugin. As with Views there are use cases where that doesn't apply and you'd want to still handle the entire result set or feed at once, and those should continue to be supported. In essence, this would make Feeds a more direct mirror of Views (and I believe that comparison is apt).

There are a number of advantages to this approach:

1. Row parsers and Feed parsers become more generic and can be interchanged more easily; It would be awesome to have a natively supported way to swap out a different payload for Atom feeds, or a way to trivially wrap NodeXML into some other envelope format.
2. Because we're separating out each row, there's no requirement that rows have to be parsed and processed in the same request. It would be easy to simply dump the rows into a queue and then process them one at a time either in the same request if it's small enough, via batch API if it's an interactive pull, or over time with the Drupal Queue module (or D7's native queue system). I believe there's another issue open on the same subject. That would allow for handling vastly larger incoming feeds as queues can scale almost indefinitely.
3. As with the benefits of the queue over the old way that update hooks run (where you have to keep track of how many items there are to process yourself), row parsers become easier to write as you need only handle one (1) record at a time. Questions of synchronization and scaling are handled "elsewhere" and a plugin author needn't worry about it.
4. Because it more closely parallels Views, it should make it easier for new developers to pick up. "Ah, it's just like Views only backwards."

The disadvantages would be additional internal complexity, especially as the system is already written, and the need to provide some way to handle the case were processing of one item does depend on a previous item. That may need to just be handled with a non-row-using parser, which should be retained. There are also UI implications to be considered, although I firmly believe those are surmountable.

Hopefully that better explains what I'm talking about. :-) Alex, let me know if it's still unclear.