Data Collection API

You probably saw this already on the other issue, but if not, look at http://drupal.org/project/remote_macro. There's no point re-inventing the wheel, so evaluate what's reusable before writing any code.

I agree there are two types of capturers; Data Streams and Event Capturers. Lets try to allow them to be both, not one or the other. I can think of cases where a capturer may want to provide both events and a stream. For example, an advanced client-side screen capturer may also send events when the user switches applications on their OS. This is useful data to have on the timeline, where all events will be displayed, and possibly streams will be viewable (not in the scope of GSoC).

I believe "Live user comments" capturer plugin would be an Event data capturer, not a stream. Task completion (when the user clicks 'next') and answers to questions in the study are events, but are a part of the UTS itself, not an optional plugin. They could be part of a required plugin, employ the same hooks as plugins, or neither.

I think the main distinguishing concept between an event and stream capturer is that streams only make sense if played back continuously. An event dataset can be read haphazardly (e.g. in the order of 1 10 5 7 2 3 9 4) and still make sense. A stream by contrast must be viewed continuously (although continuous sections can be arbitrarily small), i.e. 5.0 to 9.1 seconds.

Streamed data will potentially be so large that it cannot practically go in the DB, but on a filesystem. Also streams may be recorded locally and uploaded asynchronously to preserve quality. There are cases where S3 would be useful to store data (out of scope for now).

Event data however will be used for automated analysis, for example, in a large-scale study an analysing algorithm may try to find where most users varied from the ideal steps, or what the most common path was for advanced users. In this case it's convenient to have the data in the database. Further, Event data will always (AFAICT) be plenty-small enough for the DB, since it's usually either text-based, a custom code, or a data object like a drupal Form, or Node.

So yes, Data storage, filesystem storage, schemas, should be the responsibility of the plugin, but all events need to have generic information exposed, either by storing some of the data in UTS's schema, or through hooks. Fields like timestamp, duration (may not apply to all events), plugin and name/summary of event will appear on the timeline; an overview of all events.

Syncing timestamps from a remote capturer with the server may be challenging, although may not be required of the plugins you've decided to work on in GSoC.

Post-capture hooks; I think we need a few here;

• Study is complete -- allows plugins to transfer recorded data? Although this will probably be initiated client-side
• Cleanup -- compress streamed data
• Purge -- delete data, once it's no longer useful and/or disk-space is required for other task

Further, plugins may optionally provide hooks to merge streams and events together. For example, the screencast plugin might provide a merging hook to merge audio and screencast into one video with audio, and overlay the url, clicks, or typed text as subtitles. This post-merging will only be able to be invoked manually since it is typically resource-consuming. The merged streams and events will be exposed as another data feed, like the raw streams and event feeds, but be flagged and promoted as "analysed" or something.

Similarly, other plugins may provide hooks to look at data from multiple test sessions in the same study (most likely only event data) and analyse it to try to find [in]consistencies. This is different to a "merging hook" above, but similarly, is only invoked manually. Results are stored by the plugin but exposed in the UI as part of the study, not an individual participant's test session. "Analysing hooks" will use other plugin's standard 'get data' hook.

Simple capturers should not try to do anything analytically useful with the data other than storage, format manipulation (e.g. compression), cleanup, view/get and purging. Analysis should be handed off to the above hooks, which may be in different plugins.

The functions you have listed are a good start to defining the API. I'm sure we'll need to adapt it as we go since it is impossible to anticipate all the requirements with an ambitious project like this. Hopefully this will give you something to consider in order to do another iteration.