Create a metrics backend framework module

posting this discussion w/ drumm on IRC, regarding a more generalized approach:

hunmonk: drumm: so about this idea of putting the metrics stuff in it's own module...
drumm: hunmonk: I think a lot of it sounds separated out
drumm: hunmonk: in an ideal world other people start using this and it gets updated to D7 by the community & such.
drumm: kinda ambitious, but I think smaller modules are more workable
hunmonk: drumm: i agree.  i guess i'm just concerned about doing *another* 'metrics' module.  this whole area seems disorganized
hunmonk: drumm: i guess it's a complicated area
drumm: hunmonk: we are well past the 'no duplicates on d.o', that's part of why we are doing metrics 
drumm: hunmonk: I haven't looked at it too much in detail, but at a high level, grab numbers every x time (week), store em, and display em, isn't too project specific
hunmonk: drumm: is your proposal that we shoot for a truly general metrics framework, or more just toss this particular style of doing metrics over the fence and see what happens?
drumm: hunmonk: we won't make a truely general one
hunmonk: yeah
hunmonk: drumm: i guess if we're generalizing, i'm trying to figure out what this would generally be
hunmonk: drumm: 'time-based metrics' ?
hunmonk: drumm: period metrics?
drumm: maybe not that
drumm: hunmonk: if we simply go with the current plan and do the namespace change, are there apis and such that will get in the way?
hunmonk: drumm: yeah, there is an info hook
hunmonk: drumm: that's how other modules advertise their metrics
drumm: ie more modules pusing data in and not calling project apis from it
drumm: yeah, so it already is modular
drumm: I think technically you do the same thing, with a different name
drumm: time-based metrics is decent
hunmonk: drumm: we have a separation of logic, in that this module handles storage and decides when to process the metrics.  but the exposed metrics are calculated completely by the module advertising the metric
hunmonk: drumm: i give you a list of ids, and a time period, you give me your value
drumm: hunmonk: sounds good
hunmonk: drumm: the one non-general thing about it is that the metrics module has to have a way to know the ids to send to the other modules
hunmonk: drumm: i guess we could expose that via another attribute in the info hook
hunmonk: drumm: so one attribute would be 'tell me where i can get the list of ids you want me to handle', and another attribute would be 'tell me where to get the metric value for your ids'
hunmonk: drumm: i think if we did it that way, then we have full abstraction

hunmonk is working on what a sample info hook from a provider module would look like, and I think that'll answer alot of these questions. From my perspective, being able to change the time window, and possible have different time windows for different metrics, is a key requirement for a 'genric' solution. There may be other features that would go into making this generic, but I think that is probably the biggest, most complex that I can think of (or I have use case for) at this time.

From my perspective, being able to change the time window, and possible have different time windows for different metrics, is a key requirement for a 'genric' solution.

while this may be true, i don't think we have to solve this problem the first time around to make this module generally useful -- it would just be more useful with time flexibility. that said, it might be fairly easy to allow for time flexibility, i don't think it's totally off the table. what about if we provided a default implementation of weekly calculation native to the module, but allowed modules to specify a callback to return their own time periods? that seems like a fair solution

We don't have to make everything generic & configurable upfront. As long as the code is clean, don't have 60*60*24*7 everywhere, at least use a constant. When the time comes to make things alterable by API or configurable by UI, we can swap in what is needed.

the start of the code is now living here:

http://drupalcode.org/viewvc/drupal/contributions/sandbox/thehunmonkgrou...

so far i've got:

1. stub module file with @file and the info hook loader wrapper
2. .metrics.inc file with the info hook loader
3. install file with intall, uninstall, and schema hooks
4. a small example module that demonstrates using the info hook, and callbacks for fetching the object IDs and calculated metrics

this is basically done. updated code at the sandbox listed in #5. brief summary of what's been accomplished:

1. added README
2. add time_unit and time_unit_interval metric attributes.
3. add support functions for doing the necessary date math
4. add support functions for: calculating sample points based on unit/interval, pulling last sample time, computing metrics, storing metrics, and a high level process function that leverages all these
5. completely refactored the schema handling stuff to use the approach outlined at #889890-4: Finalize and implement data storage plan for project_metrics. The schema is now loaded from our schema state table when hook_schema() is called, and updated any time the info_load() function initially builds its information. i've also triggered the info_load() function on module enable/disable for maximum consistency

the code still needs to be tested, but it's definitely far enough along to start work on #889892: Expose metrics data to views

Looks generally okay. Any specific areas that might need closer review?

i struck on a fairly easy way make this API much more flexible, so i spend about two hours reworking the sample set logic so that it can both be used more flexibly when querying for a metric, and can accept custom sample periods and sample sets. i also updated the example module to showcase the custom sample sets. latest code in the sandbox mentioned in #5.

i've given the module an extensive workover, debugged thoroughly, and documented everything, so i think we can put this issue to bed, and open new issues for anything that comes up.

brief list of more stuff i accomplished from my sandbox commit log:

rework sample set logic so that it can both be used more flexibly when querying for a metric, and can accept custom sample periods and sample sets.
rip out some repetitive code doc, and make the example module showcase the custom sample sets.
add simple test function that allows computing/storing a metric via a GET request. always unset metrics in schema update check when the metric is already known.
add some debugging code to time/date helper functions.
make save function try update first.
Convert string timestamps into Unix timestamps if necessary in options array.
Don't calculate any values if the last sample point is in the future.
add a simple google chart API implementation to the test function.
convert example node body phrase counter metrics to use more common phrases.
ensure valid metric data is returned before taking any action.

no official project exists for this module yet. moving to redesign queue for now, and retagging

deployment will be handled over at #893912: deploy Sampler API with project metrics

moving this issue to the new module's home...

another complete rewrite has been done in my spare time... :P

now it's object-oriented, and storage, sampling method, and sampling adjustments are all pluggable.