Change usage stats to not use google charts

I understand the principle behind open source and not using black boxes, but in this situation I don't know that we gain much by not using Google charts, at least as long as that service is available and reliable.

One downside to creating these graphs as .png files is that the bandwidth usage might increase substantially. I have no idea how many hits the project usage pages get, but I just looked at the chart at http://drupal.org/project/usage/drupal (created by Google) and the size of the .png is about 20k. If these files are recreated each week (which will be a *lot* of png files, btw), caching on the local user's computer is probably not going to reduce the need to download the images.

Also, having a separate script and additional dependencies makes it more difficult to test this part of the code and for other sites that want to run the project_usage module (granted, probably very few) to do so.

We've already got all the google charts code working, so I'd prefer not to flush that code in cases where folks just want to use that. So, if/when someone works on this (/me points to the "Unassigned" value in the "Assigned" field), I'd like to see the code for generating the charts moved into .inc files and just move the existing google chart code into a project_usage_chart.google.inc (or something) and then add whatever non-google alternative to a project_usage_chart.foo.inc. That way, folks that what to use the existing solution can, and d.o can run something else.

Also, given how many other things need work, including critical bugs, and far more urgent features (e.g. for the d.o redesign launch) I'm dropping the priority on this to minor. @killes: if this is really important to you, write the code or find someone else to do so. However, if you're finding project* help, I've certainly got a lot more important things they could be working on. ;)

That said, I'm happy to find a different solution for all the new project* charts and sparklines as part of the redesign. We don't have to keep using google for those, just because we're using google for this (for now).

I would recommend taking a look at Flot. It's a jQuery plugin that creates graphs with CSS. No mass-generation of PNG files required.

http://people.iola.dk/olau/flot/examples/graph-types.html
http://code.google.com/p/flot/

I'm not so sure that's a big loss. Screenshots are easy to take. Mollom uses flash to generate its graphs, and you can't save that either, but it doesn't stop anyone from reusing the data:

http://images.google.com/images?q=mollom%20graph

speaking as the person who wrote the original code, i think swapping this out it a totally waste of time. we've got working code that's using a free, reliable web-service that doesn't put a cpu or bandwidth load on d.o. should try to gain independence from the OSL next?

pcharts doesn't seem to have any sparkline support either. http://sparkline.org/ seems like it might be a better php solution for those graphs.

I think pChart is a promising solution. It's open source, free, and the class files hosted in your server.

This might be another reason to not use Google charts: #654392: Drupal Usage Statistics Graph Broken.

At the moment only the graph for the Drupal project is broken, but as time goes by more of the graphs will break as the size of the chart's URI gets too big.

DamZ posted http://drupal.org/files/issues/654392-move-project-usage-to-flot.patch over at #654392-10: Drupal Usage Statistics Graph Broken. That patch belongs here...

Just to note that the charts are now using the Google charts extended encoding so we should be OK for another year or 2 before the URL gets too long again!

I'd just like to mention here that I'm doing some work on a views_sparkline module over at http://drupal.org/project/views_sparkline and this discussion is certainly relevant as the code there is intended to provide the per-project metrics that are part of the redesign.

I looked briefly at the PHP sparkline lib, but it hasn't had a release since 2005 and appears quite abandoned. pChart looks fairly nice, and would definitely be an option, but what I want to avoid is trying to support 2 different chart APIs in views_sparkline since they tend to all work just differently enough that they end up with lots of very similar code in each backend.

If we did pChart I think it would make sense to provide a module for it that had options for caching and storing charts in the filesystem. It seems that generating charts on demand, and then caching may be a reasonable approach.

Ideally I'd like to see charts implemented with HTML5 Canvas tag (and obviously with a fallback for bad browsers). One good example is RGraph. I would also like the ability to mouse over a node on the graph and display the values of the x and y axis on that point, as seen on StatCounter.

It's back... also, here are two other ideas for HTML / JS charting libraries:

• Raphaël JS - demos look very impressive
• Google interactive chart API - still depends on google, but creates modern vector / interactive charts that do not have limits on data size. We're planning to use this for google_analytics_reports.

This is a bit more than 'minor' now, since the graph is totally broken on the usage stats pages.

Damien's flot patch no longer applies, but I'll repost it for reference.

Would it be possible to get an SQL dump of the project_usage tables from d.o? (minus the project_usage_raw table of course, which contains private info). I think this would hugely increase the number of possible contributors able to work on the issue.

Second, I think feedback from infrastructure on a particular tool is important. Nobody wants to work on a patch only to be told by infra the needed library (such as flot) isn't acceptable.

Sure, here's a dump of the table data from Drupal.org (~17 MB):

http://dl.dropbox.com/u/10160/project_usage-2011-09-30.sql.zip

Note that both project_usage_day and project_usage_raw have no data, since on d.o we keep those in a separate MongoDB database due to the volume of data.

And I guess killes didn't follow-up here, but I spoke with him on IRC and he's +1 for Flot as well.

So, go forth! Go forth and code thine divine chartiness!

Some progress.... first here's a mock database: http://dl.dropbox.com/u/32009546/project_usage.sql.gz -- the node / term ids have been remapped to a fake drupal project. The patch can be tested at /project/usage/1. Drupal username and password is "admin".

Next, a re-roll + screenshot. We're close but legends are a bit wonky.

Hm.. legend problem seems to be caused by interaction with Garland. It looks fine in Zen; someone will need to test with the bluecheese theme.

As a nice bonus, the X axis labels now include the year!

I don't really like getting rid of the theme_project_usage_chart() function; while Project Usage module can make the default output be Flot, it's possible another site would want Google Charts or some other thing.

And maybe rather than making Flot a hard dependency, we could put an if (module_exists()) around it, and fall back to the Google Charts method if it's not found? For extra special credit, a settings page for this. ;)

I've deployed this patch + Flot module to http://usage-graphs-drupal.redesign.devdrupal.org/project/usage/drupal but I don't see a graph. :\ Am I missing a cluestick?

Also, this might be easier for you to develop if you were to apply for a Drupal.org sandbox access so you could deploy/debug these patches yourself.

What'd really be nice is if we used a ctools plugin for the charts. We could provide both the existing google charts code and the new flot code as separate plugins, and just configure which plugin we wanted. We're moving towards more plugins for this kind of thing. See #1024942: Add ctools plugin system for site-specific logic for packaging release nodes for example. Definitely not a blocker, but I think this makes more sense than a pure theme function, so if you wanted to re-roll as a plugin system, hurray. I'm too tired to review the patch right now, but hopefully in the next day or so.

Thanks!
-Derek

webchick: the link you posted is password protected, but perhaps the flot library (http://code.google.com/p/flot/) isn't installed?

dww: If we took the ctools approach, would that mean that ctools itself becomes a dependency?

I see the merit of a pluggable system, but can you elaborate on the specific benefits of retaining the google charts code? It just seems like more obsolete cruft to maintain.

@grendzy: Hrm, ctools should already be a dependency of project_release although I see it's not. ;) It only matters if you're trying to have the site package release nodes and it's certainly possible to use project_release without that, but whatever, that's for another issue. If we go a ctools plugin route here, I'm fine making that a dependency inside the project_usage.info file.

However, you raise an interesting point about the benefits of maintaining the obsolete cruft. I was just thinking since that code already exists, it'd be a *lot* easier for sites that just want to use google charts to change a setting to swap out the plugin then to dig all that code out of version control and put it in their theme. Maybe they don't want to install the flot library? I dunno. I guess I don't have a compelling argument. This just seems like the kind of thing that would be easy to make pluggable and we already have the gchart code. So yeah, I'm lukewarm on my 1am logic from last night about this. ;)

p.s. Re: password protected: the popup tells you the username/password to use. ;) You still probably wouldn't be able to log in as an admin to verify your guesses, but that's another story.

doh! I see the password hint now, and the flot library indeed seems missing: http://usage-graphs-drupal.redesign.devdrupal.org/sites/all/modules/flot...

Using google, I can only find one publicly visible site displaying project_usage graphs:
http://www.google.com/search?q=inurl:project/usage+-site:drupal.org+%22i...
While I'm sure this search misses some, to me it indicates there isn't a large installed base we have to worry about.

LOL. Well that would certainly help, wouldn't it? :) I filed #1298340: Add a hook_requirements() to check for existence of Flot library so the next person doesn't make that mistake.

Looking much better now! http://usage-graphs-drupal.redesign.devdrupal.org/project/usage/views

However, still not getting any output at http://usage-graphs-drupal.redesign.devdrupal.org/project/usage/drupal. Hrm.

Moved over to http://webchick-drupal.redesign.devdrupal.org/project/usage/drupal to consolidate on disk space.

And I do now see output at http://webchick-drupal.redesign.devdrupal.org/project/usage/drupal. Marking this back to needs review.

Also, updated the issue summary with what I think is the latest status.

http://webchick-drupal.redesign.devdrupal.org/sites/all/modules/flot/flo... is present, but http://webchick-drupal.redesign.devdrupal.org/project/usage/drupal still shows nothing.

From reading the comments, it seems like the current google chart being broken is more recent? http://drupal.org/project/usage/drupal

It would be nice if this could be completed, however it doesn#t seem to work on the testsite. No idea why.

The remaining is just a minor caching issue: the module caches the HTML, so the Javascript is not inserted when we pull from the cache.

We would be in Drupal 7, I would suggest #attached... but in Drupal 6 I will let the brave people in that thread find a better solution.

New patch, that adds flot_add_js() to make sure cached requests get the right scripts.

Updating issue summary.

The last patch was still broken, because theme_flot_graph adds the data via drupal_add_js() as well. I added a workaround that grabs the data and prepends it to the report output. This is a little ugly, because when the cache misses the data will be included in the page twice (once in head and once inline). However it's a simple assignment and so occasionally including the data twice is harmless.

New patch, new demo site:
http://drupal-org-428680.metaltoad.com/project/usage/drupal

isn't it a bug in flot that the module can't properly handle the caching layer? if so, it would be nice to fix the problem there instead of ugly hacks here.

Yes, I think the flot module is wrong to add the data to the document head. Flot.module also lacks a 7.x release, so we should rethink this (even though the broken usage graph drives me BANANAS!)

It might be smarter to use the flot JS library directly; I'm not sure the Drupal wrapper module gains us much. I might be able to work on this at the D7 upgrade sprint in April.

A fresh Git checkout of project module with this patch applied is at http://usage-drupal.redesign.devdrupal.org/project/usage/drupal. Looks like it works. Personally, I don't like the shadow.

We already depend on flot for http://drupal.org/metrics, switching to something else entirely is for another issue.

For a whole pluggable system, I don't think that is needed right now. We could make it a soft requirement, if oyu have flot, you get graphs. If you don't have flot, that's fine, no graphs.

Sounds like we want this then!

i'm not entirely comfortable with the hack introduced in #38. this line of code in particular concerns me:

$inline = $scripts['inline'][count($scripts['inline']) - 1]['code'];

that doesn't seem like a very reliable way to access that data. ;) if this was in drupalorg module i could live w/ it as a d.o specific hack, but this is going in what is supposed to be the more generally useful project module.

i've contacted the most recent maintainers of flot to see if they are able to help address the caching bug.

@hunmonk: i got your email ;p

I became co-maintainer of flot because we developed the D7 version, the hack in #44 looks ugly, i'll try to have a look at it tomorrow/friday.

This looks ugly, but personally i would not make it a blocker to get this RTBC'ed, for the D7 version we use the same so we will have a look at both D6 and D7 version (D7 needs some more cleaning to split it into sub modules), task created #1515468: Flot D7 code cleaning

+  // Include the cached inline code in the output.  Ugly, but there's no harm in assigning the JS variable twice.
+  $scripts = drupal_add_js();
+  $inline = $scripts['inline'][count($scripts['inline']) - 1]['code'];
+  $flot_graph .= "<script>$inline</script>";
+  return $flot_graph;

I had a closer look, the problem is caused by the caching inside usage/includes/pages.inc since it caches the HTML output, I adapted the code so it caches the data, not the HTML.

I personally prefer the javascript added to head/footer because it keeps the HTML cleaner.

I posted a different patch for flot at #1515468-1: Flot D7 code cleaning, in order to not sidetrack this issue too much I'll explain more over there.

I agree that #48 addresses the caching issue. I've updated my pseudo-sandbox as well:
http://drupal-org-428680.metaltoad.com/project/usage/drupal

http://usage-drupal.redesign.devdrupal.org/project/usage/drupal is updated again.

Attached in an updated patch which:

• Cleans up chart junk, no border or shadow
• Only uses flot if the module is installed, not requiring it

Graph looks better and making flot optional is working, leaving at RTBC

What is missing to deploy this change? I discovered that right now there's no Usage graphic for Drupal core at all (http://drupal.org/project/usage/drupal), so why don't we push this one, or am I missing something ?

Note: Turns out I had a bug in the stats rewrite where it wasn't discarding the oldest data every week. project/usage/drupal is now working again.

I fixed this bug when rewriting the stats processor as drush commands, and as of today the graph is showing up again for drupal.

We still need to get this in, though.

Committed & deploying.

Deployed.

Yayyyyy!!! Thanks, Neil!

Yay, thanks! Although, what were you doing working on May Day? ;)