Provide API to get the top N projects for various categories

Things are stored in {project_usage_week_project} by project nid, week-timestamp, and tid (of core version).

Summing on count and grouping by nid should suffice for getting project popularity. Haven't thought about the speed implications of this. It may be manageable if it's cached somewhat.

Timestamp:

module_load_include('inc', 'project_usage', 'includes/date_api');
$timestamp = project_usage_get_current_active_week();

mysql> describe select sum(count) as cnt, nid from project_usage_week_project where timestamp = 1260662400 group by nid order by cnt desc limit 5;
+----+-------------+----------------------------+-------+---------------+---------+---------+------+--------+----------------------------------------------+
| id | select_type | table                      | type  | possible_keys | key     | key_len | ref  | rows   | Extra                                        |
+----+-------------+----------------------------+-------+---------------+---------+---------+------+--------+----------------------------------------------+
|  1 | SIMPLE      | project_usage_week_project | index | NULL          | PRIMARY | 12      | NULL | 353591 | Using where; Using temporary; Using filesort |
+----+-------------+----------------------------+-------+---------------+---------+---------+------+--------+----------------------------------------------+
1 row in set (0.00 sec)

mysql> select sum(count) as cnt, nid from project_usage_week_project where timestamp = 1260662400 group by nid order by cnt desc limit 5;
+--------+--------+
| cnt    | nid    |
+--------+--------+
| 246598 |   3060 |
| 157231 |  38878 |
| 146942 |  48429 |
| 139175 | 106016 |
| 111770 |  17345 |
+--------+--------+
5 rows in set (0.13 sec)

Those need to go into the solr index. We don't actually need Views.

Also: usage statistics are already in the solr index. So those are already covered.

@Damien Tournoud: Can the implementers make use of the solr index of usage statistics (via some sort of query protocol?) to do things like:

"show an html unordered list of the 5 most popular modules"

"show an html unordered list of the 3 most popular translations"

"show an html unordered list of the 3 most popular installation profiles"

If so, then also, some examples on how to do this (or where to look for example code) would be really helpful too.

@jrguitar21: the first step is to get all that information into the solr index. Let's concentrate on this for now.

* "Most Installed" is already in solr,
* "New modules" is just a fancy name for node.created,
* "Top Rated" and "Most reviewed" need to be clarified (aren't those basically the same?),
* "Most Downloaded" can be derived from the download log,
* "Most Active" needs to be specified

The usage summary data are in relatively small tables that are well indexed. I'm not sure we necessarily want to use solr for the original request here. Seems like a relatively trivial patch to project_usage to expose these tables to views. I don't see what we gain by doing this via solr vs. views. @DamZ: care to elaborate why a custom solr-based solution that very few people understand is better than using views which basically all of the redesign implementors are/should be familiar with?

Also, as of comment #6, this issue is turning into a giant meta rathole that's rehashing discussions like http://groups.drupal.org/node/7191. Are we sure we want to go there? ;) Why don't we actually accomplish the specific task at hand? *shrug*

Cheers,
-Derek

As one of the implementation firms, it's going to be pretty challenging for us to implement this as a solr index. We haven't done it before, and we don't know the project module or project_usage well enough to handle this quickly/easily on our own.

Consider this a vote for exposing this stuff to views, or, if we go the solr index route, a guide on how to implement it for the redesign as so many of us need this data for our sections.

@dww: those metrics are used to power the facet-based project browsing.

The project browsing starts with the download landing page:

https://infrastructure.drupal.org/drupal.org-style-guide/prototype/downl...

And it continues at modules and themes:

https://infrastructure.drupal.org/drupal.org-style-guide/prototype/modul...
https://infrastructure.drupal.org/drupal.org-style-guide/prototype/theme...

The blocks on the right of those pages are facets and/or ordering of the current query.

I don't understand why we should do twice the work: once to design the download page and once for the browsing facets/ordering. Once we will have supplied those facets/ordering to solr, building the download page will be trivial.

This issue is related to #372243: Expose project statistics to the solr index, just linking them together here for historical purposes.

The identical effort was started at #372243: Expose project statistics to the solr index and it's pointless to have two issues about the same thing. Let's consolidate on the older issue.

Actually, upon further thought, let's repurpose this issue for the other aspect of this effort... building blocks for the top N projects in various categories like "most used", "most downloaded", whatever. We'll use #372243 for making sure everything we care about is in the solr document. This issue will be about adding some API calls in project_solr for returning these lists of the top N hits for a given constraint, and code to expose these lists as blocks that can be stuffed into the main download and extend landing page (and other places in the project browsing pages in the redesign). In many (most?) cases, this code doesn't belong in drupalorg but in project_release or project_solr.

There is a start of a module that does this on #862240: Provide additional solr facets for advanced project browsing. However that module currently duplicates part of what is in HEAD of drupalorg.module (mainly the menu callback, preprocess, and the tpl.php). Overall the version in drupalorg.module seems to be a bit more up to date and more central, and I'd recommend keeping that template and page callback there, and moving the solr related bits into project_solr, as an API described in this issue, that can be consumed by drupalorg.module.

Here's a simple version that does this sort of code, based loosely on project_solr_browse_page().

In order to make this a more strict API, no code in this function checks any URL paramaters, and no path is passed to construct ProjectSolrQuery.

With some additional refinement, it may be possible for project_solr_browse_page to become dependent on this function.

I removed some paging gunk that was left in, as well as an un-needed field alias.

adding redesign tags

the "... and expose these lists as blocks" part is actually already provided by drupalorg_order_facet.module, although that should be in apachesolr: #868794: Add an apachesolr_result_block module to render solr facet results as blocks.

Given that, it's not clear we actually care about this API function at all. We can probably do everything we need with result blocks. It seems like an API for returning these lists is perhaps duplicate effort. So, likely this issue should be "won't fix", but I'll leave it open for now...

We've solved this entirely via Solr, so I don't think we want an API like this anymore. We've got something similar enough via #868794 (which is currently functionality in drupalorg_search but it should hopefully move into apachesolr).