Add user ratings for projects
| Project: | Drupal.org infrastructure |
| Component: | Drupal.org module |
| Category: | feature request |
| Priority: | normal |
| Assigned: | eaton |
| Status: | active |
| Issue tags: | drupal.org redesign |
There's been a lot of discussion about rating projects, so I took a quick stab at adding support to Project.module. It's built on top of VotingAPI, which may be a philosophical issue for some, but it sped things up for me. If VotingAPI isn't installed, the voting options just won't show up.
Voting can be turned on and off on a project-by-project basis in the same fashion as the issue tracker. If multiple categories have been entered into the Issue Tracker 'categories' field, they'll also show up as separate criteria to rate the project on.
Initial results of the patched moduel slapped onto one of my sites can be seen at http://www.talkonomy.com/node/2. The theming of the voting widget and the voting results could use some work, but it's functional and gives us some direction. Any thoughts, feedback, improvements, etc are welcome.
| Attachment | Size |
|---|---|
| project_voting.patch | 8.77 KB |
#1
The theming function for displaying vote results also points to the images in the 'voting' directory. Attached is a zipped version of the dir/images.
#2
Jeff, thanks for doing this. Nothing like a concrete example after lots of talk.
Want to take a crack at adding a browsing option, so that we can browse by rating (as well as category, date, and name)? This is done through the 'project_sort_methods' hook. The most straightforward way is to modify
project_project_sort_methods()to add your method. (The alternative is to implement a function in your own module.)Begin by declaring your new method:
<?phpcase 'methods':
- return array('category' => 'project', 'date' => 'project', 'name' => 'project');
+ return array('category' => 'project', 'date' => 'project', 'name' => 'project', 'rating' => 'project');
?>
For
case 'sql_settings' :, put the pieces (joins, fields to select, sort order, etc.) you'll need to add to the sql select statement. This works a bit like views.module (which lamentably I hadn't yet looked at when I wrote this into project.module).You'll also want to put
<?php// Does this method page results?
case 'pager':
switch ($method) {
case 'category':
case 'date':
+ case 'rating':
return TRUE;
case 'name':
return FALSE;
}
?>
And that should do it. Email me if you're digging into this and have questions.
#3
Yep, that's next on my agenda. It doesn't look like it will take too much doing. Projects will only be browsable by the 'overall' rating, as each project might have different sub-criteria.
#4
subscribe - i am toying with this. just for fun, for now. subscribe.
#5
Bump: http://drupal.org/node/77976 is now marked duplicate with this -- it contains some discussion that might be useful. There's a lot of interest in this these days, so it'd be nice to come up with something...
#6
Subscribing and highly interested in seeing this come to reality soon.
See also related: http://drupal.org/node/166471
#7
As much as I would love to see something to improve the quality of some modules, I also see comments about rating that very much trouble me. "I want to download the 10 most highly used modules." ... "I want to use the 10 most highly rated modules." ... etc. Those statements are totally ludicrous! And yet I see them frequently as reasons to have a rating system or download count. These are people who want to choose a solution and then go looking for a problem - the tail wagging the dog.
People need to choose a module that solves a problem they are having or a function they need.
Yes, there are many modules with exceeding poor documentation (I hope none of mine are considered as such). And there are poorly written modules out there (obviously not mine). So how are we to encourage the developers to improve their product?
First, we need to make sure users are aware that they can - and should - post feature and support requests to improve the documentation and functionality. When some of us who are familiar with Drupal coding choose someone else's module, we should be welcome to offer positive suggestions on how to improve them.
Second, perhaps all projects ought to come with a pre-built queue that is for user comments on the module.
These two ideas allow a developer to fix some of the problems and actually look the better for doing it. A rating system will leave (despite the ability to change a vote) bad ratings around even long after the problems that caused them have been addressed and solved. Many people who give poor ratings will never come back, hence they will never change the rating.
#8
Nancy, I agree. Any rating system for drupal modules would need to take into account the nature of the system. Possibly, an 'overall' rating and a per-branch rating. All ratings would be cast on a per-release basis, while the 'overall' would be averaged over releases. That would give modules a chance to 'make up' for a bad release.
That's just one solution, though. I think gathering useful statistics about issue queue traffic, developer activity, and so on are all just as critical.
#9
I've posted this already on the thread about drupal 6 beta 2 and was pointed here because it was off topic there:
I won't pick a module just because it's one of the TOP TEN - but sure I would take the time to look at it and maybe play around with it (on a local test site, of course).
That's one way of learning drupal! And not "the tail wagging the dog". ;-)
There are hundreds of modules for drupal 5 and I think knowing which of them are the "most wanted" will be a good way to discover the power of drupal. As a new user I'm just learning what can be done with drupal, but I find it rather hard (and time-consuming) to dig through the overwhelming crowd of modules to discover wich possibilities lie hidden in there.
And then there are modules which (seem) to have similiar features. Ratings and download rates might be misleading sometimes, but they can't be worse than the current alphabetical order. At least the rates give you a hint which module to try first.
To guide new users through the "modules jungle" there should be:
All ratings should be bound to one version, otherwise they won't make much sense.
#10
Ooops - I changed the issue title inadvertently. :-O
#11
One technique that I haven't seen mentioned elsewhere is that the advanced search can be limited to check project (module) pages.
#12
@nancyw: See http://drupal.org/node/168819
#13
Until that is implemented, one may still limit one's search the way I suggested.
#14
Ernie,
That is a good proposal as far as I am concerned (a good first step that could be done) and there are interesting similarities to my proposal which goes further. Please check it out here: http://drupal.org/node/166471
Nancyw,
I like the idea of having a queue for user comments on the module. Let's say recent comments.
You are right in not encouraging a rating system just like a voting system. I don't think the users would like just that.
This needs more consistency in addressing the real issues like letting the users qualify things in some more details.
An idea, would be to rate the different points on a shorter span of time, like the last year (or 3 months) only, instead of since the very beginning. This would reflect more fully on the "actual" work being done by the "active" developers.
In any case, from a user's standpoint, something should be done about a ratings system fairly soon.
Others do it and I think it is really needed for Drupal.
And it would further encourage "the" developers that are doing an already great job.
Like Ernie also said, to guide new users through the "modules jungle". It is a real jungle and it would be a whole lot better to guide the new users to those quality modules instead of the contrary.
You don't know until you tried... but maybe hundreds have tried before you. Let's get their opinions...
In the end, this would certainly increase (and somewhat insure) the overall satisfaction in using Drupal.
My humble opinion.
#15
Indeed, I'd love to see something that rates the quality of certain aspects of a module. I'd love to think I'd rate fairly well. But I am still quite concerned about "the tail wagging the dog."
"I have not failed, I have simply found 10,000 ways which do not work!"
— Thomas A. Edison
#16
Nancy,
Of course, anyone, implementing some quality measurements in their systems for the first time, is always quite concerned.
For some reasons, I have seen this reaction over and over in the industry... It's quite normal.
I've implemented such things... and it has to be done smoothly and the goals have to be well explained and documented for everyone.
But after the initial fears, everything settles in 99% of cases.
And remember that adjustments can and will be made on a regular basis.
If you, Dries, and a group of good people including developers and users oversee the implementation, why would "the tail wag the dog"...?
Going ahead intelligently is much better than not taking the appropriate steps to improve quality on the basis of some fears.
#17
There's one indicator of "quality" that I always look for, but I know from various forum posts that others don't, or don't understand. That is the issue queues.
Perhaps the system could create and display something like this:
This display shows how long issues have remained for this contribution.
Status Average Longest
---------- ------------- ---------------
active 1 week, 2days 8 weeks, 1 day
active, needs more info 3 days 8 weeks, 2 days
patch needs review 3 weeks, 4 days 16 weeks, 3 days
fixed 4 weeks, 5 days 32 weeks, 1 day
won't fix 1 week, 4 days 2 weeks, 5 days
closed 6 weeks, 6 days 34 weeks, 3 days
Note that "closed" status happens automatically two weeks after an issue has been marked with one of the resolution statuses ("fixed," "by design," etc.) and all activity ceases. This can "inflate" the average times and should be taken into account.
If these figures could also show "three months ago" that would be even better.
A few other things I'd like to see are: 1) in indicator (check box) for whether the module has been checked with the Coder module; 2) an indicator showing whether a translation template is available.
#18
@nancyw: see http://drupal.org/node/79550 for automated quality metrics and statistics abou thte issue queue. also, please check out the existing "statistics" pages for the issue queues for each project, for example, start here: http://drupal.org/project/issues/project then click on "statistics": http://drupal.org/project/issues/statistics/project. pretty similar to what you're talking about.
Seems I need to make another wiki on g.d.o about "all the issues about project quality stuff" so people don't splinter the conversation. this issue is about *user ratings*. if that's not what you're interested in discussing, please look elsewhere. thanks.
#19
#20
Thank you, Derek. I was unaware of the other discussion. While I knew about some (not all) of the statistics, I doubt seriously that 1% of Drupallers do, and even fewer ever check them.
If you wish, you may have my post deleted from this thread.
#21
Subscribing to this. Just quickly I think a multiple axis vote would just get abused (10,10,10,10,10 etc.), unless it was hard to do that.
Also I like the idea of votes being aggregated per release and across releases - some kind of degrading of votes across time is very, very important.
#22
Thank you dww for pointing me to this thread. This is a great initiative! Also, thanks to Eaton for getting it started and rolling a patch for it.
That said....gah!!!!!!!!
This poor issue has been languishing for too long, and has seen its fair share of sidetracking. Please let's get it back on track asap. :-)
Let's invigorate this issue and reach the successful, and implemented, conclusion that it so richly deserves:
A) almost *anything* would be better than what we have now
B) there will never be a 'perfect' rating system
C) it's not like things can't change once *something* is committed. This has been stalled long enough.
#23
See http://drupal.org/node/192410
Voting is known to have two problems. Cummulative low quality voting leads to low quality results. "I went to Drupal and the most popular modules were crap". Second, voting can be gamed. We've see voting and even reviews gamed on Joomla and Amazon's system.
But community feedback has it's place and I think we can evolve solutions for public review.
Please read: http://pivots.cms.si.umich.edu/sites/pivots.cms.si.umich.edu/files/pivot... and http://pivots.cms.si.umich.edu/sites/pivots.cms.si.umich.edu/files/Poste...
We will get the pivots up on scratch and see what people think.
#24
@caleb: Thanks for your post. Your comments are encouraging and are right to the point.
@amazon: I read your pivots doc and it is another very interesting road.
@all... Please note... I am most probably just an average user... but I love Drupal like many.
I am not a qualified team member of Drupal at all, but still as a serious day-to-day user/admin/integrator implementing the system for about every new customer, I just feel the urge to voice my concerns which might not have been fully addressed. Just hoping, in the end, some of my inputs can be useful to make Drupal even better. And I would love to participate and/or test things out if you ever needed. Do not hesitate to get an opinion from just an average user like me. Cheers.
#25
@jeanhr - anyone who is a is a more than a
All it takes is a willingness to help.
#26
Willingness there is a lot... so do not hesitate to "use" me ;-)
Yours truly.
#27
Hm. So, I guess effectively this thread is dead in favor of the issue that Amazon posted and is actively working on (see his comment above for more)?
All further discussions there?
#28
No, this thread isn't necessarily dead, but Amazon's comment #23 shows some of the potential pitfalls this thread faces. If you're interested in the pivot stuff, go to that other thread, but don't hijack it to say "I just want to add user ratings and reviews to projects!". ;)
Probably what we need is a whole friggin' group on g.d.o about this topic, and a lot of brainstorming, mockups, proposals, etc. (@Amazon: would the existing d.o redesign group be appropriate, or should we make a separate group just for this?) Currently, the discussion is splintered in too many places. There are a lot of opposing viewpoints on this topic, many different possible implementations (some of which are complementary to each other). There's a ton of interest, but very little concrete progress, since few of the proposals are clearly a step forward and have someone actively working on implementing them.
So, we need:
A) To collect all the possible proposals in 1 place (e.g. a g.d.o wiki page with links to the appropriate issue or g.d.o post for every proposal related to project quality indicators of all kinds). I can help populate this page once it exists, since I have a bunch of these proposals bookmarked or otherwise know how to find them.
B) Meta-discussion on the proposals: which ones are clearly a good idea now, which ones need more thought, which ones aren't going to work at all, etc.
C) Volunteers to work on the "low hanging fruit" -- the proposals that we can right now agree would be worth doing, but which won't take a huge amount of code to accomplish.
D) Further discussion on the specific proposals that need to be fine-tuned or otherwise better thought-out. I consider this particular proposal for end-user ratings and reviews (issue #50605) in this category. There are a lot of potential problems this will create, so we need to have good answers for those if we want to continue exploring this.
Therefore, I'm marking _this_ thread postponed, in favor of creating the g.d.o group about the broader topic of project quality indicators (or finding the appropriate existing group to use for the purpose), and the 4 steps I outlined above.
Cheers,
-Derek
#29
Heh, one person's hijacking is another person's way of trying not to get 'stuck in committee'. ;)
I can see that the route to solving the issue is obviously a bit more circuitous than I had first imagined. Fair enough. I defer to the project maintainer. :-)
#30
@calebg: Don't just defer... ;) Feel free to start the steps I outlined above. Your help would be great, and you're clearly inspired/motivated to do something -- I just don't have time to do all of this myself. Thanks.
#31
Dww, ok with your green light I guess I will try to scoot things along as I can. It would be good to hear from Amazon about this before a new group is made though:
#32
Starting a thread on module quality measures and recommendations as part of the Drupal.org redesign is appropriate. It's the most requested feature according to the "State of Drupal 2007" survey.
#33
On the "gaming" problem, I wrote some code that I need to contribute for votingapi. It is an integration between voting and buddylist. So, in the extreme only the scores of your buddies count toward the averages that you see. The formula can be tweaked so that buddy votes are x% of the score and overall is the other 100-x%. This basically gets rid of the gaming problem, since unknown voters won't affect the scores i see. Further, it answers the "what module should i use" question in precisely the way i want it answered. Namely, I want to use the modules that Earl, Eaton, Dries, Angie, dww, etc. are using. Those are my trusted sources. I'm sure others have theirs.
One small issue with this is that new drupal people don't have a trusted network. I think thats easily remedied by showing on the site which users have been befriended often. I think this is a smaller, solvable problem.
I was thinking of running this on my own domain for a while until drupal.org got its act together. But I haven't dedicated enough time.
Anyway, thoughts on this approach are welcome.
#34
Okey dokey. Thread created within D.O. Redesign Group. :-)
#35
Ok, thread updated with initial pile of links to all the stuff about this topic I've been trying to keep track of.
#36
Thanks, that's totally awesome. :-)
#37
http://drupal.org/node/271933 was duplicate. Moving this to infrastructure - IMO it's not a project.module issue, or not yet anyway.
#38
Derek, could you point to your work on project voting for this?
#39
That work is for project issue voting, not projects themselves: http://drupal.org/project/project_issue_voting
#40
Once rating is added to projects, it might be able to feed us quality data to generate better module recommendations via the pivots system. The outcome would probably look like recommendations on amazon.com.
#41
#42
Being an active issue for the drupal.org redesign, not marking this postponed.
#43
I've asked Daniel to write a blog post summarizing research on ratings systems and how the are gamed. If this is moving forward, let's make sure it's done with our eyes wide open the manipulation that is inevitable.
Kieran
#44
Kieran: we discussed the following metrics might be easy to game: downloads, usage, rating, reviews, alphabetic (although that is a bit harder, since we look at project names and vet 00aa00__ prefixes), activity (independent of whether we measure it by cvs commits or issues). These are basically *the* metrics we are going to have, and looks like each of them can be gamed in different ways. Looking forward to all kinds of ideas to make them non-gameable in their respective issues (look at at http://drupal.org/project/issues-term/799) :)
#45
There's a long discussion about all this at Project Quality Metrics on Drupal.org (meta document) -- please let's not duplicate that discussion here, but rather contribute to that if there's more to say.
#46
My advisor is one of the leading researchers in preventing gaming in recommender systems. His recent academic publications in this area include Manipulation-Resistant Reputation Systems, The Influence Limiter: Provably Manipulation-Resistant Recommender Systems, etc. The basic idea is to solicit honest ratings because it's the best strategy for individuals under a cleverly designed mechanism. I'll write a blog about it after I digest the algorithm :)
The manipulation-resistance algorithm and other cool features will be implemented in the Recommender API module as part of my research. The Recommender API module is at the heart of the next version of the pivots module. It's designed to support generic purpose collaborative filtering tasks. It is different from the content-based algorithm used in Solr. Hope the new Recommender API module (developing) will be a contribution to the community.
#47
@danithaca: the recommendations and the ratings for projects show up explicitly in the redesign, so we need to have some kind of metric for the user ratings as well, independently or digged from under the hood of the recommendations.
#48
@goba: Thanks for the suggestion. Actually recommender systems and reputation systems are quite related. I'm marking it down in my work note about mining reputation from users activity in the Recommender API module as well.