We, the community and maintainers of d.o., need feedback on content and comments because there's old, outdated, irrelevant and spammy content on d.o. that needs to be reported and fixed.

Rather than let it rot and clog up the path to people finding useful content, let's let the community identify this for us.

Randy's original:
A simple flag or Plus1 on content and comments of "Was this useful to you" could go a long way for us. Every e-commerce site in the world uses this type of technique. We can do this.

CommentFileSizeAuthor
#34 problems-1.png138.45 KBlisarex
#15 feedbackadmin.png60.82 KBlisarex
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

lisarex’s picture

Yes, exposing a mechanism to allow any authenticated user to help us identify the good from that bad would be awesome.

On the content audit, we have several flags in use: Redundant, Inaccurate, Out of date, Needs style review, Needs technical review, Needs URL alias, Remove, Incomplete, Needs graphic and No work needed

Not saying we should use those, but we should consider a way for the community identify what "Needs work" as well, so we can prioritize it for fixing. A mechanism for adding more detail would be great too, so we aren't guessing why something didn't meet their needs. Something like this could work to allow folks to file an issue against a page: #810508: Kill handbook comments, but it's not implemented yet.

But a single flag is better than no flag :) And docs team might want this too.

davidhernandez’s picture

+1 flags. I definitely agree with a need to prioritize the doc changes and audits. We've discovered some 1200 pages in the handbooks, and this isn't even everything, and doesn't include doc pages. We definitely need some metric to prioritize work. This would also help alert people to problems before they swell.

Would we want positive flags, or negative flags? By that, I mean is it helpful to know the pages people like, or just the ones people don't like. Would the flags be reset every time a change is made?

Bojhan’s picture

Following, I think its important to set a goal here - who are we targeting this functionality at?

rfay’s picture

This is targeted at *the content*, not a who. We'd like every user to provide feedback on every piece of content. That gives us the leverage to improve both search and presentation.

So:
What content should have this? All content and comments
What users should use this? All users
What does it really do? Gives us the capability to differentiate between useful and unuseful content and present it differently in many ways.

Bojhan’s picture

I do not see how "content", can be the audience. I am asking this question because it allows us to set boundaries and focus the functionality, this will help us significantly in making something useful.

From my understanding the usecase for this functionality is to determine whether content is helpful or not. The flags proposed by #lisarex seem to be more deeper, for people to specificy "why" its not helpful. Do we want this deeper layer of functionality?

It's likely that not all people are able to answer this "why" question, because people that are still at the beginning of the learning curve are berry-picking the information to build a conceptual model. Where as very experienced users, might be able to give a very detailed description to what needs fixing. Do we want to accommodate both, or just one? How will it help the Doc team, just another metric or can we group helpfulness per book, child etc?

Additionally, it would be usefull if we have some resources on this. After doing a quick browse I saw a lot of diffrent interface patterns, from the yes/no to the more social aspect of "2 of 3 people found this answer helpful. Did you".

rfay’s picture

I'm *all* for a fancier thing. I'd just like us to do *something*. And then use the results.

lisarex’s picture

Title: Allow flagging content and comments with "Was this useful to you" » Allow flagging content and comments to determine quality

The goal is to crowd-source the 'auditing' of drupal.org since a small handful of volunteers isn't sufficient. Plus, content audits should be ongoing anyway.

Audience

A lot of folks see problems on pages but are intimidated to actually edit a page, don't know how to file an issue, or can't be bothered. Then they are frustrated.
Given them an easy feedback mechanism would empower them to note the problem without a lot of extra hassle.

Info to capture on nodes

I think this should be at the bottom of every *node*, above the comments (if enabled)

Was this helpful? O Yes O No
Feedback (optional)
_____________________________________
| |
| |
_____________________________________

http://www.ixda.org/node/24101 was a good reference

Then we'd have a view with all node titles that had been marked as not helpful, along w/ the comment. This would be so awesome for determining where d.o. content is lacking.

Flags on comments

I don't know exactly how'd we should handle those. I think that's a separate issue actually, because either we want the good comments to float up, or we want to hide / minimize the ones deemed 'bad'

davidhernandez’s picture

Why do we need flags on comments? Isn't one of the goals to get rid of comments from docs? Or is that only some docs?

I think, ideally, the flags should be as simple as possible. I don't know that a comment box is necessary. If someone is too intimidated to post a comment, they probably aren't going to give full feedback, but anyone can click a flag. At the very least, the docs team can get a report of the flagged content, with the most flagged at the top. Something that is heavily flagged as unhelpful probably has an obvious reason why. (Maybe I assume too much.) I will say, after doing much d.o auditing, most of the problems are obvious. The problem is they are usually pages that are buried down a tree somewhere. Only someone really seeking out that content would be likely to find it. We just need a notification mechanism.

lisarex’s picture

@davidhernadez, you say Docs but I really want a solution for all content nodes on d.o. The Docs team is not responsible for content in the About section or in Community Initiatives. That's why we attempted auditing those first.

The problem with just clicking a 'low quality' flag is that it's not actionable for webmasters. So even if 10% of people filled in feedback, that would be much better than what we have now (rotting, festering content that has hidden spam links).

How could we implement a simple radio + feedback box?

davidhernandez’s picture

My comments are general. I'm sure anything can be applied to guides as well as "Docs". I hate that they are considered separate, but I'll save that for #drupal-vent.

It is not clear to me why a flag is not actionable? Are you saying webmasters won't know what the problem is, or that they won't know that there is a problem? A flag clearly alerts of a potential problem.

rfay’s picture

There are lots and lots of comments besides on docs. Every issue queue, forums, lots of places.

lisarex’s picture

Yes, I'm saying that a "Needs work" flag is not always sufficient; we'd need several that are more specific like "Needs technical review", and even then, if the person who found the problem could just note down what it is, it would make getting it fixed much quicker.

I think if I were faced with a wall of node titles that were marked "needs work", it would be horribly dispiriting. More info is needed.

That said, if we were willing to use multiple flags, it might be easier for volunteers to review nodes that need reviewing, because everyone has their specialty (style review vs technical review vs. whatever)... but might look very cluttered.

At this point though, just having a single "needs work" flag would be a good place to start, if a 'feedback' field is too ambitious.

Bojhan’s picture

I dont know if a feedback field is to ambitious, it should ideally only show once the user selected "this is not useful". I am afraid that more indepth "why" options, will not help the user - because its hard to understand what they mean for someone who doesn't know d.o that well. It would be much easier to tell "what" they see is wrong, without needing them to classify it.

davidhernandez’s picture

lisarex’s picture

Status: Active » Needs review
FileSize
60.82 KB

Yes, overall I think the Feedback module could be a great "better than nothing" solution. There is currently no Views integration for 6.x, but there will be for 7.x, but the admin screen seems sufficient for now. The admin screen allows sorting by table heading, so we can see if some nodes are getting multiple complaints.

The only other issues are we might want to customize the feedback text, but it is not something I would want to hold us up ("If you experience a bug or would like to see an addition on the current page, feel free to leave us a message")

Infrastructure people, want to weigh in on whether this would be a good fit for the d.o. infra?

lisarex’s picture

I should also add, it doesn't allow people to rate/rank whether content is good or not. It is simply a feedback mechanism. Therefore it is creating work for more volunteers, but the result is we can focus on improving the content that people are looking at.

An automated ranking system that lets good content float to the top also seems good, but I think having actionable feedback sounds great.

lisarex’s picture

Another option: add a link at the bottom of nodes to allow people to create an issue re: content problems. Fields should be prefilled as much as possible, and it could go into the Webmaster queue or a separate content-only queue.

rfay’s picture

#17: I have no problem at all with making a way to create an issue. Perhaps then people would at least *do* something about bad content.

I'd like to see us just lose thousands of pages that get poor ratings though... (Never marked as helpful, or whatever)

davidhernandez’s picture

I also like #17, and think it going to its own queue is a good idea, too. Any other opinions on that?

My only concern is that a lot of people may not feel like going through the effort to create an issue. Maybe that's ok. If it isn't worth the effort, it probably isn't a real problem. On the other hand, people probably won't bother creating an issue for something like "wasn't helpful," which is something we want to know about.

jhodgdon’s picture

RE #17 - We're discussing this idea of having a "file an issue" link elsewhere:
#995292: Noderef field on issues for Documentation project

Once that is done, we'd like to completely get rid of comments on doc pages, and instead have a related issues block.
#810508: Kill handbook comments

So probably we can just keep that discussion on that other issue?

jhodgdon’s picture

Also, just as a note: We tried to use the Flag module on drupal.org for comments on issues, for a completely different purpose (issue summaries, which are about to be deployed using a different mechanism anyway).

But the use of Flag on comments was rejected by the infrastructure team, because it is currently *very* inefficient about its queries. That is being worked on, but just be aware that you'll need to get that fixed before you can use Flag on d.o. Here's the issue:
#1133956: Improve efficiency when displaying lists of flaggable comments

killes@www.drop.org’s picture

the flag performance issue is in the process of being resolved.

lisarex’s picture

Is it desirable for non-docs pages to have the nodereference solution?
e.g.
http://drupal.org/about/new-in-drupal-7
or http://drupal.org/node/27367
or http://drupal.org/about/accessibility

At this point, I say "go with one solution" because implementing two is just nuts ;) Happy to let this issue focus on Flag then (and +1 to rfay's comment in #18!)

killes@www.drop.org’s picture

yes, re-use is golden.

jhodgdon’s picture

So... For Docs pages, we don't want to have comments (eventually), and we want to encourage people to file issues.

IMO, having someone flag content as 'not all that useful' is not very helpful, because it doesn't tell us why or how it can be improved. If they are willing to take the time to file an issue, they'll put more details in (presumably, or at least we have a format where we can ask for more information).

So, what's the purpose of this whole flag idea, really? If it's for doc mantainers, it seems to be useless.

davidhernandez’s picture

Issues are great for problems like typos, incorrectness, needs updating because of major Drupal version change, etc; issues that are specific. But, it is harder (by that, I mean less likely for someone to create an issue) if the content is just plain not helpful. The handbooks have a lot of pages that are technically and stylistically correct, but are just plain "blah". It can be hard for us to determine which ones are unhelpful without people telling us, and I don't see people going through the effort to file an issue just to say, "I think this page is meh."

This is less of a concern for docs, because if any small number of people find it helpful, it is probably worth keeping. If not many people find the Getting Started guide helpful, we want to do something about that.

Also, consider that a lot of the people going through handbook pages are likely to be less experienced users, and thus, less likely to be comfortable creating an issue. Clicking a flag is easy (or some solution that is simpler than creating an issue,) and provides a way to get general feedback.

All that said, I'm in favor in multi-purposing whatever solution is being implemented for other things. Any feedback mechanism is better than none.

+1 to killing comments.

jhodgdon’s picture

Really? I would think an issue would be the perfect way for someone to flag content as wrong or useless, because rather than a simple thumbs up/down, they can explain why.

The problem I see with flags is that we don't know what is going through the person's head. So let's say we have a page that has been marked "unhelpful" -- what do we do about it? How do we know how to fix it? If they have a link that says "Please file an issue if there is something wrong with this page", and it fills out most of the issue for them (like the links on api.drupal.org do, leaving a space that says "Please describe the problem"), then we get an issue report that at least tells us why they think the page is garbage.

davidhernandez’s picture

The use case for a flag is not necessarily for specifics, or individual people to relay problems. It is more of a voting substitute. If one person finds a page unhelpful, we probably don't care. If 200 people find it unhelpful, we probably care, and don't need 200 issues created.

rfay’s picture

IMO, we need some kind of way to distinguish good content from bad. Issues are great. But probably up to 50% of pages on d.o should just be removed. 10% can be improved with issues. Let's figure out how to do the simple case. Most people won't do an issue. Many people, if encouraged, might at least rate the quality of a page.

jhodgdon’s picture

Flag will only allow us a binary choice though, not a rating, right?

davidhernandez’s picture

Yes, binary. I think that is fine. I remember reading a study a friend sent me (it might have been about when Youtube switched from a 5-star rating system to thumbs up/down) that concluded most non-binary rating systems don't provide appreciably different results than a binary system, because most respondents just choose the highest or lowest values.

arianek’s picture

Not much to add other than I think we should definitely focus on killing docs comments, and that when that's done, a vote up/down is a lot more visually useful than the feedback (which should really be an issue as noted above).

The thing with feedback is like jhodgdon said, it doesn't tell us what's needed.

Additionally, and I think a more critical issue: who is supposed to be monitoring this and how? Whenever we add another place for feedback, it means someone needs to be monitoring it so like comments it doesn't just end up floating into the ether. Getting stats is one thing, but acting on them is another (and more important?)

lisarex’s picture

Agreeing we should kill handbook comments/ get issue reporting in place first. That issue has been around a lot longer, as well.

@ariane, how about we add columns for related issues and flags to this view? http://drupal.org/documentation/manage ?
(And remove or reposition the copyright block) It seems like, for now, there's no separation of the different books, only who is looking after them....

We can add something to the header that books a, b, c are Documentation and x, y, z are Communications (or whatever we're calling it these days)

lisarex’s picture

FileSize
138.45 KB

As for this issue, are the two flags "Yes, helpful" and "No, not helpful"?

Also, let's give some thought about where these should appear. My preference is that they're prominent, so they get noticed, therefore used. See attached. If we don't want it above the book nav in the right column, perhaps right below?

davidhernandez’s picture

I agree that if there is a flag it should be prominently displayed, like where Lisa has it. Don't bury it at the bottom of the page.

As far as monitoring goes, is there to be a separation between docs and non-docs? Will docs people be annoyed if non-doc pages should up in their lists? I volunteer to help monitor stats/issues, but there should definitely be a group of people involved. Would this fall outside the purview of the docs team? If so, it will need a different group of volunteers and a separate view.

BTW, do we have an official name for non-doc pages, and official "separation of powers"? This is bugging me. They seem to get the proverbial red-headed stepchild treatment.

jhodgdon’s picture

I think that both doc and non-doc Book pages show up in the management view
http://drupal.org/documentation/manage
It's all about which book they are part of -- most of the books are officially Doc, and there is I think one that isn't (is that correct?).

So... The question about monitoring is not so much whether we can find pages that have a lot of "not helpful" votes (as lisarex said, we can add that to the View as a column and/or filter), but what do we do about it. We are already overwhelmed in the doc team with the issues that are comprehensible and specified and we know how to fix. How do we fix a "not helpful" page, and where are we going to find a new army of docs people to visit the management page, find "not helpful" pages, evaluate them to figure out what should be done, and either do it or file a proper issue?

IMO, giving people the ability to vote up/down on a page is not good, if in reality we don't have a plan for what to do with that information.

lisarex’s picture

Bottom line, we are just guessing about how the community sees content on d.o. and how we think they'd respond (though, we need to be more persuasive in the UI for them to report problems, so that's why these issues are so important.

I think organizing a survey and/or usability test would go a long way in giving us insight. This has also turned into quite a long discussion :)

Jennifer, there's actually quite a few identified non-Documentation books: http://drupal.org/node/1168704, but I may have misorganized them.

What do we do about monitoring the non-Doc books? Do we need a separate dedicated group of volunteers? But it's unclear how to organize ourselves separately from the docs team, because apart from the content, the processes are and will be nearly identical.

arianek’s picture

@lisarex that all seems fine to me - would be great if there was some way to id at a glance (if it's on the manage page) just how many votes have been negative so that it's easy to ID ones worth looking at. i don't think many people use the manage view except at sprints, etc. and as far as non-docs book content... maybe there should just be a designated maintainer(s) that *are* technically part of docs team, but liaise with the DA, etc. on what should be going on with the content there?

silverwing’s picture

Quick note re: the screenshot in #34 - some pages' sidebars get very long (ex. http://drupal.org/node/953096) and the flags definitively wont get seen. Perhaps getting it to the top of the sidebar would make sure it's seen (of course, then there may be more abuse.)

jhodgdon’s picture

Why in the sidebar at all? If you want someone to notice it after they've read the page, maybe put it at the bottom of the content (which is conveniently where Flag will want to put it anyway)?

davidhernandez’s picture

There is a concern that it will be lost on longer pages, especially if someone does not find a page helpful, and thus, stops reading it. It's a UI thing that I'm sure can be argued until we are blue in the face. Another alternative could be at the top in the yellow box that shows the tags and update info. I have no personal preference.

Bojhan’s picture

Lets stop discussing the UI, and discuss the reasons for the actual functionality - and what kind of feedback it is that we need.

lisarex’s picture

Bojhan, I explain the problem in the revised body.

#226678: Add a "Report spam/abuse" link to forum/issue comments (next to the "edit" & "reply" links). is also begging for a 'report spam' link.

If we had a "Report a problem" link it could either go to a page listing the different places to report a problem (Docs for content, Webmaster for comment spam ---ugh) or ideally it is just a simplified issue creation form that has a select field that describes the problem, and depending what they choose, directs the issue to the right issue queue.

jhodgdon’s picture

OK, you've identified two different issues:

a) Knowing the quality of comments... In the Docs team, we have pretty much decided we don't want comments on doc content at all, so we don't actually want the quality of comments rated. We want to get rid of comments, and plan to when we get the infrastructure in place to have people report issues instead, and associate issues with doc pages through a node reference (so that we can make a "related issues" block on doc pages). See issues:
#810508: Kill handbook comments
being held up by
#995292: Noderef field on issues for Documentation project

b) Knowing the quality of doc pages... As I've stated previously above in this issue, we really need the quality problems to become issues, in order to be able to address them and really understand what the problem is. I personally don't believe that having people vote up/down on "was this helpful" will be very useful, because the problem could be that the doc is of low quality, or it could be that they didn't get to the right page (which could also be a problem that we can address through a better title, or it could be that they didn't search/navigate in a fruitful way, or that the doc they need doesn't exist).

You didn't mention:

c) People looking for documentation having a crowd-sourced method for knowing whether the page was deemed helpful or not helpful.

I'm not sure whether this is part of the objective or not...

So. IMO...

1) We want to get rid of the ability to comment on content, and gradually also get rid of the existing comments. So let's not worry about rating the quality of comments on doc content at this time.

2) The solution we find for "letting the community help us identify content quality" needs to:
- Allow the docs team to understand what the problem really was, if there is a problem. For instance, giving them some choices, like "Wasn't what I was looking for", "Innacurate", "Didn't fully answer my question", etc.... rather than just "helpful/not helpful".
- Enable them to add more details and notes to their feedback easily.
- For the docs team, it would be preferable if the more detailed feedback method meant filing an issue, if this can be made easy to understand (usability test with an issue link that includes an automatic title and body??)

ericduran’s picture

subscribing to this issue. On purpose ;)

lisarex’s picture

Title: Allow flagging content and comments to determine quality » Allow flagging content to determine quality

@ jhodgdon, I'm changing the title to remove comments (and that can be discussed in a separate issue). Otherwise I think we're all on the same page now: a detailed "bug report" from the visitor is converted into an issue. As for c) above, that seems like a nice to have. C) was the original intention of this issue, I believe....

Perhaps we should leave this issue for the 'flagging' and create a new issue for 'report a problem > issue get filed.

jhodgdon’s picture

Title: Allow flagging content to determine quality » Allow marking content to determine quality

I don't think a simple on-off flag is going to satisfy the requirement of understanding what problem the user had with the page (see comment #44, item 2 at very bottom). I think at a minimum we would need several choices, not just "was this helpful".

klonos’s picture

jhodgdon’s picture

Status: Needs review » Closed (duplicate)

Hm.

Actually, at this point I think this is a duplicate of
#1300640: Add moderation flag to book pages
that other issue is more updated with what we currently want to happen, as part of
#1278256: Develop a plan to make it more clear that the current Documentation on drupal.org is community maintained.
Can we just close this issue?

jhodgdon’s picture

Issue summary: View changes

Clarify problem