From May 17 - 19, 2011, in advance of DrupalCamp Twin Cities, several Drupal community members met at the University of Minnesota usability lab in Minneapolis to perform a round of formal usability testing on Drupal 7. This is the fourth major usability testing for Drupal, and the first targeting the new Drupal 7 release.

People who were familiar with building websites (but not with Drupal) were observed while they worked through a number of site building tasks. This report contains a summary of the results.

Summary

The good news is that most of the changes that were put in Drupal 7 tested well. Compared to Drupal 6, Drupal 7 no longer confuses new users with basic conceptual hurdles like where the front-end vs. back-end of their site is and how to create an "About us" page, and for the most part the administrative interface is clear.

The bad news is that now that some of these basics have been dealt with, we've uncovered a whole new layer of challenges for first-time Drupal site builders, some of which were pretty surprising. Finding modules, creating and placing blocks, and creating content types were difficult tasks for participants to understand, and it's these site builder tools we'll want to improve for Drupal 8.

Who did we test, and how did we test them?

Eight participants (a typical number for this kind of study) evaluated Drupal by working through a series of real world tasks over a 75 minute period, "thinking aloud" while the team observed and took notes from behind a one-way mirror. The participants could phone a "help desk" (staffed by a member of the team) if they ever got stuck and needed a nudge in the right direction, and a number of open-ended questions were asked throughout the test session.

All eight of our participants are people directly in Drupal's target audience. They are site builders, already using tools such as Dreamweaver and WordPress. They know HTML and understand how the web works. All but one had no prior experience in Drupal.

What did we ask them to do?

First, we gave participants a couple of minutes to click around the interface and provide their initial impressions. Then, we provided them with a series of tasks (see the scenarios covering basic things site builders do from day-to-day, including:

  • Content creation
  • Appearance settings
  • Block management
  • Installing a module
  • Creating a new content type

Test Results

In previous usability testing against Drupal 6 and early Drupal 7, we had intended to test things such as CCK, users and permissions, taxonomy, and so on. What we ended up testing instead was how horrifically confused new users were performing even basic tasks within Drupal 6's administrative interface: the separation of front end/backend, the overwhelming number of options at /admin, confusion between Page and Story, and more all stopped people from using Drupal successfully.

Based on this testing, we made a number of changes to Drupal 7 to try to combat these conceptual problems. And in short, all of the major usability problems found in Drupal 6 and attacked in Drupal 7 appear to be either vastly improved or non-existent in Drupal 7. Hooray! :D

What tested well?

  • First impressions: After some initial exploration (around 2 minutes), participants had a favorable response to Drupal 7 and thought it was:
    • "very simple"
    • "clearly laid out"
    • "there is a lot you can do"
    • "not very cluttered"
  • Creating content: All participants—but one—were able to add content (using different links from the homepage) without difficulty. Participants thought it was "pretty simple", "straightforward" and "easy to find 'add content'".
  • Basic Page vs. Article: Participants understood the difference between "Basic Page" and "Article". All the participants who tried to add an "About us" page to the main menu chose "Basic Page".
  • Changing the color of their site: Participants instantly knew to go to "Appearance" to change the color of their site. Although some participants had minor issues finding the theme settings page, they were eventually able to complete the task. Participants also liked how of color changes were previewed live on the affected area itself.
  • Toolbar: Participants found the toolbar quickly and had a fair understanding of what it did.
  • Toggle between the administrative overlay and site: All but one participant had a clear understanding of how to toggle between the overlay and the site. Participants understood when they were performing administrative activities and frequently used the overlay's "close" button to return to their previous location or before preparing to start a new task.

What tested poorly?

Now that people can actually navigate and do basic content creation tasks without much difficulty, they encounter other parts of Drupal's administration that are still utterly baffling. This section covers the major issues, though there are many more, some of them quite easy to fix!

  • Drupal terminology is obscure.
    Terms like "Block" and "Module" and "Content types" and "Triptych" either have no meaning, or a preconceived meaning that is different from what Drupal uses these words for. For example, some think of "module" as a block.
  • The distinction between "sidebar" content and content made absolutely no sense. Content is content.
    When asked to create a sidebar block, every single participant went for the "Add content" link. Unsure which option to choose, they went for either "Article" or "Basic page" and scoured the vertical tabs options for a way to add it into the sidebar.

    A Drupal person thinks in terms of nodes, blocks, views, and panels and a normal person just sees these things as content, and expects to be able to place them where they want on the page.

  • The lack of a visually obvious way to place content on the page was a huge detriment
    People really craved some sort of "edit in place" mode as a visual way of controlling their site layout. They tried to click on region names in the block demonstration page to add content there. They tried to take a node that was outlined with the contextual links border and drag it to the sidebar. They tried to click and drag to resize sidebars. The lack of this functionality made Drupal appear extremely clunky.
  • Lack of realistic previews makes building sites in Drupal extremely challenging
    The preview as provided by Color module is what participants really were craving to see absolutely everywhere.

    Menus have no preview either, only text descriptions that give no visual clue to where on the page they will appear. Content can be previewed, but this preview is shown in the administration theme and overlay, not in the context of the actual user-facing site. This was frustrating and confusing.

    Participants liked the Color module preview, but expected to see it everywhere and were disappointed when they did not.

  • While Drupal takes a "content-first" approach to site building, many people take a structure- and/or appearance-first approach.

    When looking at a mockup of the website they were trying to build, the natural inclination for several participants was to start setting up the navigation structure first. They ran into issues when doing this, because they were required to enter something for the "Path" field of the menu item they were creating, and they had no idea what that could be.

    If you try and create a menu before a node, 'path' is mystifying.

  • These Drupal evaluators did not realize you can extend it, and so Drupal is seen as limited in functionality.
    If users found their way to the modules page at all, they did not see that they could add new options there.

    Given that the entire point of using Drupal is that you can easily extend it with modules, making this fact more discoverable should be a primary objective of Drupal 8.

    The list of options on the modules page appears as though that's all you get, because the install link is easily missed.

  • The workflow of finding and adding a new module from Drupal.org is overwhelming.
    Once a help desk call pointed them at the "Install new module" link, new problems began. Being taken off-site to Drupal.org was extremely jarring, as was the (apparent) requirement to download a module to the desktop and then upload it to the server to install it. People suggested an in-app browser for modules from Drupal.org. Additionally, the filtering options on drupal.org for modules are overwhelming. The search box, arguably the most valuable tool for when you have a general idea of what you're looking for, is located far down the list and was missed by most participants. Instead, they honed in on the "Categories" drop-down, which was completely useless to them.

    People missed the search box, and the categories options overwhelmed them

  • People were unable to make the connection between a module's version number and modules listed on Drupal.org
    They generally just picked the first thing in the list, which happened to be 7.x for Webform, but wasn't for "Announcements", which another participant searched for. (Another issue is that once people know modules exist, they want to use them for absolutely everything, including things that could be accomplished through blocks and content types.)

    A drupal.org project page which highlights the download table as being confusing, as well as the fact people clicked 'Git instructions' for help. Oops.

Overall Results

After 75 minutes of using Drupal 7, participants were asked about their overall expressions. Participants commented:

  • "This was just hard. I struggled with it. I will have my sister help me."
  • "If you know what are you doing, it's okay. But for the first time user, it is a little shaky."
  • "A little confusing at first ... but I think you need to toy around with it ... get familiar with it."
  • "Not as user friendly as I thought."

When the participants were asked to select five words they associated with their Drupal 7 experience, 60% of their comments were positive. The most prominent word was "Customizable". (See the complete results.)

Participants were also asked to rate their experience on a 5 point Likert scale in terms of ease of use and value. Half of the participants thought Drupal was "Difficult to use", and all the participants thought it was "Completely valuable" or "Somewhat valuable". (See the complete results for ease of use and value.)

Finally, it is important to note that none of the participants were able to make it through their session without needing at least some assistance from the "help desk".

Next steps

We found over 100 issues during this testing round, some of them major, and many more minor ones. These are captured in raw form in the "Issues Analysis Matrix" spreadsheet, and in the issue queue under the "UMN 2011" tag. We will discuss, design and code changes for these, involving the larger community on what it means to fix them, and how. You are most welcome to join the effort!

  • Keep tabs on high-impact issues at the Usability Community Initiatives page.
  • Participate in larger discussions in the Usability group.
  • Provide and review patches for issues with the "UMN 2011" tag.
  • Help us create Drupal.org issues for the remainder of the issues found in testing. (Ping someone in #drupal-usability for edit access)
  • Help with uploading and splicing videos of participants' experience to illustrate issues to developers in the issue queue. (Ping someone in #drupal-usability for more info)
  • Chat with the team in #drupal-usability and #drupal-contribute on IRC.

We also encourage you to help with further usability testing, both formal and informal. What we tested at University of Minnesota only scratches the surface of what Drupal core can do (for example, we didn't cover Taxonomy, Search, adding themes from Drupal.org, etc.). We need further testing to verify if our changes are actual improvements, and there are many contributed modules out there that would benefit from usability testing as well.

Join us and help make Drupal 7 and 8 a joy to use!

Sponsors

A sincere thank you to our many sponsors :

    Acquia

And our many individual sponors; Marilyn Langfeld, zerolab, Walter Ebert, Jacine Luisi, Miles Worthington, Nate Haug (Lullabot), Yashesh (Venuslabs Web Solutions), Stein Bjørklund, Brian Link, Greg Dunlap, Wunderkraut, and others. Without you, this testing, and gathering together all who took part, would not have been possible!

Resources

Comments

Anonymous’s picture

These findings all make perfect sense, but the details would have been hard to predict. Real-world data rules OK! Fixing these difficulties will make D8 a much stronger product. Kudos and thanks, and I hope much more of this sort of work can be done.

cbrookins’s picture

Sobering data for sure, but inspiring to help focus our UX efforts going forward. D7 UX was great but D8 will rock even more! As will D7 contrib efforts to prototype solutions for D8 core consideration.

A big "yeah!" to the volunteers and organizers who contributed to deliver these results.

joachim’s picture

Some of these are really interesting, but one or two I think are blind alleys.

The confusion over 'block' and 'module' is caused by Wordpress, where 'module' means more or less what we call a block. That's just a poor use of the word, and just because Wordpress is silly is no reason for us to change.

> While Drupal takes a "content-first" approach to site building...

That has always seemed to me to be one of Drupal's great strengths. I can clearly remember that when I started using Drupal it took me some effort to get my mind round it, and I can remember the 'aha!' moment when I got it and realized how much better this approach was. Let's not through the baby out with the bathwater here: our goal should be to make it easier to get the aha moment, not remove one of our strengths.

webchick’s picture

Identifying these problems doesn't mean we need to undo the way Drupal does things. We're just stating facts.

One easy way to make the word "Modules" more clear, for example, is to fix this problem:

Mouseover text for 'Modules' explains modules as 'Enable or disable modules'

See http://drupal.org/node/1167458

And we can still support content-first approach to site building, but put in some useful methods for supporting structure-first as well, to accommodate both mental models. Jeff Noyz has some cool mockups started over at http://drupal.org/node/1101600 for one such approach.

Patches welcome. :) Here's the hit-list: http://drupal.org/project/issues/search/drupal?issue_tags=UMN+2011

iandickson’s picture

And the first rule is that it's the READER that matters.

Modules - they're Apps. Call them that. [Yes Drupal was first, but language is what it is, ruthless :-) ]

firebus’s picture

but also Joomla and the various "Nukes" (and maybe others?) use "module" to mean what Drupal calls a "block".

I can't think of an objective reason why this is silly and Drupal's decision is not. When the vast majority of your competition is using a term in the same way, and you're the odd man out, and it's confusing your customers that's a reason to consider a change of some kind, or some way to make it clearer.

I'll also note that in many other CMS worlds, blocks are deployed on a per-node basis, and it's easy out of the box to a create a block that displays a node. These two features lessen the distance between "Content" and "Sidebar".

naught101’s picture

Our "blocks" are called "widgets" in wordpress, not "modules" :)

Here's what wiktionary says about the three words:

Block:
17 definitions (just counting nouns - 10 more for verbs), the most relevant of which is probably:
A substantial, often approximately cuboid, piece of any substance.

Widget:
5 definitions, including:
Any one of the components of a computer application's graphical user interface, such as a Cancel button or text input box, that a user interacts with.
However I can't imagine that this definition is very well known by anyone but programmers.

Module:
8 definitions, including:
A self-contained component of a system, often interchangeable, which has a well-defined interface to the other components.
and (computing) A section of a program; a subroutine.
Which are both perfect descriptions of drupal modules, but, again, probably only known by programmers.

Personally, I think that "plugin", "addon", and "extension" are all better terms than module, if only because so many other systems use them (ie. every common browser uses one of these).

Block is a lot harder to think about, but "box" could make sense (as in "box-out" in print layout).

Then again, I don't actually think any of this really need changing, it just explaining (as webchick points out in her comment).

jrabeemer’s picture

Personally, I think that "plugin", "addon", and "extension" are all better terms than module, if only because so many other systems use them (ie. every common browser uses one of these).

That is the clearest suggestion I've seen so far. As soon as I read that, the meaning became very, very clear. I support changing the name "module" to one of the above. :)

JamesOakley’s picture

That actually had me laughing in places! Sometimes, you've lived with Drupal just too long, so when someone new to it does what the terminology actually suggests, you realise what the labels look like. I think my favourite was "Git Instructions" :-)

That should really help improve D8. Good stuff!


This signature is currently blank
ShaneOnABike’s picture

This is a really great report actually and interesting.

I used to do a great deal of this with my first job (as I took usability
studies in Uni back in the day). And actually I totally love doing this
kind of thing...

The most important quote I think for me was: "While Drupal takes a
"content-first" approach to site building, many people take a structure-
and/or appearance-first approach."

I think this is one reason why Wordpress does well in this arena b/c
they focus on the appearance and user interaction... (and they workflow
is generally less complex).

Peace out

dcmistry’s picture

That is awesome to know you have done this before! The plan is do more usability studies going forward which means more involvement from awesome people like you. Yayy!

ShaneOnABike’s picture

Sure I'm registered in the usability group and would like to start closing / responding to tickets around usability so thanks for your encouragement @dcmistry

millaraj’s picture

Great to see these results. So many obvious points raised that are forgotten about once you've been using Drupal for a while. Great to see the community standing back and having their work critiqued openly and honestly.

However I wonder what is being doing to address accessibility issues and make Drupal and accessibly platform for everyone, including those with disabilities.

webchick’s picture

Drupal 7 had enormous work done on it to improve accessibility. While we're not perfect yet, it's light-years beyond Drupal 6 (and even a lot of other CMSes out there, period), and the work in Drupal 8 to incorporate HTML 5 has accessibility as one of its primary mandates.

If you'd like to help improve accessibility in core, here's the list to follow: http://drupal.org/project/issues/search/drupal?issue_tags=accessibility

mlangfeld’s picture

Thanks to the usability team for undertaking these tests, and for making good use of them in D7 and in D8. Already D7 has made huge improvements, can't wait to see how D8 continues the improvements.

technicalknockout’s picture

Thanks so much for the time and effort spent on this testing session. I remember thinking Drupal terminology & "the Drupal way" were not very intuitive when I first started working with it. I'm glad to see the community take a critical look at usability.
Also, I have been thinking about starting to do my own usability tests (I have not been designing for very long). This article shows nice techniques and questions to ask for getting started on usability testing.
Awesome show, great job!

- - turpana.com - -

dcmistry’s picture

If you have not, I would suggest you to join the usability group/ IRC. There are bunch of usability geeks hanging out there. I have been conducting several usability studies and have used many different kind of methodologies. I would be happy to offer my 2 cents. Looking forward to you doing more usability studies :)

gidgetk’s picture

In Wordpress, modules are analagous to Plugins, and Blocks to Widgets. WordPress doesn't use the word module anywhere, so the confusion doesn't arrive from there.

David_Rothstein’s picture

gidgetk’s picture

which would be analogous to Drupal Gardens I guess. The real self install, most common most widely utilized Wordpress does not use these terms.

That of course isn't even remotely the point. Other systems do use those terms to mean different things. The user confusion shouldn't be plopped on other systems, it should be fixed.

bmateus’s picture

I've been drupalin' for about one year now, and I've got to say that there's nothing free or bought that's even close to Drupal.

That sayd, I still remember the steep learning curve, specially in Theme making, which I think was not so in-depth in this study.
Css, pre-process, templates (lots of them), hooks... It's harder than I've ever though. Of course, it's also much more powerful than I ever though!

But a balance must be met, and the way forward is with studies like this.
This makes me feel even more confident of my choice of Drupal as a main framework, as I'm sure that new versions will be surely more powerful and intuitive.

A big thank you to all involved.

Bruno Mateus

Alphabool’s picture

These usability concerns are spot on. Since I've already taught myself the 'Drupal way' for Drupal 6, I guess this doesn't apply, but usability is always a top priority and it's good to see the Drupal team putting such an emphasis on it. Bravo!

Encarte’s picture

Don't get me wrong: I love Drupal and couldn't live without it. I love Drupal developers and the community they've founded.

It strikes me, though, that lots of the conclusions that this usability tests point to are things told by site builders in drupal.org that developers don't listen to. The tests are great but it would be worth thousands of tests if a certain culture of respect for site builders would be encouraged.

The present culture of «scratch your own itch» while being important in some aspects, has a dark side: it turns Drupal.org into a developers only area. You clearly feel that if you don't code you're like some kind of parasite using others work. You are tolerated because developers need you for testing. And that's it. Your feature requests are ignored even if you try to show how much that would improve Drupal and the module. You are made to understand that if that's not important to the developer, it should have not been even suggested. In the end you tend to quit from the community and just «subscribing» away.

I agree that it's kind of an impossible mission to try to make developers do things that are not of their interest and for witch they're not paid for. And yet, making them do just that is in the interest of Drupal and the community. It's because developers tend to only worry about their immediate goals and immediate clients that you find such «horrific» situations in usability tests.

So, there are a few things I would like to point out to some developers and that I think should be almost publicised:

- The person to witch you say «scratch your own itch» is your best usability test and improving guide.
- Even non-for-profit-small-site builders are endorsers and yes, they do remember who listened to them and who was arrogant.
- Site builders don't know how to code because they spend their time testing, using and giving sense to your work.
- An annoying site builder is giving you feedback, a silent site builder is giving nothing back to you or the community.
- Even the «stupid subscribers» are telling you what is most important.

I know that lots of people are working hard to change things and include site builders. Initiatives like this one and others are good. But day-to-day community culture is the most important thing and there's lots to do in that field.

naught101’s picture

This is a valuable point, but it's hard to gather data like this in the field. Perhaps a general drupal feedback survey could be created for this kind of info though? Especially using open-ended questions and qualitative text coding the input...

KarenS’s picture

I was there with you for the first usability study. I am visualizing you guys sitting in the dark room watching the users missing all that stuff that seemed so obvious to us :)

Yes, every time we fix something, it is just going to reveal a deeper layer of confusion. That's OK, we'll never get it perfect. And Drupal is so flexible it's hard to make it simple.

We quit using the word 'block' in Buzzr already, and already drew the conclusion mentioned above that the word 'widget' is a better fit for the way we use blocks in Drupal. Most people seem to instinctively 'get' what a widget is, and that it's different than other content.

The confusion about how to add and enable modules would be a great focus for Drupal 8. We really do need a nice, simple, front end for locating and adding new modules. And I also agree that the word 'module' has too many other meanings to be useful here. We need something like 'features' or 'functionality'.

This process is always enlightening though :)

rafamd’s picture

I agree that modules would be better identified by a new name and think "Extensions" is the better one mentioned in this thread. It's also a word that has similar translations in many other languages, so it would make more sense for those guys.

technicalknockout’s picture

I don't think the word 'module' should be changed. It's quite appropriate and the meaning of the word lies IMO at the heart of what makes drupal so awesome and flexible:

module
each of a set of standardized parts or independent units that can be used to construct a more complex structure, such as an item of furniture or a building.

Yes, usability is crucial - so I suggest making the functionality easier to discover and play with, rather than focusing on the semantics. By that I mean, make big fat buttons and graphics that people can't miss, probably the on the overlay-dashboard would be a good place.

- - turpana.com - -

dcmistry’s picture

"We quit using the word 'block' in Buzzr already, and already drew the conclusion mentioned above that the word 'widget' is a better fit for the way we use blocks in Drupal. Most people seem to instinctively 'get' what a widget is, and that it's different than other content."

KarenS: Curious, Did you guys do any usability studies around that? Is it possible to share the findings?

KarenS’s picture

No formal research, just talking to people who didn't know anything about Drupal trying to find terminology they understood.

zhangtaihao’s picture

I know the study is done on Drupal 7, but here's a Drupal 6 module that more people could look into:

http://drupal.org/project/reviser

Also, I think many Drupal developers/experts completely miss out on the usability of the commenting system. It's an excellent example of a contextual "content" creation process. In some sites at our organization, we've taken to focusing on the concept of a dashboard-centric approach to building medium- to large-scale sites. The user can then relatively easily discover a conceptual workflow to adding content to the obvious sections of the site.

Just on the first image, too many users won't even be able to distinguish between the areas of content on the same page. Many will simply attempt to edit that one page even for all the dynamic news and events that appear there. The best way to understand regular users is to suppress your Drupal sensibilities and break down all abstraction in your mind (to before the Drupal trauma).

Glenn Hilton’s picture

Do you have any of the videos available from the study that you showed at Drupal Camp Twin Cities that you could upload? Some of them were fabulous. :)

webchick’s picture

The 8 videos have been split up among the team members who were on-site and we're actively working on getting them uploaded. The main thing holding up the process is that we need to make sure any personally identifiable information is removed (for example, one participant mentioned their company, another's hands were accidentally filmed as they were filling out the survey, etc.) before throwing them on The Internets. :)

Glenn Hilton’s picture

Looking forward to seeing them. We were dialoguing about Drupal with various presenters and participants at the the Interlink Conference here in Vancouver this weekend and there was interest in the UofM usability study and any videos that would be available.

rcross’s picture

This is (another) really great piece of work.

I am curious about how soon we might see some of these changes though. It seems obvious to me that there are several simple improvements that could be implemented and some deeper issues that will need more thought/work to get right. Is it possible that we can work some of these improvements into the point releases of D7? or is everything going to have to wait until D8?

Perhaps a direct question to webchick - will there be any focus on usability / accessibility improvements in the upcoming d7 releases? or does it all have to happen in contrib before D8?

yaoweizhen’s picture

Drupal is more powerful for developers. So I think it must know clearly at first, which parts are for site builders, which parts are for developers. For example, CCK isn't for site builders because there are some database concepts.

Two roles always should be in when make new site . "Admin" has all permissions, there is in Drupal 7. "Webmaster" has some permissions specify by "Admin". I like admin_menu module for "Admin", but no perfect module for "Webmaster", that means well structured and easy understanding menu bar.

I haven't used Drupal 7.

Francewhoa’s picture

@webchick: Very interesting. Thanks for sharing the results.

And in short, all of the major usability problems found in Drupal 6 and attacked in Drupal 7 appear to be either vastly improved or non-existent in Drupal 7. Hooray! :D

I agree

Loving back your Drupal community result in multiple benefits for you  
HJulien’s picture

I've been struggling with Drupal for 4 months now and can attest to every one of the issues raised by the participants. In fact, this just touches the surface. There are many more layers of usability issues beyond the first 75 minutes!

My experience has been extremely frustrating. It's been one issue after another with very few moments of smooth sailing. Modules without documentation, some that are buggy and don't tell you. Forums with way too much crap on it you have to wade through to find the hidden gems. Only rare answers to issues or questions in the forums.

I spend 3/4 of my time searching and trying to figure things out. It is a very expensive trade-off for 'free' open-source software.

In my case, I am building a website that is really important to me. It's much bigger than most so I expect more problems than most. I am extremely patient but Drupal has tested me beyond my limits at times. For example, I have just spent 3 days on 4 different poll modules and I still haven't got it right. That's enough to piss most people off. I have a list of things I've been unable to figure out so I have no choice but to pay someone with expertise to help me just so I can get this project completed.

I have a computer programming background with 8 years experience in mainframe, mid-frame, scripting and several web programming languages. And years of schooling to go with it. I'm also certified as a web designer and been doing websites for the last 8 years.

You'd think someone with this much background with be able to pick-up Drupal easier than most and that's probably the case. I've wondered if it's just me - maybe I'm getting too old for this stuff. I know I am expecting a lot from Drupal - way more than the average user here will ever need. However, I never expected it to be so frustrating.

I think a lot of people give up on Drupal within the first month. It's definitely not for the faint-hearted or the average business-owner who wants to build a little website. It doesn't lend itself well to those who just want to touch the surface. It's niche is really for hard-core users who can understand the importance of intricacies and details, the jargon and who accepted awhile back to live with the fact that every period, dash and bracket must be just perfect. Technology is unforgiving. I know for myself, that I have become unforgiving in my own standards of excellence. I see that a similar standard exists in Drupal and where it falls short, is very critical to it's success.

A few recommendations:

1> That it be required that all modules be documented really well. The days of the 'cowboys' who live by their own rules are over. Develop some standards modules must adhere to or be removed. Give enough warning for people to either fix the modules or remove them. Leaving 'dead wood' to trip over is very detrimental to the entire community and to the Drupal brand.

2> That people who answer questions in the forums be paid or receive something of value in return - discounted membership, free Drupalcon courses, free hosting - whatever you choose. This is very valuable feature to users if you wish to keep them. If you don't support the Forums more in several different ways, Drupal can't honestly claim how valuable the 'Drupal community is'. Do a count on the number of posts that have 0 replies and you'll know what I mean. Every unanswered question is a negative in the eyes of users. Maybe do a sweep and remove 90% of them.

3> Clean out junk from the Forums or add a feature to posts like 'sticky' so that the good answers could rise to the top to cut down on time spent searching.

4> Ask people for feedback on what the issues are. Also ask for ideas on how to solve them.

5> Think up ways for newer users to participate more. The mix of users on Drupal is like a pyramid. Find ways to help people move up and widen the top of the pyramid. How about code competitions hosted on Drupal with financial gains for the winners? How about a competition for finding the best 500 answers to questions?? lol

6> Provide a place where module developers can have a 'test site' for their module for people to go and see it in action. (A similar kind of environment can also be used to reproduce bugs or problems the maintainer needs to see.) The module maintainer might just load their module and whatever other modules are required to create a 'clean' environment. This environment could include sample views that work well with that module, etc. Maybe even add code snippets in places people can just copy and paste and use on their own site. People who have made great views for specific modules can post them for others to learn how they can get similar functionality. What's important about this 'test site' is that it's also a great learning site and will eliminate a lot of questions.

These are a few ways Drupal management can make working with Drupal less frustrating. And we can all use more of that.

Hélène

HongPong’s picture

If it is a viable option I really recommend checking out a local user group. A few minute conversation can eliminate a ton of confusion. Totally agree with you, all the dead-end forum entries don't help anyone solve anything & it would be good to flush them out /archive them entirely.

iandickson’s picture

Hard to have the power without the complexity. My current project would be totally impossible in any other system other than hand cut code. And I cannot code.

1) Documented Modules. Would be good, but most people are very poor at writing. Most of my income comes from people who know their stuff but can't for the life of them write it down in a sensible way.

Developers need help.

Suggestion - Document volunteers work to Document modules as they are rolled out.

In practice this is hard to do at a distance unless the writer is also technical, but CAN be done face to face if the writer is not. I'd be happy to put my hand up and say "will work with UK module developers" - a sit down weekend will suffice to document all but the most complex modules.

2) Forums - this is an area where FREE is important, and people would probably contribute LESS if it was in any way "paid". See many studies passim...

BUT

Here's an idea for a Drupal guru looking for a biz opp. "Tenner a month and I'll answer your Forum Questions"

If 300 people like you and I ponied up, there's a 36K job there for someone. Hmmm - if the person running it was in some way approved by Drupal.org they could be "official" and some of the revenue fed back to support Drupal Org...

Actually that's not a totally nuts idea.....

It would generate First Comment Quality Answers, and they'd be there for everyone else.

3) Cleaning the junk

Yes Please. Just yesterday someone commented on a post of mine from 2007, pointing out that my solution was flawed, (true, but that's because it's now 2011).

Suggest adding tags to posts. Posts seem to fall into the following types :-

Help Requests
Bug/Issue Observations
Documenting solutions

If we started to tag the initial post AND comments it would then be easy for drupal.org to highlight (e.g. as default views) Documented Solutions.

Bolt on rating (prob limited to "official" types - invited peeps, probably invited on account of their own good participation) plus automatic downgrading of "empty after a month" Help requests and the system would be much better at helping people find what they need.

(BTW, if the Tenner Club is a success, the gurus could be mandated to post answers to questions unanswered after a month, thus delivering more value back to Drupal.org).

Cheers

dcmistry’s picture

Thank you for all the feedback, very much appreciated :)

In fact, I would love to interview you about your experience with Drupal, the challenges you face and try to understand things better from your perspective. Would you have a 30 minutes to spare?

Ops, forgot to introduce myself. I am Dharmesh Mistry, UX team member for Drupal. I am also the usability researcher for Acquia, Inc.

HJulien’s picture

Dharmesh: who specifically are you asking for an interview with?

Hélène

dcmistry’s picture

My bad, I was talking about you both @HJulien @iandickson

yeti22’s picture

# Appearance settings
# Block management
# Installing a module
# Creating a new content type

This test was for admin users only?
Can the actual users, also known as end-users, or members, or auth members that is the mass for whom we build the site and who use the site day to day be tested ?
Is it as easy to add text or image or webcam picture as in facebook with minimum of clicks? Is it as easy to find my contacts or persons I will be interested in?
For example, I could not figure out how to position the image, left right or bottom to text, I uploaded in Drupal 7.

As admin or no 1 user or someone who has to use shared hosting and pay bills for it
I find Drupal 7 takes lot of space, lot of CPU resources etc - somewhat impossible to run compared to 5 or 6 and somewhat not so good user exp for me.
May see this article http://anti-mega.com/antimega/2008/08/15/this-is-not-my-beautiful-house

dcmistry’s picture

Yeti22,

Yes, this test was particularly focused only on the site builders/admin users. I agree that we have to test much more especially with site contributors/end-users. We intend to include these users for our future studies.

Appreciate your awesome feedback!

webchick’s picture

Testing for end users is extremely challenging, because almost all Drupal site builders will customize the administrative interface and tools uniquely to whatever task is at hand for their end users. Managing content on drupal.org is completely different than managing content on michaeljackson.com, and completely different again from managing content on whitehouse.gov. So finding a distilled set of tasks that apply equally to all Drupal end users is something that I'm not sure how to do; some sites will have WYSIWYG editors, some will have sophisticated media management tools, some will use a completely different admin interface with a customized "dashboard" specifically for forum moderators, for example. Such is the power of Drupal!

OTOH, tasks that site builders have to do are much more ubiquitous across all of those sites. All Drupal site builders need to create content types and views, locate and configure modules, and make adjustments to site appearance. If we make those tasks easier, then it has "trickle down" positive effects on the experience for end users, too, because more time can be spent on making their end-users' experience awesome, and less time on being frustrated with performing basic tasks in Drupal.

yeti22’s picture

Such is the power of Drupal!

Power of Drupal or power of Drupal Modules? I started using Drupal 5 for a module which was never upgraded to next versions, so for a site I still use Drupal 5. Sure this can be done by paying $ but then user exp = bad, but then again one has to pay for D 7, 8 and so on. Point and click "update" and its updated via web within seconds like Firefox or Wordpress = supreme user exp. Why are modules that will not guarantee future versions and easy upgrade in the official repos of drupal org? They need to be in some other section or domain, and not having this = bad user exp. The above test done for admins used Drupal out of the box. Surely out of the box exp. could be bettered or extended by additional modules, language, video whatever additions for those first time dummies, isn't it? So how easy is Drupal out of the box for an end user?

It is most easy to define what a typical end user to any site does - they submit something. How easy it is to add and format a piece of text? 3 to 4 years ago http://drupal.org/usability-test-university-baltimore-community-solutions recommended WYSIWYG in core as the first recommendation. Was this for the end-user or the admin? End-user also typically and most commonly wants to share a photo, D 7 out of the box is so rigid that it does not allow to position the photo, and its 2011. Users also expect to easily import their contacts in a site today, so that they can interact and share with them. They also expect not to get their data lost while they are in the process of writing/submitting http://drupal.org/node/1123696#comment-4568606

What tasks to be tested can also be very easily found out by making a sticky post on the front page and thousands of drupal org users can contribute happily.

PS : This is not to say I am unhappy with Drupal. But this hoopla on admin user exp seems sort of useless. Which is more easy to comprehend - a small picture showing "a man with flower" or saying "Mož z rožo" to a non-Slovenian speaking person. Each one has to learn a language and its not difficult always. Drupal is a language and uses its own language, so one has to spend some time to learn. If Drupal is made a picture language like the above it won't be able to create poetry! Will these dummies be ever good or creative admins, I doubt. If newbies who were using other CMSes are tested and you want to have them that exact ease of user exp. the best way is to have D install profiles that mimic the language and actions of those CMS, which Drupal being flexible can probably easily be done. Nothing to learn then, smoothest user exp - tra la la la!

AFowle’s picture

As many here have pointed out, a module is a building block. Plugin is used by other applications for similar entities. I happen to think module is a better name, but I am more concerned that changing it will confuse existing users needlessly.

What is really needed is not a debate about what to call it, but more instruction built in to an empty installation that provides some steerage - it can disappear when no longer needed.

juampynr’s picture

We are getting ready to perform a usability testing in 3 weeks time. As we are not sure about what to test yet, we could take the word of Webchick and go for Taxonomies, Search engine or installing new themes. Is there anything else we should add to the list?

More details here: http://groups.drupal.org/node/153144

Cheers

David_Rothstein’s picture

Nice!

That's probably a good set of big topics.

For smaller, more focused topics to test, it might be worth taking a look at issues tagged with the "Needs usability testing" tag. Those aren't prioritized in any way, and not all of them are necessarily well-defined enough to test at the moment, but in general those are issues that have been identified as needing some usability testing to help decide what direction the issue should go in (or whether the change proposed in the issue is worth doing at all).

PetarB’s picture

HJulien & Encarte put things very well in my opinion.

One of my primary issues with Drupal is that modules lack documentation. A simple template txt file, without which a module cannot be distributed here, would work wonders, ie:

- What it does
- Show an example
- Path to settings
- Explain what each setting can change
- List (and this will always be incomplete, but its still very important) known incompatible modules

With this simple txt file included with each module, and documented on the modules project page, hours of responses, headache and drama are avoided. I posted a suggestion to this end a few months ago and was shot down for various reasons. I think it's madness not to incorporate standards like this.

Claims that we are not all writers, etc are not valid in my book. If you can code, you should be able to explain why you are coding, or what you intend to achieve even at a simple level. I have downloaded and attempted to install and use many modules which barely scratch the surface of my small list above. In alpha, the module release could request help with the documentation. But the seed of that documentation MUST come from the developer.

If the standard is set... I know that the Drupal module ecosystem can ONLY get healthier. As it is, it's a morass and offputting for the newbie and veteran alike.

sonar_un’s picture

I am with you on this as well. I have been in Systems Analysis, Troubleshooting, Repair, Networking and general "do-everything with computers" my whole life. Most of my experience has been working directly with end users and seeing things they way they see it, teaching them as well. I felt that I could be able to dive right into such a popular CMS without too much trouble. Of course, I expected some work, but no where near as much as I have ended up putting into it. Like many here, I have an incredibly important project that I am working on so I knew it would take me some time. I also knew that Drupal was the only CMS that could possibly be able to do what I needed done.

I have a background in HTML, CSS, Some programming, but am very familiar with all the tools of the trade. So it wasn't like I was diving into it completely blind. But boy did I feel that way when I started. The terminology was just the first stumbling block, but it is no where near the most difficult concepts to grasp. I could go on, but most of what I mean has been said to some extent.

I started 7 months ago, using Drupal 7, before it was released. So I had no prior working knowledge of how Drupal worked. Though the UI is better than Drupal 6, I still don't think it's that great. I am not sure why I have three different places to configure options for a module, it's just strange. If the D7 UI is much better than D6, then I am frightened at to what D6 was like, or even D5!

I mean, I don't want to go on a tirade complaining about Drupal. Heck no! I love Drupal and i'll defend it (as I have had to do already for my client), many times. If it weren't for Drupal the things that I am doing just simply would not be possible. I wish that I could do more to help the community, but I do understand how those who are not developers feel out of place looking for answers, or even afraid of asking questions on how a module works. I understand that the developers do these things (mostly) for free and on their own time. That in itself is what is wonderful about open source. I think that the Drupal community fosters a kind of helping attitude between developers.

I don't have all the solutions to help users be able to use and give feedback to the project in a more meaningful way. I know there are lots and lots of people who probably give up on Drupal way too early and don't ever get deep enough to explore its power. I know that UX is the way to do that and I applaud all the efforts that the team is doing in order to facilitate the experience is wonderful. Thank you so much for working to bridge the cap between developer and end user.

zkrebs’s picture

I've been training clients to use Drupal for awhile now. Something that Mac users don't get is why they can't drag and drop things (pictures, text documents, etc.) from their desktop directly into the node body, without effort.

They also don't understand why they can't copy and paste a word document into a WYSIWYG editor and have it look the same.

They don't understand how or why to create content in Drupal first, as opposed to doing it in Word first then translating it.

If it could address these things, my life would be easier.

Also, I'd just like to say: the reason people don't learn is their own personal resistance to change, not because its "hard". I experience this all the time, being a Tai Chi / Qigong instructor. I will ask someone to move their body a certain way, they can't/ won't and even after one hundred viewings/showings of "how to do it" they still can't. Is the problem that the movement is too hard? Or is it that we have no idea how to learn? Anyhow, just some non-programmer perspective here.

dcmistry’s picture

Interesting perspective! Irrespective it is hard or people are resistant to change - they are both linked to user centered design approach. Usability is based on this approach of user behavior and psychology. Ideally, a good design/ product which fits their needs, should be able to persuade the user to learn.

They also don't understand why they can't copy and paste a word document into a WYSIWYG editor and have it look the same.
This is true. I have seen this in my testing as well that users expect WYSIWYG to translate content perfectly from Word - which never happens :)

zkrebs’s picture

It would be interesting if we could translate columns from word, for instance, into "columns in a panels layout" and have the pictures be inserted as image_fields linked to the node, etc. So one could upload a word or open office file, yet it turned into a "Drupal-way" content page. That's probably not the direction anyone is going in, but it would certainly be awesome.

dcmistry’s picture

All the participant videos (8 participants) from the usability study are available to view on Archive.org. Here is the link: http://www.archive.org/details/Drupal7UmnUsabilityStudy

Hussamamer’s picture

Thanks for sharing this video