There hav been lots of discussions about if the overlay really improves Usability, and if so, in what area of Drupals admin screens. A lot of people got to the same conclusion: it needs quite some user testing. For this I started creating a focused Test plan. I keep all the content in the first comment, so it is possible to edit that post and keep it up to date by including improvements.

CommentFileSizeAuthor
#2 frontpage.jpg60.83 KBeigentor
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

eigentor’s picture

The test plan adresses the overlay together often with contextual links, as those are closely related to the short tasks the overlay is supposedly good at - and contextual links need some testing as well.
There is a nicely formatted version of the plan over here:
http://docs.google.com/View?id=dgzwvm6w_61fs6qc6cc
Which I also will keep updating as it improves.

Please upload videos you record of the tests to this vimeo group:
http://vimeo.com/groups/drupalux/videos/
Vimeo allows for videos longer than 10 minutes.

The plan needs some more tasks, so especially those but also other improvements are welcome.

Test plan for the overlay

What needs to be tested

Several tasks that show the strength: short quick jumps to the overlay and back.
This test plan tests the overlay in conjunction with contextual links, as those two are closele related, especially in performing quick Admin tasks (editing in place).
It is set up mainly for A/B Testing with overlay on / off

Recommended setup

Two identical versions of Drupal Head, one with Overlay turned off.
You need to create some Content on both:
Our Sample site represents a local bookshop (no online shop).
You find the content for the sample site at the bottom.

Tasks

How to best do the A/B testing

There are always two similar (but not identical tasks) performed in the
version that has the overlay on, and the other one that has it switched
off. The art is the fluent switching so it feels natural to the user. I have not tried out the plan yet, but would think that doing two tasks in the overlay then switching to the non-overlay for two tasks might be a good rhythm.

Tasks that result in clicking through several Admin screens

Overlay site: Change the site title to "Waterbury's Bookshop"
Plain Seven site: The background color of the site can be changed. Change it to green.

Test if Overlay helps in keeping context (and if context helps in orientation)

Overlay site: 
Send the user to the full node view of the Harry Potter post. This should be easily memorizable for the large image on there, so the user should recognize when they found their way back.
Add a user "alda" for the shop owner. If they succeed in adding the user (and also if not) try to make them find the close icon on the overlay, without direcly telling them. If they do not use the close icon (cross on red side) make them go back to the users page and ask them if they see another way of returing to the live site. (another valid option is the house symbol in the upper left corner, would be interesting if they use this).
If they still do not use the close button, make them go back once more and try to guide them to it, if still not used, tell them directly to click it. We need the close button to find out if simply closing the overlay helps in orientation.

Plain Seven site:
Send the user to the full node view of the Harry Potter post. This
should be easily memorizable for the large image on there, so the user
should recognize when they found their way back.
Send the participant to see if they can find a log, where everything that happens on the site is recorded. If they do not find it, try to guide them. When they have found it, tell them to go back to the Harry Potter full node view. As in the other task, send them back once more and ask, if they see another way in navigating back (as well as before, the home button may be an option, or the menu)


Tasks that shortly call the overlay and go back to frontend

These tasks use the contextual links (gear icon that appears on hovering a block). Try to guide the participant to it without directly telling him what to click. We want to know if the gear is findable and intuitive enough.
If they do not find out by themselves in reasonable time, give them a hint. Else they cannot continue.

Overlay site: Your phone number has changed to 955-1808. Change it in the block on the right side.
(Objective
also: find out if the gear for Centextual links is findable and people
manage to use it intuitively. If the do not manage, give them a hint.)

Plain Seven site: Change the placement of the search box from the left sidebar to the right sidebar.

Overlay site: (first make sure we are on the front page) Add a sentence in the end of the Harry potter post:  "Tents to spend the night in front of the shop will be available.". 
See if they use the gear icon.

Plain Seven site:  (first make sure we are on the front page) There has been a price drop for Dances With Wolves: change the price to £ 0,99 See if they use the gear icon.

Find a task to cancel.
(Objective: Check if people manage to find the close button of the overlay. Keep special eye on additional risk to lose data.)

Content for sample site

Only local images are allowed.

See Screenshot for how the frontpage should look like.

Four Nodes:

Pages:
- Opening hours
- Special offers of the day

Storys (also on Front Page, each with image)
- Announcement of new Harry Potter and the Garden of Eden
- Team

Add all nodes to the main menu.

One block
(right-hand-side) with phone number and adress

Nodes

News for the front page:

Waterbury's first to put out Harry Potter and the Garden of Eden

Image (Harry Potter) http://auweia.org/dateien/harry-potter-eden-small.png

The best-kept secret of the literary world has been unveiled today:
Jeanne K. Rowling wrote an unpublished predecessor of the Harry Potter
Novels. It tells about how Harry became what he is and even turns out
that his personal fate is deeply intertwined with the very history of
mankind... Waterbury books exclusively presents to you this unexpected
beauty which will cast new light upon all of Harry's adventures on next
Monday.

Offers of the day

Image (Sales symbol) http://auweia.org/dateien/special-sale.png

The catcher in the cupboard

Handsigned anniversary edition of the classic novel. Martha faces an
unexpected threat when shen wants to sort her laundry and has to face
her torn childhood to survive all the temptations that follow.

£ 12,50

Dances with wolves

A man torn between the skin color he is born into and the cruelties
that his race commits on the american natives. Will our hero make the
right decision and can he change the course of history? Special offer,
paperback.

£ 1,50

Hannah Montana's Cookbook

Does Hannah cook at all, and if so, what's in the pot? Be in for culinary surprises that make a better day. Fully colored.

£ 21,90

Pages:

Opening Times

During our opening times we take orders also by telephone. At other times you can always send orders and requests by email.

Mo - Fr: 9am - 5pm
Sa:  9am - 2pm
Su:  closed

For special occasions or events like Author signing hours or readings these hours may be changed.

Team

Our
staff loves to help you with finding your newst reading adventure. We
love books and are passionate about offering you the best service.

Alda Waterbury

Founder and owner of Waterbury's bookshop she is guiding it through
all kinds of seasons. In close contact with the purchasing team she
makes sure the choice is always appealing.

Peter Shelveshove

Responsible for presentation, Mr. Wesworth arranges shelves and
tables in a way you always find your favourite books quickly. He also
makes sure our shop-window always has a fresh look.

Susan Marketwick

Waterbury's event manager. When we have authors reading or signing
their work or release parties for the latest craze in the world of
literature, she makes sure this is really fun.

Block

Contact us

Waterbury's Books
63 Ashton Lane
Liverpool L4 2HY
England

phone:  0151 351 8196
email: sales@waterbury.co.uk

Images

Sales symbol: http://auweia.org/dateien/special-sale.png
Harry Potter:  http://auweia.org/dateien/harry-potter-eden-small.png

eigentor’s picture

FileSize
60.83 KB

Screenshot attachment

gagarine’s picture

tracking

mcrittenden’s picture

Title: Test Plan for the overlay » Test Plan for the Overlay

Sub.

arhak’s picture

Overlay site: Change the site title to "Waterbury's Bookshop"
Plain Seven site: The background color of the site can be changed. Change it to green.

careful, a shocking background color would make users hate it from the start

kaakuu’s picture

A very nicely detailed plan for the tests.

I am not sure who will be the leaders of this studies or it will be multi-point multi-user community study with feedback analyzed via these issue lists, without any so called leaders as was in the now abandoned d7ux org site.

Instead of a singularly ( or more accurately two persons headed the D7Ux project who were "experts" but had no experience of something as complex as Drupal ) attributed leadership it will be great if there are chosen leaders via announcements or selection process in the community ( previously the choice was openly yet closed door-ly done, not known to the community till the choice was over ) where community means drupal.org site front page / forum / issue lists.

Any meaningful study can prosper and gain accuracy from comparison and control study groups. When we compare A to B or X to Y - we know which is better. The more the comparison groups ( within feasibilty, of course) the better. For example, here, with respect to overlay, there can be two groups ( blind to each other, something like double blind studies ) - one who test with overlay on
and the other who test admin tasks with admin module on ( and overlay off )
Then we can compare the ease and efficiency of the groups and eliminate any bias honestly
Without comparisons, tests can lose much meaning and even give false results.

Last but not the least, tests does not and should not mean only videos. Users who have reported or are reporting in the forums against overlay should also be counted as test data. User feedback gained thus is a natural one, compared to so called tests where we go to the user and user does not come to us, as they do spontaneously in the forum or issue lists. Well organized user polls are also part of a valid test data.

I hope someone now or in future see the light in what I have written.

seutje’s picture

subscribe

SeanBannister’s picture

sub

eigentor’s picture

kaaku: valid comments.
I myself am doing the tests in a much more impromptu way. Even creating a plan is a consequence of Bojhan and Yoroy convincing me that it's better and makes results more comparable.

For the moment _any_ testing is valuable. So if you want to step up as a leader, do so, I second you that this should be coordinated. As for lack of time I won't be too much in this atm, but if the plan with some added and modified tasks is of value that could be a help.

I put some effort into making the content fun, so the users will crack a smile here and there while actually reading what's written which accounts for something very important in user tests I found: make them not feel like in a test. Users may recall awkward exam situations in school where they failed miserably, which may cramp the entire situation.

EvanDonovan’s picture

Subscribing...I'll see if I can take on this sometime soon.

/me is excited about potentially be part of user acceptance testing

My initial reactions to the overlay are in http://drupal.org/node/659488#comment-2774686.

Alan D.’s picture

In regards to #6
The only "poll" I know of on this has been in #659488: Properly test the overlay to determine if it belongs in core or contrib. From the first 100 or so comments, there were about 14 unique negatives and 5 unique positives. It was interesting to note that all of the seasoned Drupal developers were negative accept for Dries.

However, quoting Dries "I've organized idea polls and feature ratings at the beginning of every major release cycle and will continue to do so" so there may be one somewhere.

In regards to #2
"Several tasks that show the strength : short quick jumps to the overlay and back."

The tests should be based on common administrative tasks and should not be deliberately bent to show off Overlay strengths. Otherwise there is no point in even testing. There should be good tests around somewhere, but I would have thought that it would include turning on modules, changing themes and some common user administration tasks.

davidwatson’s picture

subscribe

davidwatson’s picture

Status: Active » Needs work

Awesome work so far on this! :]

If I understand it correctly (and please correct me if I'm wrong), the current test suggests tasks with and without the overlay on two different sites with the same user. However, I'm concerned that this would introduce a form of aesthetic bias, one that I fear some people have already begun to succumb to: the "Web 2.0 effect" for lack of a better term. Yes, the overlay has aesthetic appeal, and makes for an appealing experience in turn, but that doesn't inherently make it more usable.

To be clear here, I'm neither for or against the overlay at this point. My concern is that the non-uniformity of the test may introduce biases that will skew the results. Splitting users into two groups as #6 suggests, overlay and control (Seven) with more tasks each, seems like the best way to address this.

Another big concern I have is the suggestion to provide "hints" to the user. Strictly speaking, we shouldn't be interfering at all, lest we introduce biases of our own - if the system can't help the user achieve the task in $timeframe, then simply mark it as a failure (and raise it as a new usability issue). For those concerned about the user being demoralized after "not being able to figure it out in time," I've found that emphasizing the purpose of the test in advance helps wonders. Even something to the effect of "remember, we're testing the system, not you" goes a long way.

Other than that, much respect for putting all this effort in. Looking forward to seeing this come together!

mcrittenden’s picture

Status: Needs work » Active
davidwatson’s picture

@14 - If the current iteration of the work item (the plan) has issues that need to be addressed, why isn't it "Needs Work" rather than "Active?" Not trying to be snarky here, I'm genuinely curious, since I'm still a relatively new contributor and want to be sure that I'm setting status correctly.

yoroy’s picture

Status: Active » Needs work

Running into a limitation of the queue here. "Needs work" is a status for issues that have actual patches attached to them. Since this isn't about a code change that wouldn' apply but, meh. There's an outline of a plan here that needs work indeed. A wiki page in groups.drupal.org/usabiltiy might be better to continue work in?

mcrittenden’s picture

@yoroy & c.ex, yep, now that I think about it you're probably right that this should be Needs Work. Sorry for the preemptive change :)

davidwatson’s picture

@mcrittenden - No worries! :] I was just worried that I was prematurely changing something, that's all.

+1 to yoroy's suggestion, a wiki page would be a lot more collaborative. I'll be very much in and out for the next few days (and gone the following week), but if I can steal a moment between now and then, I'd be happy to jumpstart efforts there.

EDIT: See http://groups.drupal.org/node/65263 for the wiki page. I made a few minor tweaks, but have yet to do any sort of serious editing. Go nuts!

David_Rothstein’s picture

It would be great to see something happen in this issue - thanks for setting up the wiki page!

I think Alan D is correct that it is important to not only test the overlay in relation to its strengths... I'm guessing it's quite likely it will perform well when tested in combination with contextual links, but that's the definition of a small, short task. The "add a user" test mentioned on the wiki is probably a decent in-between example, but since the overlay currently appears on all administration pages, I think it would be wise to test something more in-depth also -- maybe turning on a simple module and going to configure it on a couple different screens, something like that? Also, it might be useful to test it in comparison to one of the alternative simpler proposals that have been put forward to address the same "context" issues which the overlay is designed to solve - for example, the patch at #787896: Add a link so that administrators can return to their most recently visited non-admin page (if the implementation there can be polished up a bit, at least).

Sorry to just throw out ideas there - but if it's possible, it would be great! This is a very, very important issue - having some real data and evidence here would go a long way. I'm not particularly "for" or "against" the overlay myself, but I think Alan D's statement that all seasoned Drupal developers in #659488: Properly test the overlay to determine if it belongs in core or contrib were negative on the overlay except for Dries is not correct - for one thing, there are some pretty big names in the Drupal community, both in that issue and elsewhere, that are very "pro-overlay", but more to the point, I think most experienced developers have a more nuanced opinion and are really waiting for solid usability testing to help figure this thing out.

carlos8f’s picture

Priority: Normal » Major

Marked #659488: Properly test the overlay to determine if it belongs in core or contrib as a duplicate in favor of this issue. Let's get cracking with this!

aaron’s picture

subscribe

webchick’s picture

Priority: Major » Normal

Since Acquia has some 20,000-something live Drupal 7 sites, all of which are using the Overlay, I asked Peter Wolanin to connect me with Linea Rowe for some statistics of some kind based on actual data from the field. Here's what she said:

I don't have a lot of data for you around overlays, but do have a few notes. We've only seen 2-3 questions about turning the overlay off out of thousands of forum posts. These were requests from experienced Drupal people and not from newbies. I've done a few general usability tests on Gardens with people who were seeing Drupal for the first time and they navigated very easily with the overlays. Jeff is starting a new round of usability testing next week and the topic of overlays will come up as users try to use the site. If we get any interesting nuggets of information, I'll send it your way.

Since I didn't get any follow-up, I'm assuming either there were no nuggets of information, or else she got busy, like I did. :)

But at any rate, if this was as big a problem as core developers (who are perfectly adept at drush dis) generally say it is, there should be more than 2-3 questions out of 1000s of posts about it. It should also come up way more frequently in the numerous usability tests Acquia does on new Gardens users.

It's still a great idea to test Drupal 7's administrative usability, mind you, but I don't see any particular reason to focus so much on this one specific feature.

EvanDonovan’s picture

I agree with webchick on this one. Over time, I've come around to liking the overlay. It works well in conjunction with the contextual links. I think that it is mostly hard for heavy users of Drupal 6 and previous to get their minds around.

At this stage in the game, more generalized usability testing would be beneficial rather than a focus on just the overlay.

Jeff Burnz’s picture

@22, wow that's great news. Agreed that focusing on the "overlay" per-se is wrong, the focus should fall on the features within the admin UI. I dont mind the overlay (FWIW), I prefer without it and for me the one feature request I would have is that I could use it only for contextual links and not the entire admin, say just add/edit content, manage blocks etc - but thats another story and already well hashed out arguement...

Good news, thanks for that webchick.

David_Rothstein’s picture

Doing usability tests of other parts of the admin UI is of course a good idea (I've done it and it was very interesting)! However, if we had to pick one thing to focus on, I think it would be the overlay. The reason is that it is the biggest change, and also the one that builds off of previous work the least.

Consider some of the other UI changes:

  • Contextual links: This concept has been around for a long time in various guises.
  • Dashboard: Same thing.
  • Toolbar + Shortcuts: Same thing, mostly (although there are some key differences with admin_menu, so this is another one that is worth testing).

In the case of the overlay?... Not so much (at least not the way the core overlay module does it, where all admin pages always display inside the modal dialog).

The Drupal Gardens forum post data is very useful, but I don't think it tells us anything about the usability improvement of the overlay itself. It does tell us that the overlay is not going to send thousands of users kicking and screaming away from Drupal 7 in horror, but I don't think anyone at #659488: Properly test the overlay to determine if it belongs in core or contrib ever thought that (well, maybe a couple people did, but the people posting sane, reasonable comments did not).

Rather, the concern there was that the overlay is a large and somewhat radical change, and the bar for making major changes to Drupal core like that should be very high, and that bar had not been met. In particular, although the usability problem that the overlay tries to solve is certainly a real problem (validated by testing), the idea that the overlay does a good job solving it (or a better job than other simpler solutions) has not been validated at all, and even though I personally think it is likely that usability testing would validate it, the real and serious concern in that issue was that this testing had not been performed and the lack of such testing was a big hole. For example, Moshe wrote this towards the beginning of that issue, which summed it up nicely:

We don't have to "trust the UX team". We can and should test the overlay with real users. I believe that was always part of the plan and I really want us to stick with that.

That's what this issue is about.

David_Rothstein’s picture

Also, to expand on a couple Drupal Gardens-related things:

  • ~20,000 is the number of Drupal Gardens sites that have been created (actually higher now), but that doesn't mean all those people actually used their site in any significant way. In other words, the number who actually experienced the overlay "for real" is presumably smaller (although still definitely a large number).
  • In looking at that data, we do need to be a little careful due to terminology. Experienced Drupal users are the ones most likely to even know that the overlay is called the "overlay" and therefore to refer to it as such when complaining about it :) For example, we got some Drupal Gardens feedback recently from a user (I do not know for sure, but I do not believed they were an experienced Drupal person) who said this:

    if I close the site manager, it should not pop open again every time I hit a primary nav button. A high annoyance factor.

    I believe that translated into Drupal-speak, that would say "I don't like the overlay", but you wouldn't necessarily know it unless you read it carefully :)

yoroy’s picture

I read that as someone interpreting the admin toolbar/shortcuts as the primary nav for his front-end site :)

EvanDonovan’s picture

I think yoroy is correct in his reading of that sentence, which would suggest that the person in question didn't realize the purpose of the Toolbar/Shortcuts (which is still a usability issue).

tyabut’s picture

Is anyone using an existing moderator's script for this? Just curious.

Alex UA’s picture

Just want to give a big +1 to David's comment in #25- very well said.

Sub...

galundin’s picture

Status: Needs work » Active

Wait, what? That was the end of the discussion? November 2010, everybody agreed overlay needs to be user-tested. That never really happened, so.... stick it in core!!

I know it's too late, and I know this is probably the wrong place to express this, but here's a post-mortem reflection on the criteria a feature should be required to meet before it is accepted into core:

  1. Need

    Is it clear that a significant percentage of the user base have a problem that the feature is designed to solve?

  2. Suitability

    Is it clear that the feature solves the problem, in the best way possible, considering all the factors and alternatives?

  3. Unharmful

    Is it clear that the feature does not introduce new, fundamental problems that are worse than the one it supposedly solves?

  4. Precedent

    Is there already an established user base for this feature (or something very similar) in a contrib module?

It might be reasonable to make rare exceptions to point 4, but points 1-3 must each be demonstrated, with fairly strong consensus, before we impose a feature upon all Drupal users by default in core.

galundin’s picture

Status: Active » Closed (won't fix)
David_Rothstein’s picture

Version: 7.x-dev » 8.x-dev
Status: Closed (won't fix) » Needs work

@galundin,

There's actually been quite a lot of usability testing of the overlay since November 2010 even though it did not take place via this issue (see, for example, http://drupal.org/node/1175694).

I think this issue should remain open, though, because what still didn't quite happen was targeted overlay testing (which could, for example, compare the overlay to other simpler solutions attempting to solve the same specific usability issues, e.g. #787896: Add a link so that administrators can return to their most recently visited non-admin page). But I think this is lower priority given the testing that has already been done.

I think your list of criteria is pretty good and broadly consistent with the usability "gates" that have been adopted for Drupal 8 (see http://drupal.org/core-gates). You might consider commenting on some of the related discussions for those if you think they should be tweaked a bit. In the case of the overlay, I would say, retrospectively: #1 was absolutely followed, #3 was somewhat followed (although maybe not in the best possible time frame), and #2 is what we're still talking about (although by definition I think that's worded as an ideal, since "the best way possible" can never really be known for sure).

Alan D.’s picture

Even with all of the bug fixes that have been done on the overlay, I still find the performance worse than go to an admin page and return to the source page on submit. For this reason, I'm still am religiously turning off the overlay.

IE: Without:
Click edit link
Load admin page
Save and goto page
----------------------
Two drupal page bootstraps

With:
Load page in overlay (slightly faster)
Save
Ajax return and close window
Base page reloads
----------------------
Three drupal page bootstraps

So I think #2 is still worth investigating, even if it ends up as a contrib module rather than core :)

nod_’s picture

Issue summary: View changes
Status: Needs work » Closed (won't fix)

Overlay is dead to D8 #2088121: Remove Overlay.