Jimmy,
I just completed a fairly comprehensive review of most of the UIs and many of the most-likely tasks UTEs and participants are likely to have. My notes are below. Most of them are minor to medium UI changes that will or could result in significant improvements. Some are larger issues to be considered.
Overall, progress since my last review is mind-blowing! This is getting rather close to being good enough to run tests with remote participants. The most important problem and problem area is the test session. The UI that guides a participant through a study needs to be extremely distinct from the UI that is being tested. The user must have 100% clarity about which text and interfaces are from their 'guide' and which text and interfaces are part of the tool they are testing. It is not clear at this stage and I have pointed out some specific issues below and offered some suggestions.
Further, it's important to give the user enough info to do the task, but not so much they might get overwhelmed. E.g. the time remaining or total number of tasks is too much info. Easy access to the task description, that doesn't lead the participant away from the test subject is also critical and part of that 'guide' UI.
Please consider taking one hour to listen to these podcasts to understand why this is important. The UTS 'guide' is the Lab 'moderator'. http://www.uie.com/brainsparks/2008/07/07/usability-tools-podcast-modera... http://www.uie.com/brainsparks/2008/07/22/usability-tools-podcast-modera...
- General
- 'Usability testing', 'Analyze' and 'Tasks' menu items still have 'expandable' type bullets, but no child menu items. I think you need to set a 'type' in
hook_menu()here
$items['admin/uts/studies'] = array( 'title' => 'Studies', 'description' => 'Manage studies: create, edit, copy, and delete studies.', 'page callback' => 'uts_studies', 'access arguments' => array('manage studies') ); - consider adding a study overview page (node/NID?) with UTS details
- so that the UTE doesn't have to explicitly give the most common permissions, can anonymous and authenticated users be given 'participate in study' by default?
$db_prefixis not appended with a '_' character. This might be because the code insettings.phpis out of date?
- 'Usability testing', 'Analyze' and 'Tasks' menu items still have 'expandable' type bullets, but no child menu items. I think you need to set a 'type' in
- admin/uts/configuration
- Invite and Session info emails are great!
- More help in the UI is required to let the user of the UTS know what !username, !site, !mailto, !date, !register, !participate, !session_id will be replaced with. e.g. !participate: The urls to start a session
- "You may be required to register an account on the site..." makes the UX too complicated for the participant. Instead, make menu handler for path
/utsanduts/participate/2handle user-registration, and bring the user back touts/participate/2once complete. - Can !participate in the Session Info include the session_id so that the user doesn't need to copy and paste it, and the session resumes automatically?
- These should be defaults, shich can be overridden at invite-time on admin/uts/invite
- UTS and Proctor modules show on admin/build/modules in a test site / base environment. Consider hiding these modules via form_alter so that a participant can't break his own session.
- admin/uts/invite
- I'm not sure the general invite feature is worth the effort to maintain. Can we require the Study field?
- admin/uts/environments
- After opening a base environment instance, logging in, and quiting back to host drupal instance, the user gets logged out of the host instance. Is this to solve other bugs? Can the user be re-logged in?
- admin/uts/studies
- consider adding 'number of tasks' column
- consider adding 'number of complete sessions' column
- consider adding 'status' column
- consider hiding 'analyze' operation/link if there are no sessions to analyze
- consider adding 'add task' operation/link if study is 'in development'
- consider removing 'description' column
- admin/uts/studies/2/edit
- 2 checkboxes for enable & require would make more sense as 3 radio buttons; disable, enable, require
- admin/uts/tasks/2
- meaning of 'free' link/operation is not clear
- I'm not sure if the 'free' link is very useful or necessary. consider removing, or merging with 'delete'
- There is no link here to view or edit the study. I have to go back to admin/uts/studies. Consider putting view and edit study links here as tabs?
- similarly there is no context about the study here, except in the title, which is too subtle. Consider including the study's description?
- admin/uts/tasks/2/add
- after submission, consider adding link to drupal_set_message text and/or operations to record the steps for the task, since this can't be done from node-add
- admin/uts/tasks
- consider adding 'copy' link/operation
- after clicking 'add' link (admin/uts/tasks/add) and submitting, user of UTE is returned to admin/uts/tasks/add. Return to admin/uts/tasks/2 or admin/uts/tasks instead
- make values in 'parent study' column links to study overview page (node/NID)
- admin/uts/tasks/2/import
- This looks like it's incomplete. Consider removing menu item till complete, or completing it
- Consider complementary 'export' functions.
- admin/uts/tasks/4/edit
- 'Record steps' doesn't open a new drupal instance
- After recording steps, path includes starting page, e.g. admin/uts/tasks/2
- uts/participate/2
- Since a complicated base environment could be quite large and creating a test environment could be very complicated can we show the participant a throbber while we're waiting for the server to do it's thing?
- "Please make sure you have JavaScript enabled. To confirm that it is working click here..." Automate this. See how
install.phpchecks for clean urls as an example. - As mentioned previously, this page/form really needs to split into 2 or 3 pages/steps.
- to ask 'Start new session, or continue old session?'
- (only if restarting) to ask for the code
- third to give the user the link and ask them to bookmark it. This step should include the session id in the url to facilitate bookmarking and so that bookmarks and the link in session emails allows the participant to skip the first 2 screens. e.g.
uts/participate/2/uts607831
- Session email was never received. Incomplete?
- uts_proctor/task/3 (in participant's session instance)
- It's not clear that the content of the screen is the task and the instruction. This needs to presented apart from the subject of the testing (the session's drupal instance). Possibly as a thickbox or similar?
- Feedback UI is awesome!!!
- Feedback UI should also be available when no task is active. A task may ask the participant questions or solicit feedback. Also, we need the ability to have thingies in between tasks with specific questions, and required feedback.
- Hitting the 'enter' key while typing feedback should probably submit feedback? WDYT?
- Feedback UI accepts empty text as acceptable feedback message. It shouldn't
- It's not obvious how to make the feedback window go away once the participant has finished submitting feedback. Consider hiding automatically? Time-out? After submission? Not sure. This needs some thought and consideration
- Don't display the total number of tasks to the participant. If they run out of time (for the whole study) we don't want them to feel like they haven't been helpful if they only did some of the tasks. We want them to feel helpful regardless of how many tasks they completed or didn't complete.
- The task description needs to be readily available to the participant without distracting them from the task (i.e. as a thickbox laid over the test session screen.)
- While no task is active, the menu items in the test session's drupal instance don't work but look like the should. This further supports the need to over-obviously separate task descriptions from the test subject.
- Also, the start, and done buttons need to be more closely connected and associated to the tasks.
- 'Quit' should also be available even while a task is active to allow for easy exit.
- The visible count-down timer is very unnerving. Don't show it to the participant. Just interupt them once time is up and ask them to please move on. Also encourage the task designer (UTE) to set very high time limits.
- Reconsider making time limits optional
- Browser's forward back buttons modify the "UTS bar" but should only be changing the test sessions' instance. This will be less of a problem once we separate out tasks from the test session's instance of drupal. iframe's might be useful here with javascript event handlers for browser back/forward.
- "apreciate" is misspelled
- Different themes are required for the participant to easily distinguish the start and end of a session and tell what is part of the test subject and what is part of the environment that guides them through the study. Consider setting different garland colors in the default base install and/or encouraging the UTE to consider changing the host site's theme.
- upon quiting a study early, the participant is not reminded of their restart code/link. They should be.
Comments
Comment #1
Bojhan commentedHey, Bevan
I will try to comment on some points and I do see some of these issues, should be solved with an easier workflow instead of fixing the current screens.
1.2 : We should still look at the workflow, I spoke with a few people yesterday and they had trouble finding out the process to setup a study, I still believe that a step 1 / step 2 / step 3 type process is easier for the user.
I will make some wireframes to clarify the particpant workflow.
5: We should consider the amount of tests that will be listed here, assuming that most people only have 1 or 2 modules they test, you could assume that we have to design for about 10 studies. (Only people in the usability group will have lots more studies - but we have to redesign it a bit for usability.drupal.org anyway).
5.1 : I don't think this is necessary information.
5.2 : Add % instead of number, like 70% completed.
5.3: At the moment the status is to long to nicely be fit in an column, maybe rename them?
5.4: Agreed, but make sure to display it when there is any data.
5.5: Workflow?
5.6: Agreed, usually descriptions are to long to be easily fitted.
7: Concider workflow, people dont usually want to see all task, they want to see the tasks of one study. (BTW I couldnt attach a free task to a study?)
7.1: I found the free function really funny, it kinda looked like tasks where imprisoned task. But unnecairy functionality definitly.
12.1: Agreed, make sure that it's long enough for the user to notice its creating a new envoirment, not particulairly how long the server would do it, this is to keep the mental model of the user in mind (Like 5 seconds?)
13.1: Agreed, concider loading a black layer over the complete site, with the instructions (http://www.garrettdimon.com/themes/content/images/articles/gorilla_usabi...) - With a starting button.
13.3: Question plugin, still has to be developed.
13.4: Agreed, should trigger it.
13.6: When the user is not active in the feedback window, it should just collapse again, concider a small time frame of about 2/3 seconds when the user go's back to the screen.
13.7: Agreed, dont display.
13.8: A slightly larger top bar, where the particpant can find his session information and related functionality such as feedback?
13.10: Concider start, outside this UI, and only stop or skip to be needed?
13.11: Do people really use quit, instead of just exiting on their browser?
13.12: Agreed, a couple weeks ago I pointed boomba to this, I think he was working on removing it.
13.13: I am not sure, what about secertly doubling the given time by the UTE? I dont think its wise to leave it completely up to the particpant.
13.16: Concider the purposed, black screens with info, in between tasks.
13.17: Just wondering, cant we say like "If you have to stop, just quit and we will send you a e-mail with a continue link" or atleast something simple along those lines.
Comment #2
Bevan commentedBojhan. I mostly agree. A few further notes;
> 12.1: Agreed, make sure that it's long enough for the user to notice its creating a new envoirment, not particulairly how long the server would do it, this is to keep the mental model of the user in mind (Like 5 seconds?)
It takes a few seconds with a very basic base environment. It wouldn't need to be very large to require 10 seconds. I don't think we should intentionally be making this take longer, it's just that creating and populating 50 to 300 tables takes time. We need to let the participant know we're working on it and they need to wait for few seconds/minutes. It would be nice to estimate the time it will take to create a test session but this is not easy, although needn't be accurate. And yes 300 is a realistic upper limit for the number of tables a test environment may need to deal with.
> 13.1: Agreed, concider loading a black layer over the complete site, with the instructions (http://www.garrettdimon.com/themes/content/images/articles/gorilla_usabi...) - With a starting button.
It would be nice to show the test subject through the black overlay with some transparency, rather than opaque black. This will help the participant recall the UI when giving feedback.
> 13.6: When the user is not active in the feedback window, it should just collapse again, concider a small time frame of about 2/3 seconds when the user go's back to the screen.
That's what I had in mind. I'm not sure if this will be annoying though. If we don't start the 2-3 second timer until focus is lost from the feedback form it will probably be 'smart' enough.
> 13.8: A slightly larger top bar, where the particpant can find his session information and related functionality such as feedback?
I'm not sure larger is better in this case. If the 'guide' UI 'slides' in and out of this bar, and if they are themed similarly (white text on black / grey?, so as to be unique and different to a typical test subject) then it will be clear that the bar is part of the guide. This will involve a considerable amount of jquery animation but I believe is necessary.
> 13.11: Do people really use quit, instead of just exiting on their browser?
I believe this is important in order to provide an easy exit. Clicking 'quit' will pause the timer, remember where the user was up to, take them back to the host instance of drupal, and give them important info, like how to recover the session if they want to continue later (a link / bookamrk), and thank them for their time and help. UTE's may want to add info here such as "You're payment/reward will be made once you complete the study. You have 20 minutes remaining."
Btw 'completing the study' should not require attempting all the tasks. I think a time limit on the entire study is at least as important as a time limit on tasks. I expect it will be common for UTEs to set a time limit of one hour on a study with 5 tasks with limits of 40 minutes each. Participants are likely not to do all the tasks. That is fine. The participant shouldn't know whether there were tasks they didn't get to or not. Only the UTE will know that.
> 13.13: I am not sure, what about secertly doubling the given time by the UTE? I dont think its wise to leave it completely up to the particpant.
We can force the user to move on. But I'm not sure the UTE will always want this. I'm not sure how to deal with this issue. Maybe we need to try out the UTS in the real world before we can answer some of these questions.
> 13.16: Concider the purposed, black screens with info, in between tasks.
I wasn't refering to the participant's guiding UI whilst using the test subject (doing the tasks) but the appearance of the website they land at after clicking links in emails, and after quiting or completing a study. However it would be convenient to link the theme / appearance of the guide UI with that of the host site so the participant feels they are the same, and different to the test subject.
> 13.17: Just wondering, cant we say like "If you have to stop, just quit and we will send you a e-mail with a continue link" or atleast something simple along those lines.
Participants may or may not click the Quit button if they want to finish early. If they close the browser window we should try to 'pause' the test session for them by implementing a hook on the page unload javascript event. If the browser crashes thought this doesn't work. So I think we should be sending them an email when they start the session with the recovery/restart link.
Comment #3
boombatower commentedThanks for the thorough review. I've responded to everything in your initial post and bolded items that need a response or take a look at.
$form_state['redirect']not working with node API.Items that I will work on (does not include completed items):
Comment #4
Bevan commentedSorry for the large number of nesting etc. I'm just trying to keep this relatively readable without significant scrolling.
1. General
2. admin/uts/configuration
4. admin/uts/environments
5. admin/uts/studies
7. admin/uts/tasks/2
9. admin/uts/tasks
11. admin/uts/tasks/4/edit
12. uts/participate/2
13. uts_proctor/task/3 (in participant's session instance)
Comment #5
boombatower commented7.2 - been working on figuring out an easy way to copy task as that was my intention.
13.17 - add to list (as part of resign of the starting process) 12.*
Comment #6
boombatower commentedHere is the progress I have made.
* 2.2 (DONE)
* 2.5 (NICE-TO-HAVE)
* 2.6 (DONE)
* 3.1 (WAIT)
* 5.1 (DONE)
* 5.2 (DONE)
* 5.3 (DONE)
* 5.4 (DONE)
* 5.6 (DONE)
* 6.1 (LOW)
* 7.3 (DONE)
* 7.4 (DONE)
* 8.1 (DONE)
* 9.2 (DONE)
* 9.3 (DONE)
* 11.1
* 12.1
* 12.2
* 12.3
* 12.4 (ADD issue and postpone)
* 13.1 (DONE)
* 13.3 (DISAGREE)
* 13.5 (DONE)
* 13.11 (DONE)
* 13.12 (DONE)
* 13.15 (DONE)
After reading response
* 7.1 (DONE)
** Changes have been committed so use dev tarball if you are interested. Since we are coming down to the end of SoC I think we should try to have this reviewed somewhat frequently so that we can get all the changes completed and agreed upon before the deadline.
Comment #7
Bevan commented> Since we are coming down to the end of SoC I think we should try to have this reviewed somewhat frequently so that we can get all the changes completed and agreed upon before the deadline.
IRT meeting GSoC deadlines & criteria, don't stress about it, you've already done that! :)
I'll try to review again soon, nevertheless.
Comment #8
boombatower commentedCommitted some commented out code for 11.1. It is rather complex to provide a clean solution and will require some more thought.
I have made several other improvements, but I will postpone 12.* until after SoC. A separate issue with wire-frames and discussions on how the new interface should work would be appropriate.
Comment #9
boombatower commented13.17 should become part of issue related to new participant interface.
Comment #10
boombatower commented2.3 the redirect stuff the ?destination works, but I'm not sure what a good way to deal with that is since the menu checks the permission. I could move that into the actual page function, but if the registration requires an e-mail it will fail anyway. ??
I'm thinking that participants should be able to figure out how to register, if not I'm not sure why they are being used as participants.
Comment #11
boombatower commentedBelieve I have delt with everything that was planned on being changed.
If more issues arise lets open new issues.
Comment #12
Anonymous (not verified) commentedAutomatically closed -- issue fixed for two weeks with no activity.