Usability Testing Suite Flow

Looking at 9 I'm not sure about an "I've completed the task button" - I'd be more in favour of a time limit (albeit a generous one), and an "I'm done" button - which'd be equivalent to 'go on to the next question' on some multiple choice tests.

Very similar, but tying the end of a task to 'completion' runs the risk of placing too much emphasis on actually completing it - whereas arguably some of the more valuable data comes when people don't get to the end of a task at all. If the only way out of a task is to mark it completed, then it also might be an incentive for people to 'cheat' (i.e. look things up in the handbooks, ask on the forums on in #support etc. - things that are quite natural to do remotely if you know anything about Drupal at all).

subscribing

At UMN it was per-study, although I think we intervened once or twice when we saw some poor soul bashing their face in on the same thing for longer than 45 minutes.

I'd meant to add "this is really minor, and otherwise it all looks great" to that post - and yeah, essentially the same thing. Ideally we don't want any particular study to last much longer than an hour, ideally less (although a hard time limit might be 90 minutes or something), so individual deadlines for specific tasks is probably a bit too fine grained. Maybe the ability to add 'suggested time' to each task when developing it, and have the overall time-limit as an aggregation of those when they collected into a study?

If there's a hard time limit for the study but not for individual tasks, maybe we shouldn't put the emphasis on having to complete all tasks?
They may want to "skip" a difficult task that seems to take too long (I mean push the "I'm done" button before the task is complete), if they feel they have to try every one of them.

The list looks good!

Workflow for the lifecycle of a UTS Study;

• UTE decides to do some usability testing with the UTS
• UTE creates Study in the UTS and the UTE discovers / remembers (s)he needs;
  • to think about who his/her users are
  • UT participants that typify those users
  • to work out what data feeds are needed to make testing worthwhile
  • to work out what are the most common tasks of those users
• UTE creates Eligibility Requirements, Technical Requirements and creates Tasks.
• UTE may also consider at this point;
  • Time Limits per Task
  • a Total Time Limit per Study
  • deciding and defining (by recording) ideal Steps / path to complete a Task
  • specific Feedback Questions relating to specific Tasks
• UTE invites Participants by sending an Invite email to the participants from the UTS. (Participant email address is not associated with test data). UTE may customize this email or just copy/paste elsewhere.
• Participants receive Invite and/or link and click it.
• UTS checks technical requirements in background, asking the participant to wait patiently.
• Participant is asked to install software or make changes as per Technical Requirements the UTE defined above.
• If Technical Requirements are met;
• Participant completes Eligibility Requirements, checking off yes/no questions/statements like "I have never used a CMS"
• Participant reads generic introductory blurb about UT, "You are not being tested -- we are. You have some tasks to do. You don't have to complete them." etc..
• UTS begins timing and data capturing
• Participant reads first task
• Participant attempts first task
• Participant finishes first task (clicks "Next task"), or runs out of time.
• Participant completes Task feedback questionnaire
• Participant writes some general feedback about the Task
• Participant reads second task...
• Participant completes Study feedback questionnaire
• Participant writes some general feedback about the Study
• Participant is thanked
• UTE is notified
• UTE views the session data
• UTE tells the UTS to process the data (e.g. merge video, audio and text feeds into one video)
• UTE views the session data
• Once multiple Test Session data is collected
• UTE tells the UTS to analyze data (e.g. What was the mean/median/range of time to complete each task? How many Participants completed Task 2? What were the first thing each Participant clicked on at the beginning of Task 1? What was the contents of the node-edit form that was submitted in Task 3?)
• UTE takes notes and makes actions elsewhere.

A few specific issues with boombatower's flow:

I think we need to force UTEs to make a study before creating tasks so that they must plan a bit and be sure they know what and how they are testing. Individual tasks have little meaning outside of the context and requirements of a Study.

There is no need to ever close a Study.

Some data feeds or combinations of data feeds will require post-processing of each data session. Some analytical processors will need time to analyze data across multiple Test Sessions, e.g. a Data querier that analyses the steps a Participant took and tries to find what path MOST users took. Most of that is well out of scope for GSoC, but needs to be considered.

Here is a list of generic tasks the UTE will expect to achieve in the UTS, approximately in order of importance/frequency;

• CRUD Tasks
• CRUD Eligibility Requirements
• CRUD Technical Requirements
• Email test environment link and details to participant/s
• Be notified when test is complete.
• View test data
• Analyse test data (from multiple test sessions)
• Test/check a study (test session as dry run)
• CRUD Study
• Copy Study (in entirity?)
• Test Data Capturers/Data Store

And participant tasks:

• Recieve link and info about test session
• Read and check eligibility requirements
• Check technical requirements
• Start task
• End task
• Answer survey questions
• Type general feedback
• Type thoughts in realtime

While I'm dumping my notes. Here's an inexhaustive list of Entities that need their Relationships better defined:

• Projects
• Study
• Task
  • Time limit
  • Feeback questions
  • Ideal path / steps
  • Eligibility requirement
  • Technical requirement
  • Survey Question
  • Steps
• Participant
• Test session
• Test Session Data
• Timeline
• Analysis
• Data capturer
• Data feed
• Data processor
• Data analyser
• Data store

I have taken a (big) stab at wireframes for the UI here: http://drupal.org/node/270716

Workflow described in this issue will continue at http://drupal.org/node/283528, where we will work on getting the information that has been placed here acctually into the UTS's workflow.