Study & Task scope and implemenation

That's how it went at UMN...

1. Figure out the list of tasks you want to test people on.
2. Combine that together in a "Study"
3. Choose participants from your existing pool who meet requirements for this study
4. Go do it. :)

So yeah, eligibility requirements were based on the study as a whole, not on any individual task. A task only really makes sense in the larger context of a study, anyway.

I think it makes sense to allow tasks to be re-used among Studies; both the "Setup a blog with Drupal" study and the "Create a library workshop catalog" study probably have a task "Create a vocabulary to categorize the content." But we should have the ability to (optionally) modify the tasks slightly when they're re-used in another study. For example, for the blog I want "Tags", for the library workshop I want "Departments" or something.

> Should tasks be able to be re-used between studies?
> If so what kind of UI is needed to then related them to a certain study?

Let's see some ideas and sketches.

> Could just give them names and provide a combo for each task that a study will have.

Tasks will each need a unique ID number, integer is fine, text stirng is probably overkill for this.

> Another option would be to use the export feature (to be) to keep a copy of a study and import its definition back in when tasks are to be re-used. This could also be accomplished with a copy feature.

I can imagine valid use-cases for both;
* using the SAME task in multiple studies (split testing, studies that are MOSTLY similar except for one or two tasks and elegibility requirements) by, AND
* duplicating a task to modify in a the same or a different study (using a previously created task as a template)

This might be out of scope? I think we should design the UI with this in mind and write the code with this in mind too, even if we can't implement it yet.

> Since a study may have as many participants as the UTE wants I don't see a compelling reason for separating the task pages since they could always be related to a study.

If the Task:Study relationship is many:many it will be confusing for the user that they edited a task in one study and it changed in another study. The UI needs to clearly communicate when a task is copied (a new task is created) and when a task is simply used in two studies.

> This relates to http://drupal.org/node/263968 since my vision doesn't provide for eligibility requirements on individual tasks. Instead I was thinking that tasks would be grouped into studies with eligibility requirements set accordingly. That would be simpler in many respects and makes logical sense. A user is eligible for a study then they should be able to do any of the tasks within the study.

If many:many rels on Task:Studies is overkill and we don't do that, then I agree. If we do do it or want to do it in a future iteration, then we should have separate pages to better communicate when a Task is in multiple studies so that the UTE doesn't accidentally break another study.

Remember that tasks may also have info that attempts to identify (from the Drupal log and/or other data capturers) when a task is complete.

> So yeah, eligibility requirements were based on the study as a whole, not on any individual task. A task only really makes sense in the larger context of a study, anyway.

You're right, eligibility requirements for tasks is stupid -- what was I thinking! :)

I think this comes down to "How complex are tasks?" What fields they have etc...

An "entities and rels" diagram would be real handy about now...

Off the top of my head, Title, Description, Flags to look for in Data capturers that suggest the task was complete, e.g. "Vocab was created" with title containing "Departments" from "Drupal log". This requires data capturers implement hooks and provide UIs to define and configure a "Watcher" that looks for these flags (triggers and actions?). This is probably out of scope for now.

What does Morae do here? What data does a task have?

I kind of wonder if there's any reason that tasks shouldn't be nodes and use Node Clone module to handle the copying?

Tasks as nodes would allow:
- Comments. "This task needs to be refined; users get hung up on doing x, y, and z."
- Revisions. See how a task has evolved over time.
- Taxonomy categorization. So we could build a view of all "User interface" tasks or all "User Registration" tasks.
- A bazillion other things.

Thoughts?

The comments could be sort of a development cycle and maybe that should be unrelated to this module. As in design the tasks elsewhere (or at least on another node) then copy the final product into the UTS.

Sure, makes sense. I mostly just used that as an example.

Another concern would be how the tasks are created. Should the module provide an interface that works with studies and then just create the nodes behind the sense or should the UTE create the nodes and then link them to studies?

It'd definitely be more intuitive, I think, for users to create a study first, then associated tasks (with the ability to import in an existing one).

I think what we might be missing here (or what I might be missing here ;)) is a "picture" of what it is we're actually building here. This could either be a use case diagram, or an E-R diagram, or even just a list of steps. But I think it'd be good to get that nailed down before we get too bogged down in implementation details.

Jimmy, could you take a stab at writing up some "user stories" for this project, and then Bevan/Bohjan can respond to see if that's what they're thinking as well?

Hello,

Sorry for my late response, but it took me some time to figure out all of this information and process them into work-able wireframes. There is still some information missing on the last part, of how do we want to run trough the actual study in terms of participant selection and notifications but it should provide a "picture" of what we are actually building.

The only question I have is what does a task consist of?
Tittle, Description, Maximum time (We don't want people endlessly trying to complete a task when they fail at the first step), Steps - so what actions does the user need to take to complete the task, the ideal steps.

Should tasks be able to be re-used between studies?
Yes, as we do a usability study and improve on that we want to make sure that these improvements are correct so you then do another usability study with the exact same tasks.

Should UTS indicate when a task is complete, or the user?
A user in a lot of cases has no idea if he acctually completed the task, since a lot of modules have no visual feedback that the user completed the task. Although this is an issue, this is not addressable with UTS easily. Therefor I would vote to let the UTS indicate when a task is done and communicate this back towards the user. This also allows us to more accurately track time.

What does Morae do here? What data does a task have?
None, the task is given to the participant and the observer indicates if a task is completed and if the user has any other task indicators (failing, difficulty ect), so there is essentially no data, other then what the observer finds. Morae is a tool to do usability testing, not a program that identifies stuff, you have to do that yourself almost always.

Some Wireframes
Process : I made these wireframes from looking at the information given in this topic (issue que?) and talking to 3 users(2 programmers, 1 interaction designer) that where not Drupal users, I asked them questions as how would you setup a Usability Study - Task Analyses. Important to note these where not usability experts so I went from my assumption that most people using this module wouldn't be Usability Experts.

Some interesting questions that they already asked on the first setup of the wireframe was, what are studies? Which slightly killed my confidence in that everyone would know how to roughly setup a usability study from the first place, but from that point on they gave me some information on how they would do it.

After that I spoke with yoroy (drupal user) how he would setup a Usability study and what parts of the wireframes I build he found hard to interpret.

http://bojhan.nl/Drupal/createstudy.jpg

General

Type
At the moment the UTS will only provide Task Analyses, but in the future we could/should also have A/B testing and other usability methods that we could apply. So under type Task Analyses should first(currently only - type), however hopefully in the near future we can take easy usability methods and also include them under type.

Note : I renamed it from Task Analyses to Task Completion after webchick and 1 of the particpants noted they had no idea what a task analyses is, where task completion makes a lot more sense to them.

Number of participants
You don't want to run the study on an infinitive amount of participants. Usually to encounter the obvious problems you only need 3 to 5 participants, on which you improve and do another study with 3 to 5 participants to encounter a new set of problems and identify if your improvements are correct. To note here, this amount of participants is unlikely to supply statistical significance however statistical significance in most cases inst needed anyway.

Audience
The general assumption here is that we would allow the creator of a study to set an audience he things most likely will use the module, on page 3(Run Study) I put how we will recruit this audience.

Experience Level
There are three levels of experience : Beginner, Intermediate, Expert . We have to make a difference between this because we intend to focus on beginners/intermediates in order to lower the process of learning drupal(beginner) and using it effectively early on(intermediate) and not for hardcore users who wouldn't mind an easier interface anyway. To note here, Alan Cooper (About Faces : The essentials of interaction design) notes that interfaces should focus on intermediates, however with Drupal we currently have to many beginners barriers in order to effectively focus on the intermediate (very small space between beginners and expert users).

Role
Does this label needs to be renamed? The roles would be Designer, Programmer, Administrator maybe more, however I want to keep these lists as short as possible since classification amongst roles is always hard for the user.

Questions
I took this section apart from the Audience as two people noted, they saw it although connected to audience as visually a separate level.

Well the interface should explain it, it allows the creator of the study to make questions to ask the participants before they start the study. I am somewhat concerned on how to create an interface that filters out participants when it comes to specific questions (mostly this is done by hand) so we have to figure out how we could present that.

http://bojhan.nl/Drupal/createtasks.jpg

Title, Decription, Max Time

Steps
We want to track what actions are required to complete a study (the ideal steps), so we can see at what point people fail in completing the task.

So there is an inherent part of usability we cant track, which is the visceral (processing visual aspects of a interface before an interaction occurs). So the task-analyses only focuses on the behavioural (the actual interactions) and with other methods try to access reflective processing (not important atm).

I took Views2 as case to asses the problems in this interaction, if you want to preform a usability test on Views2 like create a list with filters applied ect. You would essentially have to create all these steps from long lists of functions, so thinking about that got me to realize the best way to encounter this problem is to record all the steps required (webchick : macro recording , yoroy : photoshop batch actions) and with the data collected in the recording session put out the steps required to complete the tasks. Essentially its the same as UTS would do with a participant, but just a bit less data it has to capture.

Example of a Recording : http://www.bojhan.nl/Drupal/TaskRecording.html

A firefox plug-in that does something similar is http://dev.spikesource.com/wiki/index.php/Projects:TestGen4Web . There are also some firefox plug-ins that allow logging of mouse movements, however this would make us very platform dependant.

http://bojhan.nl/Drupal/runstudy.jpg

The purpose of this page is to finalize the study and choose some data that you cant really answer until you have created the tasks. However I am unsure if we could not just leave out this page, or change its purpose. I am unsure if we should set presentation and capture settings on this page and or if this is applicable to the task at hand.

Participant selection
Automatic (loosely)
It sounds a bit bad so we might have to change the labelling, however this pretty much means that we will try to search audience that was specified but also recruit from audiences that are not specified since a lot of tools are not to be specified really to an audience and obvious problems can be identified by any type of audience.

Select myself
Is this in scope of the project? To allow the creator of the study to select the audience from a pool of participants?

Select perfect audience
This label is also horrible. This is only really applicable if you are testing tasks for a really specific audience, such as modules that will only be used by specific developers.So the study is focused on a specific audience and you can easily (with questions) make the distinction between certain audiences.

Example : If you want to test UTS with the UTS, you need to test loosely to capture the really important issues and then recruit loosly again and again to fix important issues. And also test with a perfect audience, to understand if programmers understand the terminology used.

Notification of process
Do you want to be notified, every time a participant finishes a study and see the results, or only if all 8(number of participants finish the study.

All of what I just wrote, is just a possible way of approaching study and task. It is in no way definitive on the way we should go, please comment as much as possible and I will try to change it in the wireframes. It is to identify the what not the how(design wise) of the UTS.

As noted by webchick there is an inherent need of more information such as workflows, scenarios and personas(primary persona : developer, secondary persona : designer) to supply both (context) and clarification to what UTS should do, and how. I will try to invest some time into creating these and work together with a few other people to get the interaction's better(visual appearance of data, form workflow) but I guess it’s more essential to first figure out the steps of the whole UTS (since it’s hard to build wireframes of nothing).
I would like to thank webchick, yoroy and my three anonymous users for their feedback and webchick for clarifying stuff along the way wich helped a lot.

Best Regards,

Bojhan Somers

P.S Sorry for posting this so late, I was disconnected from the internet yesterday. Boomba I have made it a habit to be around on IRC from time to time, if you have any immediate questions you can always drop me a line.

subscribing. Bohjan: amazing work and input so far. Very good.

> I was planning that tasks would always be related to a study in a many:one relationship.

I now agree this is better.

> I was talking with webchick in IRC and we came to the conclusion that the user should indicate when they think they are done with a task (ie. push a button, etc). That being part of the test.

I agree. However it often occurs that a user believes they have not completed a task, when they actually have. Or that a user believes they have completed a task when they actually have not.

If the task completion is easily detectable via the Drupal log, we should allow UTEs to tell the UTS what Drupal log events to look for, and under what circumstances it should put a specially marked item on the timeline. From the participant's POV nothing changes. They still click a button when THEY think they've finished.

> I'm seeing tasks as having a description of the task and possible a title if needed by allowing tasks to be reused. If not reused then the only thing a task needs is a description. If the description (and possible title) is the only data related to a task then I don't see much need for creating a separate interface for managing tasks. If a task need to be used twice then simply copy the description text.

I agree, except that I think we should keep in mind a Task may have more data in the future. See the Morae videos for more examples.

> I kind of wonder if there's any reason that tasks shouldn't be nodes and use Node Clone module to handle the copying?

Let's try separate implementation discussion into a future thread and focus on design, tasks and our users (UTEs, and less directly; participants). Personally I like the "Everything is a node, even when it's not" approach, but I'm not sure Drupal's node is sufficiently abstract for such an extreme design style yet. But that's implementation. Bah!

> I think what we might be missing here (or what I might be missing here ;)) is a "picture" of what it is we're actually building here. This could either be a use case diagram, or an E-R diagram, or even just a list of steps. But I think it'd be good to get that nailed down before we get too bogged down in implementation details.

Yes. I agree. I have this in my head, but haven't had time to write up or draw prototypes or diagrams. I was also hoping Jimmy would be able to gather this info and do this as part of the GSoC project, but I'm probably communicating enough of it well enough...

Look like Bojhan has taken on some of this though. Awesome! Thanks Bojhan!

Before I look at your wireframes in detail I want to sketch my own. So no comments yet... ;)

> There are also some firefox plug-ins that allow logging of mouse movements, however this would make us very platform dependant.

I believe this can be accomplished with javascript platform-independently.

no time to read this now but subscribing.

Before I look at your wireframes in detail I want to sketch my own. So no comments yet... ;)

Bevan, do you have an ETA for this? SoC is already 2 weeks in; Jimmy needs to get rolling on implementation here.

I've been helping him with the basics of getting the study/task forms together and workflow making sense, but he's fast and will be done with that soon. :)

I've been away for 3 days but will work on those today.

Bevan I also send you an e-mail regarding a specific part, boomba is it possible to record as how I tried to display it?

ERD attached, starting on UI sketches.

Hopefully this makes sense. Ordinality of relationships are read in the direction of the arrow. I've been a little lazy with attributes and only added ones that are likely to appear in the UI and came to mind. A lot are implied. This is not the schema, since Analyser, Capturer and some related entities (orange and red) would be hardcoded. These attributes would be retrieved with hooks and callbacks, not table rows.

These are as far as I got so far with the UI. Are these understandable? Or are they too scruffy? Feedback on the direction would be appreciated. Jimmy or Bojhan if you (or your colleagues Bojhan) want to take it from here and have time that would be great. I'm struggling to find the time this needs. I'll put a few more hours in tomorrow to follow up on feedback, do drafts of other UIs (like edit-Question, edit questions, edit tasks, invite participants, View/Edit project and possibly some of the analysis and participant UIs), tidy up the Study wireframe, do a few different versions of it at different stages of a Study's life cycle (like new/empty, after created tasks (draft attached), after created Survey / Feedback Qs, after created Eligibility/Technical Requirements, after inviting participants, after participants start (or complete) a session, after analyses).

Jimmy,

Please take some time to look at http://dev.spikesource.com/wiki/index.php/Projects:TestGen4Web which may be useful for us.

Bojhan,

Sorry for taking so long to respond to your wireframes.

I think the layout makes it harder to read than necessary and don't communicate the system's model clearly. In the sketches I am working on I have taken a more overview and zoom approach to the UI, which I think/hope will help communicate the system's model better.

Also, you have included input fields for a lot of information the system doesn't need to know, such as;

* number of participants; the system can count them retrospectively and doesn't need to know in advance
* Study Type; I'm not sure how you expect the system might use this info
* Audience Role and Experience level; While this is relevant info, the System doesn't need to know
* Participant Selection

This information might be overwhelming for inexperienced UTEs. If you think it is useful for more experienced usability professionals to be able to record this info, let's put it to the side or hide it in the initial UIs.

If you think the system should behave differently according to this info, how?

I find the inline-add-form style you have for adding additional questions confusing and time consuming to understand and learn to use. Mostly because the layout and style of this UI pattern is inconsistent across all web interfaces. I went for a simple [+] button in the sketches. I also think questions/tasks need their own page to change all details, like time limit, question type, title and other added details perhaps by addons tot he UTS. Especially so if we are going to fuzzy-up the difference between Qs and Tasks.

Having edit-forms for multiple items of the same type, as in the Create Tasks UI, is confusing for the user and results in oversized forms -- this is a classic mistake made in several drupal forms, such as imagecache 1. I feel strongly that each item should have it's own page/window/form and their should be an overview of all of them in the study.

Your wireframes imply that a study has a definitive number of participants and a definitive beginning and end. I'm thinking many UTEs may prefer more blurriness here, since they will be liasing with participants virtually by IM email etc outside of the UTS, and participants may take a week to do the test in their own time. I think it makes sense for the UTE to be notified after each test session is complete, and at their discretion they can analyse data collected to date (regardless of whether all the invites were responded to or not), an individual session, or none, instead choosing to wait till a few more participants have completed a test session.

Hence the only options I would offer for Participation Notifications would be a text field;

Notify these addresses [Enter comma-separated addresses here... ]

This would be populated with the current user's email address but would allow a team of UTEs to be notified.

I do very much like the idea of recording ideal steps using the same technology we use to record the participant's steps. However I think it may be more difficult than you anticipate. We'll need to push that off to latter stages of development since we need a basic UTS to build the drupal logger, to build this feature! :)

Cheers

Well, my idea was to have good studies, the more blurrier you make this kind of basic stuff of a study the less likely you will get anything out of it that you can interpreted wisely (without starting or stopping places I am not sure how you can do any task-analyses). I think your idea of participant selection is completely different from mine (as is your idea of the UTS), which is fine but then I have to completely rethink the module.

My wire frames where not meant to communicate the system model, it was meant to throw ideas on how I thought you could get the most valuable data easily (The point of anything towards actual design, shouldn't be the implementation model).. However I do get your points, as I was acctually in the process of creating a less overwhelming UI for developers, but this is initially for a study right? Not just to facilitate in the testing process, because those two have completely different needs.

I see you pointed out some of actual design parts, I wont comment on these as I have a complete different understanding of the UTS and well specific design parts, where not what I wanted to communicate (I wanted to show ideas/directions I would find valuable), I am not a designer - so I totally agree there can be a lot of improvements on that part.

You pointed out that the UTE will do a lot outside the UTS, for me this sounds like the use of UTS could easily be taken by any recording program or turn into Morae. My vision was to make the UTS easily allow developers to test stuff, instead of going to the difficult process of finding the right participants (as then its best to go with interview-based tasks).

I read trough a lot of your comments and your workflow, but can you for example give a scenario from the UTE's view and from the Particpants view (not workflow, but a story) as I am quite confused in the direction you wish to go. Especially as a lot is pointed out as out of scope, is there an challenge for boomba in any of this?

Do you want to allow developers to easily and test a lot, or do you want to go for formal usability testing in which has to be done careful and is quite time consuming? I prefer the first one, but you might be aiming more on the second one. I want every module that is in beta, to have a UTS done in timespan of a week or 3, as the best way to improve usability is to test - improve - test - improve - test - improve....

I have taken a (big) stab at wireframes for the UI here: http://drupal.org/node/270716

> Do you want to allow developers to easily and test a lot, or do you want to go for formal usability testing in which has to be done careful and is quite time consuming? I prefer the first one, but you might be aiming more on the second one. I want every module that is in beta, to have a UTS done in timespan of a week or 3, as the best way to improve usability is to test - improve - test - improve - test - improve....

This is great, but we can't get from here to there without some intermediary goals. Also, I'm not convinced that a code-centric developer/programmer will bother or be able to do effective testing and get useful results without significant learning and/or the assistance or input from someone with more UT experience. So while it would be great, I'm still a bit skeptical about targetting that audience as a *primary* audience.