Scoring of different question types

Issue Summary

Hia, mbutcher wanted a use case of scoring and assigning points, so I'll give you my opinion here (which also is our usecase(s) the last 3 years!).

Assigning points

There are several points I feel are important here. First, you need to be able to assign points to all question types, even though some of them can only be true or false. This is as you need to be able to value one question more than others. E.g. some questions are harder and / or more important and should typically give more points.

Second, to allow for a true separation between the question database and the quizes, questions should only be given points within the setting of a quiz. This is because a question can have different weights in different quizes. This means that there should not be a field for setting points when you create a question (like it is now for short and long answer questions).

Here is how I imagine the GUI for this:

http://skitch.com/vegardjo/bsyr6/quiz-4-6.12

..the black squares are ment to be textboxes where you can set point. Usually the default value is 0, but that could be debated I guess.

So to recap, a true separation of question database and quiz means that the quiz must have control of the points, they should not be set on the questions themselves!

I realize that this might be a radical suggestion from a newcomer here, but I think this is the natural continuation of your decision to split the quizes and questions, so I hope it's allright and look forward to your thoughts on it! :)

Scoring

Here's my thoughts / experience on how different question types should be scored:

Multiple choice

This is normally quite straight forward, if you choose the correct alternative you get the points the question have. There are however "debates" whether a wrongly chosen alternative should give negative points, while you only get 0 points if you answer nothing. This has been discussed here earlier too, and is because you shouldn't be able to guess an answer and get points (say, if there are 3 alternatives and you have no clue, you have a 33% chance of getting points by clicking a random one, while this is more risky if you can get negative points).

I certainly see the logic with negative points for wrongly marked alternative, but we have never used it and had few (but some) requests on it, so I'm a bit ambivalent here. I lean towards the simpler solution where a correct answer gives point, and that's it.

True / false

Same discussion as multiple choice!

Multiple answer

This is a different ballpark. Now this is scored as an all or nothing (true or false) in the quiz module, but this is not the best way I feel. If you have a question worth 5 points, with 10 alternatives where 5 are right, I feel it is fair that the user gets 4 point if 4 correct answers are checked.

You also need to make sure a user cannot simply check all alternatives and then get the full score. We recently changed this in ATutor and you can read my blog post about this to see an example and more details! The logic is:

• The median score per alternative is calculated from the max possible score on the question (in this example it's 1 for all alternatives, but it would be a different number if you had a higher max possible score on the question)
• For this example, when median = 1: one correctly marked alternative gives 1 point, one falsely marked gives -1.
• it is not possible to obtain negative score, as 0 is the lowest you can go.

Matching

Exactly the same as multiple answer :) If you match 2 out of 3 right it is fair that you get 2 points for it (given that the max score for the question was 3).

Long answer

Can only be manually scored

Short answer

My initial reaction here is that the quiz module has atleast one alternative too many, and one missing :)

Our experience is that those who create questions are writers / content creators, not developers. These people have in most cases no clue of what regex is, and the alternative to match the answer to a regular expression (which they will have to write themselves) with a reference to the php manual will be confusing.

Granted, there are cases where this is very useful, regexes are extremely powerful, and there will be cases (in math or physics I recon) where content creators will be able to write this, but in 99%+ of the cases I believe they won't have a clue.

So, it would be nice if we could just turn off that option in the settings? Thoughts?

Also, just GUI-wise I don't think there should be two alternatives in radio boxes for case sensitivity or not, it could just be an optional checkbox behind the answer if you wanted case sensitivity, see:

http://skitch.com/vegardjo/bsyme/create-short-answer-question-6.12

The missing alternative is "accepted answers". So you could specify that you will accept both "ocean" and "salty water" as an answer. Granted, you can do this with regex, but.. :)

I don't know how you GUI-wise should specify this, maybe more fields, or comma (or semi colon) separated list? Or textfield with one alternative per line?

Likert

I have requested this question type (and we can create it!). For points this is a bit special. I believe it should have points like the others, but all checked answers would have to give you these points, as it is in the nature of the question that it isn't right or wrong.

Other use cases

Most of the stuff above is relevant for e-learning and features like Certify depend on this being right. There are however one very normal use case that isn't really related to points, and that is statistics on test results - in most surveys you only wish to have the aggregated data on what has been answered, and the points a single user got are irrelevant.

Looking forward to your comments on all this, and let us know where our developers can help! :)