Here's our mockup for the Create Multivariate page Bojhan and I created today. Suggestions are welcome.

CommentFileSizeAuthor
UTS_create_mutivariate.png71.19 KBskilip

Comments

boombatower’s picture

boombatower commented

Marking other issue duplicate: #372944: Split / A-B Testing.

boombatower’s picture

boombatower commented
Status: Active » Needs review

I think the screenshot provides a nice fiew of "A-B Testing" in that you can list the original page and multiple pages of variants. That method will require the administrator of the test to setup those URLs and perform any necessary modification, but will thus give them full control over the page.

There are a bunch of ways we could do multivariate testing, some of which could be done through a UI or fit nicely for particular situations. I have thought it over for a while and I have come up with what I think is the best all-around solution.

Through an interface the administrator will enter the number of items to vary and then for each of those items how many variations there are. In addition the URL at which those variations are to take place.

Variations:
A - 1, 2
B - 1, 2, 3, 4
C - 1, 2

When the system detects the URL it will randomly choose the variations, and possibly use a more advanced algorithm to remove the variations that have been chosen more then the rest. Once the system has chosen the variation it will store it for the remainder of the user's session (just as the A-B testing will do).

The variation will then be provided through a code interface that will allow a test administrator to define a hook_form_alter() like the following.

function variatetest_form_user_register_form_alter(&$form, $form_state) {
  $mutation = variatetesting_get_mutation();

  if ($mutation[1] == 1) {
    $form['element1']['#someproperty'] = 'new value';
  }
  else if ($mutation[1] == 2) {
    $form['element1']['#someotherproperty'] = 'new value 2';
  }
  else if ($mutation[1] == 3) {
    $form['element1']['#somenewproperty'] = 'new value 3';
  }

  // For all mutations....
}

This can then be applied for non-form related variations, but gives the administrator a very easy way to make alterations to things like forms since they can be place in a test module and turned on and off without effecting the main code base, which may be receiving updates (on live sites).

In addition, another method of making changes outside of code, but in a clean manor. (this requires theme cooperation) Special CSS classes may be added to the body of the page to specify the variations. For example, the following would represent elements 1, 2, and 3 with variations 1, 2, and 1 respectively.

<body class="variation-1-1 variation-2-3 variation-3-1">
...
</body>

Then variation elements on the page could be changed in the following manor.

#variation-1-1 #form-element-7 {
  display: none;
}

#variation-1-1 .other-stuff {
  background-color: red;
}

...

#variation-2-3 .something {
  color: blue;
}

...

#variation-3-1 .login-block {
  display: none;
}

The variations will then be numbered in a way similar to binary so that a numeric value can be assigned that is decidable. Having a numeric key will allow for easy database indexing and scalability (may end up using a type table with the variations stored). Then we just need to store the number of people served the variation and the number of people the succeeded to "success page".

catch’s picture

catch
he/him
Primary language English
commented

Bojhan asked me to take a look at this in irc. I didn't spend too much time looking at it but overall this seems like a decent approach. Is the idea that a test defines a form_alter() which this module then implements on their behalf during a test run? If so that's really nice. Seems like the classes could be done with preprocess in most cases so half-decent themes ought to allow it. Do you see a way to have a UI to define the form changes (probably just basic ones)? I'm not sure how that'd even look though.

boombatower’s picture

boombatower commented

Discussed that with Bojhan and we weren't sure either or that it would be something that we should try and accomplish during SoC. Given the API though it should be fairly simple for someone who doesn't know code to make somewhat simplistic changes (like hide one field and show alternative field) via the CSS method.

Bojhan’s picture

Bojhan commented

Would probably require documentation or like two examples, so non coders could easily do it. I believe the interface itself would not necessarily be complex, it should be able to demo the variation.

boombatower’s picture

boombatower commented

This is to be done as a separate module right?

Bojhan’s picture

Bojhan commented

yes

boombatower’s picture

boombatower commented
Project: Usability Testing Suite » Multivariate
boombatower’s picture

boombatower commented
Assigned: Unassigned » boombatower
Status: Needs review » Fixed

This has been written, except for CSS since I don't see any easy way to do that.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.