Support from Acquia helps fund testing for Drupal Acquia logo

Comments

pivica’s picture

Recent comparation of multivariate vs some other commercial solutions http://www.nonsie.com/blog/ab_testing_options_drupal_7_sites.

miro_dietiker’s picture

Nice comparision, thanks nonsie.

Hope we can catch up some of the missing pieces. A lot is possible.
The Architecture also contains some patterns that can allow caches. In case of high performance needs, this can be investigated.

I personally miss transparency in reporting / statistics most with multivariate. We can improve a lot here.

pivica’s picture

Yeah reporting/stats are in bad shape, we can improve there a lot.

nonsie’s picture

Here's my top 5 (or the reasons I would recommend an external solution over this module at the moment):
1. #2085241: Success condition - links?
2. visitor segmentation and targeting
3. reporting
4. traffic allocation/distribution
5. scheduling of tests

It also needs some documentation since there isn't a UI like in Optimizely, Visual Website Optimizer etc. - how to set up different types of tests and how to write your own plugins and success conditions.

It also needs some UI help in terms of setting up tests - some of it is as simple as reviewing the copy used. For example when you create a test you are presented with the following copy under sets/variants - "Please note that if you add/remove mutations, variants or sets any test recorded data that are already saved will be lost." I'd replace it with something more easily readable like:
Please note that any test recorded data that are already saved will be lost if you add or remove: mutations, variants, or sets.

pivica’s picture

Hi nonsie, thx for the list.

Can you prioritize this items - also including two belows first 5 (better documentation and better ui help texts).

When we have prio list we can start to work on issues one by one.

Also can you share some links or docs that explains in more 2 and 4. If you have some ideas for 3 (mockups or wireframes suggestions) - please share :)

nonsie’s picture

IMHO documentation and UI help text changes can happen first (before the 5) but will need to be revised as features get added.

Here are some screens from Visual Website Optimizer showing things like targeting, audience, traffic allocation and distribution.
vwo_segment_target.png

vwo_segment_target_advanced.png

vwo_audience.png

I don't have any running tests with numbers at the moment but this is what reporting looks like in VWO:
vwo_reporting.png

pivica’s picture

Lets not forget #1602172: visualization, comparision, refer to myna which is probably related to this one.