In October 2015 VWO is launching its new SmartStats which runs on Bayesian statistics. This will provide better understanding on how long to run a test for conversion rate optimization. They do not get into the algorithm but discuss the benefits of being able to view the data in three perspectives (below).

In the end I do not think this will affect testing to much but will allow those who do not understand statistics and the phases of an A-B test to read the data clearer and make the right decisions. Too often I see decisions made too quickly in a test and this will help in preventing inaccurate conclusions.

conversion testing over time


conversion optimization a-b testing over time

Here are the three modes VWO is providing

Quick learnings
For finding quick trends where tests don’t affect your revenue directly

You can choose this mode when testing non-revenue goals such as bounce rate and time spent on a page, or for quick headline tests. With this mode you can reduce your testing time for non-critical tests where there isn’t a risk of hurting your revenue directly by deploying a false winner.

Ideal for most tests.

This is the default mode and can be used for almost all tests. As the name suggests it is the best balance between testing time and minimizing potential loss.

High certainty
Best for revenue-critical tests when you want to absolutely minimize potential loss. Usually takes the longest to conclude a test.

Suppose you have an e-Commerce website and you want to test changes to your checkout flow. You want to be as certain as possible to minimize potential loss from deploying a false winner even if it takes a lot of time. This is the best mode for such critical tests which affect your revenue directly.


Greg Ahern
Follow Me