VWO’s New SmartStats Bayesian Statistics

In October 2015 VWO is launching its new SmartStats which runs on Bayesian statistics. This will provide better understanding on how long to run a test for conversion rate optimization. They do not get into the algorithm but discuss the benefits of being able to view the data in three perspectives (below).

In the end I do not think this will affect testing to much but will allow those who do not understand statistics and the phases of an A-B test to read the data clearer and make the right decisions. Too often I see decisions made too quickly in a test and this will help in preventing inaccurate conclusions.

conversion testing over time

 

conversion optimization a-b testing over time

Here are the three modes VWO is providing

Quick learnings
For finding quick trends where tests don’t affect your revenue directly

You can choose this mode when testing non-revenue goals such as bounce rate and time spent on a page, or for quick headline tests. With this mode you can reduce your testing time for non-critical tests where there isn’t a risk of hurting your revenue directly by deploying a false winner.

Balanced
Ideal for most tests.

This is the default mode and can be used for almost all tests. As the name suggests it is the best balance between testing time and minimizing potential loss.

High certainty
Best for revenue-critical tests when you want to absolutely minimize potential loss. Usually takes the longest to conclude a test.

Suppose you have an e-Commerce website and you want to test changes to your checkout flow. You want to be as certain as possible to minimize potential loss from deploying a false winner even if it takes a lot of time. This is the best mode for such critical tests which affect your revenue directly.

 

Follow Me

Greg Ahern

Greg Ahern Founder and President of Ometrics® is a fanatic about conversion rate optimization and lead generation. Greg has been a successful Internet entrepreneur since 1994. He speaks at conferences and webinars and has built a number of internet businesses, including web marketing, web development and internet lead generation, which have been successfully acquired. Greg is the Denver Chapter Leader for the Digital Analytics Association. You can follow Greg on Twitter @gregahern and join his CRO Hacks Groups on Facebook and Slack.
Follow Me

Leave a Reply

Your email address will not be published. Required fields are marked *