{"id":577,"date":"2017-03-16T14:14:26","date_gmt":"2017-03-16T20:14:26","guid":{"rendered":"https:\/\/www.ometrics.com\/blog\/?p=577"},"modified":"2026-04-23T16:13:14","modified_gmt":"2026-04-23T22:13:14","slug":"conversion-rate-optimization-guide-ab-testing","status":"publish","type":"post","link":"https:\/\/www.ometrics.com\/blog\/conversion-rate-optimization-guide-ab-testing\/","title":{"rendered":"A\/B Testing (Chapter 12)"},"content":{"rendered":"

The Conversion Rate Optimization Guide<\/h1>\n

Chapter 12<\/h1>\n
\n

<<Previous<\/a> \u00a0| Table of Contents<\/a> | Download the CRO Guide<\/a> | Next<\/a> >><\/strong><\/span><\/p>\n

\n

A\/B Testing<\/h2>\n

Once you think you have determined the problem and have created a solution, the current site (the control) and the new solution (the variation) need to be tested against each other to determine which solution has a higher conversion rate.<\/p>\n

A\/B testing or split testing is the process of splitting the traffic to each test page, and then analyzing the results to determine the winner with statistical confidence. The pages are tested again with other pages with the same goal. For example, the goal could be submitting the form, making a purchase or downloading a paper. This same A\/B testing process should be done to your email campaigns as well.<\/p>\n

\"split-test\"<\/p>\n

Tip: Approach A\/B testing like a scientific experiment. Only change one variable at a time for accurate results. Take detailed notes of each test and review them to learn what is working with your particular customer \/ user group.\u00a0 When creating new tests, review the old one for ideas and don\u2019t waste time doing the same test you did last year.<\/em><\/p>\n

After a page has been improved, you will need to review the whole conversion funnel again as well as look at your page metrics. Some changes can have effects on other parts of the site or conversion process. The conversion rate optimization process is a continual one.\u00a0 Sometimes a small change can make a big difference, and other times a radical overhaul to the design can create unexpected positive results.<\/p>\n

\u00a0A\/B and split testing theory and statistical confidence made simple<\/h3>\n

Statistics may sound like a scary course in school but it does not have to be. Statistical confidence or statistical significance puts a percentage of accuracy for a test result. I will try to explain this without any equations so you can get the general idea.<\/p>\n

If you had 100 visitors, 50 went to the control page A and 50 went to variation B.<\/p>\n

If the control page had 25 conversions and the variation had 24, you can say that it is too close to call.<\/p>\n

If the control page had 10 conversions and the variation had 20, you can say that the variation had twice as many because both had the same number of visitors.<\/p>\n

When testing, it is rare to have an even number of testing each variation. So, extra math is needed to evaluate each test accurately. For example:<\/p>\n

If the control had 100 visitors and 15 conversions and the variation had 80 visitors and 14 conversions, the conversion rate shows that the variation had a higher conversion rate since it has more conversions per visitor.<\/p>\n

But how long should you test? If you remember what you learned in statistics class, you had a test with a coin landing on its head or tail. If you flipped it six times you could get more heads or more tails. However, when you flipped it 100 times the results averaged out to 50% heads and 50% tails. When testing, we want to make sure we are running enough tests to get an accurate result.<\/p>\n

There are two factors to an accurate result: \u00a0the number of tests and the difference in the results. If the results are close, then you will need to test more testers or visitors. If the results are very different then you will not need as many testers or visitors to reach statistical confidence.<\/p>\n

Let\u2019s assume the control conversion rate is 35 +\/- 8% and the variation is 40 +\/- 10%. In this case, there is an overlap in the odds of getting the same answer. If the control conversion rate is 35 +\/- 5 and the variation is 50 +\/- 5, then there is no overlap in the odds of getting the same conversion rate, and we can be statistically confident that the result is accurate.<\/p>\n

In most cases, you want a statistical confidence of 90% or greater.<\/p>\n

The actual equations for calculating the statistical confidence are looking at the ratio of the control and variation(s) conversion rates and then graphing the statistical probability that they are not going to overlap. This is graphed with different types of curves like CHI Squared and Z-Score that determines the \u201cp\u201d value (probability). Some A\/B test systems use Bayesian calculations to make sure the tester does not take a result too early, even if it looks statistically accurate but is not because the test has not run long enough.<\/p>\n

Calculating the statistical confidence will tell you if you need to continue testing or not.\u00a0 Be careful, there are some rules to follow so that you do not read your data incorrectly:<\/p>\n