A/B Testing Basics: Confidence Interval

by Daniel Sims

Last updated on November 24th, 2017

Confidence Interval: A range of values calculated such that there is a known probability that the true mean of a parameter lies within it.

The science of statistics is all about predicting results by sampling a portion of a population. Since you can never be 100% certain of that prediction, the result is often expressed as a possible range of values. This range is also known as the confidence interval.

For example, you might estimate average body weight based on a random sample of 500 men and 500 women. Your sample results will vary, so you need to add a measure of that variability to your estimate. When we refer to a statistic, such as average (mean) weight, being a certain value “plus or minus” another value, that plus or minus number is the margin of error, and the entire range of possible results is the confidence interval.

confidence interval

A confidence interval can be used in an A/B test design. When you calculate the required sample size, you will sometimes be asked to select a target confidence interval, depending on what calculator or software you are using. If you want to accurately detect a small difference between A and B, you need to select a small confidence interval. In the sample size calculator below, a 2% confidence interval is selected.

determine sample size

If the means observed for A and B in your test are less than 2% different from one another, your confidence intervals will overlap, meaning the results are “within the margin of error” and inconclusive. As you can see, this small confidence interval brings with it a large sample size, at least when compared to the overall population size.

If you did not originally include a confidence interval in your sample size calculation, it can still be used after the A/B test has concluded, to help you interpret the results. It can be calculated based on your sample size, population and confidence level.

find confidence interval

For example, let’s say you ran your A/B test and the results looked like the graph below.

confidence interval conversions

Group “B” had a higher mean conversion rate than Group “A”, and the two confidence intervals, although large, did not overlap. This is a good indication that B is reliably better than A, and that your results are significant.

One Comment


Get updates on new articles, webinars and other opportunities:

Daniel Sims

Daniel Sims is a Certified Quality Engineer, part-time writer and Six Sigma practitioner who espouses the use of Engineering discipline and problem solving skills in all areas of business and life.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

  1. Blake J. Elliott Timothy says:
    December 24, 2016 at 10:56 am

    That’s a really value added post, Denial clearly shown the top notch of ab testing. Knowledgeable.

Show Me My Heatmap

I love sites with - combine that with and it gives such a great set of data to inform /

Andrew Martin


What makes people leave your website?