A/B Testing Basics: Sample Size

by Daniel Sims

Last updated on November 24th, 2017

Sample Size: The number (n) of observations taken from a population through which statistical inferences for the whole population are made.

The concept of sampling from a larger population to determine how that population behaves, or is likely to behave, is one of the basic premises behind the science of applied statistics.

For example, if you have a population of 10 million adults, and you sample 10 of them to find out their favorite television show, intuitively you will realize the sample size you have chosen is not large enough to draw valid conclusions.

Knowing just how large “n” should or must be is among the most important decisions in any test or experiment, including website A/B testing.

sample size

Image Source

If your results are not truly representative of the target population, your test loses its value and can point you towards an incorrect conclusion. An online sample size calculator will usually ask you to provide the following information in order to determine a statistically valid sample size:

determine sample size calculator

The confidence level is a measure of how certain you want to be of your results, while the confidence interval provides a range of possible values, defined by the average (mean) plus or minus the margin of error.  Ideally, you want the confidence level to be high, and the margin of error to be low.  The required sample size will increase accordingly with these measures of certainty.

margin of error

The total population size must be known as well, since you want to know what percentage or proportion of your overall population is represented by your sample. Once again, a larger proportion means more certainty and larger sample size.

In an A/B test, sample sizes for both A and B will be drawn from the same population. That means the actual number of samples required to complete the study will be twice as much as calculated sample size.

No Comments


Get updates on new articles, webinars and other opportunities:

Daniel Sims

Daniel Sims is a Certified Quality Engineer, part-time writer and Six Sigma practitioner who espouses the use of Engineering discipline and problem solving skills in all areas of business and life.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

Show Me My Heatmap

Tools like @CrazyEgg & @mouseflow are a few user testing tools to make websites more awesome: hubs.ly/H01gDZZ0



What makes people leave your website?