A/B Testing Basics: Sample Size

Sample Size: The number (n) of observations taken from a population through which statistical inferences for the whole population are made.

The concept of sampling from a larger population to determine how that population behaves, or is likely to behave, is one of the basic premises behind the science of applied statistics.

For example, if you have a population of 10 million adults, and you sample 10 of them to find out their favorite television show, intuitively you will realize the sample size you have chosen is not large enough to draw valid conclusions.

Knowing just how large “n” should or must be is among the most important decisions in any test or experiment, including website A/B testing.

Image Source

If your results are not truly representative of the target population, your test loses its value and can point you towards an incorrect conclusion. An online sample size calculator will usually ask you to provide the following information in order to determine a statistically valid sample size:

• Confidence Level
• Confidence Interval
• Population Size

The confidence level is a measure of how certain you want to be of your results, while the confidence interval provides a range of possible values, defined by the average (mean) plus or minus the margin of error.  Ideally, you want the confidence level to be high, and the margin of error to be low.  The required sample size will increase accordingly with these measures of certainty.

The total population size must be known as well, since you want to know what percentage or proportion of your overall population is represented by your sample. Once again, a larger proportion means more certainty and larger sample size.

In an A/B test, sample sizes for both A and B will be drawn from the same population. That means the actual number of samples required to complete the study will be twice as much as calculated sample size.