A/B Testing Basics: Confidence Level

Confidence Level: The percentage of time that a statistical result would be correct if you took numerous random samples.

Confidence is often associated with assuredness, and the statistical meaning is closely related to this common usage of the term. To state a percentage value for confidence in something is essentially stating a level of how “sure” you are that it will happen. In statistical terms, it is the expected percentage of time that your range of values will be correct if you were to repeat the same experiment over and over again.

Unfortunately, there is no such thing as a 100% confidence level, since this would require a sample size as large as the entire population. The most commonly selected confidence levels are 95% and 99%. In most applications where a confidence level is used, such as opinion polling and A/B testing, 95% is the default value. The 99% confidence level is usually reserved for pharmaceutical testing and other fields of interest where the consequences of an incorrect conclusion are more severe.

Understanding what the confidence level really means requires understanding what is meant by being “correct”. It does not mean that your predicted mean value is correct or spot on 95% of the time. A confidence level is associated with the confidence interval, which is the predicted range of values based on the test results. A 95% confidence level means your confidence interval is expected to include the true mean value 95% of the time.

In website testing, the confidence level is probably the first thing you will be asked to decide when you determine a sample size for the test, so it should be chosen carefully. The difference between a 95% confidence level and 99% confidence level is fairly substantial when it comes to sample size.

Confidence level is also closely related to the statistical term α (Alpha), which is the probability of creating a Type 1 error:

Confidence Level = 1 – αA Type 1 error is when you incorrectly conclude that an experiment (such as an A/B test) was successful in detecting an improvement, but it actually was not. Obviously, this could be a serious and costly mistake.

The greater the confidence level, the lower the α value, so an adequate confidence level is an important component in any meaningful website testing.