A/B Testing Basics: Confidence Level

by Daniel Sims

Last updated on November 24th, 2017

Confidence Level: The percentage of time that a statistical result would be correct if you took numerous random samples.

Confidence is often associated with assuredness, and the statistical meaning is closely related to this common usage of the term. To state a percentage value for confidence in something is essentially stating a level of how “sure” you are that it will happen. In statistical terms, it is the expected percentage of time that your range of values will be correct if you were to repeat the same experiment over and over again.

Unfortunately, there is no such thing as a 100% confidence level, since this would require a sample size as large as the entire population. The most commonly selected confidence levels are 95% and 99%. In most applications where a confidence level is used, such as opinion polling and A/B testing, 95% is the default value. The 99% confidence level is usually reserved for pharmaceutical testing and other fields of interest where the consequences of an incorrect conclusion are more severe.

Understanding what the confidence level really means requires understanding what is meant by being “correct”. It does not mean that your predicted mean value is correct or spot on 95% of the time. A confidence level is associated with the confidence interval, which is the predicted range of values based on the test results. A 95% confidence level means your confidence interval is expected to include the true mean value 95% of the time.

bottle weight grams

In website testing, the confidence level is probably the first thing you will be asked to decide when you determine a sample size for the test, so it should be chosen carefully. The difference between a 95% confidence level and 99% confidence level is fairly substantial when it comes to sample size.

Confidence level is also closely related to the statistical term α (Alpha), which is the probability of creating a Type 1 error:

Confidence Level = 1 – α
A Type 1 error is when you incorrectly conclude that an experiment (such as an A/B test) was successful in detecting an improvement, but it actually was not. Obviously, this could be a serious and costly mistake.

The greater the confidence level, the lower the α value, so an adequate confidence level is an important component in any meaningful website testing.

No Comments


Get updates on new articles, webinars and other opportunities:

Daniel Sims

Daniel Sims is a Certified Quality Engineer, part-time writer and Six Sigma practitioner who espouses the use of Engineering discipline and problem solving skills in all areas of business and life.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Show Me My Heatmap

Looking for key #Webanalytics tools? Try @CrazyEgg for the best way to track your #digitalmarketing efforts in a simple to read #data form

Brad Chuck


What makes people leave your website?