A/B Testing Basics: Margin of Error

by Daniel Sims

Last updated on April 24th, 2018

Margin of Error: An expression for the maximum expected difference between the true population parameter and a sample estimate of that parameter.

When you are analyzing a statistical experiment or study and progress from discussing the test sample results to discussing the whole population that the sample represents, there will always be a margin of error attached to any estimated values. The margin of error will be stated with a “plus or minus” (+/-) in front of it, meaning you are just as likely to be above or below your estimated value by the same amount.

Despite the word “error” in the name, the margin of error is not related to mistakes or miscalculations. It is simply an estimation of how different your result is likely to be from the “true” parameter value of the population.

Polling and survey results are almost always presented with the margin of error included. In fact, if it is not included, the results should be considered suspect. The polling results below include the calculated average as well as the margin of error.

poll results

In the example, the Democratic result was 39% ±5%, meaning the actual value could range anywhere from 39% – 5% = 34%, to 39% + 5% = 44%. The overall range of values between 34% and 44% is known as the confidence interval.

In website testing, as in other types of binary (yes/no) testing, the margin of error can be approximated using this simple equation:

MOE equals

Source

Where “n” represents the sample size.

To calculate the margin of error more precisely, many sample size calculators can also be used calculate the confidence interval, which is equal to 2x the margin of error.

Confidence Interval Calculator

Source

For example, if you are conducting an A/B test and want to conclusively detect a difference between A and B of 5% or greater, examining the margin of error is a good way to determine how meaningful the results are.

In the sample below, the test results for group B produced 10% more conversions that the test results for group A.  However, based on the sample size, you calculated a margin of error of +/-6%.

conversion percent

The graph shows the narrow band of 2% overlap between the margins of error for A and B.  This means you cannot reliably conclude that your target improvement of 5% has been achieved.

No Comments

DON’T MISS OUT

Get updates on new articles, webinars and other opportunities:

Daniel Sims

Daniel Sims is a Certified Quality Engineer, part-time writer and Six Sigma practitioner who espouses the use of Engineering discipline and problem solving skills in all areas of business and life.

NO COMMENTS

Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.

SPEAK YOUR MIND

Your email address will not be published.

Show Me My Heatmap

Playing around w/ @CrazyEgg and like it so far. Quick, nice and simple. Also easily implementable with #googletagmanager built-in tag.

Brandon K. Lee

@elBKL

What makes people leave your website?