Margin of Error: An expression for the maximum expected difference between the true population parameter and a sample estimate of that parameter.
When you are analyzing a statistical experiment or study and progress from discussing the test sample results to discussing the whole population that the sample represents, there will always be a margin of error attached to any estimated values. The margin of error will be stated with a “plus or minus” (+/-) in front of it, meaning you are just as likely to be above or below your estimated value by the same amount.
Despite the word “error” in the name, the margin of error is not related to mistakes or miscalculations. It is simply an estimation of how different your result is likely to be from the “true” parameter value of the population.
Polling and survey results are almost always presented with the margin of error included. In fact, if it is not included, the results should be considered suspect. The polling results below include the calculated average as well as the margin of error.
In the example, the Democratic result was 39% ±5%, meaning the actual value could range anywhere from 39% – 5% = 34%, to 39% + 5% = 44%. The overall range of values between 34% and 44% is known as the confidence interval.
In website testing, as in other types of binary (yes/no) testing, the margin of error can be approximated using this simple equation:
Where “n” represents the sample size.
To calculate the margin of error more precisely, many sample size calculators can also be used calculate the confidence interval, which is equal to 2x the margin of error.
For example, if you are conducting an A/B test and want to conclusively detect a difference between A and B of 5% or greater, examining the margin of error is a good way to determine how meaningful the results are.
In the sample below, the test results for group B produced 10% more conversions that the test results for group A. However, based on the sample size, you calculated a margin of error of +/-6%.
The graph shows the narrow band of 2% overlap between the margins of error for A and B. This means you cannot reliably conclude that your target improvement of 5% has been achieved.