The Crazy Egg A/B Test Planning Guide

by Daniel Sims

Last updated on September 21st, 2017

A/B Testing, or “Split Testing” as it is also known, can be one of the most useful and powerful tools available for CRO, when used correctly. Without careful planning and analysis, however, the potential benefits of an A/B test may be outweighed by the combined impact of errors, noise and false assumptions.

For these reasons, we created The Crazy Egg A/B Test Planning Guide.

Our user-friendly guide provides a roadmap through the A/B test planning process. In addition, it serves as a convenient way to record and store your testing history for future review.

What is an A/B Test?

If you are reading this guide, you are probably at least somewhat familiar with the basic premise behind A/B testing.  In simple terms, an A/B test compares the performance of two variations (A and B) of a web page, each with an equal amount of traffic. The objective is to determine if variant (B) could potentially improve conversions.

AB Testing


Whether you are new to this method or just looking for better ways to plan and document your existing testing, our guide is designed to help you accomplish your goals.

Understanding A/B testing basics can be as simple as driving to work each day:

Question and Answer

Where the elegant simplicity of the A/B concept can give way to unnecessary complexity is often related to one notorious 4-letter word:


If you didn’t already notice that “Stats” is, in fact, a 5-letter word, you have actually committed a Type 2 statistical error by rejecting the alternate hypothesis that “Stats” is not a 4-letter word!

And if the previous sentence left you scratching your head, you are in good company. Luckily, our A/B Test Planning Guide is intended to demystify the statistical aspects of testing that can sometimes lead to confusion.

The Benefits and Pitfalls of A/B Testing

A successful A/B test can benefit your business in numerous ways. Besides the obvious tangible improvements in conversion rate and revenue, less obvious benefits of a well- designed and executed test plan might include:

  • Identification of the most (and least) important website elements
  • Increase in return traffic
  • Improved brand recognition and perception

The pitfalls inherent to A/B testing are perhaps less obvious, but equally important in maximizing test effectiveness. Crazy Egg provides lots of outstanding content dedicated to the watch-outs and best practices you should be aware of.  Some of the more costly and common missteps include:

  • Ending tests too early
  • Changing more than one factor at a time
  • Misunderstanding “significance”
  • Interpreting others’ test results for your own website

Many of these common pitfalls also have a common cause at their core: Human nature.

I once attempted to envision what a Perfect Website A/B Test might look like. In theory, this test would have all the benefits but none of the drawbacks common to A/B testing. Although I ultimately concluded that this goal was impossible, our guide is intended to help point you in that direction.

Using the Guide

Following our step by step guide is fairly intuitive. To make it even easier, we have provided some additional explanation for each step.

Step 1. Hypothesis

Like most guides, we placed what we believe to be the most important step right at the top.  Your hypothesis or “theory” drives everything that comes afterwards, so it needs to be clear, concise and measurable.  The example hypothesis, “My enhanced graphics will have a statistically significant impact on conversion rate”, includes all of these elements in one brief sentence. Think about what you want to prove, what single variable you will alter, and what success looks like. The result will form your hypothesis statement.

Whether this is your null hypothesis or alternate hypothesis (it’s actually the latter) is only important in stats-speak. Just make sure you spend time thinking about and researching what you should test, and why.  Do you have evidence (such as heatmaps) that support your theory? Are you testing something that you really believe could make a significant difference or improvement? Do you know what a successful test would look like?

Step 2. URL’s

The A/B Test Planning Guide will ultimately become the A/B Test Historical Record, so we have included space to document your Control and Variant URL’s. You will probably not need to refer to these during the actual testing, but documenting them is considered a wise record-keeping practice.

Step 3. Statistics

Since this is where many of us tend to get confused, our guide is designed to consider only the most essential elements of statistical analysis. Let’s take a quick look at the stats we included, what they mean, and why they are important.

  • Confidence Level: The concept of Confidence Level is extremely important, since it determines your level of certainty in the results and also drives the required Sample Size. Being 99% confident in your results is certainly better than 95%, but be cognizant of the sample size implications and make sure you understand whether your current traffic volume can support it.
  • Significance Threshold: Notice that your significance threshold is just 1 minus your Confidence Level.  When you select the latter, you automatically select the former.
  • Significance p-value: This value is sometimes misunderstood, but there are plenty of good resources available to help get to the Heart of Significance. The important thing to remember is that significance alone is not a guarantee of success. However, the lower the p-value is, the more pronounced and meaningful is the observed difference between A and B.
  • Target Conversion Rate: We added this step to let you decide (and record) what success means to you in your test.  For example, if your current conversion rate is 1.20%, is a statistically significant 1.25% enough? The best way to attain a goal is to set (and record) it.
  • Expected Chance of Reaching Target: Some software packages can provide the answer to this question within a few mouse clicks. Otherwise, the “Eggspert Advice” included in the guide provides one solution to obtaining this answer mathematically. Just keep in mind that in this analysis, a higher p-value is actually better, since it indicates how closely your results aligned with your goal.

There are numerous other statistical tools for planning and analysis that we have not included in the guide. These basic steps provide a clear path to a meaningful answer, but we definitely encourage you to utilize more advanced statistical tools as your knowledge grows.

Step 4. Screen Shots

You may be curious why we included this step. When you finish testing your hypothesis and see the results in graphical and/or numeric form (depending on your software), you will want to capture this information somehow. A simple and effective way to do this is to simply screenshot and save the most relevant and important results. Including these images with your completed guide will provide a more complete picture for future review. You probably won’t be able to paste all of your screen shots in the four boxes provided, but that’s OK. Just think of this step as a reminder to save this important information.

Step 5. What We Learned/Analysis

If step 1 is perhaps the most important, step 5 is a close second. Similar to the hypothesis statement, your summary statement should explain and quantify the results as well as interpret what they mean in one concise sentence. The example provided includes the outcome (B performed 2.2% better than A), duration (2 weeks) and meaning (statistical significance).

Now that you have completed the test and captured the results, what will you do next?

Part 2 of step 5 is to document your decision. It may seem self-evident, based on the results statement alone, but not all winning variants are implemented, just as unsuccessful tests are not always killed. Documenting how you decided to use your results will help you close that loop and create a bridge to future testing efforts.

Finally, as people at your organization come and go, documenting tests becomes imperative. Keeping properly documented historical testing records will save everyone tons of time, energy and heartache.

Download The Guide Below

It’s free! And we hope it helps you document and stay on course throughout your testing journey. Simply click on the guide below to download it (or download it by clicking on this link).

ab test planning guide

No Comments


Get updates on new articles, webinars and other opportunities:

Daniel Sims

Daniel Sims is a Certified Quality Engineer, part-time writer and Six Sigma practitioner who espouses the use of Engineering discipline and problem solving skills in all areas of business and life.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Show Me My Heatmap

Having trouble seeing your website performance? @CrazyEgg can solve that. It's amazing what you can learn from a heatmap #tech

Sean Higgins


What makes people leave your website?