Split Testing 101: A Crash Course in Optimization

by Dale Cudmore

Last updated on August 7th, 2017

Growing up, I always wanted to be a professional split tester.

Although, I thought this meant I’d be eating banana splits all day.

Split tests in the business world involve a lot less ice cream, but a lot more profit. I’d say they about even out.

If you’re looking to improve your profits per lead or customer, understanding the fundamentals of split testing is the first, and arguably most important, step on your journey.

I’m going to break down what split testing is and what role it plays in, not only conversion optimization, but in increasing your business’ bottom line.

Think of it this way: Conversion rate optimization is the “what,” but split testing is the “how.”

Why Split Test?

Not many people enjoy creating and running split tests, but what they do enjoy are the results.

Split tests are a statistically backed method that tells you if a particular change, usually on your website, will lead to more profit.

Even small improvements in your conversion rate can lead to tens of thousands in extra profit every year for the foreseeable future.

Imagine if you could improve your revenue by 400% by simply removing an image? While that’s an extreme case, most businesses can improve their conversion rate by X% much easier than increasing their lead quantity or quality by 8%.

Going Deeper: What is a Split Test?

The most basic split test is an A/B split test.

You have an A version (the original), and a B version (the variation) of whatever you wanted to test.

You “split” your traffic, and send half to the A version of your page, and half to the B version.

Here’s a basic example, where the test is to see if a red button will convert better than the original grey button:

basic button split test

Over time, you measure and determine which version converts more visitors into sign-ups.

In the case of the image of the test above, the variation converted at 4.5%, which is much better than the original 1%.

Because variation B “won,” it becomes the new standard, and will be “variation A” in any future tests. Make sense?

An Alternative: Multivariate Split Tests

Okay, so changing button color seems like a good thing to test, but what if you also want to test a different headline?

The simplest option is to do the button test and then, after you have a winner, you can start the headline test.

However, another option is to do a multivariate test. It’s essentially the same as any other split test, but you can test multiple changes at the same time.

The downside of this is that it takes much longer to run.

Suppose you wanted to test one variation of both the button and headline. Now you have 4 total variations:

Button ColorHeadline

Variation A


Headline 1

Variation B


Headline 1

Variation C


Headline 2

Variation DRed

Headline 2

And of course, you need to split your traffic equally 4 ways, which is why reaching a significant conclusion takes longer.

A Few Important Definitions

There are a few terms I mentioned above that you may or may not be familiar with. This is a good time to define a few important terms that you will see over and over as you study conversion optimization.

Conversion rate: The percentage of people who take a desired action. For example, if 100 people arrived on your homepage and 3 people signed up for a demo, that would be a 3% conversion rate.

Overall conversion rate: Whenever possible, optimize your overall conversion rate, which is the amount of customers that actually purchase your service or product. If 1 person purchases your offer out of 100 visitors, you have a 1% overall conversion rate, regardless of the steps in-between.

Note: In many cases, it’s difficult to measure the impact that a change has on your overall conversion rate. In these cases, you have to resort to optimizing the intermediate steps (like requesting a demo) that should lead to increased sales.

Sales funnel: Your sales funnel breaks down all the actions a user takes, starting from landing on your website, all the way to making a final purchase. Note that you can have multiple sales funnels. See a basic funnel below:

basic website conversion funnel

Statistical significance (or just significance): All tests have variation. Variance will even out in the long run, so as long as you reach a high level of statistical significance (or confidence), you can be confident that you’re picking the right variation as the winner.

Significance is usually stated as a percentage. If a split test result has a confidence level (significance) of 95%, that means that 19 out of 20 times, you’ve chosen the right winner. However, that means that 1 out of 20 times you falsely picked the wrong variation as the winner.

Sample size: The sample size refers to the amount of traffic sent to each variation in your split test. The higher your sample size, the more accurate your results will be. Most split testing tools have a sample size calculator built in, but there are also many stand-alone sample size calculators you can use.

What Should You Split Test?

Conversion test hypothesis

You’re probably a bit excited to get going, but hold on a second.

What you don’t want to do is start split testing elements of your website at random.

To decide what you need to split test, you need to first develop a conversion rate optimization (CRO) plan.

With a properly thought-out CRO plan, you’ll know where the biggest opportunities for improvement are , where any big leaks are, and you’ll be able to come up with a hypothesis to split test.

Don’t worry if you don’t know how to come up with a hypothesis right now, you’ll learn that over time. In general, these are some questions that will get you started about the right things that might be having a negative effect on your conversion rate:

  • are any copywriting messages simple and clear to understand?
  • is the user experience good? (i.e., everything is where the user expects it to be)
  • do the most important elements stand out?
  • what objections do visitors have when they come to this page? Do we address them?

The more you think about and clarify user interaction in general when it comes to conversion, the better you will get at diagnosing leaks costing you customers and profit.

How Do You Split Test?

Say you do all that work and come up with a split test hypothesis… what comes next?

There are tons of split testing tools that will not only help you create a variation of a page and split traffic between your options, but will also record the results and compute the statistics to tell you which variation won.

All you’ll need to do is modify the single element that you want to test, and then follow the simple instructions on one of those tools like Optimizely or Visual Web Optimizer to implement the test on your site.

At the end, all you’ll need to do is record the result, make any necessary changes to your site, and then start the process all over again.

Remember, split testing is a long-term strategy , so even though you’ll have tests that don’t produce any improvement, overall you will increase your conversion significantly. When you first start out, don’t judge the results of your efforts for at least 6 to 12 months.

Case Study Example: Does a Security Seal Hurt Conversions?

split test case study

(image source)

Let’s look at a real-life example to help clarify the split testing process.

The picture shows a basic email optin form. Originally, it had a security seal, the TRUSTe image, right beside the submit button.

The variation was the exact same form, minus the security seal.

The results: Conversions increased by 12.6% and variation B was declared the winner.

After every test, you need to try to explain why the winning result won. In this case, it’s likely that variation B won because users were put off by the security seal.

Usually, security seals are displayed when there is some sort of monetary transaction. Even though this is just a simple email signup form, this association was likely enough to make users pause and reconsider what exactly they were filling out. As a result, it scared away a significant fraction of potential leads.

Conclusion and Next Steps

Conversion rate optimization can be intimidating when you’re first starting out, but it always comes back to the fundamental concepts we looked at here.

I recommend going through our Beginner’s Guide to Creating a Conversion Rate Optimization Plan to discover what else you need to learn about before getting started.

All this effort will be worth it, as a solid CRO plan can lead to consistent, significant growth for as long as you run your business.

Have any questions about split testing? Leave me a comment below and I’ll clear them up.

Read other Crazy Egg articles by Dale Cudmore.



Get updates on new articles, webinars and other opportunities:

Dale Cudmore

Dale Cudmore is a professional marketing writer. He focuses on actionable, exciting ideas and strategies that can be used to grow your business.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  1. Neeraj says:
    June 21, 2015 at 1:16 pm

    Surely worth a try. I would definitely implement it on my next project.

    • Dale Cudmore says:
      June 22, 2015 at 7:51 am

      Definitely Neeraj. Once you get a bit of traffic, split testing is one of the most efficient uses of your time.

      Good luck!

  2. Fulvio says:
    June 19, 2015 at 12:11 am

    Thanks for your big and clear guide.
    One day I’ll do a/b test, but every time i say: tomorrow 🙂
    Using system as whictestwon is a nice idea for you?

    • Dale Cudmore says:
      June 19, 2015 at 9:25 am

      Hi Fulvio, Whichtestwon is a great resource to use.

      Try not to make split testing harder than it is, it doesn’t take too much effort to get started. Once you get in the habit of it, you’ll start to continually see an increase in your conversion rate, which adds up a lot over time.

Show Me My Heatmap

Taking @CrazyEgg for a test run @IndiciaNL. First Plus.. Smooth #TagManager implementation!

Rogier | is hier


What makes people leave your website?