Are You Guilty of These 6 Split Testing Mistakes?

by Rakesh Kumar

Last updated on July 27th, 2017

No excuses. If you want to shorten your path to online success, split testing is something you have to do.

However, not everyone’s website is ready to be split tested.

Before you even think about split testing you need to ensure that a few other things have been taken care of first.

In this article we’re going to look at what you need to focus on before you begin split testing. Ignore the points mentioned, and there’s a chance your split testing efforts will never succeed.

Should you acknowledge them, however, you’ll soon be able to claim the coveted title of profitable website owner.

1.You need traffic


Traffic is key, if you want to split test.

Effective split testing is near impossible if your site doesn’t have enough traffic.

Traffic allows for you to gather actionable data. If no one is visiting your site, you won’t have the data needed to make effective decisions.

What are we talking about here, when referring to data?

In basic terms, you want to see if in general, people are taking the actions that you want them to take whilst on your website. Knowing that information, you can then decide if the changes you made, are effective.

There’s a baseline too. You need to make sure that you start split testing only when you have enough traffic. Small quantities of visitors rarely provide the insight you need.

What counts as enough traffic?

You essentially need enough traffic to ensure your tests are statistically significant.

And the word “traffic” is really misleading.

You should think of website traffic as your population. And really, it should be thought as the population of people going into the funnel you’re testing.

You could be testing your home page, OR a specific landing page on your site. Each page is going to be a different funnel, each with differing amounts of traffic.

So do you see how “website traffic” doesn’t really help here? It’s too ambiguous.

The question you should be asking is: How much traffic do I need going to the first page in my testing funnel?

Here’s how to really break down the traffic question:

  • Determine your typical business cycle. Think of your business cycle as your normal business week. How does your business change throughout the week? How do customers and visitors behave on Monday compared to Saturday? The amount of traffic during this cycle will be your population. Remember, your population is the traffic to the first page in your testing funnel.
  • Use a sample size calculator and enter your desired confidence level and interval. The higher you set your confidence level, the more traffic you will need.
  • Using the sample size value that’s spit out of the calculator, take that number and multiply it by two. That’s how many subjects you need to go through your funnel to perform your A/B test with the confidence requirements you’ve selected.

That’s your answer.

Which means you really need to play around with a sample size calculator to see if you’re even a fit for testing. After fiddling around with a calculator you’ll ask yourself:

  • Do I even have enough traffic to satisfy my population requirement?
  • And do I have enough traffic to satisfy my sample size?

There is a chance that you may have enough traffic to satisfy both requirements, but for some reason you’re not getting enough subjects (people) going through your funnel to hit the required sample size. You’ll end up running the test for weeks or months, which exceeds your normal business cycle. This means your test is prone to more error and you’ll have to be careful as to how you judge your results.

Finally, always conclude your test at the end of a business cycle. Don’t cut your test off midweek because you won’t get a consistent sampling of traffic.

Be sure to read these articles for more information:

2. You need an awesome giveaway/product


Do people want what you’re offering?

Your website will most likely have some kind of offering. It might be a product or maybe just a giveaway.

Either way, you need to make sure your offering is something that people want.

If people don’t want it, no amount of split testing is going to bring you the conversion rates you’re looking to achieve.

For websites selling a product

If you’re selling a product, ensure that it has been validated. Does your research suggest that the product is something people want? More importantly, has someone already paid you money for it?

Here is an article that provides some tips on how to validate a product idea. The bottom line is that people shouldn’t mind parting with their cash in order to obtain your offering.

If your product sucks, split testing won’t improve how well it sells.

For websites looking to exchange something for an email

Your website might not be selling anything right out of the gate.

You might, however, be looking to obtain email addresses in exchange for a giveaway. In doing so you may decide to sell an offer to your list later on in the future — once you’ve built trust.

Whatever your plans are for the future, your job right now is to provide a giveaway that website visitors actually want.

A lot of people make the mistake of not knowing their target market well enough, something we’ll discuss in detail later.

If you don’t know how best to serve your target market, you won’t be able to create a giveaway that is enticing.

Creating an enticing giveaway

Let’s assume you run a dog training website.

You might, for instance, offer dog owners an ebook teaching them how to get their dog to sit on command.

However, what these visitors might want more is an ebook that teaches them how to stop their dog from chewing things.

Sure it’s cool if you can teach your dog to sit. But most people want to initially stop their beloved puppy from eating their favorite shoes or nibbling computer cords. That’s a more urgent problem.

Create an ebook that addresses a pressing problem your target market faces and then teach them how to solve it.

Make it something they want… and need!

3. You need to understand your target market

Target Market

How well do you know your target market?

You need to know your target market better than they know themselves.

Whether you know it or not, odds are that not everyone is your customer. If you market to everyone, you often market to no one.

We touched on this topic lightly when discussing the importance of creating an enticing giveaway.

You need to know the psyche of your target market for reasons other than crafting an irresistible giveaway.

See, when you’re split testing, one of the most crucial elements you’ll experiment with is the copy on your site. You can only create knockout copy, if you know how to enter the conversation that’s going on in your prospect’s head.

If you don’t know their fears or wants, you won’t be able to pen persuasive copy.

You need to make it a goal, therefore, to know your target market way better than they know themselves. How can you do this?

Do what your target market does

In order to understand your target market, it helps to do what your target market does.

Hang out where they hang out. Read the books and forums that they read. Speak to some of them in person.

Really imagine what it’s like to be in their position.

What do they hate the most about their current predicament? If you had could wave a magic wand and make their life better, what would change?

If you’re providing a solution to a problem you previously faced in the past, think back to that time.

Really remember the emotions you were feeling and the concerns you had on your mind. Record all of these distinctions so that you can later use them to influence your copy.

Empathy is key here.

The better you know your target market, the better you can influence them when creating copy to split test.

4. You need to set goals


Have you set any goals related to split testing?

Before you do any kind of split testing, you need to decide on a goal.

A goal is important because it directs your split testing efforts.

For instance, is your goal to boost email subscribers?

Then you’ll know that it’s pointless split testing things that don’t contribute to this goal.

Testing things like email subject lines (only important after they hit the subscribe button), button colors and the size of your font won’t help you reach your goal.

However, opt-in box copy, opt-in box location and giveaways are things that deserve your attention above all else.

Why?  Because they are often the most influential elements on your site when it comes to improving subscriber rates.

Pick a goal and then ensure that your efforts contribute to this goal.

Side note: focus on the low-hanging fruit first

The points mentioned above make a lot of sense, assuming you already know that you need to focus on the low-hanging first.

Some people do not realize this.

They might have read a case study that said a change in font size led to a rise in emails sign-ups. Trying to emulate the case study, they proceeded to change font sizes.

The result?

Little to no change. Maybe even a negative result.

What is often ignored is that such case studies are often based on companies that fixed the low hanging fruit first.

Remember: Focus on the low hanging fruit first — and then implement the cool tactics.

Here is an article that details some of the low hanging fruit you need to focus on. These will help you reach your goal quicker, when you’re split testing website changes.

You should not be scared to make big changes. For instance, getting rid of a sidebar might change the way your entire website looks, and it might scare you to make such a big change. Though there is evidence out there of it improving conversions.

Big changes often help you get results from split testing in a shorter space of time. It shouldn’t be hard to make a few radical changes on your site in order to boost conversions. If it is, you have bigger organizational issues to tackle first before you’re ready to get into split testing.

5. You need to track what current visitors are doing


Are you watching what’s going on?

Earlier we touched on how you need to make sure your split tests eliminate low hanging fruit first.

But what if you’ve already captured your low hanging fruit? Should you just randomly change things and hope for the best?

Of course not.

Thanks to platforms such as Crazy Egg, you can now collect data based on what people are doing when they’re on your site.

That means you can collect data based on user experience and therefore make informed split testing decisions.

Does it look like a lot of people are clicking on your ‘contact us’ button? Split test a version of your website where the contact button is front and center, thereby making it easier to find.

Are people even watching that video that uses up prime webpage real estate? If the numbers are disappointing, ditch the video and then see what happens.

You could even disregard the low hanging fruit section above and collect data from the get go. You’d then be able to ensure that all your tests are run with some highly relevant context.

Don’t take stabs in the dark — use data.

Side note: handling the truth

The beautiful thing about data is that it can reveal some unexpected surprises. Sometimes these surprises might be hard to stomach.

For instance, you may have spent a great deal of time creating a long-form copy page, only to find out that the data tells you your visitors don’t care for it.

When that happens, you may feel utterly dejected. You might even convince yourself that you should keep it. After all, you spent forever on it!

Now, in such a situation, you can ignore the data and hope that in the future, visitors to your website will have a change of heart and develop an interest in long-form copy.  After all, you read a case study which showed long-form copy working for another site.

Or you can accept the fact that not all websites are equal and get rid of the long-form copy.

Remember: what works for another site might not work for yours.

If you’ve spent a lot of time implementing a new change on your site, and it does not work, that’s just the nature of the beast. Don’t limit your websites potential by keeping something just because you think it will eventually work.

Unless you’re interpreting them wrong, the numbers don’t lie.

6. You need to focus on the process, not the page

Split testing can improve the number of people who click on a given button.

However, unless that button is the buy button, you probably don’t care about button clicks.

My point is this: If you eventually want to sell stuff, you need to make sure you have a sales funnel in place. You need to set this up before you split test.

That means you need to avoid focusing on the wrong metrics.

A lot more people clicking through to your sales page?


Only an incremental increase in the number of people buying?

Not so great!

It might be nice to know that you’re increasing your clickthrough rates on web page elements.  But if you can’t figure out a way to convert button clicks into profit, your time spent split testing might be wasted.

Split testing has to be an all-encompassing process. In fact it’s often the last mile of an entire process.

If you don’t have something to sell (or more importantly, something that people want) before you start split testing, you’re doing things the wrong way around.

Side note: there is not one page to rule them all

Magic bullets are rare in the split testing world. One change rarely leads to an overall rise in profits.

You need to remember that each web page is dependent on the one before it and the one after it. After all, a sales process is often made up of several pages.

An amazing home page only helps your bottom line if your checkout page can properly convert people once they get there.

An amazing checkout page will be limited by how well your home page converts.

If 4/10 people on the checkout page buy, but only 1/1000 people reach the checkout page, something may be wrong.

If there aren’t enough people reaching the right pages, you won’t be selling as much as you’d like.

Don’t plan on split testing the same page five times in a row. Divide your focus. Address other parts of your sales process too.

You’re ahead of most people

Now that you’ve read this post, you should know that you’re ahead of most people in the world of split testing.

Your first encounter with split testing can be dizzying. Though the points mentioned should help you maintain a sense of focus.

The split testing results you’ve read about elsewhere are only possible if you take care of a few fundamentals first.

Should you acknowledge those points, you’ll be able to cut through the fluff and see some great results with split testing pretty quick.

Good Luck.

Image Source: Laptop, Crowd, Traffic Goal

No Comments


Get updates on new articles, webinars and other opportunities:

Rakesh Kumar

Rakesh Kumar is an online marketer and freelance writer. He occasionally works with other freelancers, helping them to grow their ventures.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

Show Me My Heatmap

Tools like @CrazyEgg & @mouseflow are a few user testing tools to make websites more awesome:



What makes people leave your website?