How to Create, Track and Rank CRO Hypotheses So You Know What to Test

by Richard Bayston

Last updated on July 3rd, 2018

CRO makes big promises. But the way people get to those 300% lifts in conversions is by being organized. Otherwise, you find yourself in the position that a lot of marketers do: you do a test, build on the result, wait a while, do another test, wait a while… meanwhile, the big jumps in conversions, leads and revenue never really seem to manifest.

That’s because only a structured approach can get you in position to make the best use of your testing time and budget. This isn’t something you want to be doing by the seat of your pants.

In this post, we’re going to look at how to create and use that structure. You will need:

  • 1: A spreadsheet.
  • 2: That’s it.

There are a bunch of tools that will help you do this, but you can do everything you need to do to build killer CRO hypotheses in a simple spreadsheet.

When people think of tracking tests, they often think about something like this:

call to action test series


And you should absolutely do this. But this is tactics, not strategy. And it’s for when you already know what to test.

Before you move on to spreadsheets that track individual tests, you should track the testing process itself. This will let you select the most cost-effective tests first, building your individual testing process on a solid foundation.

You should have a list of hypotheses, ranked by the trade-off between how much they’re expected to move the conversion needle and how much they’re expected to cost.

Start creating hypotheses

Your hypothesis is the idea you’re testing.

I think a blue button with rounded corners will convert better than a hot pink one in the shape of a star.

There’s a bunch of evidence that it’s true on other people’s websites, but presumably you’re most interested in conversions on your site. Best practice doesn’t always carry over – check out this list of counterintuitive A/B test results.

That’s why you have to do this stuff yourself.

Building a list of hypotheses lets you identify the ones that will deliver big wins and not break the bank or eat too much precious time.

How do you build hypotheses?

Biochemist Linus Pauling thought that ‘the best way to have good ideas is to have a lot of ideas, and throw away the bad ones.’ He’s the only person ever to win two Nobel prizes he didn’t have to share, so he may have been onto something.

Start in a Sheet. Throw in as many ideas as you can about what might change conversion rate. Here are a couple of places to get those ideas:

  • Look through great CRO blogs and copy the variables they tested. You’ll see headline copy, CTA copy, CTA placement, button design – all pretty common A/B test fodder. There’s nothing wrong with starting with the obvious, and these are the go-to hypotheses for obvious reasons: they’re usually conversion elements, and the changes they can produce are big.
  • Go through your customer feedback, looking for comments on your website from the user’s perspective. There’s often gold dust waiting to be unearthed here, with customers telling you about barriers to conversion that you’d never think to look for.
  • Talk to your support team. They’ll hear customer enquiries at the point where using your site stops being intuitive and easy.
  • Talk to your sales team too. Especially if you have inbound reps fielding inquiries directly on the site, you can find out from them which questions they’re always getting asked, and redesign website flows to reflect customer interests.
  • Run surveys among current customers, asking them why they chose to buy from you. Ask them if there was a particular part of the site or buying experience that they loved, and if there’s anything they would change.
  • Survey visitors who don’t convert, and ask them why not.
  • Heatmap! Use heatmap and user recording tools to see user behavior. If your visitors don’t normally scroll down past two screens, then ‘shorter copy and CTAs near the top of the page will increase conversions’ seems like a sensible hypothesis.

Look at all the variables on your site, and consider which of them are likely to have any effect on consumer behavior. If you can control it and consumers interact with it, come up with a hypothesis for it: a way to see if you can make it better.

conversion rate optimization hypothesis

At this stage, don’t try to assess them. Some of your ideas will be terrible. Write them down and move on; you can throw them away later.

What does a good hypothesis contain?

Say you think that changing your CTA from a hot pink star to a big blue button will increase conversions.

So far, so good – and so, so likely to be true. But there’s still way more to test. Like, how big do you expect the change to be? You might not have a solid figure to put on this, but you can begin with a ballpark idea based on what you already know of the importance of CTAs and the effects other businesses have had testing theirs. It’s not 1%. Major changes to CTAs have big effects. More likely it’s 25%, 50% or more.

Build this into your hypothesis:

build this into your hypothesis


So in your sheet, your hypothesis looks like:

In this instance, you can look at a list of benchmark conversion rates by industry and compare yours. Then compare standard industry practice with yours. If your industry shows average 5% conversions compared to your 2%, and a big blue button is industry standard, you could be on a winner with this hypothesis.

Include this in your sheet. Don’t just list the expected result: list the reasons why you expect that result. That lets you and others on your team assess whether there’s a better way to calculate the likely effects. If you’ve already run plenty of A/B tests you might want to use your own past testing data as a source of efficacy predictions.

For instance, if testing major changes on conversion elements have produced conversion rate changes in the 75%-150% range in the past, it’s reasonable to think that 100% is a likely outcome.

After you complete a test, come back to your sheet and fill out the ‘outcome’ column too. It takes a couple of minutes, and it gives you a lot of data that’s useful when you’re creating and ranking hypotheses. You can compare your predictions to the real outcomes and assess the quality of the rationale behind your conversion hypotheses.

But it should go into even more detail.

Olivia Williams recommends building hypotheses that have both specific results and reasons:

  • Changing _______ into ______ will increase [conversion goal], because:
  • Changing _______ into ______ will decrease [conversion goal], because:
  • Changing _______ into ______ will not affect [conversion goal], because:

Building hypotheses with reasons forces you to speculate about how the change you’re making will affect your conversions.

That helps you to stay away from the hand-waving and woo and focus in on stuff that has a solid rationale behind it.

Ranking hypotheses

Using the expected rationale and the expected change, you now have a way to rank your test hypotheses in terms of payoff. Say these are your hypotheses:

Which of these should you test first?

I’d say the first one, right? Partly because the rationale sounds more solid, and partly because the payoff is 20 times bigger. That big payoff is what we’re looking for. If the reasoning behind why we expect it to be effective is solid, we should be basing A/B testing strategy around big wins.

If you don’t find this approach is structured enough, you can use a scoring system that lets you quickly rank hypotheses. Conversion XL’s Alex Birkett suggests a system that looks like this:

Hypothesis scoring spreadsheet

If that looks good, you can get it for free here and tweak it to suit your data.

That big payoff is what we’re looking for. If the reasoning behind why we expect it to be effective is solid, we should be basing A/B testing strategy around big wins.

How do you know if a hypothesis makes sense?

It’s pretty obvious that ‘change the button to look like a button’ is a sensible idea. And ‘use haikus for product descriptions’ is a bit off the wall. But we’re going to all this effort to avoid doing stuff without thinking it through.

To see if the reasoning behind a hypothesis is sound, we can check to see if there are sources for the statements we’ve based it on. If you’re considering testing button color because you think color makes a difference to whether a button gets clicked, you can easily find tons of evidence from marketing and other sources to back you up.

There are essentially two types of data to use for this decision.

  • Quantitative data – data about quantities, about numbers. If you can easily turn up 5 studies from businesses in your space where they did X and got Y, there are 5 good reasons to think that if you do X, you might get Y too.
  • Qualitative data – data about quality. This is less easy to come by, but you can ask users on social. Bear in mind that what people say they want and what they want are not the same!

What about the cost?

Say these are your hypotheses:

CRO hypothesis with costs

Which is better?

Both offer the same payoff.

But one is relatively easy and simple; the other will cost many dollars in designer and developer time. This is the factor that hardly gets mentioned in blog posts on the subject: what you’re looking for here is big wins with good ROI.

Consider the total cost of implementation

Figure the hourly rate of everyone who’s working on the project. If you’re going to use an hour of developer time, an hour of your time and an hour of designer’s time, add those rates up. Factor in the cost of any tools you have to use too.

Remember this is about the total cost – every hour of effort from everyone involved, from the first conversation about it to checking through the results. It’s all time away from other projects.

If you’re working alone or with a small team, cost is actually even more of an issue: your time is the most important and non-elastic resource you have. Cost it out by your hourly rate, because every hour you don’t get paid is money you spent.

What about business metrics?

Conversions at the level of a website signup button aren’t sales. If you optimize for conversions and don’t link them to sales, you can wind up getting great conversion numbers and poor sales or high churn. If you’re getting more clicks and fewer dollars, that’s not a win.

Going into this in detail is beyond the scope of this post, but it makes sense to track business metrics – sales, churn, revenue – along with conversions and in the same sheet.

Consider making predictions about sales as well as conversions in your sheet, and measure the success of your CRO efforts by business metrics, not conversions alone.


Doing this will seriously throw some light on what is and isn’t a good bet. If you approach CRO in a positive mindset you’re likely to focus on the potential rewards – especially if those rewards look big. With the total cost of implementation to consider, lower rewards that cost less to grab might look like a better option.


You don’t test your website. You test hypotheses. If they’re well-constructed and sensible you get reliable results. And if you know in advance which are likely to deliver the biggest payoffs, you won’t wind up with inconclusive results that don’t tell you anything you can use.

No Comments


Get updates on new articles, webinars and other opportunities:

Richard Bayston

Richard Bayston creates copy and content at, mostly about tech, digital marketing and content strategy. When he’s not doing that, he’s arguing amicably with his wife and Googling the answers.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Show Me My Heatmap

Ah, @CrazyEgg I really do love you! So useful evaluating how users are interacting with all aspects of our redesign

Mike Halligan


What makes people leave your website?