Just How Proven Are *Those* Proven CRO Tips?

In an article I wrote for Kissmetrics last month, I called for marketers and copywriters to be wary of so-called “best practices” and I believe you should treat CRO case studies with the same skepticism.

Space constraints in that article prevented me from exploring the idea further, but I knew that I wanted to write something else about how blindly following best practices can be dangerous.

I’m not the first to do so on this blog, so I need to tip my hat to Sherice Jacob for her post on incidences in which CRO “best practices” failed. In this post, I’ll be expanding not just on the how but also why best practices often fail.

Borrowing Hypotheses -> Conflicting Evidence

Image Source

One of the most common problems with CRO case studies is that testers read them, then immediately implement exactly the same test on their own website. This doesn’t always go well.

The problem with this is that borrowing a test from another site also borrows the hypothesis behind it. That might not be a problem for very general tests, but it definitely can be for something more specific.

For example, let’s say that a marketing team develops the following hypothesis:

Research tells us that many of our visitors are hesitant to sign up because they think the software will be too difficult for them to use.

As a result of that hypothesis, they decide to test the following headline:

Effective, easy to use CRM software for agencies

This results in a 60% improvement in leads, and the company writes a CRO case study about it. Another marketer might read it and think to borrow the structure of that headline to test on their own website: Effective, easy to use X for Y.

However, it’s very unlikely that this marketer is facing the same issue that inspired the test. Their visitors, for example, might have an issue with the cost of the software, its performance compared to competitors or something else entirely.

Bonus reason for doubt: If the second business also writes CRO case studies, they might write up the experiment and put out conflicting evidence that will confuse the matter even further.

Different Audiences React Differently

Image Source

Different people like different things. That’s a no-brainer, right? Judging from the way a lot of testers treat case studies that I just looked at above, maybe it’s not.

There’s a nice post on the Kissmetrics blog that deals with obvious A/B tests to try, and the good thing about the tips contained within is that they make suggestions about what to test without making assumptions about your audience.

And exactly what constitutes an audience depends on a huge number of different factors, like:

• B2B vs. B2C
• Age
• Gender
• Technical knowledge
• Browsing habits (mobile, tablet, desktop etc.)

A lot of best practices and CRO case studies don’t take these into account, despite the fact it’s pretty unlikely that a B2C company will have success if they try to apply principles taken from a case study by a B2B organization.

When I look at split testing case studies, I try to look at what they changed rather than how they changed it. While the latter probably won’t be relevant to the crowd I’m working with, the former may represent an interesting aspect of the site to play around with.

Not Going Far Enough

Copying a test verbatim can be dangerous because it can lead you to discount things that might actually work if done just a little differently. I know this because it happened to me.

I was working with a company and we decided to implement a compact signup form on the homepage. We copied the format directly from a successful test we’d seen elsewhere:

Name:
Email address:
[Sign up]

The results were pretty ineffectual and, if we’d taken the case study we’d read as gospel, we might have concluded that our visitors needed more information before signing up.

But I wasn’t ready to give up yet, so I tried the following instead:

Desired username: 
Email address:
[Quick signup]

We saw an improvement of around 150% despite the tests appearing, on the face of it, very similar. The second form implies, in a way that the first doesn’t, that users can be face to face with the software after just a couple of clicks.

In this case, it seems that our audience was looking for something that emphasized the speed element of the form. Had we given up on the form after the original (copied) test, we never would have known that.

Other Factors

Image Source

In addition to the above, there’s a whole bunch of other factors that affect the results of split tests. I recommend that you watch out for the following before you even think about trying to replicate a test:

• Seasonality – When was the test conducted? Buying habits change throughout the year.
• Motivation – What led the company in question to conduct the test? Did they want more conversions? Social recommendations? Leads?
• Metrics – What was being measured? Clicks? Signups? The metrics others want to improve might not be what you want to work on.
• Statistical significance – A 95% chance of improvement means that there’s still a 5% chance that the Original is actually better. How long did the test run for? What was its statistical significance?
• SEO – One to watch out for. Messing around with your headlines, even if they perform better in tests, can unseat you from your position in Google.

Is It Worth Reading CRO Best Practices At All?

In a word, yes. And I’ll get to why below, but first I’m going to put my cynical hat on.

CRO testing is big business, and blogging (guest or otherwise) is still viewed as an important traffic driver. Because of this, some companies will look for ways to frame tests to make them look more impressive than they are.

If a case study draws you in with a title like “WE CHANGED THIS BUTTON COLOUR AND GREW 9000%!!!”, then spends paragraph after paragraph talking about what their business does, I’d get out of there quick.

If, on the other hand, a case study or an article about best practices is thought-provoking and backed up with evidence I’d say it’s absolutely worth reading provided you take the context into account.

You may not come out the other side with a quick fix, but it will get you thinking about elements of your own site that you could be testing. In my eyes, that’s what gives CRO case studies their true value.

Any interesting stories about attempts to replicate something you’ve read in a CRO case study? Pop them in a comment below!