In an article I wrote for Kissmetrics last month, I called for marketers and copywriters to be wary of so-called “best practices” and I believe you should treat CRO case studies with the same skepticism.
Space constraints in that article prevented me from exploring the idea further, but I knew that I wanted to write something else about how blindly following best practices can be dangerous.
I’m not the first to do so on this blog, so I need to tip my hat to Sherice Jacob for her post on incidences in which CRO “best practices” failed. In this post, I’ll be expanding not just on the how but also why best practices often fail.
Borrowing Hypotheses -> Conflicting Evidence
One of the most common problems with CRO case studies is that testers read them, then immediately implement exactly the same test on their own website. This doesn’t always go well.
The problem with this is that borrowing a test from another site also borrows the hypothesis behind it. That might not be a problem for very general tests, but it definitely can be for something more specific.
For example, let’s say that a marketing team develops the following hypothesis:
Research tells us that many of our visitors are hesitant to sign up because they think the software will be too difficult for them to use.
As a result of that hypothesis, they decide to test the following headline:
Effective, easy to use CRM software for agencies
This results in a 60% improvement in leads, and the company writes a CRO case study about it. Another marketer might read it and think to borrow the structure of that headline to test on their own website: Effective, easy to use X for Y.
However, it’s very unlikely that this marketer is facing the same issue that inspired the test. Their visitors, for example, might have an issue with the cost of the software, its performance compared to competitors or something else entirely.
Bonus reason for doubt: If the second business also writes CRO case studies, they might write up the experiment and put out conflicting evidence that will confuse the matter even further.
Different Audiences React Differently
Different people like different things. That’s a no-brainer, right? Judging from the way a lot of testers treat case studies that I just looked at above, maybe it’s not.
There’s a nice post on the Kissmetrics blog that deals with obvious A/B tests to try, and the good thing about the tips contained within is that they make suggestions about what to test without making assumptions about your audience.
And exactly what constitutes an audience depends on a huge number of different factors, like:
- B2B vs. B2C
- Age
- Gender
- Technical knowledge
- Browsing habits (mobile, tablet, desktop etc.)
A lot of best practices and CRO case studies don’t take these into account, despite the fact it’s pretty unlikely that a B2C company will have success if they try to apply principles taken from a case study by a B2B organization.
When I look at split testing case studies, I try to look at what they changed rather than how they changed it. While the latter probably won’t be relevant to the crowd I’m working with, the former may represent an interesting aspect of the site to play around with.
Not Going Far Enough
Copying a test verbatim can be dangerous because it can lead you to discount things that might actually work if done just a little differently. I know this because it happened to me.
I was working with a company and we decided to implement a compact signup form on the homepage. We copied the format directly from a successful test we’d seen elsewhere:
Name:
Email address:
[Sign up]
The results were pretty ineffectual and, if we’d taken the case study we’d read as gospel, we might have concluded that our visitors needed more information before signing up.
But I wasn’t ready to give up yet, so I tried the following instead:
Desired username:
Email address:
[Quick signup]
We saw an improvement of around 150% despite the tests appearing, on the face of it, very similar. The second form implies, in a way that the first doesn’t, that users can be face to face with the software after just a couple of clicks.
In this case, it seems that our audience was looking for something that emphasized the speed element of the form. Had we given up on the form after the original (copied) test, we never would have known that.
Other Factors
In addition to the above, there’s a whole bunch of other factors that affect the results of split tests. I recommend that you watch out for the following before you even think about trying to replicate a test:
- Seasonality – When was the test conducted? Buying habits change throughout the year.
- Motivation – What led the company in question to conduct the test? Did they want more conversions? Social recommendations? Leads?
- Metrics – What was being measured? Clicks? Signups? The metrics others want to improve might not be what you want to work on.
- Statistical significance – A 95% chance of improvement means that there’s still a 5% chance that the Original is actually better. How long did the test run for? What was its statistical significance?
- SEO – One to watch out for. Messing around with your headlines, even if they perform better in tests, can unseat you from your position in Google.
Is It Worth Reading CRO Best Practices At All?
In a word, yes. And I’ll get to why below, but first I’m going to put my cynical hat on.
CRO testing is big business, and blogging (guest or otherwise) is still viewed as an important traffic driver. Because of this, some companies will look for ways to frame tests to make them look more impressive than they are.
If a case study draws you in with a title like “WE CHANGED THIS BUTTON COLOUR AND GREW 9000%!!!”, then spends paragraph after paragraph talking about what their business does, I’d get out of there quick.
If, on the other hand, a case study or an article about best practices is thought-provoking and backed up with evidence I’d say it’s absolutely worth reading provided you take the context into account.
You may not come out the other side with a quick fix, but it will get you thinking about elements of your own site that you could be testing. In my eyes, that’s what gives CRO case studies their true value.
Any interesting stories about attempts to replicate something you’ve read in a CRO case study? Pop them in a comment below!
6 COMMENTS
Comment Policy
Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.
Good Article. Thanks for sharing.
This is something good”” “Just How Proven Are” Hey, what a decent substance. This is essentially what I was looking of late.
Rolling out improvements in any site is not a simple errand and CRO rehearses regularly fizzle much of the time. The reasons you said are entirely consistent.
The second point is so genuine, diverse individuals like distinctive things. Every individual varies from the other in his decision, recognitions and mindset. So they respond distinctively to changes. These responses are more terrible now and again when they don’t care for or acknowledge the change. In this way, I will say that all the CRO chiefs must consider these variables while performing any sort of transformation rate advancement.
Much appreciated once more!
Hey, what a good content. This is pretty much what I was looking lately.
Making changes in any site is not an easy task and CRO practices often fail in many cases. The reasons you mentioned are quite logical.
The second point is so true, different people like different things. Each person differs from the other in his choice, perceptions and way of thinking. So they react differently to changes. These reactions are worse sometimes when they don’t like or accept the change. So, I will say that all the CRO managers must take these factors into account while performing any kind of conversion rate optimization.
Thanks again!
@Art
Reward purpose behind uncertainty: If the second business likewise composes CRO contextual analyses, they may review the test and put out clashing confirmation that will confound the matter considerably advance.
regards
rakhi
Let me throw a dovetail question in.
There are thousands of CRO products on the market – that all promise 10% – 20% conversion bumps.
Lets say my site converts at 5% – a 20% conversion boost will now yield a 6% conversion
that means all i need is 50 or 60 of these modules/plugins/email things and my site will convert at 100% – right?
I posed the question in almost every booth at IRCE this year with the caveat that I would buy their overpriced conversion service on the spot if they could explain why this plan wont work – left without one ding on my debit card
Nicely put, Mitch. Seems to me that, in some respects, CRO is the new SEO – no shortage of snake oil salesmen who claim guaranteed results, even though it’s impossible to guarantee results. I wonder if there’s another article in amongst these ideas.
Thanks for reading,
A