We are all united in a common struggle: designing products, apps, and websites that engage, entice, and delight our customers.
So how do you know for sure that you are actually providing what they’re looking for?
A/B testing can be a lightweight, sustainable, and time-saving activity that helps you get the biggest returns on the experiments you’re running on your site.
While the value of A/B testing is no secret, we’re finding that people still struggle to know where to start, and are unsure about whether they’re conducting experiments correctly. They’re afraid they’ll mess up or get an unexpected result.
For this, we tend to take the Bob Ross mentality:
There are no mistakes, only happy accidents.
Even failed tests are learning experiences, so don’t be intimidated! We are here to help.
In this beginner’s guide to A/B testing, we’ll walk you through:
- Launching your first A/B tests
- How to optimize an A/B test you’re already running
Let’s dive in.
Why Invest in A/B Testing?
The goal of using a conversion rate optimization (CRO) tool like Crazy Egg is to build a better website.
And how do you build a better website?
- You start by defining what “better” means for you, your business or organization, and most importantly: your online visitors and customers.
- You learn what motivates your visitors to click, engage with your content, and move through all of your different pages until they reach your desired goal(s).
- Then you test, test, and test some more to see how you can help visitors get to your goal faster and with fewer distractions. If you follow steps 1 and 2, but fail to act on your findings, you’re leaving money on the table!
There are many reasons why people choose not to A/B test, ranging from worries about time and monetary investment to not being sure how to get started, and being uncertain about how to interpret results.
One of the reasons why A/B testing is a part of our product offering is to keep everything in-house.
You can conduct high-level research on the behavior of your site visitors with our Snapshot Reports, drill down into individual customer experiences with Recordings, and then easily act on your findings by making changes to your site without the assistance of a developer, using our Editor tool.
While there are a variety of heatmap, recordings, and landing page editing services out there, keeping your research and tests in one platform is an easy way that you can streamline your CRO process and reduce any potential barriers to entry.
Before you start an A/B experiment on your site, you’ll need to have an idea of what conversion you’d like to improve.
- Are you trying to increase the number of CTA submissions on your homepage?
- Do you want to motivate a higher number of people to submit a contact form?
- Are you interested in increasing your sales, signups or downloads?
- Is there a specific page you’d like visitors to reach more quickly, or more often?
- Do you want to improve the length of time people remain on your page?
- Do you want to decrease your bounce rate?
Once you’ve got a goal in mind, we recommend setting up some sort of heatmapping or user behavior software on your site. If you choose Crazy Egg, our site reports are called “Snapshots.”
They’re highly visual, and let you slice your visitor data five different ways:
- Heatmap: See hotspots of click activity at a glance
- Scrollmap: Learn how far people are scrolling down your page
- Confetti: Segment your visitors by 22 different filters
- Overlay: Find out the exact click count and segment visitors on each element
- List: Discover the popularity of each of your elements by seeing them in a tabular view
Using this data-driven approach will help you decide what changes could impact your goal, based on the actions of your actual website visitors.
The Biggest Bang for Your Buck: Multi-Armed Bandit Testing
Even when you conduct due diligence using the tools we mentioned above, we understand it can be nerve wracking to launch a major (or even minor) modification to your website. After all, it’s your online castle!
While the traditional A/B testing approach is a 50/50 split testing method (where half of your website visitors see the control and half see the variant), we at Crazy Egg use the multi-armed bandit approach.
The reason for this is that we’re aiming to minimize risk on losing variants and to help you reach a verdict more quickly.
What is Multi-Armed Bandit Testing?
If you’re not familiar with the multi-armed bandit methodology, start by picturing a casino with a row of slot machines. You’re on a tight schedule and have a fixed budget, so you can’t play every single machine until you earn a payout. Your goal is to find the machine that will give you the most money for the least amount of attempts.
So, what do you do? Start by picking three or four machines to focus on and then begin to experiment.
As soon as you notice that one machine is paying out more money, you start investing more of your funds into that machine and scale back on the machines that aren’t delivering as much of a return on investment.
We take the same approach to website updates.
When you launch a variant as a part of your test, we start by splitting the traffic 50/50. This is known as the exploration phase.
Once visitors start converting on one variant over another (or the control), more traffic gets diverted to the most successful version.
This reduces the likelihood of you missing out on conversions from unsuccessful variants and helps you verify whether success can be sustained.
Over time, more and more traffic will get funneled to the successful variant. This is known as the exploitation phase.
If a variant that had been performing poorly starts to perk up, then more traffic will be diverted over to see if it will become the most effective variant over time.
How Hands-On Do I Need To Be?
The best news about this methodology is that it’s a bit of a “set it and forget it” approach.
On the Crazy Egg platform specifically, our algorithm handles the calculations behind the scenes and checks the data multiple times a day to determine whether the control or a variant has the better conversion rate.
Since we take care of adjusting the traffic split for you, you don’t need to monitor the results until we’ve reached statistical certainty. At this point, 95% of your website traffic will be diverted to the winning version of your site.
If the variant wins, you can either have a designer on your team make the changes to your website code, or leave the test running and launch a new variant from there.
Ideally, you will constantly be swapping in new updates so you can keep making improvements to your conversion rates and learning about the navigation preferences of your site visitors.
Every Website is Unique
While we often offer best practices in terms of how to run your A/B test and give you suggestions for where to start testing, the value of running an A/B test is to see what changes work with your audience.
The value of general best practices is to give you a starting point for where to begin, but A/B tests are what ensure that your website is being optimized to your specific audience.
What may move the marker for one website may have the exact opposite effect on another.
We could tell you that changing a CTA color from red to green resulted in a 78% conversion rate for a handful of customers, but that doesn’t mean you will have the same conversion rate on your site.
This is where personalized research comes in. If you’re interested in learning more about how you can gather insights on the website behavior of your visitors, check out our getting started guide!
In the next article, I’ll cover how and where to start your analysis so you’re not shooting in the dark when it comes to your A/B test runs.