Here at Crazy Egg, we want you to get the most out of all the conversion optimization tools you use – not just ours. That’s why we’re proud to present this complete guide to getting up and running with Optimizely. As one of the most popular A/B split testing solutions, Optimizely has helped millions of people improve their conversion rates with its straightforward, easy-to-use system. Here’s how to make it work for you.
What Does Optimizely Do?
Optimizely bills itself as an “Experience Optimization” platform, meaning it helps you optimize between two different variations of a webpage (often split 50/50 between your visitors – hence the name “split testing”) to determine which one converts best. A “conversion” is any action you want your users to take as a result of seeing the page. It can be as simple as signing up for a free trial or subscribing to your newsletter, or as involved as adding an item to the shopping cart and proceeding to checkout.
You can also run tests that let you test many different items on a page – known as multivariate testing. In Optimizely, these are called “multi page tests.” Although this process can take longer to see definitive results from, it’s ideal for getting relevant information on what resonates with your target audience, quickly.
In short, no matter what you want to test, Optimizely helps you do it.
What Is Split Testing?
Split testing involves taking two versions of a page, with generally a single change, and pitting them head-to-head against each other. Typically, fifty percent of your users will see one, and fifty percent will see another. You can gauge the effectiveness of which page (the one with or without the variation) by checking important numbers like clicks, subscribers or conversions – all of which Optimizely reports and presents to you via its dashboard interface.
There are other kinds of tests available too, such as multivariate (testing multiple elements on a single page) and multi-page (testing a path that a customer would typically follow to order). More on those further down in the guide.
Getting Up And Running
After signing up for an Optimizely account and logging in, you’ll be at your home page. If it’s your first time logging in, you’ll be asked to define your very first project. An account can have multiple projects but for now, we’ll simply concentrate on one. Within each project, you have the split test itself (called the Experiment) as well as the changed version of the page that some of your visitors will see (called the Variation).
The first step is to put the Optimizely code onto your page so that it can start tracking. It’s a good idea to put this code on every page of your site, as well as external landing pages.
This code is a single line and is found in the Settings tab on the Home page:
If you don’t feel comfortable editing HTML pages, you can email the code to your developer and have them add it. For proper tracking, make sure to add this snippet of code as high up near the beginning of your code as you can, within the <head> tag. If you’re using a third -party e-commerce platform or content management system like WordPress, Optimizely also has integration plugins you can install that will handle it for you.
You’ll see that there are several other tabs here that you’ll be getting a great deal of use out of.
Let’s take a closer look at what each tab does.
Each tab in Optimizely has several sub-categories under it. For example, clicking the Overview tab lets you access your Experiments, define your Audiences, change Dimensions of your test and see Change History over time.
Overview > Experiments
To the right of the Overview/Experiments section is where you’ll start a New Experiment. If you have other Experiments running, you’ll see them below this tabbed area and also see if they’re currently running or paused.
You can start or pause a campaign, as well as duplicate it or archive it. Once you click on an experiment, it will highlight, and you can make changes via the sidebar, such as changing the title, how much traffic to allocate to it, etc.
You can also filter your experiment by name, description or status (such as if an experiment is running or paused).
Experiments > Goals
Once you click on Experiments, you’ll see the goals you’ve defined for that particular experiment:
This only lets you see what goals you’ve defined. If you want to create a new goal or edit an existing one, you’ll need to open the editor and click on the Flag icon from the Goals menu:
Overview > Audiences
Here you define audiences that will see your Experiment. Using #hashtags will make it easier for you to segment them based on specific segmentation points like which mobile #device they use, #browser they prefer or even #location. Here’s an example of an Audience defined as Google Chrome users in New York:
Overview > Dimensions
You can also define specifics about each user that visits the Experiment, such as whether or not the user is logged in. You can also segment users based on where they clicked – for example, recipes for lasagna versus recipes for enchiladas and then use this data to optimize future campaigns tailored to their likes (Italian versus Mexican food). The Enterprise plan of Optimizely allows you to create custom dimensions when segmenting your users.
Overview > Change History
As the name implies, Change History will let you see changes made over time. You can see who made the change, when and to what experiment. Certain enterprise plans will also show you what, precisely was changed.
Moving on to the next tab, integrations, you’ll see all the different possible integrations you can have. These integrations range from analytics and heat mapping to third party audience segmentation and much more. You can find a full list of which integrations can be made according to your account and pricing level by clicking here.
You can also see which experiments are using which integrations, and optionally toggle them on and off.
Collaborators allows you to see who on your team is working with you on various conversion goals, as well as their roles within the experiment. You can also delete collaborators if you have sufficient privileges.
Determining What To Test
Before you can even begin to create your first split testing experiment, it’s a good idea to consider the possible ideas to test first. Of course, if you were to ask your team for ideas, you’ll probably get more answers than you can reasonably conduct at any one time. That’s why it’s a good idea to be smart about your testing and choose the things that will give your business the biggest gains in terms of customer acquisition, retention, and ROI.
Of course, how do you know which changes will bring those gains? It’s all about creating a company culture that’s centered on optimization, and understanding which metrics will bring you those “quick wins” over and over again.
The Goal Tree
Optimizely itself recommends creating a “goal tree” where you outline the metrics that define success for your company. Here is one such example:
As you can see, each step can be broken down into bite sized, “testable” pieces, with corresponding indicators that you can measure via Optimizely. They have some similar templates you can download if you’d like to create your own goal tree.
Gathering Feedback And Prioritizing Test Ideas
It’s also a smart idea to create hypotheses based on what visitors are currently doing on your site and interacting with employees. Foster a culture of open communication and optimization. Work with your staff at all levels of the customer experience – from tech support to reception, marketing and sales to IT. This will ensure everyone is on the same page and limit the number of technical bottlenecks, miscommunications and fragmented data that often sabotages even the most promising tests.
Once you have a solid range of ideas to test, the next step is to actually go in and create your first experiment.
Trying Out Your Experiment Hypothesis
Depending on whether or not your proposed idea is simple or more involved, it may be a good idea to use a design testing template to keep all the changes documented and uniform among you and your staff. Optimizely has a simple design template as well as more downloadable templates that you can use to document the experiments you’re running.
Go through this process with your team to have a set of testing goals and outcomes you’re looking to reach, as well as a history of what works, and what may not, with your target audience.
Creating Your First Experiment
Creating your first experiment in Optimizely is easy (and exciting!). Once you’ve prioritized what you want to change, it’s time to start making changes. The good news is, you don’t have to sift through mountains of code to make real, actionable changes on your site. You can do it with Optimizely’s visual editor.
The Visual Editor
Optimizely’s Visual Editor is a WYSIWYG editor (What You See is What You Get), meaning the changes you make on the screen are an accurate reflection of what your website users will see.
You can edit your experiment title, edit the variations, add new goals, define the audience or pause the experiment. If you’d like to edit the code directly, you can do that as well. More important though, is the context menu that you have access to.
The context menu tells you what kind of element you’re working with on the page, whether it’s an image, text, etc. Edit Element brings you to the next screen:
Here, depending on what kind of element is, you’ll be able to make changes to the HTML, stylesheet and text (to name a few).
Clicking on Edit Style, for example, allows you to change the CSS properties of the element:
Oftentimes elements on websites will overlap with each other, so it can be difficult to narrow down the one you want to change or track. If that’s the case, the Select Container option will help.
Using The Select Container Option
Here you can edit the parent (or area) of an element if you’re having trouble selecting it. Just choose Select Container, and a list of the containers will appear.
There you can choose the one you’d like to edit. But what if the item you want to edit only shows under certain conditions, such as a drop down menu that changes color when you hover over a specific section? Optimizely’s Interactive Mode lets you change those points.
Interactive mode loads the site in much the same way as a browser would, but still lets you have your contextual menu and selective editing options. Here you can highlight the item you want to edit and then continue through the Visual Editor to change it.
You can switch back and forth from editing mode to interactive mode by clicking the button in the top right of the screen:
Of course as you use the Visual Editor to move, change, resize and edit items, it can be very tempting to want to rearrange things on the page. One of the commands in the menu, “Move and Resize” lets you do this – but don’t get too carried away.
Move and Resize is ideal for changing things in small distances (around 100 pixels or less) but large moves are better using the Rearrange feature to help avoid browser and mobile compatibility problems.
Working With Editor Settings
When loading Editor Settings, a common mistake is to assume that the page that you’re editing is going to be the one your experiment runs on.
This isn’t always the case. You’ll need to use Options > URL Targeting to set up the URLs you want to use in the experiment. Editor settings only lets you choose the page (or template) you want to make changes to (but not necessarily use as the experiment.
If you’re making changes across multiple pages, such as product description pages for an e-commerce site, you’ll want to load the core template into the editor first and then use URL targeting to direct customers to the edited pages accordingly.
Creating A Variation
After you set up your first experiment, you’ll then need to create a variation. There are two main ways to do this – through A/B or “split” testing or multivariate testing. You can also run multi-page (funnel) tests.
When you first create an experiment, you’ll be asked to choose the type of test you want to run:
Each one has their own pros and cons. An A/B test is the simplest, where you make one change to a page and test it (for example, the headline, or the color of the call-to-action button).
The problem here is that you are limited to just two types. Of course, people can do A/B/C/D split tests (also called A/B/n tests) but the traffic segment that gets funneled to the subsequent pages becomes smaller and smaller – and thus you get less statistical significance to work with (more on that further in this guide).
With a multivariate test, you can make a combination of changes on your pages and see which ones resonate best with your customers. An example would be changing the headline AND the call to action on a page.
You can mix and match different variations (hence the name “multivariate”) to create unique options.
The issue here, as you might expect, is that you’ll need to have a substantial amount of traffic coming to your pages in order to properly test each factor. Even if you do have considerable traffic, it may be difficult to test more than a dozen or so concurrent changes and get the kind of accuracy you need to make definitive decisions moving forward.
Finally, there’s multi-page or “funnel” testing, so called because it’s often used in testing the sales funnel “path” that your visitors go on. This type of testing is useful for testing multiple pages such as a landing page > product description detail > checkout > thank you funnel.
As with A/B split testing, multi-page tests are very easy to set up and track, but again are limited on how many points can change within the test. Too many changes across too many pages and you risk muddling the data that you’re collecting without ever knowing which points lead to measurable increases in clicks or conversion rates.
Choose The URL(s) To Experiment With
URL targeting allows you to set the URLs you want Optimizely to run on. This is ideal if you’d like to split test only a certain page or group of pages before you roll out the changes across your entire site. To access this page, simply go to Options > URL Targeting and you’ll be presented with the screen below:
Here you can add or remove the URLs you want your experiment to run on. Each URL can be one of four types:
- Simple match is the default and is best for testing a single page.
- Exact match is best for adding query or hash parameters to the URL. It should NOT be used to target visitors who fall under a certain query (i.e. visitors who searched your site for “red shoes). That would fall under the “Defining Your Audience” section below.
- Substring match only runs the experiment on pages that match a certain string of text (for example, when you’re changing a single element, like a footer, across the entire site).
- Regular expression generally includes everything else that doesn’t fall into one of the aforementioned categories.
More information on the different types of URLs, including examples can be found here.
Defining Your Audience
The next step after you’ve determined which pages your experiment will run on is to define who will see the changes. Optimizely gives you many different ways to segment your audience. For instance, you can show your experiments to:
- Users who are on different browsers
- Users on different mobile devices
- Users who follow certain parameters (such as logged in versus not logged in)
- Users from certain countries or who have certain language settings
- Users who come to your site in the morning versus during the evening
- And so on (this is by no means an exhaustive list!)
Your home page shows all the audiences you have targeted for specific experiments. You can also edit these parameters or create an entirely new audience group.
To change an audience, simply go to Overview > Experiments and click the Edit link in the sidebar:
Or, if you’re in the editor, you can click the Audiences icon at the top right:
If you’ve already created an audience for a previous experiment (such as users who come to your site using Firefox), you can still use that audience on a new experiment. If you’d rather create an audience “from scratch”, you can do that too by choosing the Create New Audience button:
Create a name and description for your audience that’s easy to remember, so you can use it in later experiments if you wish.
Once you click Create a New Audience, you’ll see all the different options you can incorporate for segmenting them. You can further divide them with And/Or operators.
Keep in mind that Optimizely checks for these particular conditions you set on every page you’re running the experiment on. So, for example, you could target an experiment to all audiences using Firefox on Windows, but if someone visits the page using Firefox on their PC in the morning, but then comes back using Chrome on their mobile phone, they won’t see the experimental page.
It’s also worth noting that different account plans have different lists of what you can define an audience by. To see which ones are included in your particular plan, click here.
Working With Personalization
Personalization in Optimizely is a multi-faceted set of conditions that can be overwhelming for the first-time or novice user. Thankfully, we’ll be breaking it down into small, actionable chunks to help you understand how to perfectly target your audience with precision while delivering the kind of customized experience customers will love.
To get started, navigate to the Implementation link:
Next, you’ll want to add a new page. This tells Optimizely where the personalization experience should take place:
After adding your new page, you’ll be asked to choose a category for it. The categories you can choose from include things like Home, Article, Checkout, etc. If the page category you want to deliver the personalized experience on doesn’t really fit any of the category types, you can choose Other. This is done so that Optimizely can recommend Audiences for you based on real-time personalization details.
You’ll then tell Optimizely where the URL is located. Use URL matching to determine, for example, if you want a single page, an entire directory of pages, and so on. Use Manual if the page can’t be identified with a unique URL (for example, dynamic pages where the URL always changes). You may need to work with your development team on implementing manual URLs as there is some code editing involved.
After adding your URL, you’ll then see the page you defined under the Pages tab. Here is an example of a Product Detail page:
Once you click on the page name, you’ll be taken to the Events section.
Working With Events
Events are the actions people take on your website. Optimizely tracks these events and they aren’t just clicks. Whenever someone searches, adds a product to the shopping cart, shares on social media and so forth, these items are tied to and tracked automatically. There are many different options, including:
- Click Events – which measure clicks on buttons, offers and so on.
- Pageviews – tracked automatically for each page you set up under Implementation
- Custom Events – events you define which can’t always be tracked by click, such as submitting a form.
To set up a custom event, click on the Events tab and then New Custom Event:
Here, just like you did with pages, you’ll define a category. This will create a call to the API (a set of advanced programming instructions) with the name EventName, which you can then copy and paste into your website to enable the custom event tracking.
Working With Tags
Tags are the final part of the interaction tracking system. Tags allow you to specify parts of the page(s) that visitors interact with. These include things like the type of product and what it costs. Altogether, pages, events and tags allow you to create a multi-pronged “tracking system” that lets you get down to granular details on who’s interacting with what, and how.
To set up tags, simply click on an element in your page and choose Add a Tag. You can choose to tag things by category, subcategory, price and other.
Creating The Audience
Now that you’ve created the page you want the personalized experience on, the items you want to track and the interactions you want to count, it’s time to bring in the audience, because it’s nearly SHOWTIME! To do this, select Audiences from the navigation menu:
Next click “New Audience”. There, you’ll be able to define all the specifics of the people you want to direct to your personalization-enabled pages. You can get deep into the details if you wish, and direct a number of different audiences to each page.
Creating A New Campaign
Now is when the magic happens. The campaign is what ties the whole process of pages, events, tags and audiences together. Fortunately, now that you’ve set up all the pieces, all that’s left is to bring everything together. Click on Campaigns from the main menu, then, as you’ve done previously, create a New Campaign and give it a memorable name.
You’ll then be asked to choose the page(s) you want to fall under the campaign:
As well as the audience you want to show the personalized pages to:
If a user happens to qualify for more than one audience – only one will be shown to them. To prioritize which experience they should see, simply drag and drop it to the top.
Next you’ll choose the metric you want to measure (Revenue, Add to Cart, etc.) Even if you have multiple metrics you’re measuring, the one you choose here is the most important:
But, of course, if everyone got a personalized experience, there’d be no way to tell how well the personalization aspect is performing. That’s why you then set a Holdback – a percentage of visitors who, even if they fit the requirements, would not see the personalized campaign. Optimizely recommends this percentage be around 5%.
Working With Dynamic Customer Profiles
There are lots of other ways to extend Optimizely’s audience profiling, tracking and targeting goals – some of which may require more advanced programming. For example, you can:
- Create dynamic customer profiles
- Use those profiles to create 1-1 personal interactions (for example, if a user was a Gold Member of your service, it would show “Welcome, Gold Member!” rather than simply “Welcome, Member!”)
- Understand the metrics of personalization
As of the time this guide was written, personalization options were just rolling out and are only available on select plans. You may not have the option to create a personalized experience if you do not have the appropriate account level.
Bringing It All Together With The Experience
The stage is set. The audience is ready. The show is poised to delight. Aaaand – action! The Experience is where everything comes together.
To create The Experience you want this personalized segment of users to have, simply click the Page in the Pages detail to open the Editor.
As before, when you were first introduced to the Visual Editor, you’ll be able to select a container and see the HTML and CSS that make it up. You can then edit them just as you would creating a new split test option from scratch:
When everything is good to go, you can preview your test before it goes live to make sure everything works as you want it to. Then simply hit Publish to make it live.
For more information on Experiences and Personalization – check out this how-to on Optimizely.
Tracking And Measuring Goals
Goals are the lifeblood of conversion optimization. Getting people to take the action you want them to take should be the focus of all of your tests, no matter which type they are. So once you have everything set up correctly, how do you track and measure those goals so that you’ll know what to improve and what’s worth keeping?
First, it’s important to understand what Optimizely categorizes as “goals.” There are many different types, including Click goals, Revenue goals, Pageview goals and even custom events. Let’s go over each one step by step.
Working With Goals
To understand your way around goals, let’s take a look at the broader options of the goals page. To create a new goal or edit an existing one, simply click on the flag icon in the editor:
You’ll be able to selectively edit goals according to the experiment(s) you’re running as well as prioritize different goals and add new ones. Clicking the icon above opens up a page where you can take all of these actions:
If you click on Create a New Goal, you’ll also be able to choose between goals you’ve already created for past experiments:
You can also drag and drop goals listed here to prioritize them and tell Optimizely which is more important.
It’s a good idea not to add a goal to a campaign that’s already running since that could skew the data you’re gathering.
Now let’s look at the different types of goals you can create and track in Optimizely.
Click goals are, as their name implies, goals which count the number of times a particular element was clicked on the page.
You can set up click goals by clicking the Goals icon or by opening the Visual Editor directly to select the element you want to track.
What if you create a variation to test, and want to track the change as a new goal (such as the addition of a new button?). You can do that by creating the element in the Visual Editor, then clicking it and choosing Track Clicks > Create New Click Goal from the context menu.
As with other items in the Visual Editor, you can always click an element and expand it to enter the Interactive Mode and edit its particular state (a button change on hover, for example).
You can also track multiple elements on a page as part of your goal tracking. While in the preview mode, simply click the element(s) you wish to track. You’ll see them highlighted with a red border:
Pageview goals track how many pages have been viewed. You’d track pageviews, for instance, to see how many times your “thank you” page had been viewed, to analyze each page in your funnel to determine where visitors may be dropping off, and so on.
As with any goal, you’ll start by clicking the Goals flag icon and then “New Goal” if that’s what you wish to create.
Under the “What to Track” section, choose Pageviews:
There are different types of URL matching types that you can set. See the section on URL Targeting to understand what these mean. You can always click the + button to add more URLs.
But what if you have goals that don’t really fall under pageviews or clicks? For you, there are custom event goals.
Custom Event Goals
Custom event goals are for goals that can’t ordinarily or easily be tracked using other methods. For instance, a form that always leads to a unique confirmation page, or a pop-up that appears based on when certain conditions are met (such as a coupon code for free shipping on orders over $XX). For those points, you’ll want to set up custom event goals.
Custom event goals do require some knowledge of jQuery in order to be able to create and execute the underlying code. For more information, see this link. You’ll also find details for testing custom goals based on your website or app.
What Defines Engagement?
You’ve likely noticed when creating goals that the default in Optimizely is set to Engagement. But what exactly does that mean?
To put it simply, an engagement is the opposite of a bounce. It measures the number of visitors who click on something, whether that be a form submission, a navigation link, etc. Not every click can be counted because of the delay when the user actually clicks the link, and the browser communicating with the server to display the result. That’s why it’s recommended that you set a Primary Goal in addition to Engagement.
Of course, even if you don’t have any other goals, engagement will do a general job of showing you the success or failure of your experiment. But it’s a smart idea to create other goals to test with as well.
It’s also best to implement the Optimizely tracking snippet sitewide so that you can see a broader range of engagement across all your pages — not just the ones you happen to be experimenting on at that point in time.
Advanced Goal Tracking
For developers and those with experience in leveraging API calls and jQuery, you can extend Optimizely goals tracking features to apply to more than just websites. Click below to learn how:
- How to Track Views on YouTube and Vimeo Videos – This page discusses how to track how many times a video was paused, played or watched to completion.
Understanding Statistical Significance
Finally you have your tracking in place, your variations set up and tracking enabled. You’re starting to gather data. Now, how do you determine a winner? It’s easy to look at the percentages and think you have a clear decision on which choice won out with your visitors — and most of the time you’d be correct in the assumption.
But understanding how Optimizely calculates this result will directly affect major changes and decisions you make. It’s one thing to see how a variation compares in an experiment. But what about when that variation is applied sitewide? Will the results still hold true?
On your results page, you’ll see the statistical significance of the test. Optimizely doesn’t declare a winner or a loser until you have at least 100 visitors and 25 conversions on each variation you’re testing. Oftentimes you’ll see results only when Optimizely has determined their statistical significance.
Essentially, we’re making inferences about an audience’s behavior that aren’t due to pure chance. So if Optimizely reports having 90% statistical significance on a particular experiment, there’s less chance of random actions and errors polluting the data you’re gathering.
It’s also important to look for the three possible outcomes – particularly when your original and your variation experiments have the potential for false positives and negatives:
- Accurate Results shows that there’s a clear winner and little room for error (it’s impossible to have 100% statistical significance on any test).
- False Positives show a difference between the data in your original and in your variation which, on the surface, looks to be significant but is actually just random noise.
- False Negatives – There’s no difference in results between the two, but your variation is the winner.
By default, Optimizely sets the statistical confidence level at 90%. You can lower this if you wish, however, this may increase the likelihood of error affecting your results. Higher significance levels decrease this probability but you will need a larger sample size to determine which is the true winner.
Optimizely will tell you whether or not a variation is outperforming (or underperforming) the baseline at some level of confidence. The range between the original and the variation are known as difference intervals. You can think of this as your “margin of error.”
It will also tell you how long you have to wait (approximately) to see statistical significance:
In some cases, you may actually see significant numbers of improvement, but no statistical significance yet – like this:
This happens when the sample size is still not large enough to be able to make a decision with confidence. In this case, Optimizely needs at least 300 visitors to be exposed to a variation in order to be able to calculate its confidence in winning or losing. As of the time this screenshot was taken, only 155 visitors had been exposed to any variation, which is not enough to be able to accurately predict a winner.
Keep in mind that adjusting your statistical significance slider will immediately affect your current experiments, in that either more time/traffic would be needed to gauge an accurate winner/loser, or changes in the difference interval would affect Optimizely’s ability to clearly predict it.
Be sure to read these articles that go over sample size, pollution and statistical significance:
- When Are You Ready For A/B Testing?
- 4 Surprising Sources of A/B Test Pollution
- When Should You Stop an A/B Test?
Working With Mobile
The mobile optimization section of Optimizely’s Knowledge Base details how to integrate Optimizely with various services for iOS and Android, as well as test ideas, creating mobile audiences and much more.
Integrating Third Party Products
There are a host of other platforms (including Crazy Egg) that integrate nicely with Optimizely. Learn more on integrating them by clicking the appropriate link here.
A More Technical Explanation
Of how Optimizely works can be found here
A Final Word On Working With Optimizely
As you can see, there’s a lot that goes into split testing, goal tracking and conversion optimization. Being able to accurately test, track and predict which changes you make that ultimately lead to improvements is just as much about hypothesizing ideas as it is about crunching raw numbers. In the end, Optimizely does a phenomenal job of the latter, while giving you the tips and support you need to do the former well, and create the kinds of tests that will lead to lasting improvements on your site.
Of course, any platform you use to conduct split testing is only as good as the data you feed it, so be certain that every test will have a definitive start and end, a conclusive result and have a dramatic impact on improving your conversion rate. Small, even “micro” changes like font color and size may seem easy to implement, but it’s the larger changes that often chart the course for the business’ development and growth. With this guide at hand and a platform to springboard from, you’ll have the foundations of the tools you need to drive that growth. Good luck, and be sure to let us know your thoughts!
*Featured Image Source