How to Choose the Right Testing Software For Your Business

by David Zheng

Last updated on August 1st, 2018

Every customer base is unique.

Your ability to win your own marketing victories depends on understanding the users visiting your website each and every day.

If you’ve spent any amount of time on The Daily Egg or similarly focused blogs, you know that thorough testing is paramount to your web business’ success.

You might be intuitive (or lucky) enough to correctly read your customers on any given campaign, but over the long run, success WILL depend on data analysis.

Despite the hundreds of voices claiming that marketing is an art, you’ve heard of Neil Patel, Avinash Kaushik, Danny Sullivan, and Brian Clark for one reason: They understand marketing is a science.

Unfortunately for the small business owner, data analysis is decidedly technical.

Simply reading a few blogs won’t be enough to turn data into dollars.

You’ll need a handful of powerful tools at your disposal.

Fortunately, there is an abundance of such tools available.

If you have a marketing team ready to pounce on this, these programs will provide an effective setup for efficient A/B testing.

If the burden falls on you, the business owner, there will be a slight learning curve.

But don’t worry, each of the programs listed here is highly successful, fully supported, and come with a dedicated team invested in making your experience flawless.

Choosing the Right AB Testing Software For Your Business

Keep reading for our review of seven A/B testing tools and examples we think can help your business.

1. Google Analytics

As the only free option on our list, Google Analytics is the baseline of all Internet data analysis.

It offers users the rare opportunity to draw directly from Google’s extensive data pool.

If you’ve never used Google Analytics before, or if you’ve wandered around the dashboard a few times without really grasping its potential, here’s a great place to start:

Any website owner can connect their site to Google Analytics and start receiving valuable data in minutes.

Take some time when you first set up your profile to click around and familiarize yourself with the layout of this tool.

It’s fairly intuitive, which is always a plus when you’re starting out.

Once you’ve learned the basics, it’s time to use this new-found data to drive revenue for your business.

This requires actionable steps, and you can begin here.

You’ll find that you have a wide array of options to choose from.

To get the biggest bang for your (nonexistent) buck here, I recommend starting by checking out the Behavior tab.

The overview itself is fairly helpful in breaking down simple analytics like page views, exit %, or average time on page:

This gives you a quick read on the health and overall performance of your site.

A lot of page views and low bounce rates means you’re doing something right.

So keeping tabs on this page is a practice to start implementing from the start.

But you also need to see how your content is performing, which means diving a little deeper into the Behaviors tab.

By clicking on the Site Content submenu and then selecting All Pages, you’ll see a readout like this:

What you’re looking at is a graph display of pageviews for a seven day period on your site.

This means you can see a day to day performance review of your entire site.

Better yet, when you scroll down you’ll see a bigger breakdown of individual pages on your site:

A summation of performance like this is extremely helpful in finding troublesome pages on your site.

From here, you can also take a more in-depth look at the Behavior Flow report in your menu.

When you click this tab, you’ll see a visual representation of how users behave on your site:

What you’re looking at is a compilation of pageviews, drop-offs, and where users decided to go when they were interacting with your site.

If they started on your homepage and clicked your “About Us” page, that would be represented in the next column.

If they interact further, they get moved into the “2nd Interaction” column, and so on until they leave your site.

This means you can see where users are leaving your sales funnel in correlation to any other actions they may have taken on your site.

A visual representation like this is an immeasurably valuable tool for understanding your user’s train of thought when they visit you.

You can optimize the flow of your site much better with just this one tool.

But there are other great ways to use Google Analytics too. Take the Conversions tab for example.

This requires you to set up specific goals that you want visitors to accomplish:

You get to pick which action qualifies as a conversion for your site.

You can allow Analytics to track conversions by acquisition sources like creating an account, referrals, or social media connections:

Or, you can set yourself up to track conversions by a secondary dimension, like page views or time one page:

Let’s say for example you want to set up a goal that involves users arriving at a specific location like a “thank you” page for a newsletter subscription.

All you’ll have to do is put in the URL for your final page as the destination for your goal.

This gives you a hit every time someone subscribes, which means you have an accurate number of how many subscriptions per day, week, or month you’re getting.

You can also see an overview of Multi-channel funnels and how they perform on your site,

This makes Analytics a very nuanced tool for understanding how users behave in certain situations on your site.

But keep in mind that analytics are limited in their usefulness in certain scenarios.

The reality is that your customer’s journey is more nuanced than pageviews or on-site behavior.

Google Analytics doesn’t quite show you that, which makes it a lopsided tool in some ways.

Specifically, most businesses find that attributing actions to certain elements on your website are an issue with Google Analytics.

While you can see whether a user is a referral, direct traffic, or organic traffic on the Acquisition Overview, this information only scratches the surface.

The issue I’m talking about is called last-touch bias, and you’ll have to look elsewhere to find a way around it.

This issue specifically deals with the fact that Google Analytics only tracks the “last click” a user makes.

Here’s a scenario to help you understand what I mean:

Let’s say you sell mid-range furniture to middle-class families. A typical flow might look like this:

  1. Your prospect gets a new job in a new city and can’t bring their old furniture with them.
  2. Prospect starts looking for options that meet a certain price range.
  3. Once they do some digging, they start finding specific items they like.
  4. They read reviews, articles, and start using specific keywords and phrases in their search.
  5. They finally register for an account on your site because you offer 20% off a first purchase.

The issue here is that Google Analytics only tracks the fifth step here.

Attributing your success to a specific action on your part is entirely impossible by using Google Analytics alone.

This is because Google Analytics is largely a session-based tracking platform.

When a user comes to your site, Google assigns them an “ID” that identifies that use to your analytics.

The result is that you’re really only getting information about individual visits, and you can’t see what happened before or after your user left.

Plus, if a user leaves and then comes back a week later, there’s really no way to know.

That means your data is inflated in many instances, which doesn’t help at all.

It also lends itself toward confirmation bias. As in, what you’re doing seems to be working, so why change it?

That’s a dangerous place to be with digital marketing.

Because what’s likely to happen here is that you can start to ignore steps one through four from my illustration above.

If that happens, you’ll start losing conversions, and your SEO will take a hit.

But Analytics starts to make up for a pitfall like this with some helpful A/B testing.

Analytics offers a feature called “Content Experiments,” which allows users to split-test page variants within the program.

It’s a useful feature with advanced reporting options but unfortunately lacks a visual editor, posing a major stumbling block for many would-be users.

To get started, you’ll have to create the experiment itself.

Once you’ve selected to start an experiment, you can name it and start to configure the specific elements of the actual test:

You can select an existing Goal such as purchases or opt-in, or create a new objective if you don’t have one set up already.

You can also select a usage metric like bounce rate or exit % to be the focal point of the experiment.

This means you have some flexibility in your testing here, and the reusability is high.

Once you pick a metric, it’s time to add the variations of your website:

Input the two (or more) different versions you’ll be testing, and then move on to the final step of adding your experiment’s code.

As you can see, you’re given a block of code that will allow Analytics to track these different pages.

You’ll be able to run your experiment with minimal hassle, which is always a plus.

And once your experiment is running its course, you’ll be able to examine the experiments report as time goes on:

All in all, this makes it a very useful feature to have for testing basic elements on your site.

While it’s not the most intuitive A/B tester out there, it’s very convenient for many marketers.

Specifically, because it can be paired with another strong point of Analytics, the Cohort Analysis tool.

Over time, you’re going to make changes to your site based on your testing.

When those changes take place, you need a way to keep your metrics in the right context.

Simple comparisons like traffic on a blog post from today vs. six months ago might not be relevant if you’ve changed something like how quickly your site loads.

That’s where cohorts come in. They layer your data to add context and help show change over time.

You can find the Cohorts section under the Audience tab of your Dashboard.

When you click it, you’re able to quickly view a variety of information about your audience’s behaviors on your site.

Elements like goals completed, pageviews, or even revenue per user are just a few of the data points you can search for.

But this is fairly limited in its usefulness because it’s still just showing one giant block of data.

As I mentioned, the purpose of cohorts is to show you layers of data.

To do that, you’ll have to add a second segment at the top of your screen.

As you can see, I was able to break down my traffic further by adding a mobile segment to the results page.

This gives me an idea of how many people are viewing on a desktop vs. a mobile device, and I can take action according to what I find.

When you do this, the usefulness of Google Analytics really starts to prove itself.

One of the main drawbacks to Google Analytics is the steep learning curve.

As you can see from some of the examples I’ve given, learning how to use these tools can be a bit overwhelming at first.

But I would say that you shouldn’t quit because there’s a learning curve.

There are plenty of online guides that walk you through each element of Analytics.

In fact, I think a more notable issue for marketers is that Google doesn’t allow you to analyze any personal information.

In fact, Google has been pretty adamant about not allowing any type of PII:

“To protect user privacy, Google policies mandate that no data be passed to Google that Google could use or recognize as personally identifiable information (PII). PII includes, but is not limited to, information such as email addresses, personal mobile numbers, and social security numbers.”

From a behavioral analysis standpoint, this is a tremendous downside.

No emails, phone numbers, or even names means you’re limiting the usefulness of Analytics as an audience analysis tool.

And it means you’re essentially only getting information about pageviews.

You don’t get any insight about the actual people who are visiting your site.

And in the long run, you need to know who you’re selling to, not just the analytics of how they act on your site.

But now let’s wrap up Google Analytics by talking about price.

One of the biggest upsides here is that the basic Analytics tool is completely free.

So not using this to help you improve your site just doesn’t make any sense.

It’s well rounded and helpful, and the price tag is unbeatable.

Plus, as your business grows, you can upgrade to premium.

Premium is a more viable option for large operations, but you’ll have to talk to Google about the specifics and pricing.

There’s a wealth of information on Google Analytics, but you’ll need to be a pretty proficient analyst to take advantage.

Advanced use of Google Analytics should be an aspiration of every business owner, but it’s definitely not the easiest place to start.

The Pros

  • It’s free!
  • Simple interface
  • “Goals” feature allows for easy tracking of business targets
  • Offers quicker, more efficient multi-armed bandit testing
  • Directly integrated with other analysis features and huge data pool

The Cons

  • Raw data – interpretation and use are on you
  • Lacks multivariate testing and other testing options offered by premium competitors
  • Difficult to use on dynamic pages
  • Lacks a visual-based editor
  • Unable to integrate additional data sources

2. Optimizely

Optimizely is a great first stop for business owners wanting to start testing.

Installation is remarkably simple, and the WYSIWYG interface is highly intuitive, even for newbies.

With plans starting at $17, Optimizely is a very cost-efficient entry point.

But don’t mistake simplicity and budget pricing for an inferior product.

Optimizely’s testing software is good. Very good.

With split, multivariate, and mobile testing options, you’ll be teaching data analysis by the time you’ve outgrown this product’s utility.

One of the strongest points of Optimizely is the tools they have to help with landing page optimization.

This specialized tool lets you take two or more variations of pages on your site and see which one performs better.

In the long run, you’ll be able to increase engagement and conversions with just Optimizely, which makes it a great choice.

But let’s take a look at how this tool actually works.

When you start using Optimizely, the first thing you’ll have to do is determine what type of experiment you want to run.

On their current platform, you have three options:

Simple A/B tests let you put two pages head to head for simple changes.

Multivariate tests allow you to change multiple elements of a page and test performance.

And multi-page tests incorporate a variety of different pages that give a wider view of how certain changes can affect your sales funnel.

When you’re starting out, I recommend sticking to a simple A/B test.

The more complicated options are fairly limited in their usefulness, because you can’t accurately attribute any performance change to a single element.

So even though it’s tedious, just change one element at a time.

And with Optimizely, it’s fairly easy to change every element of your experimental page without having to dig too deeply into your sites code.

This image is showing off their visual editor tool, which as you can see isn’t too cumbersome.

It gives clean options and lets you change every element of your site you can imagine without hassle.

That means you’re spending less time getting pages just-so and more time with the actual experiments.

And don’t worry, if you like to keep track of what’s going on in your code, you can still see that too.

All things considered, it’s a great tool for beginner and expert when it comes down to making simple edits to your site.

But Optimizely isn’t just about editing your site. It’s about experimentation, which carries a few more elements.

One of the most important parts of your experiment that Optimizely makes simple is setting goals.

As you can see, you’re able to test a variety of different goals with Optimizely that makes for an infinite array of testing options.

E-commerce sites can track when an item is added to a cart.

Or, you can set your goal for simple clicks on your tracked element.

Whatever you’re attempting to change, Optimizely will let you track how well your experiment is performing.

You can even determine how much of your site’s traffic gets sent to your original page versus your experimental one.

While you might not want to change this setting at first, this feature comes in handy for more advanced tests where you feel less inclined to risk a dip in revenue from a page.

You can also change this mid-experiment if you wanted, which allows an “on-off” style of testing instead of a split audience.

And when all is said and done, you’ll get an easy to read performance summary of your experiment.

It’s hard to not like the thoroughness of this report.

You get a visitors count, and then data about your control page juxtaposed with your experiments.

As you can see in the example above, the first experimental page performed much better than the second, which means the site owner is on to something.

And what’s more, you get a readout of the statistical significance of your experiment as well.

That means you can know with certainty that the results you’re seeing aren’t just a one time fluke.

You can rely on this data to make impactful changes to your site that will improve engagement.

Plus, as your company grows and you become more familiar with testing, advanced features at the higher price point allow for continued use without needing to learn completely new software.

One such feature is the Site Personalization tool that Optimizely offers.

This image might be a little confusing, so let me explain.

Site Personalization is when you change your site based on seasonal changes or upcoming events.

So the image above is just showing you how simple it would be to change your site from an autumn-themed shoe sale to a winter-themed jacket sale.

And that type of personalization is huge in improving your site’s performance as the year progresses.

What’s more, you can also save and edit audiences on Optimizely, which means you can create even more specialized pages that allows you to experiment on specific demographics.

It’s hard to get much more personalized that showing custom pages to members of your audience that match exact criteria.

That way you can optimize for certain products that meet the interests of your visitor, which can improve performance all-around.

This plays very well with the account-based marketing approach, where you cater specific messages for a highly curated audience.

And to make the picture even better, Optimizely plays nice with other testing software by allowing simple integrations with dozens of other platforms.

So not only do you get top-notch landing page optimization, you can integrate with other specialized tools that can help you paint a better picture of your site’s performance.

The only thing better than one powerful tool is multiple powerful tools, right?

And Optimizely has recently been stepping up their game to allow program management for team workflow.

That means you can easily share work across a large team without sacrificing efficiency or jeopardizing your improvements.

But be warned – If you’re starting completely from scratch, Optimizely might not be your best option.

You’ll need passable landing pages to test in the first place, and you are entirely responsible for creating your own A/B tests.

This is a pretty big prerequisite for someone whose marketing knowledge consists of nothing more than a few blog post reads.

But Optimizely is also oriented price-wise to incentivize growth.

It comes in three tiers, and matches a wide variety of needs depending on the size of your business:

If you’re just getting started with testing software, stick to the standard package while you get your bearings.

And since you get to request a trial, you can really see if this particular software meets your business’s needs.

If you’re already working with a team of marketers instead of flying solo, you might even consider starting on the Business package. Again, it’s up to you.

Just be aware that sticking with Optimizely as you grow does come with a price.

Estimates for the Enterprise level of Optimizely put it at ~$2,000 per month minimum, which starts you off with a $36,000/year commitment.

That’s not starter money by any means, so if you’re new to experimentation, I would stick with the basic or business package at first.

As your brand grows, hopefully, you’ll be able to up your budget if Optimizely is the platform for you.

The Pros

  • Cost-efficient premium option for entry-level users
  • Easy, intuitive interface
  • Effective A/B, multivariate, and mobile-focused testing options
  • Solid basic offering with quality, advanced features at a higher price point

The Cons

  • Lacks outside data integration
  • No algorithm alternatives for judging results, such as bandit or adaptive testing
  • Multivariate testing requires expensive upgraded membership

3. Visual Website Optimizer

Visual Website Optimizer (VWO) is a full-featured testing suite that’s achieved tremendous popularity in recent years.

Overall, it’s very similar to Optimizely but with a few key differences.

At $49 per month for the starting price point, it’s not exactly the “Why not?” pricing of Optimizely, but you can expect a much wider feature set for the money.

VWO boasts a simple, easy-to-use interface and comes with an impressive array of options.

They break their platform down n an easy to swallow five-step system:

And within this system, you get a wide suite of features that can help as much or as little as you want them to.

Probably the most attractive element of VWO when starting out is their visually pleasing and highly intuitive funnel analysis system.

As you can see, you’re given a step-by-step breakdown of conversion rates that helps you see where leads are dropping out of your sales funnel.

You get a big picture readout of site performance without having to piece this information together yourself.

The result is that you’re constantly told where you need to focus your conversion testing with 100% accuracy.

And that’s just the start.

Visual Website Optimizer is also well known for its ability to let you create individual session recordings of individuals who come to your site.

This function lets you review a user’s path on the pages you’ve created.

The recording actually traces mouse movements, which means you can see where they clicked and how often they move around on your page.

In a sense, this works much like a heat map, but it actually gives you a look into the visual journey of your visitor.

You can use this information to change elements of your website that seem to be drawing attention away from your actual goal.

Or, you can use it to create multiple paths for visitors to go where you want them to.

How you ultimately use it is up to your resourcefulness.

On top of the user data you get from recording sessions, you can actually use visitor behavior to define simple or complex visitor segments for your testing.

This means you can truly know who is doing what on your site with a high degree of certainty.

And by using this information over time, you can craft your user’s journey with an enormous degree of optimization.

You’ll even be able to create and store cookies that let you help draw visitors back into your funnel.

What’s more, you can couple this approach with strategically places on-page surveys to gather additional data about why your visitors do what they do.

These surveys are highly customizable, and you can put them on any version of your website.

This approach lets you directly ask your users why they took certain actions.

When you word and place these surveys appropriately, you can gain direct feedback from users that gives you actionable data to improve your site with.

And you won’t have to sort through all of the answers, because VWO creates easy to read visuals for you.

Bar charts and Net Promoter Score are just a few of the ways your user data will be presented to you.

Whatever your mission, you can rest assured that Visual Web Optimizer will give you an easily understood readout of what’s going on with your website.

But the ultimate takeaway is that the data you collect from your audience will be easy to collect and apply.

And if all that isn’t enough, VWO allows you to conduct multivariate testing in a way that’s easier to understand than most platforms.

Since it’s a visual editor, you can make changes to the various elements of your landing page with ease.

All you have to do is pick which element to amend for your tests, and then take a few seconds to create those changes.

But where VWO stands out is when you get done making changes.

Once you edit your page to your liking, VWO will create all possible combinations of your changes to help you find the option that will work best.

You’ll be able to run a full round of testing on a single element without having to make changes manually every single time.

And then, you’re able to take your testing one step further with VWO’s intuitive testing plan management system.

In short, you’ll be able to use the tools at your disposal to observe your audience’s actions with tools like heat mapping and A/B or multivariate testing.

With the information that you gather, you’ll be able to manually input your observations and hypotheses into VWO’s system.

This gives you a single database that documents all of the steps of your experimentation from start to finish.

And speaking of finishing, once you’re done hypothesizing and setting up tests VWO’s platform will allow you to measure your results against your predictions and previous tests.

All of this culminates in a tool that lets you prepare, test, and then take action according to what you find.

Since it’s so flexible, you’d be hard-pressed not to find something your business can use with Visual Web Optimizer.

The software also includes a handful of idea-generation tools.

This is a unique selling point for business owners unsure of where to start in creating and running A/B tests.

Overall, VWO’s benefit to your company depends on your needs.

If you need something dirt cheap to get started, or something very advanced, Optimizely is probably the way to go.

If you need beginner- to intermediate-level testing options at a traffic level greater than 1,000 visitors per month, VWO is the perfect tool for your business.

But what about the pricing?

Since Visual Website Optimizer is such a vast system, you actually have a few options based on what you intend to use it for.

If you’re planning to simply use the testing platform, your options start at $49 per month.

That might work if you’re still a growing business, but if you get more than 30,000 monthly visitors, you’ll have to step up to the enterprise package.

I think if you have that many visitors, you can probably afford the heavier price tag that comes with it.

But if you want a little more out of VWO, you’ll want to take a look at the listed prices for their conversion optimization platform.

These start out a little higher, which might scare off the timid buyer.

But to get all of the powerful functionality we’ve been looking at, you’ll have to be willing to give a little.

Very few platforms are as all-inclusive as VWO when it comes to true behavioral analysis.

So if you can bite the bullet pricewise, you can’t go wrong with this option.

The Pros

  • Feature-rich for the price – full feature set available at all price points
  • Intuitive, easy-to-use interface
  • Idea-generation tools
  • Direct visitor feedback makes this platform unique

The Cons

  • Lacks the same level of real-time support options compared to competitors
  • Lacks multi-page testing options
  • Hit or miss online documentation
  • Pricing isn’t exactly beginner-friendly

2nd Opinion

“Overall, Visual Website Optimizer has the slight edge for casual A/B testers, but it’s more expensive than the basic version of Optimizely. For serious A/B testers, I definitely recommend Optimizely – more comprehensive offering, even though you have to pay extra for premium features.” – Rich Page, Website Optimization Consultant

Unbounce

Unbounce has carved out a niche for itself by focusing solely on landing pages.

That’s it. Just landing pages.

And they’ve managed to make themselves somewhat of an authority on the topic as a result.

No matter how you look at it, landing page optimization is one of the cornerstones of successful digital marketing and e-commerce.

And Unbounce has spent years perfecting their approach in order to pass their expertise on to you.

Here’s how simple it is to build a landing page on this platform.

First, choose a template from the thousands available to you.

Once you’ve picked a template that works for you, it’s time to customize and personalize for your brand.

You’ll be taken to the editing platform for the entire process, so you’ll want to take some time to familiarize yourself with all of the options at your disposal.

And since Unbounce works as a drag and drop builder, you shouldn’t have to spend too many hours on mastering the editor tool.

You really can do just about everything with this editor.

Add buttons, images, or copy with the drag and drop placement.

And then edit your creation with an array of fonts, color schemes, and effects.

If you can imagine it, you can do it on Unbounce.

But the good news doesn’t stop there, because Unbounce also knows that not everyone will be visiting your site on a desktop.

That’s why they’ve also included a tool to help you make your landing pages mobile responsive.

This means you can create custom pages that optimize conversions for any potential visitor.

When time is money, having an intuitive editing tool like this means you can easily make changes and

And all of these features I’ve been showing you make Unbounce the perfect tool for dynamic text replacement on your landing pages.

If you succeed at creating high-quality, personalized landing pages, you’ll see more leads, lower costs, and save tons of time.

It’s almost all you could ask for with a testing system.

But the good news doesn’t end there.

Another one of Unbounce’s unique distinctions is that they offer 80+ pre-designed, optimized landing page templates.

This is a fantastic resource for business owners starting from scratch or who have seen little success with their own designs.

Each template is engineered by the Unbounce team for the purpose of conversions and can be quickly customized for your company’s use.

No matter what you sell, you can find a custom template for your business that will be easy to set up and start testing.

And if you don’t like any of the templates, you can still start from scratch.

But I don’t recommend that. Chances are high that you can find at least one template that works.

In addition to templates, an Unbounce subscription will allow you to perform A/B testing on your landing page.

The only catch is that Unbounce limits you to extremely simple A/B testing, which isn’t the best selling point.

While it’s arguable that any A/B testing is still good, Unbounce really limits what you can get out of their testing.

You’ll be given a few metrics for interpreting the data, and that’s it.

Other than a few periphery extras for the higher-priced options, the full Unbounce experience consists of landing page templates and A/B split testing.

That means if you’re looking for an all-inclusive, heavy duty testing and editing system, Unbounce might not be your best option.

If you’re a true marketing rookie and want results fast, this is probably your best bet.

Few other programs cater to ground-zero quite like Unbounce, and you’ll be able to publish conversion-optimized landing pages in a matter of hours.

But the downside there is you can quickly outgrow the functionality of Unbounce, which may saddle you with a less than optimal editing system in just a few years.

The only major upside that Unbounce has to combat their limited testing is their inbuilt agency and client management functionality.

And truth be told, this is one of the best features in a “testing system” to look for.

Since Unbounce is so focused on creating beautiful and functional landing pages, it’s natural that you would want to pass this unique functionality on to your agency clients.

Unbounce makes this easy for you.

With their editor, you can easily match your landing page to your client’s branding and add it to their website.

You can also add multiple users, which means it’s easy to share and collaborate with team members or clients.

So if you’re an agency that creates a lot of landing pages for clients, Unbounce is an attractive option for pumping them out.

Your clients will receive high-performance landing pages with a minimal amount of work, which is better for everyone.

But again, since testing is fairly limited, it’s also comparable to playing hot potato with a live hand grenade.

Your agency gets ease-of-use and shareability, but proving long-term value will become an issue unless you pair Unbounce with an alternative testing platform.

And that makes you question the overall value to begin with.

But thankfully, if you’re on the fence with Unbounce, you can preview how it works to try before you buy.

It’s always a good idea to do a trial run of any testing platform if you can, so this is another tick in Unbounce’s favor.

And since it’s user-friendly, you might find yourself very attracted to this platform if you’re newer to marketing.

Then you get to the topic of Unbounce’s pricing, which matches their beginner-friendly approach:

At only $79 per month, the Essential package makes Unbounce hard to pass up.

It’s cheaper than some of the other options on this list, and it’s cheaper to keep in your toolbox as our business grows.

To sum things up, Unbounce is limited in its goals but accomplishes those goals admirably.

And you can’t fault them at all for this. They do one thing, and they do it well.

The Pros

  • Landing page templates provide great start-from-scratch resource
  • Solid split testing capabilities
  • Simple, easy-to-use interface
  • Designed to integrate easily with other marketing and testing software
  • Able to share capabilities with agency clients
  • Affordably priced to grow with your business

The Cons

  • Limited to simple A/B split testing
  • Relatively high cost for a limited feature set
  • If used with an agency, limited testing can affect long-term client performance

2nd Opinion

When creating a new landing page, Unbounce is our company’s go-to for A/B testing conversions. It is quick, gets the job done, and the analytics are actionable. We can’t live without its confidence score rating, which tells you how likely a good conversion rate is, or whether it is mere luck.
Brian Curliss, CEO MailLift

5. LeadPages

LeadPages is a newer testing software that has gained quite a bit of popularity in recent years.

And they’ve gained notoriety for good reason. You’ll be hard-pressed to find a more well-rounded landing page editing software for your business.

It makes this list because it’s intuitive, very well equipped, and has a lower threshold for entry than some of the other options I’ve shown you.

If you’re looking for a landing page builder that works well with social media and integrates with other software you use for business, LeadPages is a great place to start.

But be warned – this isn’t a traditional testing platform.

You’ll want to approach LeadPages with a wider gameplan in mind.

The reason I say this is because while there are many customizable options within LeadPages, testing isn’t its strongest suit.

In fact, you’re limited to simple A/B testing.

And while this functionality is good, it doesn’t quite measure up to the flexibility of some of the other options in this list.

But don’t let that scare you away. There are still plenty of good reasons to consider using LeadPages for your business.

Let’s start looking at a few features.

To start with, you’ll be given a huge selection of pre-built templates to help you streamline landing page creation.

No matter what you’re selling, you’ll be able to find a template that can be molded to your branding.

And since these templates are based on years of testing, you can rest assured that they’ll start pulling their weight immediately.

As you can see below, all you’ll need to do is add your logo, change some copy, and then add the page to your site.

Speed is always an attractive option when it comes to testing, and LeadPages is well aware of this fact.

The original model of LeadPages focused solely on the swift creation of landing pages, which is why they limited you to modifying their original templates.

But after listening to quite a bit of feedback, they now also offer a simple drag and drop page builder for users who want a little more power over their landing page design.

You’ll have to choose which option you want to go with when you start using the platform.

And don’t worry about choosing the wrong option. You can always change your mind later if you want more customization.

Let’s take a quick sneak peak at the new drag and drop builder.

The first thing you’ll notice is the Widgets bar that you’ll be dragging your options from.

If you’ve used a drag and drop builder before, this is nothing new.

The only difference is that this is completely catered to landing page creation, so your options are specifically catered to helping you boost your conversion rates.

And since it’s a drag and drop system, creating a visually pleasing and high converting page is as easy as point and click.

But the uniqueness of LeadPages doesn’t end there.

One of the most attractive options with this platform is how easy it is to insert built-in payments to any page on your website.

Since it’s a drag and drop builder, all you have to do to start creating optimized sales pathways is add a simple checkout widget on a landing page or website popup.

This is fairly unique as landing pages go because you’re being more direct with your visitor and creating an easier path for them to make a purchase.

So when your goal is for a visitor to make a purchase, this type of functionality is essential.

It allows you to create a relatively frictionless conversion pathway that doesn’t require a whole bunch of clicking or extra copy.
g
You just get straight to business.

And you aren’t just limited to your landing pages either.

You can add these widgets to any page on your site.

Or, you can add it to your website’s popup overlay:

There’s no limit to where you can strategically place this simple checkout option.

As long as your offer is good enough, this is by far the easiest way you can optimize for revenue conversions on your website.

For websites that have a shorter sales funnel, this type of customization is perfect.

But if your sales funnel is a little longer, I would hold off on adding the built-in payments too often.

You can just as easily scare a visitor away by asking for their credit card too soon.

And since there are no real testing capabilities on this platform, you’ll be left in the dark about the true performance of a function like this.

But there’s still more unique aspects to consider about LeadPages that makes them an attractive option.

For one, you’re actually able to create custom Facebook Ads directly on the LeadPages platform.

In my experience, this makes LeadPages truly unique.

It takes website optimization a step further than its competitors by finding innovative ways to “play nice” with social media.

Most website builders make it easy to send users from your site to your social media profiles, but how many do the opposite?

LeadPages stands out from the crowd with this unique functionality.

And they really do make the process simple for you.

All you have to do to get started is just click over to the Facebook Ads option on their publisher tool.

You’ll be sent to a simple editor that lets you add the target URL, imagery, and copy for your ad.

It doesn’t get any easier than this when it comes to creating Facebook Ads that match the imagery and branding of your website.

You’re able to upload your ad quickly, and then toggle between desktop and mobile options to ensure that everything looks and feels how you want it.

And you’re not having to switch back and forth between any conceptualizations and your finished product.

It’s all in one place, which means you can create and publish quicker than ever before.

What’s more, you can also cater your audience selection with the editor, just like you would on Facebook.

So when all is said and done, you’ve created a beautiful landing page that’s optimized to generate revenue.

And you’re pushing visitors toward that landing page with easy to use Facebook Ads that took you minutes to create.

That’s why LeadPages made this list.

If you’re looking for a testing platform that optimizes your social media integration and boosts sales, you’ll be hard-pressed to find a better option.

But as I’ve mentioned, the testing side of this platform isn’t the best option.

You’ll need to beef up your overall approach by utilizing one of the simple integrations that LeadPages has to offer.

This image shows you just a few, but there’s a mile long list of possible platforms you can pair with.

All you have to do is initiate the integration from their huge selection, and then you’ll be able to find your added application in the drag and drop editor later on.

That means you get more bang for your buck by being able to direct where your integrated application draws its information from.

If you’re not able to tell these applications what to do and where to look, they wouldn’t be any good.

As you can see though, you’re able to directly input where you want another testing software to look from a simple dropdown menu.

That means you can pair the sales-boosting functionality of LeadPages with the testing abilities of a different platform.

The result is that you don’t have to sacrifice form for function.

It really is the best of both worlds, provided you can foot the bill for multiple high-performance platforms.

And speaking of bills, let’s talk a bit about LeadPage’s pricing.

At only $25 per month for the Standard package, you can start using LeadPages for a mere $300 commitment.

This gives you access to the basics elements of LeadPages that lets you start creating and testing your visually oriented landing pages.

If you decide you want a bit more muscle, upgrading doesn’t require too much of an additional monetary commitment.

And when you consider how much money you can pull in with an editing and testing system like this, it’s a no-brainer for your business.

The Pros

  • A wide variety of templates makes this an attractive option
  • Drag and drop builder means it’s easy to edit
  • Allows you to create custom Facebook Ads
  • Priced affordably for any business size
  • Simple integration with other platforms

The Cons

  • No true testing functions, meaning you have to integrate with another system
  • Not the best option for businesses with a longer sales funnel

6. Kissmetrics

Unless you’ve been living in a box for the last few years, you’ve heard of Kissmetrics.

This popular testing tool focuses on the human side of data.

Reports and testing data are connected to real people, who are tracked across their interactions with your website.

Kissmetrics is definitely a premium service, but it comes with the advanced abilities you’d expect at that price point.

You won’t find software offering deeper or more diverse sets of data.

Kissmetrics lets you keep a pulse on the entire conversion funnel, from start to finish, and identify where and how customers are disconnecting from your website.

Very few platforms are as robust as Kissmetrics when it comes to analyzing your site, and you’re about to see why.

First off, it’s incredibly easy to examine the key performance indicators of your website with Kissmetrics.

You’re easily able to define which metrics are most important to your business, and then optimize your results feed accordingly.

The results get regular refreshes, which means you can dive deeper into the available information whenever you need.

Better yet, all of this information is highly reliable and easy to use.

You can even color code the information to help you focus on the information that really matters in certain circumstances.

From this information, you’re also able to easily pinpoint the exact location where your funnel is leaking.

With a chronological map of your website visitors’ behavior, you’ll be able to make decisions about where you need to focus your website optimization efforts.

By identifying the actual areas where your site is bottlenecking your funnel, you can start creating simple tests to help improve your customer journey.

And to help you dive into the details of each potential bottleneck, you can find an exact breakdown of your visitor’s activity on your site.

You can track any and all pages on your site, much like you would with Google Analytics.

Where Kissmetrics stands out is in the ability to analyze more than simple page views or clicks though.

You can dig deeper and find specific insights into what’s working or not on your site.

By testing optimization with Kissmetrics, you’re able to find actionable ways to improve your site without distraction.

And with the ability to segment your audiences based on their behavior, you’ll get a deeper insight into the minds of your potential customers.

This feature is even more useful than most because it also allows you to create an export various segments of email addresses and other personally identifiable information.

That means you can create customized email campaigns based on insightful data instead of just guesswork.

You can also optimize your retargeting campaigns with this feature, or even find the right time of day to make a phone call to a specific customer.

That amount of insight is incredibly useful for any business, but it’s especially powerful for brands that have a longer sales process.

And the driving force behind all of this is an intuitive A/B testing platform.

With Kissmetrics’ A/B tester, you’ll be able to create experiments and evaluate their performance for the outcome of your choosing.

You can easily test conversions like subscriptions, sales, or simple page views.

And then you’ll be able to dive deeper and find ways to further optimize your website for any outcome you choose.

It’s by far one of the most well-rounded, data-driven, and insightful testing software on the market.

And if the A/B testing isn’t enough for you, you’ll also be able to conduct a cohort analysis of your website.

This will help you visualize how various elements of your site perform over time.

You might get good results from a change initially, but a cohort analysis from Kissmetrics will help you see if it truly performs well as time goes on.

This lets you understand the best times to engage with your audience and keep your marketing efforts moving in the direction you want them to.

And you aren’t limited to a specific page or element of your site. You can track any segment or page you choose, which gives you an unlimited amount of control.

And then if all of these features and testing capabilities weren’t enough, Kissmetrics also allows you to store complete profiles on your website’s users.

This aspect makes Kissmetrics stand out in the greatest possible way, especially up against Google Analytics.

You’re able to see the historical details of every single visitor to your site, which other testing platforms don’t quite do.

You can examine the journey they take on your site, which features they decide to use, and get an overall idea of what kind of user your visitor is.

This means you can finally differentiate a user who came to your site for research from a user who came to buy.

When you know which is which and have the tools to contact them, you can cater a message that lets you optimize your conversion chances.

Other testing software might give you an idea of their behavior, but Kissmetrics lets you know who they are and gives you valuable insight into their intentions.

Once you’ve gathered this information and feel certain about its accuracy, you can also create custom populations to get the most out of your findings.

This feature lets you monitor different segments of customers who are vital to your business and see how they change over time.

You’ll get a curated overview of the highlights, which means you can get a constant readout of the health of your online audience.

This overview lets you know what’s happening at a glance, which means you don’t have to waste time chasing endless analytics to know what’s actually going on.

With real-time updates, you get an instant readout of the effects of any changes you make on your site.

Then, you can dig into the specific details of these populations to see how they change over time.

With this information, you can get an idea of positive or negative effects of your changes as it relates to a specific demographic.

This is helpful for a few reasons.

First of all, you might make a change that increases the overall click-through rate on a particular landing page but ultimately causes a drop off on trials for your product from a segment of your audience that generates a lot of revenue.

You’d be better off to reverse those changes, and these population details help you see that.

And with the ability to segment each population, you can truly get a deep understanding of what elements actually create changes in your audience.

When you pair this type of customization to a powerful A/B tester and insightful data, you have a recipe for success.

And to make the picture even better, Kissmetrics also allows you to generate customized campaigns based on the information you gather.

For example, you can use this testing and information to personalize emails using the internal editor on the platform.

This lets you create and send a compelling message to anyone in your database.

And instead of having to spend hours emailing each individual user, you can blast a personalized email with the perfect message in minutes.

Since these campaigns are also integrated with Kissmetrics’ powerful analytics engine, you also get more data than just opens and clicks from these emails too.

In fact, you get a smorgasbord of useful information from these campaigns, like what happened directly before and after your email was delivered to your visitor.

This means you get an even better understanding of the behavior of your customer and can use this information to help you sell to them again and again.

More importantly, you can create and send personalized emails in the future when your customer starts making the same behaviors as when they first visited or purchased from you.

This means you’re not only created custom-made sales pitches with a high degree of success, but you’re also sending them at the perfect time.

All you have to do is automate your message to send when the right trigger comes along.

And you can line up as many of these emails and campaigns as you want, for as many populations as you want.

It really is the ultimate tool for data-driven optimization and sales efforts.

The only catch is that it comes at a high price.

That’s right. If you want to use Kissmetrics, you’ll be looking at a fairly steep $6,000 tab for a year’s worth of use.

And to make the picture a little more complicated, you’ll end up shelling out a bit more each month if you go over your MTP allowance.

But if you are serious about data, Kissmetrics is one of the best tools around.

Its ability to connect data to actual users might also be a great alternative for business owners struggling to wrap their heads around more traditional reporting styles.

The Pros

  • Highly flexible data reporting options
  • Simple, intuitive interface
  • Can trace data to real people
  • Excels at funnels with extremely accurate funnel-based data

The Cons

  • Lower powered A/B testing compared to other testing software
  • High cost – won’t be affordable for many small businesses
  • More advanced data reporting requires a higher learning curve

2nd Opinion

“I use other split-testing tools, such as Optimizely, but Kissmetrics is a must if you’re serious about getting accurate results. The problem with many of the tools out there is they don’t measure the impact of the test throughout your entire funnel. That’s why I love Kissmetrics; it will show you exactly what the impact is for a test at every step of the buying process (not just the next step).” – Ruben Gamez, Founder Bidsketch

7. Crazy Egg

Of course, our list wouldn’t be complete without the home team’s offering.

Crazy Egg specializes in click-tracking.

If you’re a visual learner, Crazy Egg will be your go-to source for data.

The available reports frame data in an easily digestible way that is intuitive and provides immediate insight into user activity, regardless of your analytical skills.

Take the highly popular recordings feature for example.

This tool allows you to see the exact movements of every user on your site.

Not only does this tell you what they looked at and where they clicked, but it also tells you any potential trouble spots on your site.

Here’s one example of what a recording can look like:

You’re given an exact playback of every single action the user takes.

When they red circle pops up, it means the user is clicking.

Any by mapping out where the mouse moves, you can start to optimize your site for that type of behavior.

This is especially useful for strategically placing your call to action buttons in a place where users are more inclined to click it.

But recordings aren’t the only feature to talk about on Crazy Egg. One aspect that we’re particularly proud of is our robust A/B testing system.

It easy for users to create simple and complex A/B tests that give them valuable data and a clear path of action, just like other testing platforms.

But where Crazy Egg really stands out is in the snapshot features of the software.

The most notable of these features is the heat mapping system that gives you an incredibly accurate analysis of user behavior on your site.

This is a great way to get direct information on how people use your site.

When you know how they interact with the various elements you’ve included, it’s easy for you to find out what your visitors want.

And when you know what they want, you can eliminate any needless clutter that might be bogging down their user experience.

That means you can optimize for clicks, conversions, or whatever you’re actually after.

What’s more, you can also see a scroll map report to get additional insight beyond the heat map.

The scroll map report lets you see how far into a page a visitor is actually scrolling.

This is a great tool for using on landing pages, as you can determine if your current page is too long or too short for your user.

It also helps you know where to put forms, call to action buttons, or other important elements you want your visitor to see or click.

And then you can take things a step further with an even more precise look at where users are clicking on your site by using the confetti tool.

This feature lets you generate a report that shows you exactly where users have clicked on your site.

The result is that you’re given powerful insight on how users are actually using the different elements of your webpage.

And if you want to further pick apart how each individual element is performing, you can also use an overlay report.

As you can see in the image above, this gives you the option to zoom in on individual aspects of your page.

You can look at images, copy, or even links to see how well they perform.

And if all the features you’ve seen so far aren’t enough for you, you can also generate a list report with Crazy Egg.

This final report will give you a list of every part of your site that’s received a click.

Instead of making you sort through all of the clicks and visual information, we’ve put it into a tabular and easy to share format this time.

You’ll find that all of the information in each of the columns is pulled directly from your page, which means you can make decisions accordingly.

So you get a tremendous amount of insightful data from a variety of different reports, and it’s all in one convenient location.

What’s more, you’ll find that the pricing for Crazy Egg won’t leave you too winded if you’re looking for a cheaper starter option.

While it’s not exactly the free price tag of Google Analytics, you get much more functionality and a lot more useful data for an affordable $29 per month to start with.

That means you can do an unlimited number of tests and record months of user activity without too much of a commitment.

If you’re at least capable of building a landing page, Crazy Egg is the ultimate entry-level option. It is one of the cheapest testing software you’ll find anywhere.

While it is decidedly limited in scope and test variety, you’ll find it much easier to use than Google Analytics and most other options on this list.

It is, perhaps, the only offering on this list that allows business owners to immediately read and understand test data with zero context for web analytics.

The Pros

  • Highly visual data – enjoyable to view
  • The benchmark of interface simplicity
  • Extremely low learning curve with intuitive reporting
  • Cheapest pricing you’ll find for testing software

The Cons

  • Limited feature sets and testing options
  • Limited utility for dynamic pages

2nd Opinion

“It’s a great tool considering how inexpensive it is. If used correctly and you have a plan for tracking your websites, this tool can make a world of a difference.”
– Amber Steinert, Marketing Analyst 3M

Conclusion

Whichever A/B testing software you choose from this list, you’ll be able to better optimize your site and improve your long-term revenue simply by taking the time to test.

If you use Google Analytics, you’ll be able to know the overall health of your site and find actionable ways to grow.

Optimizely is a user-friendly introduction to anyone new to testing, but be careful that you don’t outgrow it too quickly.

And Visual Website Optimizer can help you see your website and users in a new light, meaning you’ll be able to create a better overall journey for your visitor.

Then Unbounce will allow you to create beautiful landing pages for both yourself and your clients, which is always a plus.

If you focus on social media integration and have a shorter sales funnel, LeadPages can give you simple testing abilities and easy to use sales pages.

And if you’re into a data-driven approach, Kissmetrics will keep you on course and help you create powerful campaigns.

Then last, but not least, Crazy Egg will help you map out your website visitor’s actual behavior and give you ways to further optimize your site.

Each of these platforms has unique strengths and weaknesses, but all of them can help you grow.

Now all you have to do is decide which AB testing tool is best for you.

24 Comments

DON’T MISS OUT

Get updates on new articles, webinars and other opportunities:

David Zheng

David Zheng is the Editor in Chief at CrazyEgg, Founder of GrowthWit and WiseMerchant and the Head of Growth at BuildFire. He helps influencers, ecommerce brands, venture backed startups, and Fortune 500 companies grow their traffic and revenue online.

24 COMMENTS

Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.

SPEAK YOUR MIND

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  1. John says:
    June 17, 2015 at 8:57 am

    In order to take customers onboard your modification team and to minimize the chances of revenue and customer loss, A/B testing has become the sure shot way for various business organizations. A/B testing not only helps in smooth transition of the changes but also helps in improving the user experience and conversion rates of a website. Some of the tools that can help in this direction are Google Analytics, Unbounce, Optimizely and such others. I would also like to mention Mocking Fish as it is lower in cost, provides more accurate real time results and is easy to implement.

    • Kathryn Aragon says:
      June 17, 2015 at 2:06 pm

      Thanks, John. We’ll check Mocking Fish out.

  2. Irit Gillath says:
    April 29, 2015 at 3:29 pm

    Another tool that I like for heat maps and visits stats is luckyorange.com

  3. sandybrown says:
    April 27, 2015 at 6:53 am

    google Analytics is best traffic analyser.

  4. Scotty Truong says:
    April 26, 2015 at 1:17 am

    Thanks for this detailed review. Would love to add 3MinuteOptimizer (disclaimer: I’m the founder of the business) which focus on exclusively on setting up split URL tests fast. Hope we could make it to the list.

    • Kathryn Aragon says:
      April 26, 2015 at 8:55 am

      Thanks for letting us know, Scotty.

  5. Peter says:
    April 23, 2015 at 8:39 am

    I would like to know if this article still holds good? or is it outdated as some of the A/B testing tools mentioned here may have worked on the cons?

  6. Pete says:
    January 23, 2015 at 7:08 am

    Brilliant, mission accomplished!!! We have researched kissmetrics and find that, although the functions are amazing, its not financially viable for us. Thanks.

    • Kathryn Aragon says:
      January 23, 2015 at 9:46 am

      Glad you arrived at a decision, Pete. Most businesses have a wish list of products. KISSmetrics may be a good for one down the road.

  7. sandybrown says:
    January 15, 2015 at 12:40 am

    software is save your time.

    • Kathryn Aragon says:
      January 15, 2015 at 9:15 am

      Agreed, Sandybrown. The best testing software DOES save you time.

    • Neil Patel says:
      January 15, 2015 at 12:01 pm

      Sandy, it definitely does!

  8. Andrei says:
    January 13, 2015 at 5:15 am

    You also forget such tool as a Changeagain.me
    It’s the easiest A/B testing toll based on Google Experiments API, so it’s fully integrated with your account in Google Analytics. So all the data is gathered with maximum accuracy.

    • Kathryn Aragon says:
      January 13, 2015 at 9:22 am

      Thanks, Andrei. Appreciate the addition.

    • Neil Patel says:
      January 13, 2015 at 12:06 pm

      Andrei, thanks for adding that one and providing the additional feedback.

  9. Chris says:
    January 9, 2015 at 1:06 pm

    How is Adobe’s Test & Target tool not on this list? In the testing community it is arguably THE a/b testing tool.

    • Kathryn Aragon says:
      January 9, 2015 at 2:15 pm

      Thanks for the addition, Chris. What’s the cost? I couldn’t find it.

  10. June 26, 2014 at 10:18 am

    Hi Jacob,

    Hope Convert.com makes it to the next list.

    Dennis

  11. Tia Kelly says:
    June 26, 2014 at 1:53 am

    Hey Jacob! Really appreciate you mentioning Unbounce in this list. Have to give props to the other 5 you chose to highlight here 🙂 All do a great job in their specialty. I also liked the “second opinion” quotes—something different that I haven’t really seen done before in a top tools type of post that adds credibility. Thanks!

    Social Strategist, Unbounce

    • June 26, 2014 at 2:26 pm

      Thanks Tia. My goal is that it will make choosing software easier for business owners.

      • Neil Patel says:
        June 26, 2014 at 5:59 pm

        Jacob, Im sure it will. Thanks for the feedback 🙂

  12. Zac Aghion says:
    June 25, 2014 at 2:40 pm

    Nice review Jacob, thanks!

    Google Content Experiments can also be used to run automated A/B tests. From a high level, the way this approach works is to start showing variations that work better more often, automatically. It’s a much better solution to the question of ‘What works better: A or B?”

    In the most basic example of an A/B test, you have a variation A and a variation B each shown to 50% of your user base. By definition, this approach will be sending half of your users to a worse performing version during the entire duration of the test!

    The automated approach is based on a bandit algorithm that updates the percentage of your user base which is shown a given variation dynamically. With each new piece of data that you collect on the conversion rate and statistical confidence of the variations in a test, it adjusts the percentages automatically so that better performing variations are promoted and worse performers are pruned away.

    This leads to:

    1) faster results, because your directing test resources (i.e.: users and their data) to validate what you actually care about (i.e.: confidence in the best variation’s performance)

    2) a higher average conversion rate during the test itself, because relatively more users are being sent to the better performing variation automatically, and

    3) less time and effort required to actively manage your experiments.

    Though the math behind this approach is slightly more complex than a traditional A/B test, it’s worth talking about for audiences that are interested in making data-driven decisions because of how much better the results are that it produces.

    For anyone interested, here’s a post we put together on how it works: http://splitforce.com/resources/auto-optimization/

    • June 25, 2014 at 2:52 pm

      Great insights, Zac. Thanks for sharing your expertise.

    • Brian Lang says:
      June 26, 2014 at 10:36 am

      Great comment Zac – I’m convinced minimizing regret is the way to go, but I think there are limitations/concerns with existing offerings – Auto-optimization sounds great in theory, but the reality is it isn’t as simple.

      For example, most of the available tools haven’t implemented contextually aware auto-optimization methods, so you ultimately end up creating a lot more work implementing the same experiment for many different user segments (a time consuming pain point), or you settle for, “what’s the best option for most”.

      Separately, overlapping experiments with an auto-optimization setup, where one experiments better-performing arm can inadvertently skew (positively) the results of another experiments inferior arm, is a real problem, and most tools available don’t offer a solution to it.

      There are also issues with traffic quality and returning customer bias (through things like the recency/novelty effect) in-advertently skewing results of one arm, when it in fact might be the inferior one – in this situation, you would in-advertently be maximizing, not minimizing regret, and depending on the method of your “stopping point criteria”, might lead to wrong conclusions being made.

      A frequentist approach *typically* doesn’t suffer the above issues, since you are typically doing a 50/50 split of traffic, and can then segment results. There are, of course, ways to address these issues in an auto-optimization setup, but as of yet I haven’t really found a tool that does so.

      Thanks again!

      Brian

Show Me My Heatmap

@AndreaFine @CrazyEgg is an awesome tool! #semrushchat

Netvantage Marketing

@netvantage

What makes people leave your website?