What’s In Store For The Future Of Split Testing?

by Art

Last updated on January 9th, 2018

I started writing this post on October 21st (2015), also known as Back To The Future Day. For those not in the know, this is the day that Marty McFly and Doc Brown travel through time to—at the time of filming—the distant future.

This seemed like a nice time to start thinking about the future of split testing, particularly in relation to how technology that already exists might be used for A/B testing. Oh, and I promise I’ll try to avoid any dodgy hoverboard puns.

Dynamic Content

There’s nothing particularly new about dynamic content. You’ve probably already encountered at least one of the following today:

  • E-commerce recommendations based on your browsing history
  • Tailored Twitter trends
  • Banner ads triggered based on your location and/or cookies

In fact, dynamic content is now so much the norm that those who fail to take advantage of it risk losing customers:

A survey conducted in 2013 found that almost 75% of consumers get annoyed when websites serve them irrelevant ads and 67% said that they’d leave a site if they advertised a political party they dislike.

vintage political ad

But is dynamic content being used to its full potential?

Split Testing And Targeted Content

Despite the above, you won’t find much if you try to Google dynamic content split testing. One of the few results I was able to find explained how existing pages with dynamic content can be split tested. And it sounds pretty difficult.

There’s very little mention, however, of testing additional dynamic content on landing pages…although you could argue that any split test is dynamic content if you’ve tinkered with the target audience in some way.

But it seems to me that when most marketers talk about split testing they often refer to site-wide or landing page specific changes without delving into the audience to whom its delivering, which seems like an opportunity missed.

Optimizely, for example, allows you to create audience segments using any of the following criteria:

  • Browser/device
  • Source (e.g. AdWords, organic search, Facebook)
  • Certain cookies
  • Parameters such as UTM codes
  • Geographical location/language
  • Time of visit

Are enough split testers taking advantage of this segmenting capability?

UTM Codes

When you talk about paid search campaigns, you start to get an idea of the potential of split testing, segmentation and dynamic content.

Consider how a paid search marketer might use UTM codes and split testing to direct traffic to a single landing page, implementing targeted content for each query.

Campaigns with different headers, sub-headers, calls to action and even different copy can be used for each different paid search query, and the results measured to see what works most effectively.

But what if those campaigns weren’t *just* split tests, but ongoing experiments that could be used to cherry pick the most impactful headers and content for future visitors based on how they compare to past conversions?

Could this data then be spliced in such a way that would create the ultimate landing page for someone in a particular audience segment in real time?

Split Testing Is Dead

split testing tombstone Oregon trail

In a future where content is truly dynamic AND based on the results of previous tests, we might theoretically reach a point where split testing as we know it becomes irrelevant.

As more and more outcomes are tested on a site, we approach maximum conversion…although it’s true that, particularly on sites with low traffic, it would take a long time before that happens.

But, it’s possible that we could eventually find ourselves in a position where almost every visitor from a certain tightly defined niche converts.

Once that happens, there’s really no need to test content for that segment anymore. At that point, A/B testing as we know it is dead.

Or Is It? (Long Live Split Testing)

Ok, let’s rein things in a bit and be realistic.

Is it really possible to define a target audience that tightly? Probably not. What happens when you want to add new content? You need to go back to the drawing board and start testing again.

The tone of voice and calls to action used in highly optimized sales content for a particular segment, for example, are unlikely to have the desired effect when used on an informational page for that same segment.

Dynamic content, cookies and detailed information about visitor segments can do a lot, but they can’t do everything. Someone still needs to analyze the data and figure out what (and where) to implement and what to discard.

Plus, I haven’t even begun to talk about what to do with people who use ad blockers (which some claim already has over 15% penetration in the U.S. and 21% in the UK) and opt out of personalized advertising.

Finally, as we mentioned last week, you need to have enough of a sample size AND be able to run tests long enough, to really be confident in your testing results. Dynamic split testing would need to be carefully checked for:

  • Sample size
  • Statistical significance
  • Duration (to capture complete business cycles)

…and doing that on the fly (or should I say “McFly”) is really tricky. You would need someone carefully checking each test throughout the day to make sure so-called winners aren’t ruining your business.

So What Does This All Look Like?

I’d love to provide a picture below of exactly what the future of split testing looks like with a breakdown of exactly how some piece of next gen software works. I may be talking about the future, but I’m not actually a time traveler.

If I knew what the split testing tool of the future looked like, I’d be locked away somewhere building it myself. But, were I to take a guess, I would say that in the next few years one of three things is going to happen:

  1. Split testing software (such as Optimizely, VWO etc.) will place more emphasis on dynamic content that adjusts automatically
  2. Landing page software like Unbounce and LeadPages will promote dynamic content for different segments harder*
  3. Hybrid landing page/split testing software will emerge, either as the result of a merger between two companies or from a new provider

* Marketo already mentions this on one of their landing pages. A sign of things to come, perhaps.

Split testers as they currently exist will probably find themselves transitioning to overseers and analysts of various automated processes that will be chopping and changing content far more rapidly than we do today.

Final Thoughts

You may feel like we’re reaching “peak split testing”, and I can understand why. While Split Testing 101 type guides are great for teaching the basics, they don’t always encourage a holistic *bigger picture* approach to A/B tests.

What I’m really getting at with the above post is that I think we’ll begin to see the following in the future:

  • Tighter audience definition and more focus on maximizing conversions for key audience segments
  • Broader, holistic goals—and tests that work towards them—rather than a random color change here or different sub-header there
  • A focus on quicker, easier implementation of successful tests, possibly through deeper integration with landing page software

As with any prediction, I could be completely wrong but I might just make a note to revisit this post a few years down the line. Unless anyone happens to have a DeLorean handy…?

Does the above ring true for you, or do you think the future of split testing will bring something totally different? Let us know in the comments.

No Comments


Get updates on new articles, webinars and other opportunities:


Art is a freelance copywriter/content marketer based in the UK. He's worked with both startups and large corporations on topics ranging from fitness to tech and business issues.


Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.


Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Show Me My Heatmap

@AndreaFine @CrazyEgg I love using it. That's when my clients really get it. More than any @Google analytics data.

Dagmar Gatell


What makes people leave your website?