Glossary: Multivariate Testing

by Daniel Sims

Last updated on November 24th, 2017

A type of hypothesis testing where multiple variables are tested simultaneously to determine how the variables and combinations of variables influence the output.

If several different variables (factors) are believed to influence the results or output of a test, multivariate testing can be used to test all of these factors at once.  Using an organized matrix which includes each potential combination of factors, the effects of each factor on the result can be determined.

Also known as Design of Experiments (DOE), the first multivariate testing was performed in 1754 by Scottish physician James Lind as a means of identifying (or eliminating) potential causes of scurvy in sailors.  Today, it is widely used in Science, Engineering and website testing.

A/B testing is a form of hypothesis testing which utilizes the most basic matrix.  If you want to test 2 different colors of a website order button, you are testing 1 factor (the button color) at 2 levels, so the number of combinations required is:

21 = 2

Order Button Output Conversion

There could be several website elements influencing conversions, revenue, or whatever output you may be seeking to optimize.  For example, if you believe your order button size, font, position (top or bottom of page) and color may all have an impact on conversion rate, using a multivariate test allows you to study all of these elements simultaneously. Since you are now studying 4 factors at 2 levels, your multivariate test would require:

24 = 16 combinations

mock chart

If you want to study additional levels for each factor, this would require more combinations.  For example, if you wanted to test green, red, blue and yellow order buttons, this would require a 4 level experiment and 44 = 256 combinations!

Since an A/B test splits traffic in 2 directions, the calculated sample size required to obtain statistically significant results is doubled in order to adequately test the population.  In multivariate testing, this same calculated sample size is required for each combination in the matrix, meaning you may require 4X, 8X, 16X or more samples (visitors) than an A/B test.  For this reason, multivariate is typically recommended for websites which already have a high level of established traffic.

Red Green Buttons

The statistical software analysis of multivariate testing results will tell you which combination of factors had the best outcome, which individual factors had the most impact and whether or not these results were significant.  Just as importantly, the analysis will tell you which individual factors and combinations of factors were not significant.  This knowledge can help guide future testing efforts and eliminate unnecessary testing.

2 Comments

DON’T MISS OUT

Get updates on new articles, webinars and other opportunities:

Daniel Sims

Daniel Sims is a Certified Quality Engineer, part-time writer and Six Sigma practitioner who espouses the use of Engineering discipline and problem solving skills in all areas of business and life.

2 COMMENTS

Comment Policy

Please join the conversation! We like long and thoughtful communication.
Abrupt comments and gibberish will not be approved. Please, only use your real name, not your business name or keywords. We rarely allow links in your comment.
Finally, please use your favorite personal social media profile for the website field.

SPEAK YOUR MIND

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  1. Russell Moore says:
    April 9, 2017 at 11:36 am

    Thanks for sharing this post – I’m a firm believer in multivariate testing and I think it can really help a business grow its revenue.

    I have to say, somtimes i feel that multivariate testing can be overwhelming. Virtually everything on your website can be changed and tested. Whether it be font color, font family, font size, images, CTAs’ button colors, ect. The list is endless.

    I’ve recently switched to having 1,000 people as a sample size to gauge the effectiveness of a change. I’ve also started focusing on changing the items that MOST visitors see. So text,images,colors above the fold.

    Anyways, just wanted to throw in my thoughts. Thanks for posting this article!

    • Daniel Sims says:
      April 12, 2017 at 12:17 am

      Thanks for the feedback Russell. I don’t know if multivariate testing is for everyone, but it is definitely an interesting and powerful tool!

Show Me My Heatmap

Click tracking, heat maps, and without a spreadsheet? Yes, please. is one solut...

Aimee Graeber

@AimeeGraeber

What makes people leave your website?