If several different variables (factors) are believed to influence the results or output of a test, multivariate testing can be used to test all of these factors at once. Using an organized matrix which includes each potential combination of factors, the effects of each factor on the result can be determined.
Also known as Design of Experiments (DOE), the first multivariate testing was performed in 1754 by Scottish physician James Lind as a means of identifying (or eliminating) potential causes of scurvy in sailors. Today, it is widely used in Science, Engineering and website testing.
A/B testing is a form of hypothesis testing which utilizes the most basic matrix. If you want to test 2 different colors of a website order button, you are testing 1 factor (the button color) at 2 levels, so the number of combinations required is:
21 = 2
There could be several website elements influencing conversions, revenue, or whatever output you may be seeking to optimize. For example, if you believe your order button size, font, position (top or bottom of page) and color may all have an impact on conversion rate, using a multivariate test allows you to study all of these elements simultaneously. Since you are now studying 4 factors at 2 levels, your multivariate test would require:
24 = 16 combinations
If you want to study additional levels for each factor, this would require more combinations. For example, if you wanted to test green, red, blue and yellow order buttons, this would require a 4 level experiment and 44 = 256 combinations!
Since an A/B test splits traffic in 2 directions, the calculated sample size required to obtain statistically significant results is doubled in order to adequately test the population. In multivariate testing, this same calculated sample size is required for each combination in the matrix, meaning you may require 4X, 8X, 16X or more samples (visitors) than an A/B test. For this reason, multivariate is typically recommended for websites which already have a high level of established traffic.
The statistical software analysis of multivariate testing results will tell you which combination of factors had the best outcome, which individual factors had the most impact and whether or not these results were significant. Just as importantly, the analysis will tell you which individual factors and combinations of factors were not significant. This knowledge can help guide future testing efforts and eliminate unnecessary testing.