Hypothesis Testing

Hypothesis Testing: A systematic way to select samples from a group or population with the intent of making a determination about the expected behavior of the entire group.

Part of the field of inferential statistics, hypothesis testing is also known as significance testing, since significance (or lack of same) is usually the bar that determines whether or not the hypothesis is accepted.

A hypothesis is similar to a theory

If you believe something might be true but don’t yet have definitive proof, it is considered a theory until that proof is provided. Turning theories into accepted statements of fact is the basis of the scientific method, which consists of basic 4 steps:

Like many commonly used statistical tools today, A/B testing and multivariate testing are forms of hypothesis testing, so it is important to begin your website testing with a strong hypothesis statement.

For example, if you had reason to believe that the color of your landing page might be having a detrimental effect on conversions, your hypothesis statement could be:

“Changing my landing page color from black to blue will have a statistically significant impact on conversions.”

Once this hypothesis is established, you need to run your test to prove (or disprove) it. Including the words “statistically significant” in the hypothesis statement is important, since it means your sample sizes need to be adequate to analyze it as such.

The Null Hypothesis

The word “null” comes from the Latin word “nullus”, meaning not any or nothing. Perhaps this definition is helpful in understanding this often confusing term. In hypothesis testing, your null hypothesis is that nothing will change or improve between the two groups of data. Obviously, this is not want you want to prove, but rather what you want to disprove. For example, your null hypothesis might be that your landing page color change will have no impact on conversions.

The Alternate Hypothesis

Also known as the experimental or “research” hypothesis, this is what you are really aiming to prove through testing. The alternate hypothesis is simply the opposite of the null hypothesis. In our landing page color change example, the hypothesis statement is actually the alternate hypothesis. Therefore, you want to disprove your null hypothesis in order to prove the alternate.

An analogy that is often used to describe hypothesis testing is a defendant on trial, since he is presumed innocent until proven guilty. This is equivalent to the null hypothesis being presumed true until proven false. In the courtroom, the jury decides whether or not there is enough evidence to disprove innocence. In an A/B or multivariate test, the tester sets a significance p-value threshold (such as .05 or 5%) for the test that determines how unlikely the null hypothesis needs to be before we can confidently reject it.