Quick Reference: A/B Test Statistics In Plain English
Recall the practical meaning of null hypothesis, MDE, power, alpha, p-value, and confidence interval.
Hypothesis What is the null hypothesis in a typical A/B test? The claim that the treatment has no meaningful effect on the primary metric. You reject or fail to reject this claim using the pre-planned statistical test. Sample size What does minimum detectable effect mean before launch? Minimum detectable effect: the smallest change the team wants the test to reliably detect. Smaller MDEs require more sample; unrealistic MDEs create underpowered tests. Compare Power versus alpha: which risk does each control? Both are chosen before launch and both affect sample-size requirements. Objection "The p-value is below 0.05, so this is definitely…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in