One primary metric earns the verdict; guardrails keep the verdict honest. Statistical significance assumes you know what question you are testing. When a team checks many metrics after the fact and celebrates whichever one crosses 0.05, the false-positive risk quietly rises. A clean A/B test names one primary metric tied to the hypothesis, then uses guardrails to catch unacceptable side effects. Sample size should be calculated for the primary metric, not the easiest metric to move. If checkout completion needs 120,000 sessions to detect a meaningful lift, open-rate significance at 8,000 sessions does not rescue the decision. Guardrails are interpreted…
Read the full lesson
Sign up free: one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in