Sort: Spot The Experiment Anti-Pattern
Recognize behaviors that break hypothesis, sample-size, or significance discipline.
Sort each team behavior into valid practice or the statistical risk it creates. Valid practice Hypothesis risk Sample-size risk Significance risk Write the primary metric and guardrails before launch Use a pre-defined harm threshold for stopping early Rewrite the hypothesis after seeing which users responded Declare a subgroup winner that was not planned or confirmed Call a test directional even though it had traffic for only 20% of the planned sample Choose an MDE after learning the observed lift Stop the test on the first day the p-value dips below 0.05 Search 12 metrics and report only the one that…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in