Quick Reference: Readout Battlecards
Respond to common A/B testing readout objections using hypothesis, sample-size, and significance principles.
Close enough "p=0.07 is basically significant, right?" The team wants to round a miss into a win. Your line It missed the threshold we chose before launch. We can call the signal promising, but the decision should be iterate or rerun, not ship as a confirmed win. Do not move alpha after seeing results; that is how false positives become policy. It respects the pre-committed significance rule and still gives a useful next step. Directional "Can we call it directional and launch carefully?" The test was underpowered for the stated MDE. Directional is fine for discovery language, not a rollout…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in