Zum Hauptinhalt springen

Decision Walk: Diagnose A Broken Randomization

Decide how to investigate and respond when observed traffic split undermines an A/B test readout.

ab-testing5 min read

Pre-readout check Expected traffic split is 50/50, but observed assignment is materially imbalanced before the readout. The p-value assumes the experiment design actually happened. Framework Assignment Validity Before Outcome Significance Before interpreting lift, confirm the randomized groups were formed and logged as planned. Shortcut Read p=0.032 and ship The readout is based on valid groups, not just a green number. Statistical significance is downstream of valid randomization and measurement. Flag Step 1 The split is 61,200 control vs 54,100 treatment in a 50/50 design. What is the first move before anyone trusts the significant p-value? Diagnose Step 2 A log…

Read the full lesson

Sign up free: one personalized lesson every day, matched to your role and goals.

← Back to library

Kontakt aufnehmen