Decide how to investigate and respond when observed traffic split undermines an A/B test readout.
Pre-readout check Expected traffic split is 50/50, but observed assignment is materially imbalanced before the readout. The p-value assumes the experiment design actually happened. Framework Assignment Validity Before Outcome Significance Before interpreting lift, confirm the randomized groups were formed and logged as planned. Shortcut Read p=0.032 and ship The readout is based on valid groups, not just a green number. Statistical significance is downstream of valid randomization and measurement. Flag Step 1 The split is 61,200 control vs 54,100 treatment in a 50/50 design. What is the first move before anyone trusts the significant p-value? Diagnose Step 2 A log…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in