Experiment Result Reversal Walk
Navigate an experiment readout where aggregate results may hide a segment reversal.
The readout The overall experiment lift is positive, but returning users were much more common in treatment than control. If the product ships globally on a mix-driven lift, new users may get a worse experience. Simpson check Overall -> segments -> mechanism -> rollout Read the aggregate, then ask whether the same story appears inside the segments that will receive the rollout. Shortcut Ship because the overall result is green. A rollout decision that survives the subgroup read. Simpson's paradox is a signal to compare like with like before trusting an overall average. 01 Mix 02 Segments 03 Rollout Step…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in