Aller au contenu principal

Experiment Result Reversal Walk

Navigate an experiment readout where aggregate results may hide a segment reversal.

data-skepticism6 min read

The readout The overall experiment lift is positive, but returning users were much more common in treatment than control. If the product ships globally on a mix-driven lift, new users may get a worse experience. Simpson check Overall -> segments -> mechanism -> rollout Read the aggregate, then ask whether the same story appears inside the segments that will receive the rollout. Shortcut Ship because the overall result is green. A rollout decision that survives the subgroup read. Simpson's paradox is a signal to compare like with like before trusting an overall average. 01 Mix 02 Segments 03 Rollout Step…

Read the full lesson

Sign up free: one personalized lesson every day, matched to your role and goals.

← Back to library

Nous contacter