Recall the highest-risk contamination traps that weaken the treatment-versus-control comparison.
What is the simplest contamination test? Ask whether control users are receiving any experience element that belongs only to treatment. If yes, the baseline is already blurred. “Let's move our most important accounts into treatment halfway through, but keep calling it the same experiment.” Your line We can override the experiment for business reasons, but once we do that, we should stop pretending the comparison for those accounts stayed randomized. It keeps the tradeoff honest and protects the meaning of the holdout. Which launch habit is stronger? Deliberate classification prevents accidental leakage and makes the result easier to interpret.
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in