Diagnose suspicious validation performance using leakage checks and time-aware validation.
Suspicious result The AUC spike appears immediately after a feature refresh. The timing suggests the evaluation might be seeing information the live model will not have. Trustworthy validation Symptom -> boundary -> time split Do not debate the metric until the evaluation design is clean. Shortcut Ship because the metric improved. A lower but believable metric. Validation must mimic the live prediction task. 01 Diff 02 Cutoff 03 Split Step 1 AUC jumps after new features are added.
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in