Metric Battlecards
Recall when to use common recommender metrics and how to handle common objections.
Offline vs online NDCG@k vs task-success metric: which decides launch? Use offline metrics to screen; use online metrics and guardrails to launch. Objection CTR is up, so why are we holding launch? The experiment also increased hide rate and reduced week-two retention. Your line CTR tells us the items attracted attention. The guardrails tell us whether that attention was healthy enough to keep. Treating attraction as satisfaction can reward clickbait or repetitive recommendations. It acknowledges the win while explaining why the decision needs guardrails. Coverage What does item or supplier coverage tell you? Whether the recommender is concentrating exposure into…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in