Turn a Ticket Into an Eval Case
Transform a production RAG failure into a durable eval case with evidence and pass criteria.
Support ticket: AI told Acme they could upgrade mid-cycle without proration. The customer is on an annual plan with 7 months left. Eval case = capability claim + required evidence + answer requirements + scoring rule. Copying the messy ticket text into the eval set preserves ambiguity but not a reliable measurement. The case must be self-contained and scoreable. Name the failed capability Capability: answer upgrade eligibility with billing-cycle proration rules. This prevents the case from becoming a one-off Acme story. Identify required evidence Required sources: upgrade_policy_annual, proration_faq, billing_terms_current. A retrieval check can now verify whether the system found the…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in