Explain how to design test oracles for AI outputs that do not have one exact expected answer.
The oracle is part of the test design. An oracle is how a test knows whether the observed result is acceptable. For a calculator, the oracle may be one exact number. For an AI summarizer, chatbot, reranker, or classifier, one exact expected answer may be the wrong mental model. Do not collapse the problem into vibes. Build a practical oracle from smaller checks. Required facts confirm the output used the source. Forbidden claims catch hallucination or policy breaches. Ranges handle length, confidence, latency, or score thresholds. Rubrics capture judgment that exact match cannot. Metamorphic relations test whether behavior changes in…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in