Keep Four AI Oracle Moves Handy
Recall four oracle strategies for different AI testing situations.
Structured output When is an exact oracle still appropriate for AI output? Use exact or schema checks when the expected shape is deterministic, such as tool-call JSON, required fields, policy IDs, or refusal codes. Do not make open-ended prose exact-match unless exact wording is part of the requirement. Rubric or relation Rubric versus metamorphic relation: which one fits? Rubrics judge quality. Relations test behavioral consistency across controlled changes. Objection We cannot automate this AI test because there are many acceptable answers. Your response Agreed, exact match is wrong here. Let's define required facts, forbidden claims, and a review threshold instead.…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in