The move: write the eval as the spec, then train against the spec. An eval suite is more than a quiz. It is the operating definition of model quality for a task. Task statement Describe the behavior in observable terms. "Sound more expert" is weak. "Return a structured risk summary with cited source snippets and no uncited legal advice" is testable. Test inputs Use realistic prompts, including messy source material and edge cases. Keep a holdout set the training job never sees. Scoring and analysis Decide what counts as pass, partial pass, and failure. Then analyze by slice, not just…
Read the full lesson
Sign up free: one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in