Turn a Failed Attack Into a Regression Test
Convert a successful red-team attack into a repeatable regression test with pass criteria.
A hostile PDF can make a procurement assistant redirect an approval email. The fix has landed, but the team needs a test to keep it fixed. Finding -> fixture -> expected safe behavior -> scorer -> release gate The common shortcut is to paste the original transcript into a ticket and call it documentation. That does not run when the retriever, prompt, or model changes. Fixture Save a minimal injected PDF with one hostile instruction and no real confidential content. The fixture should be safe, small, and focused on the failed boundary. Prompt Use a stable user task: summarize this…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in