Respond to an AI canary failure using symptoms, causes, and rollback criteria.
Canary room Traffic is at 5%. Latency is green, but escalations double and tool denials jump to 19%. The launch window closes in two hours. The service is up, but user-visible behavior says the new release may be unsafe or unusable. SRE canary response Symptom first, cause second Honor user-visible symptoms before debating the internal cause. Preserve enough evidence to learn, then reduce harm. Explain it away Launch pressure turns signals into opinions. The team learns without using all users as the test set. When a canary hits a stop condition, continuing is no longer measurement. It is exposure. 01…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in