Canary releases with a measurable rollback condition

A canary lets a candidate control a small share of real decisions while operators monitor technical and product guardrails. The rollback rule must be set before launch, including the denominator and minimum sample. Otherwise teams can rationalize a bad signal or overreact to one early request.

A five-request teaching canary

The deployment lab routes 5 of 100 invented requests to a candidate. One canary request fails.

from deployment_cases import canary_case

result = canary_case()
assert result["canary_requests"] == 5
assert result["canary_error_rate"] == 0.2
assert result["rollback_triggered"] is True
print(result["policy"])

The policy permits at most 1% errors after a minimum of five canary requests. At 20%, rollback triggers. These values demonstrate logic and are not recommended production thresholds.

Define routing and guardrails

Use stable assignment where repeated requests must stay on one version. Exclude unsupported cohorts explicitly. Monitor availability, latency, schema errors, decision distribution and safety or business guardrails. Mature performance labels may arrive too late for initial rollback.

Specify who can stop the canary, how traffic returns to the last approved version and how compatibility is verified. Preserve candidate logs and artifacts for diagnosis. A rollback should not require rebuilding the fallback.

Increase traffic in stages

Advance only after a minimum duration and sample pass. Recheck every stage. Separate “no evidence of harm yet” from “proven improvement.” Rare failures need larger samples and targeted tests.

The Data Science course connects canary math to a concrete rollback runbook.

Exercise

Design stages at 1%, 5%, 25% and 100%. For each, set minimum requests, technical guardrails and owner. Simulate zero, one and burst errors and verify deterministic rollback.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Shadow deployment before switching model traffic.
• Continue with Monitor model input drift without treating every alert as failure.

Reference: Kubernetes deployment rollout documentation.