Create a model rollback runbook

Rollback must work while an incident is happening. Name the trigger, last approved version, traffic action, compatibility checks, owner and success conditions before release. “Redeploy the old model” is incomplete if its schema or dependencies no longer work.

Eight ordered actions

The deployment lab defines a teaching runbook triggered when canary error rate exceeds 1% after at least five requests.

from deployment_cases import rollback_case

result = rollback_case()
assert result["complete"] is True
assert result["steps"][0] == "freeze_new_traffic"
assert result["steps"][1] == "route_to_last_approved_model"
print(result["fallback_version"])

The fallback is risk-2026-09-01.3. The runbook freezes new candidate traffic, routes to fallback, verifies health and schema, checks metrics, records the incident, preserves failed artifacts, communicates status and opens follow-up.

Verify recovery

Success requires fallback health, schema compatibility and cessation of new errors. Monitor business and safety guardrails too; a technically healthy fallback may be unsuitable after a policy change.

Keep fallback artifact and environment ready. Test rollback in staging and practice permissions. Define database and feature-store compatibility. If the model writes side effects, reverting predictions cannot undo completed actions; include compensation procedures.

Preserve learning

Do not delete the candidate or its logs. Record request IDs, version, time window, trigger and operator actions with controlled access. Conduct a blameless review and update tests or thresholds based on evidence.

The Data Science course connects model versioning and canaries to an executable operating response.

Exercise

Run a tabletop exercise for schema failure, latency spike and performance regression. Time each rollback, verify the fallback and update the runbook where operators hesitate.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Model retraining triggers based on evidence.
• Continue with Dockerize a model service with a reproducible environment.

Reference: Google SRE incident response guidance.