Design a recovery exercise for lost workflow state

A backup existing is weaker evidence than a restore exercise. Stateful AI workflows also need semantic recovery: after restoring bytes, can the system rebuild the correct state without repeating an external action?

Define the measurable control

Define recovery point and recovery time objectives from business need. Snapshot durable state, preserve an ordered versioned event log and record downstream idempotency identifiers. Simulate loss, restore the snapshot, replay later events, reconcile external effects and verify user-visible state.

The AI reliability lab makes the decision reproducible with authored data:

from reliability_cases import state_recovery_case

result = state_recovery_case()
assert result["replayed"] == ["approval_requested", "approved"]
assert result["recovered"] == "approved"
assert result["duplicate_effects"] == 0
assert result["external_system_contacted"] is False

The fixture restores sequence two and replays approval-requested plus approved to reach the approved state. It records zero duplicate effects and contacts no external system. A fuller drill injects corruption, missing events and an ambiguous downstream acknowledgement.

Challenge the result

Replaying code from today against events written by yesterday’s schema can corrupt state. Version event handlers and retain migration tests. Protect backups as sensitive data, test access independently and record how deletion obligations interact with retention.

Keep a reviewable evidence pack:

• RPO/RTO and state inventory
• snapshot and event version manifest
• restore/replay/reconciliation log
• duplicate-effect and missing-event tests

This work aligns with the evaluation, security, cloud operations, reliability and FinOps sequence in the FDE for Professionals course. The linked course describes the learning pathway; this article’s numbers are synthetic and do not report a model, client, audit or production result.

Practice task

Delete a local workflow projection, rebuild it from a snapshot and four events, and compare every field with the known expected state.

Continue learning

This article is part of the Advanced AI reliability and assurance sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate private inference against a managed model API.
• Continue with Write an AI assurance report with reproducible evidence.

Reference: Google SRE Book: Addressing Cascading Failures.