Measure retry amplification across an agent workflow

Retries recover transient failures, but each retry consumes capacity and can trigger more downstream retries. In an agent workflow, this multiplication can hide behind a single user request.

Define the measurable control

Assign a run id, logical step id and attempt number to every call. Count original plus repeated tool calls and divide by logical steps. Report amplification by tool and failure reason. Bound attempts, add exponential backoff with jitter, respect retry-after signals and keep a total run deadline.

The AI reliability lab makes the decision reproducible with authored data:

from reliability_cases import retry_amplification_case

result = retry_amplification_case()
assert result["logical_steps"] == 16
assert result["tool_calls"] == 24
assert result["amplification"] == 1.5
assert result["within_budget"] is False

Four authored runs contain sixteen logical steps and twenty-four calls: 1.5 calls per step. That exceeds the fixture’s 1.25 budget. The result says where to investigate; it does not prove retries are wrong. Some may be necessary, while duplicate or nested policy retries should be removed.

Challenge the result

Averages hide retry storms. Inspect the tail and simultaneous bursts, and distinguish a retry from intentional polling. Any write retry needs a stable idempotency key and reconciliation for an ambiguous timeout-after-success.

Keep a reviewable evidence pack:

• trace fields for run, step and attempt
• amplification histogram by dependency
• retry budget and total deadline
• failure test for timeout after side effect

This work aligns with the evaluation, security, cloud operations, reliability and FinOps sequence in the FDE for Professionals course. The linked course describes the learning pathway; this article’s numbers are synthetic and do not report a model, client, audit or production result.

Practice task

Instrument one five-step workflow. Inject a transient failure at two layers and prove that only one layer owns the retry policy.

Continue learning

This article is part of the Advanced AI reliability and assurance sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Define an SLO for successful AI task completion.
• Continue with Design a circuit breaker around unreliable tools.

Reference: Google SRE Book: Handling Overload.