Handle an LLM refusal in an application workflow

An application should not treat every non-answer as the same error. A safety refusal, rate limit, timeout, invalid schema and missing evidence need different responses. Blind retries can increase cost, repeat a refusal or turn a recoverable validation problem into a confusing user experience.

Classify before retrying

The offline foundations lab defines three authored failure classes.

from foundation_cases import refusal_case

result = refusal_case()
assert result["blind_retry_on_refusal"] is False
assert result["retryable_count"] == 2
print(result["classes"])

The refusal receives a bounded explanation and safe alternative without automatic retry. A rate limit allows bounded backoff. Invalid structured output allows one schema-repair attempt before review. This table is workflow design evidence, not a provider response log.

Preserve the user’s task

Return a stable application status rather than raw provider prose alone. Explain what could not be completed and what safe path remains. If part of a request is allowed, offer that bounded portion without pretending the complete task succeeded.

Do not rewrite a disallowed request repeatedly to bypass a refusal. Do not switch providers silently as a safety fallback. Record provider/model, request ID, failure class and handling outcome under appropriate privacy controls.

Separate policy, availability and evidence

A refusal may be a model response; an authorization denial should come from application policy; an abstention may come from missing evidence. Keep those states distinct in telemetry and user messaging. This distinction helps teams see whether failures require prompt changes, capacity controls, policy review or better sources.

Test refusal handling with multi-turn conversations, tool calls, streaming responses and partial structured data. Enforce maximum attempts and total time. Confirm that downstream actions cannot run on a refused or unvalidated output.

The Generative & Agentic AI course connects refusal handling to typed agent state, bounded retries, evaluation and human escalation.

Exercise

Design an error taxonomy for one application. Simulate a refusal, timeout, rate limit, malformed JSON and missing source. Verify retry count, user status, log record and that no side effect occurs after each failure.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Separate model knowledge from current external information.
• Continue with Design an abstention response for missing evidence.

Reference: OpenAI error-code guidance.