Diagnose a RAG failure from its intermediate evidence

“The chatbot hallucinated” is not a diagnosis. A RAG answer can fail because the source was absent, filtered out, poorly chunked, not retrieved, dropped from context, ignored during generation or cited incorrectly.

Find the first failed stage

The retrieval evaluation lab stores an authored trace.

from retrieval_cases import diagnosis_case

result = diagnosis_case()
assert result["first_failed_stage"] == "retrieval"
assert result["generation_should_have_abstained"] is True
assert result["diagnosis"] == "eligible source existed but retrieval returned none"
print(result["trace"])

refund-v2 was eligible, but retrieval returned no chunks. Generation then invented passport acceptance instead of abstaining. This is a fixture, not a live agent trace.

Inspect in order

1. 1Inventory/ingestion: Was the correct version processed completely?
2. 2Eligibility: Was it allowed for this user and date?
3. 3Retrieval: Did labelled evidence appear at *k*?
4. 4Assembly: Did truncation/order keep it in context?
5. 5Generation: Did claims follow supplied evidence and abstention rules?
6. 6Citation: Do source spans support each claim?

Capture stable IDs, versions, scores, filters and validation results without logging unnecessary sensitive content. A trace needs enough evidence to reproduce the decision, not every secret.

Repair the responsible stage

Do not change the prompt to fix a missing index record. Do not change embeddings to fix a wrong authorization filter. Add the failure to the appropriate regression suite and rerun downstream evaluation after the repair.

The Generative & Agentic AI course uses stage-level traces for debugging, observability and release review.

Exercise

Inject six failures, one at each stage. Give traces to a reviewer without the answer key, measure diagnosis accuracy and add a regression that fails at the earliest responsible boundary.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate retrieval on numerical tables.
• Continue with Measure the effect of stale documents on answer quality.

Reference: OpenTelemetry tracing concepts.