Test a knowledge assistant on unanswerable questions

Every corpus has limits. A knowledge assistant must recognize when eligible evidence cannot support an answer, even when retrieval returns topically related text.

Define the expected abstention

The retrieval evaluation lab asks for a meal allowance absent from its synthetic policies.

from retrieval_cases import unanswerable_case

result = unanswerable_case()
assert result["question"]["relevant"] == []
assert result["response"]["status"] == "abstain"
assert result["response"]["answer"] is None
assert result["appropriate_abstention"] is True

BM25 may still return candidates sharing common words. The contract abstains because none is labelled evidence. No model generated the response.

Build several unanswerable types

Include information absent from the corpus, outside the requested date, present only in unauthorized content, contradicted by equally valid sources, and questions too ambiguous to resolve. Distinguish “not found” from “not permitted” without revealing restricted source existence.

Label the desired behaviour: abstain, ask a clarifying question, route to an owner or provide a bounded partial answer. Supply a reason code and safe next step.

Measure both error directions

Track unsupported-answer rate on unanswerable cases and over-abstention on answerable cases. Break down by type. Tune retrieval thresholds on development data; a universal similarity cutoff may behave differently across topics and languages.

Review citations and claims, because an answer can avoid an explicit hallucination yet imply unsupported certainty. Preserve source eligibility in the evaluation trace.

The Generative & Agentic AI course requires unanswerable questions and a tested abstention contract in RAG projects.

Exercise

Add 20 unanswerable cases to 50 answerable ones. Label response policy, measure unsupported answers and over-abstention, and inspect every case where a related but insufficient document caused an answer.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Create hard-negative examples for a RAG benchmark.
• Continue with Calibrate an LLM judge against human-labelled cases.

Reference: NIST Generative AI Profile.