Measure the effect of stale documents on answer quality

Stale documents may rank highly because they share vocabulary with current policy. Measure the harm directly instead of assuming a freshness filter is cosmetic.

Compare identical cases

The retrieval evaluation lab stores two authored answer pairs.

from retrieval_cases import stale_effect_case

result = stale_effect_case()
assert result["accuracy_with_stale"] == 0.5
assert result["accuracy_current_only"] == 1.0
assert result["difference"] == 0.5
print(result["scope"])

Including the old refund rule produces “receipt only” instead of current “receipt or order ID.” The travel answer is unchanged. These are authored answers; no model ran and the difference is not a production estimate.

Build version-sensitive labels

Create questions whose expected answer changes by effective date, plus controls whose answer remains stable. Label the applicable version and source span. Run retrieval/answering with stale sources eligible and with correct as-of filters, holding all other configuration fixed.

Measure current-answer correctness, stale citations, abstention and historical-query correctness. A blanket removal of old sources can break legitimate as-of questions; version selection should follow the query contract.

Add freshness to regression

Test missing dates, overlapping effective intervals, future versions and unchanged titles. Record the owner and review policy. When an updated document arrives, run impacted questions even if the model and prompt are unchanged.

If a source has exceeded its review date without a replacement, decide whether to warn, abstain or request owner confirmation. Do not silently treat age as proof of invalidity.

The Generative & Agentic AI course links freshness metadata to measurable answer impact and historical retrieval.

Exercise

Create three versions of five rules and 30 as-of questions. Compare unfiltered and version-aware systems, report stale-source errors and retain historical accuracy.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Diagnose a RAG failure from its intermediate evidence.
• Continue with Compare chunking strategies without changing multiple variables.

Reference: NIST Generative AI Profile.