Build a keyword retrieval baseline before vector search

Keyword retrieval is fast, inspectable and often strong for identifiers, policy terms and exact phrases. Build it before dense retrieval so added infrastructure has a fair baseline to beat.

Execute BM25 on labelled questions

The retrieval evaluation lab implements BM25 over six synthetic documents.

from retrieval_cases import keyword_case

result = keyword_case()
assert result["executed"] == "local BM25 implementation"
assert result["answerable_questions"] == 5
assert result["recall_at_3"] >= 0.8
print(result["rankings"])

The denominator excludes one explicitly unanswerable question, leaving five labelled answerable cases. This small synthetic result is a unit-scale baseline, not production retrieval evidence.

Define relevance before ranking

For each question, label the minimum document or span that supports the answer. Include exact terms, paraphrases, numbers, identifiers, ambiguous words and hard negatives. Apply effective-version and authorization filters before scoring.

Record tokenizer, normalization, stemming, stop words and BM25 parameters. Test how punctuation, plural forms and domain abbreviations affect results. Preserve per-query rankings; a mean can hide one missing critical policy.

Compare incremental value

Dense or hybrid retrieval should use the same corpus, labels, filters and *k*. Compare recall, rank, duplicate results, latency, cost and answer support. Keyword retrieval may remain the selected approach or a hybrid component when exact matches matter.

Avoid tuning directly on a tiny final set. Use development questions for parameter changes and keep a release set. Re-evaluate after corpus or tokenizer changes.

The Generative & Agentic AI course requires this baseline before vector search in its RAG evaluation path.

Exercise

Build BM25 for 50 labelled questions. Report recall@1/3/5 and failed evidence spans. Add one synonym strategy, evaluate on the same set and record both gains and new false matches.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with BM25 versus dense retrieval on the same question set.
• Then apply it in Hybrid retrieval with reciprocal rank fusion.

Reference: BM25 paper.