Choose an embedding model with task-specific evidence

Public benchmarks can narrow candidates, but your documents, questions, language mix and filters determine retrieval quality. Evaluate with labelled question-to-evidence pairs before reindexing a corpus.

Apply quality and latency gates

The RAG ingestion lab uses authored metrics for candidates A and B.

from ingestion_cases import embedding_choice_case

result = embedding_choice_case()
assert result["quality_gate"] == 0.75
assert result["latency_gate_ms"] == 25
assert result["eligible"] == ["A"]
assert result["selected"] == "A"
print(result["scope"])

B has fixture recall@3 of 0.80 but p95 42 ms, above the 25 ms gate. A meets both at 0.75 and 18 ms. These are invented values; no embedding model or service ran.

Build a representative retrieval set

Label the minimum source spans that answer real task questions. Include terminology variants, identifiers, tables, multiple languages, unanswerable queries and access filters. Keep a keyword baseline; dense retrieval should earn added cost and complexity.

Compare recall@*k*, rank, duplicate results, filter correctness and downstream answer support. Measure end-to-end ingestion/query latency, vector dimensions, storage, provider cost and deployment constraints. Report per-slice denominators.

Control the migration

Pin model and configuration. Re-embed into a new versioned collection, validate count and lineage, run shadow comparisons and switch through an approved pointer. Retain rollback evidence. Check data handling and regional requirements for external embedding APIs before sending content.

Hybrid retrieval may outperform either dense or lexical alone, but it adds fusion/reranking choices that need their own evaluation. Select the simplest candidate that clears declared gates.

The Generative & Agentic AI course makes task-specific retrieval evidence part of the RAG deliverable rather than relying on a generic leaderboard.

Exercise

Compare two current embedding candidates and BM25 on 50 labelled questions. Predeclare quality, language, latency and cost gates; report per-case transitions and write a migration/rollback record.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Version embeddings alongside the source text.
• Continue with Build an ingestion error queue for failed documents.

Reference: MTEB benchmark repository.