Measure ingestion completeness with a source manifest

A pipeline can finish successfully after skipping a required document. Completeness compares what should have been ingested with what the candidate index actually contains.

Fail on a missing output

The RAG ingestion lab expects one page and one chunk from each of four synthetic sources.

from ingestion_cases import completeness_case

result = completeness_case()
assert result["complete_documents"] == 3
assert result["incomplete_ids"] == ["travel-v1"]
assert result["publish_allowed"] is False

The observed travel source has zero chunks, so the publish gate fails. This does not say every real document must yield exactly one chunk; expected values belong to the manifest for its document type.

Reconcile stages

Track expected files/versions, bytes, pages or records, extracted units, chunks, lexical entries and vectors. Add quarantined, failed and intentionally excluded counts with reason codes. Reconcile unique source IDs and digests, not only totals that can cancel missing and duplicate records.

Define criticality. A missing obsolete appendix may allow a warning; a missing current safety policy should block release. Keep that rule in the contract and name the owner who can approve an exception.

Link completeness to retrieval

Count agreement does not prove useful content. Sample source spans, run labelled queries and verify filters. Conversely, strong retrieval on tested questions does not prove untested sources were ingested. Keep both gates.

Version and sign the manifest according to project practice. Save the exact source snapshot and ingestion configuration. During incremental updates, reconcile additions, updates and deletions separately; a stale vector is also a completeness failure.

The Generative & Agentic AI course requires an ingestion manifest before the RAG evaluation report can support a release decision.

Exercise

Build a ten-document manifest with variable page counts. Inject one skip, one duplicate and one stale vector. Produce reasoned failures and prove totals alone would miss at least one defect.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Reindex a corpus without disrupting active readers.
• Continue with Handle contradictory document versions explicitly.

Reference: OpenLineage facets.