Deduplicate documents without erasing valid versions

Duplicate files can crowd retrieval with repeated passages. But two documents with the same title may be legitimate versions with different effective rules. Deduplicate from content and source identity, not title similarity alone.

Separate exact copies from versions

The RAG ingestion lab adds an exact copy of the travel policy.

from ingestion_cases import dedup_case

result = dedup_case()
assert result["input_documents"] == 5
assert result["exact_duplicate_groups"] == [["travel-v1", "travel-copy"]]
assert result["refund_versions"] == ["refund-v1", "refund-v2"]
assert result["valid_versions_preserved"] is True

The exact travel texts share a digest and can be grouped. Refund v1 and v2 have different rules and remain separate. The corpus is synthetic.

Define duplicate levels

Byte-identical files, normalized-text duplicates, near duplicates and versioned revisions need different actions. Keep a canonical source plus aliases for exact copies if access and provenance match. For near duplicates, calculate similarity as a review signal and inspect changed clauses.

Never merge across access classes, owners or tenants just because text matches. One copied public document does not make a restricted source public. Retain source relationships for deletion and audit.

Control retrieval redundancy

At query time, group near-identical chunks by source family while preserving the effective version. Prefer current approved content for current questions, but allow historical queries to filter by date. Measure redundant top-*k* results and whether deduplication removes any labelled evidence.

Hash canonical text with documented normalization. Changing whitespace rules or extraction tools can alter identity, so version the process. A hash match proves identical processed bytes; it does not establish that the source is approved.

The Generative & Agentic AI course treats deduplication, version selection and contradictory evidence as separate ingestion responsibilities.

Exercise

Create exact, normalized and near-duplicate pairs plus two true policy revisions. Write grouping rules, preserve access/source aliases and test that historical and current questions retrieve the intended version.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Attach metadata that supports retrieval filters.
• Continue with Track document freshness in a RAG pipeline.

Reference: Python hashlib documentation.