Create a document intake workflow with duplicate detection

Email retries, users and connectors can submit the same file more than once. Deduplicate before expensive extraction and side effects, while preserving a trace of each received event.

Hash exact content

The automation lab processes three in-memory fixtures.

from automation_cases import duplicate_case

result = duplicate_case()
assert result["accepted"] == ["D1", "D3"]
assert result["duplicates"] == ["D2"]
assert result["duplicate_of"] == {"D2": "D1"}
assert result["content_stored_externally"] is False

D1 and D2 have identical bytes. No external storage received them.

Use layered identity

Record trigger event ID, source system/file ID, size, checksum, uploader, tenant and receipt time. An atomic uniqueness constraint prevents concurrent workers from accepting the same key. Exact hashes do not catch rescanned, recompressed or slightly edited duplicates; add a reviewed near-duplicate stage using normalized text, perceptual hashes or business keys.

Do not silently discard. Link the duplicate event to the canonical record and tell the owner whether processing was skipped. Version genuinely revised documents and invalidate dependent extraction/cache artifacts.

Test zero-byte files, hash collisions by simulated key, concurrent submissions, deleted canonical records and cross-tenant copies. Deduplication must not reveal that another tenant holds the same document.

The Generative & Agentic AI course joins ingestion manifests with idempotent automation.

Exercise

Create ten local files containing exact and near duplicates. Implement atomic exact deduplication, quarantine near matches and prove two parallel events create one canonical record.

Continue learning

This article is part of the Business automation with AI sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in AI email drafting with a mandatory human send step.
• Continue with Connect a form to an AI workflow without trusting raw inputs.

Reference: NIST AI RMF Generative AI Profile.