Incremental ingestion with deduplication and watermarks

Incremental ingestion reads new or changed records rather than rebuilding all history. A watermark limits the next read, while a stable event key makes retries idempotent. The watermark must match source ordering; using a maximum ingestion timestamp over data sorted by event time can skip records.

Recover all rows and replay safely

The local lab sorts a 1,000-event source by ingestion time. The first pull reads 800 rows. The second overlaps 50 old rows and includes the remaining 200. It then replays the second pull.

from data_engineering_cases import incremental_case

result = incremental_case()
assert result["final_rows"] == result["final_unique_events"] == 1000
assert result["idempotent_replay_added"] == 0
for run in result["runs"]:
    print(run)

The second pull accepts only rows after the watermark, and replay adds none. Primary-key deduplication guarantees one stored row per event ID in this append-only teaching case.

Define update semantics

If events can change, choose the winning version using a source sequence or update timestamp and deterministic tie-break. A watermark on maximum time may miss records arriving with an older timestamp. Use an overlap window, change-data-capture offset or source cursor appropriate to the system.

Commit output and watermark atomically. If a job fails after writing rows but before saving progress, retry should converge. Track input, accepted, deduplicated and rejected counts.

Test gaps and retries

Inject duplicate IDs, out-of-order delivery, same-timestamp events, corrections and failures between steps. Reconcile source and destination counts by window. Keep event time separate for feature logic.

The Data Science course connects incremental reliability to late events and point-in-time features.

Exercise

Add updates to existing event IDs and a record arriving below the watermark. Design an overlap or version policy and prove that repeated runs converge to the same table.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Parquet versus CSV: measure size and read behaviour.
• Continue with Handle late-arriving events in a training dataset.

Reference: Apache Beam programming guide on watermarks and late data.