Batch versus streaming data for an actual prediction need

Streaming is appropriate when a decision loses value before the next batch can arrive. It also adds state, ordering, late-event and operational complexity. Choose from a maximum tolerable data age and recovery requirement, not from a vague desire for real time.

Three decisions, two architectures

The local lab records an authored requirements table.

from data_engineering_cases import batch_stream_case

result = batch_stream_case()
assert result["streaming_required"] == 1
assert result["batch_sufficient"] == 2
for need in result["needs"]:
    print(need["decision"], need["maximum_data_age"], need["mode"])

Monthly model retraining tolerates 24-hour data and weekly inventory planning tolerates six hours, so scheduled batch is sufficient. A fraud-review queue with a 30-second usefulness window requires a streaming path under the authored assumption.

Specify more than latency

Define volume, event order, duplicate behavior, correction, replay, availability and cost. A streaming system still needs durable history and backfill. Exactly-once business outcomes require idempotent sinks and keys; a delivery guarantee alone cannot undo duplicate side effects.

For batch, define cutoff, completion objective and rerun behavior. A reliable hourly batch can beat an unreliable “real-time” system for many decisions.

Measure end-to-end freshness

Track event-time-to-feature and feature-to-decision delay, not only compute duration. Set alerts and fallback when streams lag. Backtest features with point-in-time logic identical to serving.

The Data Science course relates data movement to the actual model action and its value window.

Exercise

Write freshness, ordering, replay and downtime requirements for one prediction. Estimate batch and streaming complexity and select the simplest design that meets the decision.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Design a training dataset contract.
• Continue with Data warehouses versus lakes for analytical workloads.

Reference: Apache Beam programming guide on event time and windows.