# Local data engineering for model inputs

Run `python data_engineering_cases.py` from the repository root. The lab generates an original synthetic account-event fixture. Identifiers, regions, amounts and timestamps are invented and carry no external or learner data.

The verifier covers equivalent ETL/ELT outputs, a training contract, latency requirements, SQLite/Parquet reconciliation, partition pruning, measured CSV/Parquet behavior, idempotent ingestion, late events, point-in-time joins, lineage, SQL/pandas equality with Spark unexecuted, skew, Spark and Airflow evidence states, two-stage quality gates, schema evolution, bounded backfill, platform concepts, local pipeline reproducibility and critical dependencies.

Local elapsed times vary by machine. PySpark and Airflow are not installed or executed. Static plans are teaching artifacts rather than distributed-system evidence.

The 20-check verifier passed on 20 September 2026. Its saved JSON records all cases, including the corrected ingestion invariant: 1,000 unique source events are stored once and replaying the final batch adds zero rows. The generated format timings and Python hash-partition distribution can vary between processes or machines; the articles use the stable size, row-count and skew evidence instead.