Partition a dataset around common access patterns

Partitioning places rows into directory or storage units so queries can skip irrelevant data. A useful partition key appears in common filters and has manageable cardinality. Partitioning by a nearly unique account can create thousands of small objects and expensive metadata work.

One weekly query, one monthly partition

The local lab distributes 6,000 events across January, February and March 2026.

from data_engineering_cases import partition_case

result = partition_case()
assert result["partition_count_scanned"] == 1
print(result["partitions_scanned"], result["rows_returned"])
print(result["bad_high_cardinality_partition_count"])

A query from 10 February through the exclusive end 17 February touches only the February partition and returns 446 rows. Partitioning by account would create 600 values in this small fixture, a warning sign for tiny partitions.

Choose granularity from volume

Daily partitions may fit large event streams; monthly partitions may fit smaller data. Estimate bytes and files per partition, common query range and backfill unit. Compact small files. Do not partition on every filter column; columnar statistics can prune within files.

Use an explicit half-open time range to avoid end-of-day errors. Decide whether partition date comes from event time or ingestion time. Model features usually need event time, while incremental discovery may use ingestion metadata.

Test pruning and completeness

Inspect engine plans or read logs to confirm pruning. Verify a date query returns the same rows with and without the partition filter. Backfills should write bounded partitions and update an approved pointer after validation.

The Data Science course connects storage layout with model cutoff and replay behavior.

Exercise

Estimate file counts for hourly, daily, monthly and account partitions at your scale. Choose one layout and test three common date/entity access patterns.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Data warehouses versus lakes for analytical workloads.
• Continue with Parquet versus CSV: measure size and read behaviour.

Reference: PyArrow partitioned dataset documentation.