Parquet versus CSV: measure size and read behaviour

CSV is plain text and broadly interoperable. Parquet is columnar, typed and compressed, allowing engines to read selected columns and skip row groups. The practical difference depends on data, compression, filesystem, cache and query.

A local projected read

The data-engineering lab writes 12,000 invented event rows to CSV and Snappy-compressed Parquet, then reads only account ID, amount and region.

from data_engineering_cases import format_case

result = format_case()
assert result["results"]["csv"]["rows"] == result["results"]["parquet"]["rows"] == 12000
print(result["size_ratio_csv_to_parquet"])
print({name: row["read_ms"] for name, row in result["results"].items()})

In the recorded run, CSV is 893,188 bytes and Parquet 372,902 bytes, a 2.40× ratio. Live read times are printed rather than treated as portable facts; one cached local read is not a benchmark.

Compare semantics

CSV readers infer types unless a schema is supplied. Dates, nulls and leading zeros can change. Parquet stores types and column metadata, but writers and readers still need compatible schema and timestamp conventions.

Columnar formats help analytical scans and projection. CSV remains useful for small interchange, debugging and systems that require it. Compression can reduce storage while adding CPU; measure complete workload cost.

Benchmark responsibly

Use realistic rows, columns and filters. Repeat cold and warm reads, randomize order, include parse/serialization time and report environment. Validate equal row counts and aggregates before comparing speed.

The Data Science course joins file formats to schema, partitioning and reproducible measurement.

Exercise

Compare uncompressed/gzipped CSV and several Parquet compressions. Measure full scan, three-column projection and date-filter read over repeated trials with correctness checks.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Partition a dataset around common access patterns.
• Continue with Incremental ingestion with deduplication and watermarks.

References: Apache Parquet overview and pandas Parquet API.