PySpark joins: identify skew before changing configuration

Join skew occurs when a few keys hold much more data than others, forcing one partition or task to do disproportionate work. Before changing executor memory or shuffle settings, measure key frequency and inspect the execution plan and task distribution.

A hot-key diagnostic without Spark claims

The local lab creates 10,000 join keys with an authored hot value.

from data_engineering_cases import skew_case

result = skew_case()
assert result["top_key_share"] > 0.70
assert result["pyspark_executed"] is False
print(result["top_key"], result["top_key_rows"], result["top_to_median_ratio"])

HOT owns 7,175 rows, 71.75% of the table, versus a median of 29 rows for other keys—a 247× ratio. The Python hash-to-eight-partition illustration varies by process and is not a Spark partition measurement. PySpark is unexecuted.

Inspect the real join

Measure both tables: row counts, null keys, top keys, distinct keys and estimated joined expansion. A many-to-many duplicate can explode rows even without cluster skew. Inspect explain output and Spark UI task durations, shuffle reads and spills when Spark exists.

Choose a remedy for the cause

Broadcast a genuinely small side when memory and freshness allow. Pre-aggregate duplicates if semantics permit. Salt only hot keys and reconcile afterward. Filter irrelevant rows early. Adaptive query execution may handle some skew, but verify on the actual plan.

Configuration cannot fix an invalid join grain. The Data Science course starts from row semantics and measured evidence.

Exercise

In an authorized Spark environment, build a skewed join and capture plan/task metrics. Apply one remedy at a time and reconcile output keys and sums.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in SQL versus pandas versus Spark: choose from measured constraints.
• Continue with Spark transformations and actions with an execution plan.

Reference: Spark SQL performance tuning and adaptive query execution.