Joins, broadcast joins, and handling data skew

What we're doing

You'll build DataFrames with realistic skew, run different join strategies against them, measure the difference, and see exactly why join optimization is one of the biggest performance wins in Spark.

Step 1: What is a join in Spark

A join combines two DataFrames on a shared column.

The most common join types you'll use:

• Inner join — only keep rows where the key exists on both sides
• Left join — keep all rows from the left side, matching where possible
• Right join — keep all rows from the right side
• Outer join — keep everything, filling in nulls where there's no match

Step 2: Broadcast joins

A broadcast join eliminates the shuffle entirely.

• Take the smaller DataFrame
• Send a full copy of it to every worker in the cluster
• Now every worker has the small side locally
• Each worker can join its partition of the big side against the local copy

Step 3: Data skew

The other big performance killer with joins is data skew. Skew happens when one join key has way more rows than the others.

Skew often shows up as:

• A Spark UI stage where one task takes 10 minutes and every other task takes 30 seconds
• A job that used to be fast suddenly gets slow as your data grows
• Out-of-memory errors on one worker while others have free memory

Step 4: Start PySpark

pyspark --master spark://localhost:7077

Step 5: Build a big DataFrame with realistic skew

Let's create a large DataFrame simulating transactions, where 80% of them come from just 3 users and the remaining 20% are spread across other users.

from pyspark.sql.functions import col, when, rand, lit

big_df = spark.range(0, 1000000).select(
    col("id").alias("txn_id"),
    (when(rand() < 0.8, (rand() * 3).cast("int"))
     .otherwise((rand() * 100).cast("int"))
    ).alias("user_id"),
    (rand() * 1000).alias("amount")
)

• spark.range(0, 1_000_000) — a million rows to simulate transactions
• txn_id — the id column renamed
• user_id — this is where we inject skew. rand() < 0.8 means 80% of the time we pick a random number from 0–2 (three big users). The other 20% of the time we pick from 0–99 (all users). Result: users 0, 1, and 2 get most of the data
• amount — a random dollar amount

Step 6: Build a small users lookup table

Now a small DataFrame representing user metadata.

from pyspark.sql.functions import concat, lit

small_df = spark.range(0, 100).select(
    col("id").alias("user_id"),
    concat(lit("user_"), col("id")).alias("name"),
    when(col("id") < 5, lit("premium")).otherwise(lit("standard")).alias("tier")
)

small_df.show(10)

Step 7: Run a broadcast join

Now let's try the broadcast version:

from pyspark.sql.functions import broadcast

start = time.time()
broadcasted = big_df.join(broadcast(small_df), on="user_id")
broadcasted.count()
print(f"Broadcast join: {time.time() - start:.2f} seconds")

Exit with exit().