Tuning memory and executors

What we're doing

You'll learn how executors, cores, and memory work together to determine job performance, understand Spark's internal memory model, and see how different configurations impact a real workload.

Step 1: Executors, cores, memory

Every Spark job's performance comes down to three settings you control:

• Number of executors — how many worker processes run your job
• Cores per executor — how many tasks each executor can run in parallel
• Memory per executor — how much RAM each executor has to work with

Step 2: How executors and cores relate to parallelism

Parallelism in Spark is determined by two multiplied numbers:

total parallel tasks = number of executors × cores per executor

Rules of thumb:

• Number of partitions should be at least 2x to 4x the total parallel task capacity, so the scheduler always has work ready
• Cores per executor above 5 often hits diminishing returns because JVM garbage collection gets slower on very wide executors
• More small executors give better parallelism and easier failure recovery. Fewer large executors give better memory efficiency for wide operations

Step 3: The memory model — where each executor's RAM actually goes

Every executor has a fixed amount of memory, but that memory is split into regions with specific purposes.

The default breakdown of an executor's memory:

• Reserved memory (~300 MB) — for Spark internal metadata. Untouchable.
• User memory (~40% of the rest) — for your Python code, UDFs, and any data structures you create outside DataFrames
• Execution + storage memory (~60% of the rest) — the "unified region" that Spark uses for shuffles, joins, sorts, and cached DataFrames

Execution and storage share a single pool called the unified memory region:

• Execution memory — used for in-flight operations like shuffles, joins, sorts. Non-cached, gets released after each operation.
• Storage memory — used for cached DataFrames (from .cache() or .persist())

Step 4: Common sizing patterns

There are two main strategies for sizing executors, each with tradeoffs.

Small and many — lots of executors, each with modest resources (say 2-3 cores, 4 GB memory).

Large and few — fewer executors, each big (say 5-6 cores, 20+ GB memory).

Step 5: Off-heap and overhead memory

Executor overhead memory — RAM used by the JVM outside the heap, for things like network buffers, VM internals, and native code. Default is around 10% of executor memory, minimum 384 MB. When you request 8 GB per executor, YARN or Kubernetes actually reserves ~8.8 GB. If you don't have enough overhead, containers get killed during heavy shuffles.

Off-heap memory — memory allocated outside the JVM heap for storing serialized objects. Can be enabled with:

• spark.memory.offHeap.enabled = true
• spark.memory.offHeap.size = 2g

Step 6: See it on a real job

Let's actually change executor settings and watch the impact.

pyspark --master spark://localhost:7077 \
  --executor-cores 1 \
  --executor-memory 512m

Now build a workload:

from pyspark.sql.functions import col, rand, sum as spark_sum
import time

df = spark.range(0, 5000000).select(
    col("id"),
    (rand() * 100).cast("int").alias("category"),
    (rand() * 1000).alias("value")
).groupBy("category").agg(spark_sum("value").alias("total"))

start = time.time()
df.count()
print(f"With 1 core / 512m: {time.time() - start:.2f} seconds")

Now exit and restart PySpark with different settings:

exit()

pyspark --master spark://localhost:7077 \
  --executor-cores 2 \
  --executor-memory 2g

Run the same workload:

from pyspark.sql.functions import col, rand, sum as spark_sum
import time

df = spark.range(0, 5000000).select(
    col("id"),
    (rand() * 100).cast("int").alias("category"),
    (rand() * 1000).alias("value")
).groupBy("category").agg(spark_sum("value").alias("total"))

start = time.time()
df.count()
print(f"With 2 cores / 2g: {time.time() - start:.2f} seconds")

Compare the two timings. You should see meaningful differences — sometimes 2x or more depending on your cluster.

Step 7: Rules of thumb for production tuning

Practical starting points when you're tuning a real job:

• Cores per executor: Start with 4. Go to 5 max. Below 3 wastes coordination. Above 5 hits GC issues.
• Memory per executor: 4-8 GB is a safe range for typical jobs. Go higher only for jobs with heavy caching or large joins.
• Overhead memory: Leave the default unless you see container-killed errors. Then bump spark.executor.memoryOverhead up.
• Number of partitions: Aim for 2-4x total parallel task capacity. Use .repartition() or .coalesce() to adjust.
• Off-heap: Enable for jobs over 20 GB per executor to reduce GC pauses.