Caching and persistence

What we're doing

You'll learn what caching does, see the two APIs Spark provides for it, understand the different storage levels, and measure the speedup live on real data.

Step 1: What caching does

Caching tells Spark to compute the DataFrame and keep the result in memory.

The API is simple:

df.cache()

Every subsequent action on that same DataFrame skips the recomputation entirely and uses the stored version.

Step 2: cache() vs persist()

Spark has two APIs for this:

• cache() - the simple one. Stores in memory. Uses the default storage level of MEMORY_AND_DISK.
• persist() - the flexible one. Lets you choose exactly where and how to store the data.

from pyspark import StorageLevel

df.persist(StorageLevel.MEMORY_ONLY)
df.persist(StorageLevel.DISK_ONLY)
df.persist(StorageLevel.MEMORY_AND_DISK)

Step 3: Storage levels

Storage level controls where the cached data lives and in what format. The main ones:

• MEMORY_ONLY - keep the DataFrame entirely in memory. Only use when you know your data fits comfortably in cluster memory.

• MEMORY_AND_DISK - try memory first, spill to disk if there's not enough room.

• DISK_ONLY - always store on disk. Slower to read than memory, but great when your DataFrame is too big to fit in RAM but you still want to avoid recomputing.

• MEMORY_ONLY_SER and MEMORY_AND_DISK_SER - serialized versions. Use less memory but require deserialization on every read.

Step 4: Start PySpark and set up

pyspark --master spark://localhost:7077

Wait for the Python prompt.

from pyspark.sql.functions import col, rand 
import time

df = spark.range(0, 3000000).select(
    col("id"),
    (rand() * 100).cast("int").alias("category"),
    (rand() * 1000).alias("value")
).groupBy("category").agg({"value": "sum", "id": "count"})

3 million rows with a random category and value.

Step 5: Measure without caching

Run three actions on this DataFrame and time each one:

start = time.time()
df.count()
print(f"First count: {time.time() - start:.2f} seconds")

start = time.time()
df.count()
print(f"Second count: {time.time() - start:.2f} seconds")

start = time.time()
df.count()
print(f"Third count: {time.time() - start:.2f} seconds")

You'll notice all three times are similar.

Step 6: Measure with caching

Now cache the DataFrame and repeat:

df.cache()

# Trigger caching
start = time.time()
df.count()
print(f"First count (caches now): {time.time() - start:.2f} seconds")

# Reuse cached result
start = time.time()
df.count()
print(f"Second count (from cache): {time.time() - start:.2f} seconds")

# Reuse cached result again
start = time.time()
df.count()
print(f"Third count (from cache): {time.time() - start:.2f} seconds")

The first count should take about as long as before. But the second and third counts should be dramatically faster, often 5x to 20x, because Spark isn't recomputing anything.

Open the Spark Master UI, click into the application, and go to the Storage tab. You'll see your cached DataFrame listed with its memory usage and partition breakdown.

Step 7: Release the cache

When you're done with a cached DataFrame, release it so the memory can be used for other work:

df.unpersist()

unpersist removes the DataFrame from cache immediately.

Exit with exit().