Running on clusters (YARN / Kubernetes)

What we're doing

You'll learn what a cluster manager actually does, how YARN and Kubernetes fit in, and how to submit jobs with the right flags. Then you'll run a real spark-submit job on the VM to see the full flow end to end.

Step 1: What a cluster manager does

When you submit a Spark job, the cluster manager:

• Finds machines with available capacity to run your work
• Allocates resources — memory, CPU cores, containers
• Launches executor processes on those machines
• Reports back if executors die or need replacement

Spark supports four cluster managers:

• Standalone — Spark's built-in manager, what we've been using on the VM. Simple, no external dependencies.
• YARN — Hadoop's cluster manager. Dominant at enterprises with existing Hadoop infrastructure.
• Kubernetes — the modern container-based option. Increasingly the default for new deployments.
• Mesos — older, being phased out. Rarely used in new projects.

Step 2: The Hadoop-native cluster manager

YARN stands for Yet Another Resource Negotiator. It's the resource manager built into Hadoop, and it's still what most large enterprises use for Spark.

How Spark works on YARN:

• Your Spark driver requests resources from YARN's ResourceManager
• The ResourceManager finds worker nodes with capacity
• Executors launch as YARN containers on those nodes
• HDFS is usually the storage layer underneath

Why YARN is still common:

• Massive existing Hadoop clusters at banks, telcos, and older tech companies
• Integrates natively with HDFS, Hive, and the rest of the Hadoop ecosystem
• Well-understood by operations teams

Step 3: Kubernetes/the container-based option

Kubernetes has become the default cluster manager for new Spark deployments. Instead of YARN containers, Spark runs as Kubernetes pods.

How Spark works on Kubernetes:

• Your driver runs as one pod
• Each executor runs as its own pod
• Kubernetes handles scheduling, restarts, and resource limits
• Storage typically comes from cloud object storage (S3, GCS, Azure Blob) via connectors

Step 4: spark-submit/ the universal entry point

The command to submit any Spark job to any cluster manager is spark-submit:

spark-submit \
  --master <cluster-manager-url> \
  --deploy-mode <client-or-cluster> \
  --executor-memory <memory> \
  --executor-cores <cores> \
  --num-executors <count> \
  your_spark_job.py

The --master flag is what changes between cluster managers:

• Standalone: spark://master-host:7077
• YARN: yarn
• Kubernetes: k8s://https://kubernetes-api-server

Step 5: Deploy modes/client vs cluster

The --deploy-mode flag decides where your Spark driver runs.

Client mode — the driver runs on the machine that submitted the job. If you run spark-submit from your laptop, your laptop is the driver. When your laptop closes or disconnects, the driver dies and the job dies with it.

• Good for interactive work — spark-shell, PySpark, notebooks
• Good for development and debugging
• Bad for production — driver is tied to a machine that isn't part of the cluster

Cluster mode — the driver runs inside the cluster, on one of the worker nodes. You submit the job, and even if your terminal closes, the driver keeps running on the cluster.

• Good for production — job runs independently of the submitter
• Standard for scheduled and long-running jobs
• Slightly harder to see logs (they're on the cluster, not on your machine)

Step 6: Resource allocation flags

Three flags control how much of the cluster your job uses:

• --executor-memory - memory per executor. Common values: 2g, 4g, 8g. Include the unit.
• --executor-cores -CPU cores per executor. Common values: 2, 4, 5. Above 5 hits diminishing returns.
• --num-executors - total number of executors. On YARN it defaults to 2 if unset — usually not enough for real work.

Total resources requested = num-executors × (executor-memory + overhead) for memory, and num-executors × executor-cores for CPU. If your job asks for more than the cluster has, it'll queue or fail depending on the cluster manager's policy:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 4g \
  --executor-cores 4 \
  --num-executors 10 \
  my_job.py

Step 7: Submit a job on the standalone cluster

In VS Code, create ~/spark_job.py:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, rand

spark = SparkSession.builder \
    .appName("SparkOpsDemo") \
    .getOrCreate()

df = spark.range(0, 1_000_000).select(
    col("id"),
    (rand() * 100).cast("int").alias("category"),
    (rand() * 1000).alias("value")
)

result = df.groupBy("category").agg({"value": "sum"}).orderBy("category")
result.show()

spark.stop()

Save it. Now submit it:

spark-submit \
  --master spark://localhost:7077 \
  --deploy-mode client \
  --executor-memory 1g \
  --executor-cores 2 \
  ~/spark_job.py

You'll see Spark log lines scroll past as executors start, the job runs, and results print. Open the Spark Master UI at port 8080 while it's running under Running Applications, you'll see your submitted job with the resources you allocated.

Once the job finishes, it moves to Completed Applications. That's the same flow that would happen if you ran this on a real YARN or Kubernetes cluster, you'd just change the master URL and possibly the deploy mode.