Executors in Airflow: Local, Celery, and Kubernetes

What we're doing

You'll learn what executors are, see the three main types, and then build a small DAG and watch it run through the CeleryExecutor.

Step 1: What is an executor

The scheduler doesn't actually run tasks itself. It hands them off to the executor. That is the piece of Airflow that decides:

• Where a task runs
• How many tasks can run at once
• How they're isolated from each other

There are three executors in general: LocalExecutor, CeleryExecutor, and KubernetesExecutor.

Step 2: LocalExecutor

LocalExecutor runs everything on one machine. The scheduler and the tasks share the same host. Tasks run as separate Python subprocesses spawned by the scheduler.

Good for:

• Local development and small teams
• Workloads under a few hundred tasks per day
• Teams that want minimal infrastructure overhead

Not good for:

• Anything that needs horizontal scaling
• Workloads with bursty resource demands

Step 3: CeleryExecutor

CeleryExecutor splits the work across multiple worker machines. The scheduler queues tasks, and a fleet of workers pulls them off the queue and runs them.

How it works:

• A message broker (for example Redis) holds the task queue
• Workers run on separate machines, polling the broker for new tasks
• Add more workers to scale horizontally

Good for:

• Medium to large teams
• High task volumes
• Scaling without rewriting the pipeline

Step 4: KubernetesExecutor

KubernetesExecutor runs every task in its own Kubernetes pod. Each task gets a fresh container with exactly the resources it needs, completely isolated.

Good for:

• Teams already on Kubernetes
• Per-task isolation needs, conflicting dependencies
• Bursty workloads with no idle workers between bursts

Trade-offs:

• Requires a Kubernetes cluster
• Pod startup adds latency per task
• Most complex to operate and debug

Step 5: Choosing the right executor

Quick guide:

• Single machine, simple DAGs → LocalExecutor
• No Kubernetes → CeleryExecutor
• Already on Kubernetes, want full isolation? → KubernetesExecutor

Most companies start with LocalExecutor, outgrow it, and move to either Celery or Kubernetes depending on whether they already have a Kubernetes platform.

Step 6: Check the executor on this VM

Open the terminal and run:

docker exec airflow-airflow-scheduler-1 airflow config get-value core executor

You'll see:

Celery Executor

Now look at the containers:

docker ps

You'll see:

• airflow-airflow-scheduler-1 — the scheduler
• airflow-airflow-worker-1 — the Celery worker that actually runs tasks
• airflow-redis-1 — the message broker
• airflow-airflow-apiserver-1 — the web UI
• airflow-postgres-1 — the metadata database

Step 7: Build a small DAG to run through the executor

Click VS Code in the environment panel. Right click on the dags folder and create a new file called executor_demo.py.

Paste this in:

from airflow.decorators import dag, task
from datetime import datetime
import socket
import os

@task
def show_worker_info():
    print(f"Running on host: {socket.gethostname()}")
    print(f"Process ID: {os.getpid()}")
    return socket.gethostname()

@task
def show_environment():
    print(f"Executor info — task running inside container")
    print(f"Working dir: {os.getcwd()}")

@dag(
    dag_id="executor_demo",
    start_date=datetime(2024, 1, 1),
    schedule=None,
    catchup=False
)
def executor_demo():
    show_worker_info() >> show_environment()

executor_demo()

Save with Ctrl+S.

Step 8: Trigger it and trace what the executor does

Trigger executor_demo from the Airflow UI. While it runs, open two terminal tabs.

Terminal 1 — watch the scheduler:

docker logs -f airflow-airflow-scheduler-1

You'll see lines about queueing the task. The scheduler doesn't run it — it hands it off.

Terminal 2 — watch the worker:

docker logs -f airflow-airflow-worker-1

Now in the UI click the task and open Logs. You'll see the print output:

The hostname matches the worker container, not the scheduler. That's CeleryExecutor. Scheduler queues the work, worker picks it up, runs it, reports back.

After hibernation

If the VM hibernates, reconnect and run in the VS Code terminal:

cd ~/airflow
docker compose up -d