Airflow Executors: Local, Celery, and Kubernetes

Executors in Airflow

The executor is the component of Airflow that actually runs tasks. The scheduler decides what to run. The executor determines where and how it runs.

Every executor controls three things about every task:

• Where the task runs (which machine or container)
• How many tasks can run at once (parallelism)
• How isolated tasks are from each other (shared memory vs separate containers)

Executor Types

Airflow supports three main executors used in production.

Local executor – runs everything on one machine. The scheduler creates a Python subprocess on the same host as itself. No extra infrastructure required, just limited to one machine.

Celery executor – distributes tasks across multiple worker machines. The scheduler puts tasks on a message queue (typically Redis or RabbitMQ). A fleet of worker processes pull tasks off the queue and execute them. Results go back to the metadata database.

Kubernetes executor – launches every task as its own Kubernetes pod. When the scheduler decides a task is ready, it calls the Kubernetes API to create a pod. When the task finishes, the pod is destroyed. Every task is completely isolated with its own container, memory, and CPU limits. Most powerful but also the most complex to operate.

Checking the Current Executor

To check which executor your environment is using, open a terminal and run:

docker exec airflow-scheduler airflow config get-value core executor

Key parts of this command:

• docker exec airflow-scheduler – runs the command inside the running airflow-scheduler container.
• airflow config get-value core executor – reads the executor value from the [core] section of Airflow's config.

Creating a Minimal DAG to Verify the Executor

Create a new file in the DAGs folder called executor.py:

from airflow.decorators import dag, task
from datetime import datetime
import socket
import os

• dag and task come from the TaskFlow API for a cleaner decorator-based syntax.
• socket provides the host name of the machine or container running the task.
• os provides the process ID and working directory.

Defining the Tasks

First task – reports which host and process is running the task:

@task
def get_host_name():
    host_name = socket.gethostname()
    print(f"Host name: {host_name}")
    process_id = os.getpid()
    print(f"Process ID: {process_id}")
    return host_name

Second task – reports the working directory:

@task
def show_environment():
    working_directory = os.getcwd()
    print(f"Working directory: {working_directory}")
    return working_directory

Defining the DAG

@dag(
    dag_id="executor_demo",
    start_date=datetime(2023, 1, 1),
    schedule=None,
    catchup=False
)
def executor_demo():
    host_info = get_host_name()
    env_info = show_environment()
    host_info >> env_info

executor_demo()

Key fields:

• schedule=None – the DAG runs only when triggered manually.
• catchup=False – Airflow will not backfill previous intervals.
• host_info >> env_info – enforces dependency: show_environment runs after get_host_name.

Running the DAG

Save the file. In the Airflow UI, navigate to the DAGs section, search for executor_demo, and trigger a single run.

Expected output:

• get_host_name logs the container host name and process ID.
• show_environment logs the working directory as /opt/airflow, which is the Airflow container path.