Airflow Executors: Local, Celery, and Kubernetes

Checking the Current Executor

To check which executor is currently configured in your environment, open a terminal and run:

docker exec airflow-scheduler airflow config get-value core executor

Creating a Minimal DAG to Inspect the Worker Environment

Create a new file executor.py in your DAG folder with the following content:

from airflow.decorators import dag, task
from datetime import datetime
import socket
import os

@task
def get_host_name():
    host_name = socket.gethostname()
    print(f"Host name: {host_name}")
    process_id = os.getpid()
    print(f"Process ID: {process_id}")
    return host_name

@task
def show_environment():
    working_directory = os.getcwd()
    print(f"Working directory: {working_directory}")

@dag(
    dag_id="executor_demo",
    start_date=datetime(2023, 1, 1),
    schedule=None,
    catchup=False
)
def executor_demo():
    show_worker_info = get_host_name()
    show_env = show_environment()
    show_worker_info >> show_env

executor_demo()

Viewing the Results in the Airflow UI

Save the file and go to the Airflow UI. Navigate to the DAGs section, search for executor_demo, and trigger a single run. Open the logs for the get_host_name task to see the host name and process ID. Then check the logs for the show_environment task to see that the working directory is /opt/airflow, which is the Airflow path inside the Docker container, not on the host VM.