Airflow Operators & Dependencies: Multi-Task DAGs

Tasks and Operators

A task is a single unit of work inside a pipeline. Airflow tracks the state of every task separately, which means individual tasks can succeed, fail, or be retried on their own.

An operator defines what a task actually does. The task is the step in the DAG and operator is the code executed by that step.

Airflow provides many built-in operators. Three common ones:

• PythonOperator – runs a Python function.
• BashOperator – runs a shell command.
• EmptyOperator – does nothing. Used as a marker for the start or end of a pipeline.

Dependencies

Dependencies define the order in which tasks run. A task cannot start until its upstream dependencies finish.

Tasks can be chained in three patterns:

• Sequential – one after another.
• Parallel – multiple tasks run at the same time when they do not depend on each other.
• Branching – the pipeline splits into different paths based on a condition.

Setting Up the Environment

Open VS Code, navigate to the Airflow directory, then to the DAGs folder. Create a file named pipeline.py.

Imports and Functions

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
from airflow.operators.empty import EmptyOperator
from datetime import datetime

• DAG – the class that defines the pipeline.
• PythonOperator, BashOperator, EmptyOperator – the three operator types used in this DAG.
• datetime – required for the DAG's start_date.

Define three Python functions that will run inside the pipeline:

def validate():
    print("Validating incoming data.")

def transform():
    print("Transforming data.")

def report():
    print("Sending prepared statement.")

DAG Definition

with DAG(
    dag_id="pipeline",
    start_date=datetime(2023, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:

Key fields:

• dag_id – the name shown in the Airflow UI.
• start_date – the date the DAG becomes active.
• schedule="@daily" – runs the DAG once per day.
• catchup=False – prevents Airflow from backfilling runs between start_date and today.

Creating Tasks

Define each task with the operator that matches what it does.

    start = EmptyOperator(task_id="start")

    validate_task = PythonOperator(
        task_id="validate",
        python_callable=validate,
    )

    check_logs = BashOperator(
        task_id="check_logs",
        bash_command="echo 'Checking system logs.'",
    )

    transform_task = PythonOperator(
        task_id="transform",
        python_callable=transform,
    )

    report_task = PythonOperator(
        task_id="report",
        python_callable=report,
    )

    end = EmptyOperator(task_id="end")

Key points:

• task_id – the identifier shown in the UI. It must be unique within the DAG.
• python_callable – the function to run for a PythonOperator.
• bash_command – the shell command to run for a BashOperator.
• EmptyOperator – has no work to do. Used here to mark the start and end of the pipeline visually.

Setting Dependencies

Chain the tasks with the >> operator. Use square brackets for tasks that run in parallel.

    start >> [validate_task, check_logs] >> transform_task >> report_task >> end

The pipeline runs in this order:

1. start runs first.
2. validate_task and check_logs run in parallel.
3. transform_task waits for both to finish.
4. report_task runs after transform.
5. end runs last.

Save the file.

Running the DAG

Open the Airflow UI and sign in with username and password airflow. In the search bar, type pipeline to locate the DAG. Click Trigger, choose Single Run.

Expected task output:

• start and end – EmptyOperator tasks. Logs contain only Airflow lifecycle messages.
• validate – "Validating incoming data."
• check_logs – "Checking system logs." from the shell command.
• transform – "Transforming data."
• report – "Sending prepared statement."