AIRFLOW

Airflow Reliability: Retries, SLAs, and Alerts

Tasks fail in production. Airflow has three built-in tools to handle this: retries, SLAs, and failure callbacks. We'll build a DAG that fails on purpose so you can see each feature react and understand how they work together.

Setting Up Retries

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
import random

The Task Function

def random_failure():
    if random.random() < 0.7:
        raise Exception("Random failure occurred")

The Callback Function

def failure_alert(context):
    dag_id = context['dag']['dag_id']
    task_id = context['task']['task_id']
    print(f"Alert: Task {task_id} in DAG {dag_id} failed after all retries")

The DAG Definition

The DAG block sets the configuration, including retry behavior:

with DAG(
    dag_id='retries',
    start_date=datetime(2023, 1, 1),
    schedule=None,
    catchup=False,
    retries=3,
    retry_delay=timedelta(seconds=10),
    sla=timedelta(minutes=5),
    on_failure_callback=failure_alert
) as dag:
    task = PythonOperator(
        task_id='random_failure_task',
        python_callable=random_failure
    )

Running the DAG

Save the file and go to the Airflow UI. Navigate to the DAG section, find the retries DAG, trigger it with a single run. In the Task Try section, you can see the DAG failed on the first try, waited 10 seconds, then succeeded on the second try. The logs of the first try show the random number that caused the failure.

What's next

Now go and try this out in a live environment — boot a fresh cluster and play with the manifests above.

Start Airflow
Spec 4 CPU / 8 GiB ·Disk 25 GiB
Sign in to launch this environment
Required 1 VM · 4 CPU · 8 GB · 25 GiB disk
Available 1 VM · 1 CPU · 2 GB · 10 GiB disk
Sign in

Suggested tutorials

Keep the momentum going, pick a related topic.

AIRFLOW

Airflow Executors: Local, Celery, and Kubernetes

The scheduler decides what to run, but the executor decides where and how. We'll walk through the three main executor types - Local, Celery, and Kubernetes and understand when to reach to each.

Open tutorial
AIRFLOW

Airflow: Branching and Trigger Rules

Not every pipeline runs the same path every time. Sometimes the next step depends on a condition and Airflow handles this with branching. We'll build a DAG that picks between two paths using a branch operator and see how trigger rules control what happens when paths merge back together.

Open tutorial