Airflow Reliability: Retries, SLAs, and Alerts

Reliability in Airflow

Airflow provides three built-in reliability features that let a DAG recover from failures without human intervention.

Retries - Airflow automatically re-runs a failed task a configurable number of times before marking it failed. Useful for transient errors like network glitches or temporary rate limits.

SLAs (Service Level Agreements) – A time limit for how long a task should take. If it exceeds the limit, the task is marked as an SLA miss. SLAs do not stop the task, they flag it.

Alerts (failure callbacks) – A function that runs when a task fails after all retries are exhausted. Typically used to send notifications to Slack, PagerDuty, or email.

Setting Up Retries

Create a new DAG file called retries.py in the Airflow DAG folder. Import the required modules:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
import random

The random module simulates failures. timedelta defines retry delays and SLAs.

Simulating a Task Failure

Define a function that fails randomly:

def random_failure():
    if random.random() < 0.7:
        raise Exception("Task failed due to random chance")

random.random() returns a value between 0 and 1. If it is less than 0.7, the task raises an exception. This gives roughly a 70% chance of failure per attempt, enough for retries to actually kick in during testing.

Defining a Failure Callback

A failure callback runs after all retries are exhausted. It receives a context dictionary containing DAG and task metadata.

def failure_alert(context):
    dag_id = context['dag'].dag_id
    task_id = context['task'].task_id
    print(f"Alert: Task {task_id} in DAG {dag_id} has failed")

Callbacks are where real alert logic goes, could be sending Slack messages, PagerDuty pages, or emails. In this example, it just prints.

Configuring Retries, Callbacks, and SLA

default_args applies to every task in the DAG unless overridden at the task level.

with DAG(
    dag_id='retries',
    start_date=datetime(2024, 1, 1),
    schedule=None,
    catchup=False,
    default_args={
        'retries': 3,
        'retry_delay': timedelta(seconds=10),
        'on_failure_callback': failure_alert,
        'sla': timedelta(minutes=5)
    }
) as dag:
    task = PythonOperator(
        task_id='random_failure_task',
        python_callable=random_failure
    )

Key fields:

• retries: 3 – Airflow attempts the task up to three times before marking it failed.
• retry_delay: timedelta(seconds=10) – wait time between attempts.
• on_failure_callback: failure_alert – runs only after retries are exhausted.
• sla: timedelta(minutes=5) – flags any task that runs longer than five minutes as an SLA miss.

Running the DAG

Save the file. Go to the Airflow UI, open the retries DAG, and trigger a run.

Check the task instance under the Task Try section. You will typically see the first attempt fail, followed by a 10-second wait, then a second or third attempt succeed. Opening the failed attempt's logs shows the random number that triggered the failure - for example 0.17.