Airflow Scheduling: Intervals and Catchup

DAG Scheduling

Airflow provides two ways to define a schedule for a DAG. The first is preset shortcuts such as hourly, daily, weekly, or monthly, which are useful for simple cases. The second is cron expressions, a standard format from Linux with five asterisk fields.

Catch Up and Backfill

• catchup=True: Airflow runs every missed interval between the start date and today.
• catchup=False: Airflow skips missed intervals and only runs going forward from today.

Creating a Scheduled DAG

Create a new file called schedule.py in the Airflow DAG folder. Open it and import the necessary modules:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

Define the main function for the DAG:

def process_data(**context):
    data_interval_start = context['data_interval_start']
    data_interval_end = context['data_interval_end']
    print(f"Data interval start: {data_interval_start}")
    print(f"Data interval end: {data_interval_end}")

Define the DAG block:

with DAG(
    dag_id='schedule_dag',
    start_date=datetime(2024, 1, 1),
    schedule='0 9 * * *',
    catchup=False,
) as dag:

Create one task that wraps the process_data function:

    task = PythonOperator(
        task_id='process',
        python_callable=process_data,
    )

Save the file.

Running the DAG

Go to the environment panel and use the URL to enter the environment. Trigger the DAG. In the task section, the process works. In the details section, you will see the start and end dates, confirming that the context executed correctly.