Airflow: Write and Run Your First DAG

Writing the DAG file

Open VS Code and navigate to the Airflow directory on the left side. Go into the dag folder, right-click, and select New File. Name it first_dag.py and press Enter.

Start by importing the necessary tools from the Airflow library:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

Defining the task functions

Define three Python functions that represent the extract, transform, load pattern:

def extract():
    print("Data fetched from the API.")

def transform():
    print("Data cleaned and transformed.")

def load():
    print("Data loaded into the database.")

Creating the DAG

Use a with block to define the DAG:

with DAG(
    dag_id="first_dag",
    start_date=datetime(2024, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:

Creating tasks and defining order

Inside the DAG block, create three tasks using PythonOperator:

    t1 = PythonOperator(
        task_id="extract",
        python_callable=extract,
    )

    t2 = PythonOperator(
        task_id="transform",
        python_callable=transform,
    )

    t3 = PythonOperator(
        task_id="load",
        python_callable=load,
    )

    t1 >> t2 >> t3

Save the file and close VS Code.

Running the DAG in the UI

Go back to the tutorial page. In the environment panel, click the URL to the Airflow UI. Sign in with username airflow and password airflow.

On the home page, use the search bar to find your DAG by typing first_dag. Click the blue button to trigger the DAG, select "Trigger DAG" for a single run, and press Trigger again.

Check the logs for each task:

• The extract task log shows "Data fetched from the API."
• The transform task log shows "Data cleaned and transformed."
• The load task log shows "Data loaded into the database."