Airflow XComs: Sharing Data Between Tasks

XComs: Cross-Communication

XCom stands for Cross-Communication. It is a small storage layer inside Airflow's metadata where tasks can leave values for other tasks to pick up. When a function used by a Python operator returns a value, Airflow automatically pushes it to XCom.

Creating the DAG

Create a new file called xcom.py in the DAGs folder.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

Create an extract function that returns a value:

def extract():
    record_count = 42
    print(f"Extracted {record_count} records")
    return record_count

Create a transform function that pulls the value from extract:

def transform(ti):
    count = ti.xcom_pull(task_ids='extract')
    print(f"Transforming {count} records")
    return count * 2

Define the DAG and tasks:

with DAG(
    'xcom_dag',
    start_date=datetime(2023, 1, 1),
    schedule='@daily',
    catchup=False
) as dag:

    extract_task = PythonOperator(
        task_id='extract',
        python_callable=extract
    )

    transform_task = PythonOperator(
        task_id='transform',
        python_callable=transform
    )

    extract_task >> transform_task

Save the file.

Running the DAG

In the Airflow UI, go to the DAGs section. Search for xcom_dag. Click on it and trigger a single run.

For the extract task, go to the XCom section and see the value is 42. For the transform task, the XCom value is 84, confirming that the data was shared between tasks.