Data Engineering Tutorials
Step-by-step walkthroughs with live environments — read, run, learn.
All Tutorials
AIRFLOWWhat Airflow is, why orchestration, install & first look
Apache Airflow is an open-source platform for orchestrating data pipelines. Learn what it is, why orchestration matters, and familiarize yourself with the UI on a real VM.
Open tutorial AIRFLOWWrite & run your first DAG on Airflow
You know what Airflow is and its main concepts. Now make it do something. Write a real DAG with tasks, drop it into Airflow, and watch it run to completion.
Open tutorial AIRFLOWTasks, operators & dependencies in Airflow
Go beyond a single DAG. Learn how operators work, how to set dependencies between tasks, and build a multi-task pipeline with real task ordering.
Open tutorial AIRFLOWScheduling, intervals & catchup in Airflow
Learn how to schedule DAGs, understand data intervals, and control catchup and backfill behavior in Airflow.
Open tutorial AIRFLOWXComs and passing data between tasks in Airflow
Tasks in a DAG run in isolation and XComs let them share data. Learn how to pass values between real tasks and inspect them in the UI.
Open tutorial AIRFLOWSensors & external triggers in Airflow
Sometimes a pipeline has to wait for a file to arrive, an API to respond, another DAG to finish. Sensors are the way how Airflow waits.
Open tutorial AIRFLOWThe TaskFlow API in Airflow
TaskFlow lets you write DAGs as plain Python functions. Rewrite a real DAG with TaskFlow and see the difference.
Open tutorial AIRFLOWConnections, hooks and providers in Airflow
DAGs need to talk to real systems such as databases, APIs, S3. Connections store the credentials, hooks let you use them in Python, providers ship pre-built integrations for everything common. Connect a DAG to a real Postgres database on the VM.
Open tutorial AIRFLOWDynamic DAGs and task mapping in Airflow
Sometimes you don't know how many tasks you'll need until runtime. Task mapping lets Airflow generate those tasks dynamically. You'll fan out real tasks over a list and watch Airflow spawn them.
Open tutorial AIRFLOWBranching and conditional flows in Airflow
Pipelines don't always run in a straight line and sometimes you need to take one path or another based on a condition. Branching is how Airflow handles that. You'll build a DAG that picks a path at runtime.
Open tutorial AIRFLOWExecutors in Airflow: Local, Celery, and Kubernetes
The executor decides where and how your tasks actually run. Local executor on one machine, Celery on a worker fleet, Kubernetes spinning up a pod per task. Learn the differences and when to use each
Open tutorial AIRFLOWRetries, SLAs, and failure notifications in Airflow
Real pipelines fail, APIs time out, networks blip, databases get slow. Airflow has built-in tools to handle this. Configure all three on a real DAG.
Open tutorial AIRFLOWMonitoring and logging in Airflow
Knowing how to find the right log fast is what separates 5-minute fixes from 5-hour debugging sessions. Learn to read task logs, navigate the grid, graph views and find the metrics that matter.
Open tutorial
SPARKWhat Spark is and why distributed compute matters
Apache Spark is the engine that powers most modern big-data pipelines. Learn what it is, why distributed processing exists in the first place, and get hands-on with a real Spark cluster on the VM.
Open tutorial SPARKYour first Spark analytics pipeline
Time to actually use Spark for what it's built for. We'll take a raw movie ratings dataset, clean it, aggregate it, and produce a top-10 movies list.
Open tutorial SPARKRDDs, DataFrames, and how Spark actually executes your code
Every Spark operation you write turns into a plan, gets optimized, and runs across the cluster. Learn the difference between RDDs and DataFrames, why one is preferred, and how Spark actually executes what you write.
Open tutorial SPARKThe DataFrame API and Spark SQL
DataFrames aren't just a way to hold data, since they come with two full APIs for querying it. Python method chains and actual SQL strings. We will learn both, understand when to use each, and query real data.
Open tutorial SPARKPartitions and shuffles in Spark
Every Spark job is fast or slow because of two things you can't see in your code, that are partitions and shuffles. Learn what they are, why some operations trigger massive data movement, and how to inspect what Spark is actually doing on the cluster.
Open tutorial