Module 07
Orchestration
Build reliable, observable, and maintainable data pipelines using modern workflow orchestration patterns.
7 lessons · 9 videos · 1h 49m- 07.01
Workflow Orchestration Fundamentals
Describe DAGs, tasks, dependencies, schedules, sensors, retries, backfills, SLAs, and idempotency.
- 07.02
Airflow DAG Authoring
Create an Airflow DAG that runs an extract, transform, validate, and publish workflow on a schedule.
- 07.03
Dagster Software-Defined Assets
Model a pipeline in Dagster as assets with dependencies, partitions, metadata, and materialization checks.
- 07.04
Prefect Flows and Deployments
Build and deploy a Prefect flow with task retries, parameters, logging, and a scheduled run.
- 07.05
Pipeline Design Patterns
Apply fan-out, fan-in, dynamic mapping, partitioned execution, staging, and checkpointing patterns to a data pipeline.
- 07.06
Retries, Idempotency, and Backfills
Design a pipeline task that can be retried or backfilled without producing duplicate or inconsistent outputs.
- 07.07
Orchestration for ML Training
Orchestrate a training-data generation, model training, evaluation, and model registration workflow with clear dependencies.