07.01 · Concept
Workflow Orchestration Fundamentals
Describe DAGs, tasks, dependencies, schedules, sensors, retries, backfills, SLAs, and idempotency.
Workflow orchestration coordinates data jobs as ordered, observable, repeatable workflows. A DAG captures task order without cycles, while schedules, sensors, retries, SLAs and backfills control when work runs, how it waits, how it recovers, and how historical data is repaired safely through idempotent task design.
What this lesson answers
- what is a DAG in workflow orchestration
- how do retries and sensors work in pipelines
- why idempotency matters for data backfills
Notes
Workflow orchestration is the practice of coordinating data work so it runs in the right order, at the right time, with clear visibility into success and failure. Most orchestrators model a workflow as a DAG, or directed acyclic graph: a set of tasks connected by dependencies, with no cycles. Each task represents a unit of work such as extracting data, running a SQL transformation, validating a dataset, or publishing results.
Common questions
- What does a workflow orchestrator actually do?
- A workflow orchestrator runs data work in a controlled order, tracks each task, applies schedules, handles dependencies, and records success or failure. Instead of relying on scattered cron jobs or manual steps, it gives teams one model for when work should start, what it depends on, and what happened during execution.
- Why are DAGs used for data pipelines?
- DAGs make dependencies explicit while preventing circular execution paths. Each node is a task, and each edge says what must complete first. That structure lets independent work run in parallel, while steps that consume upstream outputs wait until those outputs are ready.
- What makes a task idempotent in orchestration?
- An idempotent task can run again for the same input without damaging the result. In data pipelines, that usually means deterministic logic, safe writes by partition, upserts, or replacing a target slice before writing it again. This is essential for retries and historical backfills.
Short definition: what is Workflow Orchestration Fundamentals?