07.06 · Concept
Retries, Idempotency, and Backfills
Design a pipeline task that can be retried or backfilled without producing duplicate or inconsistent outputs.
Retry-safe pipeline tasks produce the same stored result when rerun with the same logical inputs. The core design is deterministic output ownership, usually by partition or time window, with staged writes, validation, and replacement or keyed merge instead of blind appends. Backfills then become controlled reruns, not data repair emergencies.
What this lesson answers
- how to make pipeline retries idempotent
- how to avoid duplicates during data backfills
- safe write patterns for orchestrated data tasks
Notes
Reliable orchestration assumes that tasks will fail, be retried, and sometimes be rerun for historical dates. A pipeline task is safe to retry when running it more than once with the same inputs produces the same final state as running it once. This is the idea of idempotency. In practice, idempotent tasks avoid blindly appending duplicate records, avoid generating new random identifiers on every run, and avoid depending on “current time” unless that time is explicitly part of the task’s input.
Common questions
- What makes a pipeline task idempotent?
- A pipeline task is idempotent when repeating it with the same intended inputs leaves storage in the same final state. That means no duplicate appends, no fresh random identifiers on each attempt, and no hidden dependency on wall-clock time unless that time is an explicit input.
- Why are blind inserts dangerous in retried jobs?
- Blind inserts assume a task either completed fully or did not run at all. In reality, a worker can crash or time out after writing partial output. A retry then writes the same records again. Staging results and replacing a known partition, or merging by a stable key, avoids that failure mode.
- How should backfills differ from normal scheduled runs?
- They should not differ in write safety. A backfill is a rerun for an older logical window, so the task should read data for that requested interval and write only the output slice it owns. Code, input paths, and transformation rules should be stable enough to reproduce or intentionally repair history.