05.07 · Concept
Backpressure, Replay, and Failure Recovery
Diagnose lag, backpressure, poison messages, and replay requirements in a streaming architecture.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Backpressure is a symptom of a streaming pipeline producing work faster than some downstream part can complete it. Diagnosis means finding whether lag comes from capacity, bad data, dependency latency, unsafe retries, or replay design, then choosing recovery tactics that preserve correctness rather than merely hiding delay.
What this lesson answers
- how to diagnose consumer lag in streaming pipelines
- what makes replay safe after stream processing failure
- how should poison messages be handled in Kafka
Notes
Backpressure happens when one part of a streaming system produces or forwards data faster than a downstream component can process it. In practice, this shows up as growing consumer lag, fuller queues, increasing end-to-end latency, rising memory or disk usage, throttling, or timeout errors. A working engineer should diagnose where the bottleneck is: the source may be producing too quickly, the broker may be under-provisioned, consumers may be too slow, or a sink such as a database, warehouse, or external API may be unable to keep up.
Common questions
- How do I tell whether streaming lag is temporary or dangerous?
- Look at the direction of the backlog, not just its existence. A temporary spike should drain once input falls or capacity catches up. Dangerous lag keeps growing, often alongside higher queue depth, sink latency, retries, timeouts, or resource pressure. Break the view down by partition and dependency to find the constraining stage.
- What has to be true before replaying old stream events?
- Replay needs durable source data, a known recovery point, compatible event formats, and processing that gives the same intended result when run again. The sink also matters: duplicate writes must be handled through idempotency, deduplication, upserts, or transactional behaviour, otherwise replay can corrupt downstream state.
- What is a poison message in stream processing?
- A poison message is a record that fails every time it is processed because the data or application logic is incompatible with it. Treat it differently from a transient outage: capture the record and context in quarantine or a dead-letter path, then continue processing rather than blocking the stream indefinitely.