10.02 · Lecture
Retry Storms
Explain why backoff alone does not stop a herd and why jitter is the part that decorrelates it.
Retry storms happen when many clients retry a struggling dependency in lockstep, adding burst load exactly when the service is least able to absorb it. Exponential backoff spaces attempts out per client, but jitter is what spreads clients apart and stops recovery from becoming another overload event.
What this lesson answers
- why exponential backoff still causes retry storms
- how jitter prevents synchronised client retries
- where should retries live in a call chain
Notes
Retry Storms — Retry storms exist because many clients retry the same failing dependency at the same time, and without decorrelation they can multiply load until the recovering service is immediately overwhelmed again.
Key Concepts: - If clients each retry once after a timeout, the dependency receives up to extra requests at the same retry boundary. - Exponential backoff uses delays like , but if every client starts at the same time, retries remain synchronized at those exact times. - Jitter decorrelates retries by randomizing delay, e.g.
References
Common questions
- Why is exponential backoff not enough to prevent a retry storm?
- Backoff changes when each client retries, but if many clients started failing at the same moment, they can still share the same retry schedule. The result is quieter periods followed by coordinated bursts. The dependency may recover briefly, then be hit again by a herd of retries arriving together.
- What does jitter add to a retry policy?
- Jitter adds randomness to the sleep duration before a retry. Instead of every caller waiting for the same calculated delay, each caller waits for a slightly different interval. That spreads retry traffic across time, reducing spikes and giving a recovering service a better chance to drain work.
- Can retries make an outage worse?
- Yes. Retrying a saturated service increases the amount of work it must do, especially when retries happen at several layers of the same request path. Without limits, budgets and jitter, a short failure can turn into sustained overload because the system keeps manufacturing extra attempts.
Short definition: what is Retry Storms?