05.04 · Concept
Delivery, Retries & Provider Failures
Handle a throttling provider without amplifying the spike, and say why a naive retry makes an outage worse.
Naive retries can turn provider throttling into a self-inflicted outage by multiplying traffic when capacity is already constrained. Durable queues, capped worker concurrency, backoff with jitter, retry budgets, circuit breakers and dead-letter queues let notification delivery slow down safely without losing work or starving new messages.
What this lesson answers
- how to handle notification provider throttling safely
- why naive retries make outages worse
- how jitter prevents retry traffic waves
Notes
A notification provider is another service with limits, outages, and changing latency. When it throttles you, the safe response is not to immediately try harder; it is to reduce pressure, preserve work durably, and retry later at a controlled rate. Think of delivery as a queue plus a worker pool, not as a direct function call to the provider.
Retries are useful only when they are bounded, delayed, and coordinated. Use exponential backoff, jitter, retry budgets, rate limits, circuit breakers, and dead-letter queues.
Common questions
- Why are immediate retries dangerous during provider throttling?
- Immediate retries add more requests at the exact moment the provider is asking you to send less. Each failed delivery can create additional attempts, so a modest spike becomes amplified by your own system. The safer pattern is to persist the work, reduce send rate, and retry later under explicit limits.
- What should a notification delivery system do when a provider slows down?
- It should treat delivery as queued work processed by controlled workers, not as a direct provider call. Cap concurrency per provider, apply rate limits, back off on throttling, add jitter to retry timing, and open a circuit when errors rise. Messages that keep failing should move aside rather than block the pipeline.
- Why does jitter matter in retry logic?
- Without jitter, many workers can retry at the same interval after a shared failure, creating synchronised bursts against the provider. Jitter spreads retries over time, which lowers peak pressure and gives the provider room to recover. It is a simple guard against turning coordinated failure into coordinated overload.
Short definition: what is Delivery, Retries & Provider Failures?