Delivery, Retries & Provider Failures
Delivery, retries and provider failures are the reliability mechanics around sending work to an external service that may be slow, throttled or unavailable. Instead of treating delivery as a direct call, robust systems persist the work, limit how fast they send it, retry selectively, and shed or park failures without overwhelming the provider.
The problem is that notification providers, payment gateways and similar dependencies are not infinitely elastic. They have quotas, latency shifts and outages. If callers immediately retry every failure, one user action can turn into many provider calls. During throttling, that extra traffic lands exactly when the provider has the least spare capacity, so a local recovery mechanism becomes retry amplification and can prolong the incident.
A safer design treats delivery as a queue drained by workers. The application records the message durably, then workers send to the provider under explicit concurrency and rate limits. On throttling or transient errors, the item is scheduled for a later attempt using backoff and jitter, so retries are spread out rather than synchronised. Circuit breakers can pause a provider path when failures rise, and dead-letter queues separate messages that keep failing.
The trade-off is that delivery becomes asynchronous and operationally visible. Users may see delayed notifications, engineers must reason about duplicate sends, ordering, idempotency and message expiry, and queues need monitoring. Backoff settings are not universally correct: they depend on provider limits, business urgency, error type and downstream recovery behaviour. Being too aggressive amplifies failure; being too conservative can leave useful work waiting too long.
Engineers meet this in email, SMS, push, webhook and integration pipelines. Practical controls include per-provider worker pools, capped retries, retry budgets, jittered schedules, rate limiters, circuit breakers and dead-letter inspection. A common misunderstanding is that adding retries automatically improves reliability. Retries help only when they are bounded, delayed and coordinated with the rest of the fleet.
Common questions
- Why do naive retries make outages worse?
- They multiply traffic during the worst moment. If each failed request is retried immediately, the provider receives the original spike plus the retry traffic. Across a fleet, those retries can align and arrive in waves. The system then spends capacity reattempting old failures while new work queues or fails.
- What should happen when a provider starts throttling?
- The sender should slow down, not push harder. Persist new work in a queue, cap provider concurrency, apply backoff with jitter to failed sends, and respect any provider retry guidance. If errors continue, a circuit breaker can pause sends briefly while the queue preserves work for later delivery.
- What is jitter in retry logic?
- Jitter is deliberate randomness added to retry delays. Without it, many workers that fail at the same time may retry at the same later time, creating another surge. Jitter spreads those attempts over a wider window, reducing synchronised pressure on the provider and making recovery smoother.
- When should a message go to a dead-letter queue?
- A message belongs in a dead-letter queue when repeated attempts fail, or when the error looks permanent or malformed rather than transient. Moving it aside prevents one bad item from blocking the delivery pipeline. Engineers can inspect, repair, replay or discard it without starving other messages.