Is cascading failure just one service being down?

No. A single-service outage becomes a cascading failure when its effects propagate through shared resources or caller behaviour. The downstream service may only be slow, not dead, but callers block, queues grow, retries increase demand, and unrelated endpoints fail because they use the same pools, limits, or infrastructure.

Why do retries often make a cascade worse?

Retries are useful when failures are rare and short-lived, but they add work exactly when the dependency is least able to handle it. If several layers retry independently, one user request can become many backend attempts. Without deadlines, retry budgets, backoff, and jitter, retries convert latency into amplified load.

How do bulkheads and circuit breakers differ?

Bulkheads prevent one path from consuming all shared capacity by partitioning resources such as worker pools or connections. Circuit breakers stop sending work to a dependency after recent failures or timeouts cross a threshold. Bulkheads limit how far damage spreads; circuit breakers reduce pressure on the failing dependency and fail callers quickly.

How do you find the broken independence assumption after an incident?

Trace the resource, not just the code path. Ask which endpoints, jobs, and services shared the same database, cache, resolver, proxy, queue, thread pool, connection pool, quota, or deployment mechanism. Then compare deadlines, timeouts, retries, and queueing at each hop to see where waiting work consumed capacity needed by supposedly independent traffic.

Cascading Failure

Cascading failure is a distributed-system failure mode where one slow or broken component causes other, initially healthy components to fail by consuming shared capacity. It happens when callers, dependencies, retries, queues, pools, limits, or infrastructure are coupled more tightly than the system design assumes.

The problem starts with a false independence assumption. Teams often reason as if checkout, login, reporting, background jobs, and admin tools can fail separately because their code paths are different. In production, they may share the same database, cache, load balancer, DNS resolver, queue, thread pool, connection pool, NAT gateway, or autoscaling quota. A reporting query that merely becomes slow can therefore exhaust shared database connections and make unrelated user flows appear broken.

Mechanically, latency turns into load. By Little’s Law, more time spent waiting means more in-flight work at the same arrival rate. If requests that normally return quickly begin blocking on a downstream dependency, workers, sockets, memory, and queue slots stay occupied. Retries multiply the effect: each caller that times out may send more attempts, and each service layer may retry again. The original dependency is now receiving extra traffic while the callers are losing the capacity needed to serve anything else.

The usual protections trade availability, correctness, and simplicity against blast radius. Timeouts free resources, but a timeout longer than the caller’s own deadline can leave abandoned work running. Circuit breakers fail fast, but they also deliberately reject requests that might have succeeded. Bulkheads isolate capacity, but fixed pools can strand spare capacity in one area while another is overloaded. Retries help with transient faults, but without budgets, jitter, and deadlines they are a failure amplifier.

Engineers meet cascading failure in incident reviews, load tests, deployment rollouts, and platform defaults. In Kubernetes, probes can help remove unhealthy pods from service discovery, but badly chosen liveness checks can restart pods during a dependency latency spike and add cold starts. In service meshes and proxies such as Envoy, circuit breaking limits connections, pending requests, and retries per upstream. In serverless systems, high concurrency can still collapse a shared database connection limit.

Go deeper

DeploymentCascading FailureTrace how one slow dependency becomes a whole-system outage, and where the independence assumption broke.Open the lesson

Common questions

Is cascading failure just one service being down?: No. A single-service outage becomes a cascading failure when its effects propagate through shared resources or caller behaviour. The downstream service may only be slow, not dead, but callers block, queues grow, retries increase demand, and unrelated endpoints fail because they use the same pools, limits, or infrastructure.
Why do retries often make a cascade worse?: Retries are useful when failures are rare and short-lived, but they add work exactly when the dependency is least able to handle it. If several layers retry independently, one user request can become many backend attempts. Without deadlines, retry budgets, backoff, and jitter, retries convert latency into amplified load.
How do bulkheads and circuit breakers differ?: Bulkheads prevent one path from consuming all shared capacity by partitioning resources such as worker pools or connections. Circuit breakers stop sending work to a dependency after recent failures or timeouts cross a threshold. Bulkheads limit how far damage spreads; circuit breakers reduce pressure on the failing dependency and fail callers quickly.
How do you find the broken independence assumption after an incident?: Trace the resource, not just the code path. Ask which endpoints, jobs, and services shared the same database, cache, resolver, proxy, queue, thread pool, connection pool, quota, or deployment mechanism. Then compare deadlines, timeouts, retries, and queueing at each hop to see where waiting work consumed capacity needed by supposedly independent traffic.

Every term