10.03 · Lecture
Cascading Failure
Trace how one slow dependency becomes a whole-system outage, and where the independence assumption broke.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Deployment lesson by lesson; 29 of 56 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Cascading failure is a resource exhaustion problem disguised as a dependency problem: latency rises, blocked work accumulates, retries multiply load, and shared pools make unrelated paths fail together. The key skill is finding where components thought to be independent are actually coupled through databases, queues, proxies, caches, limits, or worker capacity.
What this lesson answers
- how does slow dependency cause cascading failure
- why do retries make outages worse
- where do independence assumptions break in distributed systems
Notes
Cascading Failure — Cascading failure exists because distributed systems share hidden dependencies, so one slow or failed component can consume shared resources and trigger outages in otherwise healthy components when independence assumptions are false.
Key Concepts: - A dependency slowdown from to can increase concurrent in-flight requests by Little’s Law: , so at requests/sec, concurrency rises from to .
Common questions
- How can one slow dependency take down healthy endpoints?
- Slow calls occupy workers, connections, memory, proxy slots, or request queues for longer than normal. If unrelated endpoints use the same shared capacity, they start waiting behind the blocked work. The failing dependency has not directly broken those endpoints, but it has consumed the resources they need to serve traffic.
- Why are retries dangerous during a partial outage?
- Retries convert latency and errors into extra load on the same struggling dependency. When several callers or service layers retry independently, the backend sees far more attempts than user requests. Without budgets, jitter, backoff, and circuit breaking, retries can turn a recoverable slowdown into sustained overload.
- What breaks the assumption that services fail independently?
- Services stop being independent when they share infrastructure or quotas: a database, cache, queue, resolver, load balancer, connection pool, thread pool, gateway, proxy, or autoscaling limit. The application graph may look separated, but the runtime capacity graph is often shared, which is where the cascade starts.
Short definition: what is Cascading Failure?