10.05 · Concept
Correlated Failure
Find the shared dependency that makes two 'independent' components fail together.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Deployment lesson by lesson; 29 of 56 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Correlated failure is the reason redundant-looking systems still go down together: the replicas, services, regions, or pipelines share something that fails once and affects them all. Reliability work means tracing those shared dependencies, such as nodes, databases, DNS, identity providers, limits, routes, keys, and release machinery, before treating components as independent.
What this lesson answers
- why do redundant services fail at the same time
- how to find shared dependencies in production systems
- what causes correlated failure in Kubernetes deployments
Notes
Correlated Failure — Correlated failure exists because components that look independent often share a hidden dependency, and without identifying that dependency, redundancy fails when both components break at the same time.
Key Concepts: - Two replicas in different pods are not independent if both run on the same Kubernetes node; one node reboot can make availability drop from replicas to . - If service A and service B both require the same Postgres primary, the Postgres outage is a shared dependency even if A and B have separate application deployments.
References
Common questions
- What is correlated failure in distributed systems?
- Correlated failure happens when components that appear separate depend on the same underlying resource or control path. If that shared dependency breaks, the components fail together. Common examples include replicas on one node, services using one database primary, multiple APIs behind one DNS zone, or separate deployments using the same release pipeline.
- Why is correlated failure dangerous for redundancy?
- Redundancy only improves availability when failures are independent. If both copies share a failure domain, the extra copy may not help during the incident that matters. Two pods, services, regions, or databases can look redundant while still sharing a node, route table, identity provider, encryption key, account limit, or deployment workflow.
- How do you reduce correlated failure in production?
- Start by mapping dependencies and looking for shared paths between components that are meant to back each other up. Then spread replicas across meaningful failure domains, separate critical control planes where appropriate, avoid fail-closed reliance on shared caches, and review release, DNS, identity, network, and quota dependencies as part of resilience design.
Short definition: what is Correlated Failure?