Correlated Failure
Correlated failure is a reliability failure mode where supposedly separate components stop working together because they share an underlying dependency, limit, operator action, or failure domain. It is the reason redundancy often disappoints: the backup was not truly independent of the primary in the way the failure actually occurred.
The problem is that system diagrams often show independence at the wrong level. Two services, replicas, regions, or databases may have separate names and deployments, yet still depend on the same node, DNS provider, identity system, route table, key manager, deployment pipeline, or human process. When that shared thing fails, the components fail together. The common misunderstanding is to count copies, rather than checking whether those copies can be broken by the same event.
Mechanically, correlated failure is found by tracing dependency paths and looking for convergence. If component X and component Y both need Z to serve traffic, Z is a common-mode failure source. The dependency may be runtime, such as a database primary or cache used in fail-closed mode. It may be control-plane, such as DNS or service discovery. It may also be operational, such as one release workflow pushing a bad configuration to many services at once.
The trade-off is that removing correlation usually costs complexity, money, or operational discipline. Spreading replicas across nodes, zones, providers, or identity systems can reduce shared failure domains, but it creates more networking paths, replication concerns, configuration drift, and test cases. The honest answer is not “make everything independent”. It depends on which failures matter, how long you can tolerate them, and whether the extra independence is itself reliable.
Engineers meet correlated failure in production reviews, incident analysis, architecture diagrams, and deployment policy. In Kubernetes, anti-affinity and topology spreading exist to avoid placing redundant pods on the same failure domain. In cloud systems, multi-zone databases and node groups reduce some infrastructure correlation, but not shared application dependencies. Tracing, service maps, and dependency graphs are practical tools for spotting one database, egress gateway, DNS zone, or pipeline on many critical paths.
Common questions
- How is correlated failure different from a single point of failure?
- A single point of failure is an obvious component whose loss breaks the system. Correlated failure is often subtler: you may have multiple components, but they share something that makes them fail together. The system looks redundant until the common dependency, limit, or operational path is the thing that breaks.
- Does running in multiple zones or regions eliminate correlated failure?
- No. It reduces some infrastructure-level correlation, such as shared power, cooling, or local networking, but other dependencies can still span all locations. A global DNS configuration, central identity provider, shared deployment pipeline, account-wide quota, or common database design can still make otherwise distributed components unavailable at the same time.
- How do you find correlated failures before an incident?
- Draw the request path and control-plane dependencies for each critical operation, then compare paths across supposedly independent components. Look for shared databases, caches, DNS zones, certificates, keys, quotas, routes, CI workflows, operators, and third-party services. Failure injection and game days help validate whether the diagram matches the real system behaviour.