Incident Response
Incident response is the organised process for detecting, coordinating, mitigating, and learning from a production failure. It turns a live service problem into shared facts, user-impact assessment, recovery decisions, and follow-up changes, so the team improves the system rather than relying on memory, heroics, or blame.
Incident response is necessary because production failures are confusing while they are happening and easy to misremember afterwards. Alerts, dashboards, chat messages, deploy logs, and user reports each show only part of the event. Without a structured response and review, teams often reduce the story to the person who made the last change, miss weak safeguards, and leave the same failure mode ready to recur.
In practice, responders establish what is currently true, coordinate mitigation, and record a timeline as the event unfolds. A later review separates user impact from contributing factors: failed requests, lost orders, or error-budget burn are impact; a feature flag default, missing canary, unclear ownership, or weak alert are causes. The review asks what happened, how users were affected, why detection or mitigation took time, and what will change.
The trade-off is time and emotional cost. A review that only reconstructs chat history, hunts for a single root cause, or produces vague advice such as being more careful is not worth much. Blameless also does not mean consequence-free; it means judging decisions in the context of the information, tools, and pressures present at the time. Good outcomes are owned, dated, and verifiable.
Engineers meet incident response through on-call rotations, severity processes, incident channels, status pages, rollback decisions, and post-incident reviews. The artefacts are familiar: timelines, impact summaries, detection gaps, mitigation notes, and action items. A useful action item changes the system in a way someone can check later, such as adding a test, changing an alert, improving a rollout guard, or clarifying ownership.
Common questions
- Is incident response the same as a postmortem?
- No. Incident response includes the live work of noticing, coordinating, diagnosing, communicating, and mitigating a failure. A postmortem or post-incident review is the learning step after service is restored. The review should use evidence from the response, but its purpose is different: convert the event into durable system improvements.
- What makes a post-incident review worth doing?
- It is worth doing when it produces shared facts and at least one concrete, owned, checkable change. A good review explains user impact, detection, mitigation, and contributing factors. A weak one spends the meeting retelling chat logs, assigning personal fault, or writing action items that cannot be tested or closed.
- What does blameless incident response actually mean?
- Blameless means the review studies why the system allowed a reasonable human action to cause or worsen an outage. It does not excuse negligence or forbid accountability. It asks what signals were missing, which guardrails failed, what pressures existed, and how the environment can be changed so the next engineer is less likely to repeat it.
- Should an incident review identify a single root cause?
- Usually that is too simple. Production incidents commonly come from several contributing factors lining up: code behaviour, deployment automation, monitoring thresholds, unclear ownership, weak tests, and slow rollback paths. Naming one root cause can hide the useful fixes. The better question is which conditions made the incident possible or harder to recover from.