09.05 · Concept
Incident Response
Run a post-incident review that produces learning rather than blame, and say what makes one worth the hour it costs.
A post-incident review should turn an outage into specific system changes: a factual timeline, clear user impact, contributing factors, and owned follow-up work. The point is not to find someone to blame, but to explain why the system behaved that way and what will be different next time.
What this lesson answers
- how to run a blameless incident review
- what makes a post incident review useful
- how to write good incident action items
Notes
Post-Incident Review — A post-incident review exists to convert a real failure into concrete system learning and follow-up work; without it, teams repeat the same outage pattern because the only artifact is memory, blame, or a vague apology.
Key Concepts: - A useful review answers 4 factual questions: what happened, how users were affected, why detection or mitigation took minutes, and what will change before the next similar event. - Use a timeline with timestamps such as alert fired, on-call acknowledged, rollback started, error rate…
Common questions
- What should a post-incident review produce?
- It should produce a shared factual account of what happened, how users were affected, how detection and mitigation unfolded, and what will change. The most important output is not the write-up itself, but specific follow-up work with an owner, a deadline, and a way to verify that the risk has actually been reduced.
- Does blameless mean nobody is accountable?
- No. Blameless means judging decisions in the context people had at the time, rather than turning the review into a performance discussion. Accountability still exists through clear ownership of fixes, better safeguards, improved alerts, and completed remediation. The focus moves from personal fault to system conditions that made the failure possible.
- What makes an incident action item good?
- A good action item is concrete, testable, and assigned. “Improve monitoring” is weak because nobody can prove it is done or sufficient. A stronger item names the signal to alert on, the condition that should trigger it, who will implement it, and when completion will be checked.
Short definition: what is Incident Response?