What is the simplest test for a paging alert?

Ask whether the on-call engineer can take a specific useful action as soon as it fires. If the answer is no, it should not page. Keep it as a dashboard, turn it into planned work, route it as a ticket, or remove it entirely.

Should alerts be based on causes or user symptoms?

Paging alerts should usually track user-visible symptoms, such as failed requests or missed latency targets. Cause-based signals like CPU, queue depth, or disk use are only suitable for paging when they map directly to a known urgent action and predict real user impact.

What information should a good alert include?

A paging alert should tell the responder what service is affected, how severe it is, the current measured value, who owns it, and where the runbook is. Without that context, responders waste time reconstructing the situation instead of reducing impact.

Detecting Failure

09.04 · Concept

Alerting

Apply one test to every alert - is it immediately actionable - and delete the ones that fail it.

No video curated for this lesson yet

This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Deployment lesson by lesson; 29 of 56 have their video so far.

The written notes below cover this idea in full - you lose nothing by reading instead of watching.

Actionable alerting means paging only when a human can take useful action immediately to reduce user impact. Alerts should describe symptoms users feel, carry enough context to act, and avoid duplicating the same failure. Everything else belongs in dashboards, tickets, reports, or deletion.

What this lesson answers

how to decide if an alert should page
what makes an alert immediately actionable
how to reduce noisy production alerts

Notes

Alerting — Alerting exists to interrupt a human only when immediate action can prevent or reduce user-visible failure; without this filter, teams drown in noisy pages, ignore alarms, and miss real incidents.

Key Concepts: - An alert should pass the test: “If this fires at 03:00, can the on-call take a specific action now?”; if the answer is no, delete it or convert it to a dashboard/weekly report. - Page on symptoms, not causes: “HTTP 5xx rate $> 5%$ for 5 minutes on checkout” is actionable; “CPU $> 80%$ ” is not unless it directly maps to a runbook action.

References

Common questions

What is the simplest test for a paging alert?: Ask whether the on-call engineer can take a specific useful action as soon as it fires. If the answer is no, it should not page. Keep it as a dashboard, turn it into planned work, route it as a ticket, or remove it entirely.
Should alerts be based on causes or user symptoms?: Paging alerts should usually track user-visible symptoms, such as failed requests or missed latency targets. Cause-based signals like CPU, queue depth, or disk use are only suitable for paging when they map directly to a known urgent action and predict real user impact.
What information should a good alert include?: A paging alert should tell the responder what service is affected, how severe it is, the current measured value, who owns it, and where the runbook is. Without that context, responders waste time reconstructing the situation instead of reducing impact.

Short definition: what is Alerting?

09.03 · Detecting FailureSLIs, SLOs, and Error Budgets 09.05 · Detecting FailureIncident Response