Alerting
Alerting is the production mechanism that turns selected telemetry into an interruption for a human operator when immediate action can reduce user-visible harm. A good alert is not just a detected abnormality; it is a routed, deduplicated, contextual signal with an owner, severity, current value, and runbook.
The problem alerting solves is not lack of data, but lack of attention. Modern systems emit far more metrics, logs, traces, and events than a person can interpret during an incident. If every unusual condition pages someone, teams learn to mistrust the pager. The useful test is blunt: if this wakes the on-call, can they take a specific action now? If not, it belongs in a dashboard, ticket, or report.
Concretely, an alerting system evaluates rules over telemetry, such as error rate, latency, saturation, missing signals, or error-budget burn. When a rule crosses its threshold for the configured window, the alert is labelled, grouped with related alerts, routed to the owning team, and often enriched with links and current values. Good rules page on symptoms users feel, such as failed payments, rather than internal causes unless those causes map directly to a runbook.
The trade-off is sensitivity versus noise. Alert too early and people spend nights investigating harmless spikes, retries that worked, duplicate signals, or capacity trends that could wait. Alert too late and users suffer longer before anyone acts. Thresholds therefore depend on the service, its SLOs, traffic shape, failure modes, and recovery options. There is no universally correct CPU or queue-depth page; it is actionable only if it predicts or explains user impact.
Engineers meet alerting in Prometheus Alertmanager, Datadog, Grafana Cloud, PagerDuty, Opsgenie, cloud monitoring tools, and OpenTelemetry-based observability stacks. In practice, the work is mostly rule hygiene: choose paging versus non-paging alerts, add ownership and severity labels, suppress duplicates, handle expected no-data states, and review incidents for pages where no action was taken. Those alerts should usually be deleted or demoted.
Common questions
- What makes an alert actionable?
- An alert is actionable if the person receiving it can do something specific immediately: roll back, scale, disable a feature, fail over, restart a dependency, or follow a runbook. If the only response is to watch it, investigate later, or create planned work, it should not be a paging alert.
- Should alerts be based on causes or symptoms?
- Usually symptoms. Page on user-visible failure such as elevated errors, missed deadlines, or unacceptable latency. Cause metrics such as CPU, memory, queue depth, or exception count are useful when they reliably point to a known fix, but they often create noise when users are unaffected or the system self-recovers.
- What is alert fatigue?
- Alert fatigue is the loss of trust caused by too many low-value interruptions. Once engineers expect pages to be false alarms, they acknowledge more slowly, ignore context, or miss real incidents. The cure is not better discipline; it is deleting, deduplicating, or demoting alerts that do not require immediate action.
- How are alerting and dashboards different?
- Alerting interrupts someone; dashboards support investigation and planning. A dashboard can show slow capacity growth, unusual traffic, or dependency behaviour without waking anyone. An alert should be reserved for conditions where waiting until normal working time would likely increase user-visible damage or operational risk.