What does it mean to assign a probability to an event?

It means that, after defining the possible outcomes and the events of interest, the model gives a number to one such event. For example, the event might be “more than 5 errors in 1 minute”. The probability is not a property of the sentence alone; it is relative to the chosen outcome definition and sampling process.

Why are tail probabilities harder to estimate than averages?

Averages use information from the whole sample, while tails depend on relatively few extreme observations. If the event is rare, most samples contain little direct evidence about it. More data helps, but so does modelling the right process: correlated failures, changing traffic mixes, and retries can make a simple independent-request model misleading.

Is p99 latency the same thing as a tail probability?

They are closely related but not identical statements. A tail probability asks how likely latency is to exceed some chosen threshold. A p99 value asks for a latency threshold that the model or sample puts near the upper end of the distribution. One fixes the threshold; the other estimates the threshold from a probability level.

What is the most common modelling mistake?

The common mistake is writing a probability for “failure” before saying what a single outcome is. A request, a user session, a probe sequence, and a time window can all define different outcomes. They produce different events and different probabilities, even if the same production incident appears in the underlying logs.

Sampling and Tails

Sampling and tails are the discipline of describing reliability data with an explicit probability model, then asking how often extreme events occur. A sample is the observed draws from a process; a tail is a low-frequency region such as unusually slow latency, many failures, or repeated probe errors.

Production systems fail in uneven ways: the median request may look healthy while a small fraction time out, retry together, or overload a dependency. To reason about that, you need more than phrases like “rare” or “usually fine”. You must say what one outcome is, which sets of outcomes count as events, and what probability law is assumed to generate them.

Concretely, define a sample space of possible outcomes, a collection of events you are allowed to measure, and a probability function over those events. For requests, an outcome might record success or failure for each request in a window. “At least 10 failures” is then a subset of outcomes. A tail probability asks for the chance of landing in an extreme subset, such as latency above a threshold.

The trade-off is that the model is only as honest as its assumptions and data. Independence is often convenient, but retries, shared hosts, queues, and regional incidents make observations correlated. Tail estimates are also data-hungry: small samples say little about rare behaviour. Common misunderstanding: an observed sample is not the event itself; it is evidence used to estimate probabilities of events under a model.

Engineers meet this in dashboards, alerts, tracing, and rollout analysis. Readiness probes turn into success or failure observations; latency dashboards report upper percentiles; trace sampling includes some requests according to a configured law; error alarms ask whether a count in a window crossed a threshold. The practical question is always: what process is being sampled, and which tail event is being estimated?

Go deeper

DeploymentSampling and TailsState a probabilistic model properly: the sample space, the probability law, and what it means to assign a probability to an event.Open the lesson

Common questions

What does it mean to assign a probability to an event?: It means that, after defining the possible outcomes and the events of interest, the model gives a number to one such event. For example, the event might be “more than 5 errors in 1 minute”. The probability is not a property of the sentence alone; it is relative to the chosen outcome definition and sampling process.
Why are tail probabilities harder to estimate than averages?: Averages use information from the whole sample, while tails depend on relatively few extreme observations. If the event is rare, most samples contain little direct evidence about it. More data helps, but so does modelling the right process: correlated failures, changing traffic mixes, and retries can make a simple independent-request model misleading.
Is p99 latency the same thing as a tail probability?: They are closely related but not identical statements. A tail probability asks how likely latency is to exceed some chosen threshold. A p99 value asks for a latency threshold that the model or sample puts near the upper end of the distribution. One fixes the threshold; the other estimates the threshold from a probability level.
What is the most common modelling mistake?: The common mistake is writing a probability for “failure” before saying what a single outcome is. A request, a user session, a probe sequence, and a time window can all define different outcomes. They produce different events and different probabilities, even if the same production incident appears in the underlying logs.

Every term