01.04 · Short-concept
Why Averages Lie
Explain why a good average hides a bad tail, and why p99 at scale is the size of your support inbox.
A healthy mean can coexist with a painful tail: most requests look fine while the slowest users generate retries, timeouts and tickets. Percentiles show where those users live. At production scale, a small p99 problem is not rare operational noise, it is a steady stream of bad experiences.
What this lesson answers
- why average latency hides slow user requests
- how does p99 latency affect support tickets
- when should I use percentiles instead of averages
Notes
Why Averages Lie — Averages exist to compress many measurements into one number, but they break system understanding because a “fast” mean can hide a slow tail where real users, retries, timeouts, and support tickets accumulate.
Key Concepts: - The arithmetic mean is , so 99 requests at and 1 request at produce while one user waits . - The p99 latency is the value below which of requests fall; in 1,000,000 requests/day, p99 represents the slowest requests/day.
Common questions
- Why is average latency misleading in production systems?
- Average latency compresses all requests into one value, so it can make a service look healthy while a small fraction of users wait far too long. Those slow requests matter because they trigger retries, timeouts, abandoned flows and support cases. A percentile view separates typical behaviour from the tail where reliability work often starts.
- What does p99 latency tell me that the mean does not?
- p99 tells you the boundary for the slowest requests after the faster bulk has been accounted for. The mean tells you the centre of mass, which can be pulled around by distribution shape and still hide user pain. p99 is useful because it describes the experience of the users most likely to complain.
- Why does tail latency get worse across service dependencies?
- Every extra dependency adds another chance for a request to hit a slow path. Even when each backend is usually fast, a page or API call that waits on many backends can be dominated by the slowest one. That is why end-to-end latency often looks worse than the individual service averages suggest.