Hockey Stick
A hockey stick is the sharp bend in a system’s latency curve as utilisation approaches saturation. It happens because queued work grows roughly with the inverse of remaining spare capacity, not in a straight line, so a service that feels comfortable at 70% utilisation can become unusable near 90%.
The problem is that capacity planning often treats utilisation as if it were linear. Engineers see a host, database, worker pool, or downstream API at a high but not full load and assume there is still room. In queueing systems, that remaining headroom is what absorbs bursts and variance. As it disappears, small increases in traffic create large increases in waiting time, even when average throughput still looks acceptable.
The concrete mechanism is a queue in front of a finite server. Requests arrive, each takes some service time, and utilisation is arrival rate multiplied by average service time. In a simple single-server model, mean response time is service time divided by one minus utilisation. The waiting part grows especially fast because each delayed request sits behind other delayed requests. At saturation, arrivals match or exceed completions, so the backlog no longer reliably drains.
The trade-off is wasted-looking capacity. To keep latency predictable, you deliberately run below the theoretical maximum, add replicas before the cliff, shed load, cap queues, or reserve concurrency. That costs money or rejects work earlier. The common misunderstanding is that high utilisation is always efficient. It is efficient only if latency, burst tolerance, and downstream saturation do not matter, which is rarely true for user-facing or dependency-heavy systems.
Engineers meet the hockey stick in autoscaling targets, CPU alarms, database connection pools, worker queues, load balancers, serverless concurrency, and circuit breakers. It is why scaling at 90% can be too late if new capacity takes time to appear. It is also why short overload periods can hurt despite harmless-looking averages: a minute of saturation can build a queue whose latency impact persists after traffic drops.
Common questions
- Why does latency rise faster than utilisation?
- Because requests do not only pay their own service time; they also wait for earlier work. As utilisation rises, there is less idle time to absorb randomness in arrivals and service durations. The queue becomes harder to drain, so response time grows with the shrinking spare capacity rather than with utilisation itself.
- Is the hockey stick only about CPU?
- No. CPU is a common signal, but the same shape appears anywhere work queues behind limited capacity: database connections, thread pools, message consumers, disk I/O, rate-limited APIs, or serverless concurrency. The relevant utilisation is of the bottleneck resource, which may not be the metric shown on the main dashboard.
- What utilisation target should a service run at?
- It depends on burstiness, service-time variance, autoscaling delay, retry behaviour, and the cost of latency or rejection. A steady batch worker can often run hotter than an interactive API. For user-facing systems, targets around the safer side of the curve are common because spare capacity is what prevents queues from exploding.
- Why can average utilisation hide the problem?
- Averages smooth over the periods when the system is actually saturated. If a service alternates between idle and overloaded, the average may look healthy while queues form during the overloaded interval. Users experience the waiting caused by those queues, not the arithmetic average over the whole reporting window.