Curve, Collected
The collected curve is the recurring queueing shape where latency, backlog, and retry load grow like 1/(1-utilisation) as a shared resource approaches full use. It explains why a system can look healthy at moderate load, then degrade abruptly when a small extra arrival rate pushes it into saturation.
The problem is that engineers often reason about spare capacity linearly. If a service is busy but not completely full, it feels as if there is still room. Queueing systems do not behave that way near saturation. A small increase in traffic, a slow dependency, or a retry policy can move the service onto the steep part of the curve, where waiting time dominates actual work and failures feed themselves.
Picture one worker serving requests from a line. When arrivals are well below service capacity, most requests find the worker free or wait briefly. As utilisation rises, each small burst is more likely to find earlier work still present. In an M/M/1 approximation, response time is service time multiplied by 1/(1-rho), and queue length has the same asymptote. Little’s Law then connects the growing in-flight population to observed latency.
The trade-off is that the curve is a model, not a prophecy. Real systems have many workers, bursty arrivals, locks, caches, schedulers, garbage collection, cold starts, and downstream limits. The exact formula may not fit, especially for tails. The useful lesson survives: high average utilisation is not efficient if it destroys latency margin, and retries without budgets can turn partial failure into overload.
Engineers meet this curve in autoscaling, load shedding, connection pools, circuit breakers, database queues, and retry configuration. A CPU graph, queue depth metric, p99 latency chart, or pending-request counter is often showing the same shape through a different instrument. Commonly misunderstood: scaling after queues are already full may arrive too late, because the backlog accumulated during the delay must also be drained.
Common questions
- Is this only about M/M/1 queues?
- No. M/M/1 is the clean teaching case because the curve is easy to see algebraically. Production systems are messier, but the same mechanism appears whenever shared capacity is approached faster than work can be completed. The exact multiplier depends on arrivals, service-time variance, parallelism, and bottlenecks.
- Why do retries make the curve worse?
- A retry is extra offered load sent to a system that may already be failing because it is overloaded. If many clients retry at once, they increase arrivals precisely when service capacity is least able to absorb them. Without backoff, jitter, and retry budgets, retries can convert a local slowdown into sustained saturation.
- Does high utilisation always mean danger?
- It depends on the workload and the latency objective. Batch systems may tolerate high utilisation because waiting is acceptable. Interactive services usually need headroom because users care about response time and tail latency. The danger signal is not utilisation alone, but utilisation combined with growing queues, rising p99 latency, and retry amplification.