The Hockey Stick

Notes

The Hockey Stick — The hockey stick exists because queues amplify small increases in utilization into disproportionate latency, so without spare capacity a system that looks healthy at $70\%$ utilization can become effectively unusable near $90\%$.

Key Concepts: - In an $M/M/1$ queue, mean response time is $R=\frac{S}{1-\rho }$, where $S$ is service time and $\rho$ is utilization. - If service time is $S=10\text{ ms}$, then at $\rho =0.70$, $R=\frac{10}{0.30}=33.3\text{ ms}$, but at $\rho =0.90$, $R=\frac{10}{0.10}=100\text{ ms}$. - The queueing delay portion is $W_q=\frac{\rho S}{1-\rho }$, so delay grows hyperbolically rather than linearly. - Utilization is $\rho =\lambda S$, where $\lambda$ is arrival rate and $S$ is average service time; with $S=20\text{ ms}$, one worker reaches $\rho =1$ at $\lambda =50$ requests/second. - Moving from $80\%$ to $90\%$ utilization doubles response time in $M/M/1$: $\frac{S/(1-0.9)}{S/(1-0.8)}=\frac{10S}{5S}=2$. - Moving from $90\%$ to $95\%$ utilization doubles response time again: $\frac{S/(1-0.95)}{S/(1-0.9)}=\frac{20S}{10S}=2$. - At $\rho \ge 1$, the queue is unstable because arrivals are at least as fast as completions, so expected queue length grows without bound. - The “vertical” part of the hockey stick is caused by variance and backlog: even if average capacity equals average demand, bursts create queues that take time to drain.

Watch For: - Treating $85\%$ CPU as only $15\%$ worse than $70\%$: with $R=\frac{S}{1-\rho }$, $70\%\to 85\%$ doubles latency from $3.33S$ to $6.67S$. - Averaging utilization over 5 minutes hides overload: a service at $50\%$ for 4 minutes and $100\%$ for 1 minute reports $60\%$ average while accumulating a queue during the saturated minute. - Scaling only after $\rho =0.90$ can be too late: if autoscaling takes 60 seconds and arrivals exceed service rate by 100 requests/second, the system accumulates about 6,000 queued requests before new capacity arrives. - Ignoring tail latency: if mean response time is $100\text{ ms}$ at $\rho =0.90$, $p99$ can be seconds when requests have variable service times or share a saturated dependency.

Production Connection: - Kubernetes Horizontal Pod Autoscaler commonly scales on CPU utilization targets such as $60\%-70\%$ to avoid running pods near the queueing cliff at $\rho \approx 1$. - AWS Lambda reserved concurrency prevents one function from consuming all regional concurrency and pushing downstream systems such as Amazon RDS or DynamoDB into saturation. - Envoy circuit breakers use limits such as `max_connections`, `max_pending_requests`, and `max_requests` to cap queues before latency enters the hockey-stick region. - Google Cloud Run’s container concurrency setting controls how many simultaneous requests share one instance, directly affecting per-instance utilization and queueing delay.