07.12 · Walkthrough
Agents per Megawatt
Size an agent fleet the way the workload demands: fix the SLOs first, then ask how many concurrent agents the machine sustains, not how many tokens per second it can emit. Build the per-session cost model, and identify the two levers that actually move it.
Agent serving capacity is a concurrency and latency problem, not a raw token-rate problem. Start from the task SLO, model each session’s model calls, context residency, decode work, tool waits and queueing, then size replicas by sustainable active agents per unit of power.
What this lesson answers
- how to size an agent serving fleet
- why tokens per second misleads agent capacity
- what drives cost per agent session
Notes
“Agents per megawatt” sizes serving capacity by closed-loop sessions, not open-loop token throughput. For an agent session with SLO latency budget , average model calls per step , steps per task , input/prefill tokens per call , generated tokens per call , and inter-tool/non-LLM time , the session residence time is .
Common questions
- Why is tokens per second a poor capacity metric for agents?
- Agents run closed-loop sessions with planning calls, tool waits, queueing and long-lived context. The same token rate can support many tiny calls or only a few context-heavy workflows. What matters operationally is how many sessions can stay within the latency SLO without exhausting KV memory or pushing queues past the utilisation knee.
- What should I measure before buying more inference capacity?
- Measure session-level behaviour: calls per task, prompt and output sizes per call, live context size, tool latency, decode batching, queue delay and tail latency. Then compare the resulting residence time and KV footprint with the replica’s usable memory and power draw. GPU headline throughput is only useful after that model exists.
- Which optimisations actually improve agents per megawatt?
- The main gains come from reducing live KV-token residency and improving useful decode batching while preserving the SLO. Shorter prompts, shared prefixes, prompt caching, summarised traces and lower-byte KV reduce memory pressure. Continuous batching, separated prefill and decode, and speculative decoding help only when they increase useful work without adding tail latency.
Short definition: what is Agents per Megawatt?