Agents per Megawatt
Agents per Megawatt is a capacity metric for LLM agent serving that counts how many end-to-end agent sessions a power budget can sustain while meeting latency SLOs. It treats an agent as a closed-loop programme with model calls, tool waits, queues, and KV cache residency, rather than as a stream of output tokens.
The problem is that token throughput is the wrong unit for many agent systems. A server can report impressive tokens per second and still run only a small number of useful agents if each session holds a long context, performs many short calls, waits on tools, and returns to the model unpredictably. What matters to the user is programme completion within an SLO, often at a tail percentile, not the average rate at which a GPU emits text.
Concretely, you model one agent session as a sequence of LLM calls and non-LLM waits. Each call consumes prefill work for the prompt, decode work for generated tokens, queueing delay, and KV cache memory that may remain live between steps. Little’s Law connects session residence time and arrival rate to the number of concurrent agents. Dividing that sustainable concurrency by cluster power gives the power-normalised fleet size.
The trade-off is that this metric forces you to account for the resource that is actually scarce. For long-context agents, KV cache can cap concurrency before compute does. For decode-heavy agents, memory bandwidth and batching dominate. Better packing, prefix sharing, lower-byte KV, speculative decoding, and prefill/decode separation can help, but none of them removes the need to satisfy tail latency while keeping enough sessions active to batch efficiently.
Engineers meet Agents per Megawatt when sizing inference fleets, setting admission control, choosing model and context limits, or comparing serving stacks such as vLLM, SGLang, TensorRT-LLM, Dynamo, and llm-d style deployments. The practical workflow is to fix the task SLO first, estimate per-session calls, context, outputs, tool time, and KV footprint, then test how many concurrent sessions a replica can carry before queueing or memory pressure breaks the SLO.
Common questions
- Why not use tokens per second to size an agent fleet?
- Tokens per second is an open-loop model-serving metric. Agents are closed-loop workloads: they call tools, branch, issue multiple prompts, hold KV cache, and re-enter queues. The same token rate can support many short planning calls or only a few long-context sessions. Capacity has to be measured against session completion latency.
- What are the main levers for improving Agents per Megawatt?
- The two important levers are shrinking how much live context each session keeps in GPU memory and improving effective decode batching without violating the SLO. That means shorter prompts, summarised traces, prefix caching, shared prefixes, efficient KV formats where acceptable, continuous batching, and sometimes disaggregated prefill and decode or speculative decoding.
- Does PagedAttention solve the concurrency problem for agents?
- No. PagedAttention reduces waste from variable-length KV allocation and fragmentation, so it can fit more live sequences into memory. It does not change the underlying KV bytes required by long contexts. If the real working set exceeds GPU memory, spilling attention state elsewhere usually adds too much latency for interactive decode.
- What is commonly misunderstood about this metric?
- A common mistake is treating it as a hardware benchmark. It is workload-specific. The answer depends on call count, prompt length, output length, tool latency, prefix reuse, batching behaviour, queueing policy, model size, KV format, and the chosen SLO. A single advertised number cannot describe agent capacity.