Module 07
Serving Agents
An agent is not a chatbot. It is hundreds of turns against one enormous shared prefix, with a tool call stalling the loop in the middle of every one. That workload breaks the batching and cache assumptions modules 3 and 4 were built on, and turns the KV cache from a buffer into a storage tier.
12 lessons · 12 videos · 1h 49m
- 07.01
The Agent Workload
Describe the shape of agent traffic: long multi-turn trajectories, input sequences an order of magnitude larger than the output, near-total prefix overlap between turns. Name which specific assumptions from modules 3 and 4 stop holding.
- 07.02
Session Affinity
Treat a session rather than a request as the unit of scheduling: measure prefix-cache hit rate across the turns of one real trajectory, and show what it costs when the second turn lands on a replica that has never seen the first.
- 07.03
Cache-Aware Routing
Route on cache residency instead of load: send a request to the replica that already holds its prefix, and explain why round-robin, the correct answer for a stateless service, is the worst possible policy here.
- 07.04
The KV Memory Hierarchy
Place the KV cache on a storage hierarchy: HBM, host DRAM, local NVMe, a remote pooled store. Explain why production fleets now hold more cache off the GPU than on it. This is module 2's memory hierarchy, one level up.
- 07.05
KV Offload and Reload
Do the arithmetic that decides between reloading a cache and recomputing it: bytes to move against the link bandwidth, versus the prefill FLOPs to rebuild the same tokens. Find the prefix length where the answer flips.
- 07.06
Tool Calls in the Loop
Account for the pause: while a tool runs, the sequence is neither prefilling nor decoding, and its KV cache is occupying memory it is not using. Compare holding, offloading and evicting it, and say which the tool latency distribution argues for.
- 07.07
The Tool Schema Tax
Measure what a tool definition costs before any tool is called: a handful of schemas adds hundreds of tokens to the prompt of every single request, which is real time-to-first-token unless that block is prefix-cached. Then show the fix: put the schemas where the cache can hold them.
- 07.08
Structured Output
Explain constrained decoding as a mask over the logit vector from module 1: at each step a grammar decides which tokens are legal, and sampling happens only over those. Distinguish it from asking a model nicely for JSON, which is not a guarantee.
- 07.09
What Grammars Cost
Separate the two costs of constrained decoding and find which one is actually hurting: applying a mask per step is cheap and constant, while COMPILING a schema into an automaton is neither, so a fleet with many distinct one-shot schemas pays a tail latency a fleet reusing a few cached ones never sees.
- 07.10
Long-Context Prefill
Take module 4's chunked prefill to agent scale: schedule a hundred-thousand-token prompt without starving every decoding request behind it, and show why the cache hit rate matters more here than any kernel does.
- 07.11
Context Compaction
Cut the prompt instead of speeding it up, summarising, pruning or retrieving rather than resending. Understand the trap that pays for it: rewriting history shifts every token position after the edit, so a compaction that saves tokens can destroy the prefix cache it was meant to help.
- 07.12
Agents per Megawatt
Size an agent fleet the way the workload demands: fix the SLOs first, then ask how many concurrent agents the machine sustains, not how many tokens per second it can emit. Build the per-session cost model, and identify the two levers that actually move it.