Module 04
The Batch
Turn a model server into a scheduler. One request at a time wastes the machine; this module covers continuous batching, chunked prefill and prefill/decode disaggregation, then the latency numbers that say whether any of it worked.
12 lessons · 12 videos · 1h 22m
- 04.01
One Request at a Time
Explain why batch size 1 is the worst case for a GPU: the whole weight matrix is read from HBM to produce a single token, so arithmetic intensity collapses and the device runs at a few percent of peak.
- 04.02
Static Batching
Describe request-level batching and name its two costs: padding to the longest sequence, and head-of-line blocking where every finished request waits for the slowest one.
- 04.03
Continuous Batching
Explain iteration-level scheduling: admitting arriving requests and retiring finished ones at every decode step rather than every request, and why that alone is the largest single throughput win in LLM serving.
- 04.04
The Scheduler Loop
Trace what a serving engine does on one step: the waiting and running queues, admission against the KV budget, preemption and recomputation when memory runs short.
- 04.05
Chunked Prefill
Explain how splitting a long prompt into token-budgeted chunks lets prefill and decode share a batch, so a single long prompt stops stalling every other user's stream.
- 04.06
Prefill/Decode Interference
Show why one GPU doing both phases does neither well: prefill saturates compute, decode saturates bandwidth, and mixing them means each phase pays the other's stall.
- 04.07
Disaggregation
Describe running prefill and decode on separate GPU pools so each scales independently, and state honestly when it does not pay: short prompts, low concurrency, or fewer GPUs than it takes to keep both pools busy.
- 04.08
Moving the KV Cache
Explain why the KV transfer between a prefill worker and a decode worker is the engineering crux of disaggregation, and how NVLink, RDMA and a transfer layer decide whether the split is a win or a regression.
- 04.09
TTFT, TPOT, ITL
Define time to first token, time per output token and inter-token latency, and say which one a user actually feels in a streaming chat versus a batch job.
- 04.10
Throughput vs Latency
Read the batch-size curve: larger batches raise tokens per second and raise per-user latency at the same time. Pick a point on it deliberately instead of inheriting a default.
- 04.11
Goodput and SLOs
Use goodput instead of raw throughput: requests per second that met both the TTFT and the TPOT target. Set a latency budget a serving change can be measured against.
- 04.12
Benchmarking Your Server
Run a load test that is not self-deception: a realistic prompt-length distribution, controlled concurrency, percentiles rather than means, and a warm cache reported separately from a cold one.