Module 01 · Free
The Generation Loop
Understand how an LLM produces text token by token --- from the forward pass and logits through sampling and stopping to the serving stack that puts it all in production.
11 lessons · 11 videos · 1h 10m
- 01.01
Forward Pass vs Generation
Distinguish training forward passes from autoregressive generation and explain why inference is a fundamentally different workload.
- 01.02
The Generation Loop
Trace the autoregressive generation loop: forward pass → sample token → append → repeat. Read Karpathy's generate function in nanoGPT.
- 01.03
Prefill & Decode
Explain the two distinct phases of LLM inference: compute-bound prefill and memory-bound decode. Understand why they must be treated differently.
- 01.04
Logits
Read the raw logit vector from a forward pass, understand how it maps vocabulary to scores, and trace the unembedding step.
- 01.05
Sampling
Implement temperature scaling, top-k, top-p, and greedy sampling. Explain what each hyperparameter controls and when to use it.
- 01.06
Stopping
Implement stopping criteria: EOS token, max tokens, stop sequences. Understand why stopping is a correctness concern, not a convenience.
- 01.07
Tokenisation at Serving Time
Diagnose tokenisation issues in production: token count vs character count, non-English compression, and special token handling at the serving boundary.
- 01.08
The Context Budget
Calculate KV cache memory from model parameters and context length. Reason about the quadratic prefill cost and why longer contexts change the inference profile.
- 01.09
First Batch
Understand batching as a throughput lever: how multiple sequences share a forward pass, why batch size is a latency/throughput tradeoff, and how a scheduler picks requests to batch together.
- 01.10
The Serving Stack
Map the components of a production serving system: model runner, scheduler, KV cache manager, request queue, and REST/gRPC frontend.
- 01.11
Your Baseline
Run a benchmark on your own hardware: throughput, time-to-first-token, inter-token latency. Establish a reproducible measurement before any optimisation.