Module 02
The Hardware Floor
Measure and model GPU performance from first principles --- throughput vs latency, FLOPs and bytes, the memory hierarchy, arithmetic intensity, the roofline model, and why decode is memory bound. By the end you can read a profiler trace and name exactly which regime your bottleneck lands in.
12 lessons · 12 videos · 2h 26m
- 02.01
Why the GPU Waits
Understand the GPU as a throughput machine working against a latency-optimized CPU. Explain the three performance regimes: compute bound, memory bandwidth bound, and overhead bound.
- 02.02
FLOPs and Bytes
Calculate the FLOP count and memory traffic for a single transformer forward pass. Understand why resource accounting is the first step in any performance investigation.
- 02.03
The Memory Hierarchy
Trace data through the GPU memory hierarchy: DRAM, L2 cache, shared memory, registers. Explain why each level exists and what it costs to cross.
- 02.04
Arithmetic Intensity
Compute arithmetic intensity --- operations per byte --- for a kernel. Use it to predict whether a workload is compute bound or memory bound before running it.
- 02.05
The Roofline
Draw and read a roofline plot. Place a kernel on it, identify the ridge point, and determine which side of the ridge it sits on.
- 02.06
Why Decode Is Memory Bound
Prove from FLOP and byte counts that a single decode step reads far more bytes than it computes operations. Explain why larger batches are the only lever that helps.
- 02.07
Where the Memory Went
Build a memory budget for a transformer inference run: weights, KV cache, activations. Identify which component dominates at which batch size and context length.
- 02.08
Precision
Compare FP32, FP16, BF16, INT8, and FP4. Understand the memory bandwidth savings and the accuracy tradeoffs. Read a dtype annotation on a model checkpoint and know what it implies for throughput.
- 02.09
Reading the Device
Interpret nvidia-smi output. Understand what GPU-Util actually measures --- and what it does not. Diagnose why a GPU reporting 100% utilisation can still be memory bound.
- 02.10
Profiling a Forward Pass
Use the PyTorch profiler to capture a trace of a forward pass. Read the trace: identify CPU-GPU synchronisation gaps, kernel durations, and memory transfers.
- 02.11
Launch Overhead
Measure kernel launch overhead. Apply the batch doubling test: double the work and if runtime barely rises, the kernel is overhead bound. Recognise when CUDA graphs eliminate the problem.
- 02.12
Naming Your Bottleneck
Synthesise: given a profiler trace, compute arithmetic intensity, place the kernel on a roofline, and name whether it is compute, bandwidth, or overhead bound. Defend the answer with numbers.