02.05 · Walkthrough
The Roofline
Draw and read a roofline plot. Place a kernel on it, identify the ridge point, and determine which side of the ridge it sits on.
A roofline plot compares arithmetic intensity with achieved throughput to show whether a kernel is limited by memory bandwidth or compute. The ridge point separates those regimes. For inference workloads, decode often sits on the bandwidth slope, while large prefill can move under the compute ceiling through weight reuse.
What this lesson answers
- how to read a roofline plot
- what is the roofline ridge point
- is transformer decode memory or compute bound
Notes
A roofline plot is the upper bound model , where is achieved throughput in FLOP/s, is the processor’s compute ceiling, is sustained memory bandwidth in byte/s, and is arithmetic intensity. On log-log axes, is the x-axis and is the y-axis: the sloped bandwidth roof has gradient , while the horizontal roof is . Their intersection is the ridge point, .
References
Common questions
- What does a roofline plot tell me?
- It shows the best throughput a kernel can plausibly reach given its arithmetic intensity, the hardware memory bandwidth, and the hardware compute peak. If the point is on the sloped part, moving fewer bytes matters most. If it is under the flat roof, more compute capability or better compute utilisation matters most.
- How do I decide which side of the ridge a kernel is on?
- Compute the kernel’s arithmetic intensity as FLOPs divided by bytes moved, then compare it with the ridge point, which is compute peak divided by bandwidth peak. Lower intensity than the ridge means bandwidth-bound. Higher intensity means compute-bound, assuming no other ceiling such as communication, launch overhead, or latency dominates.
- Why is LLM decode often bandwidth-bound while prefill is not?
- Single-token decode tends to stream model weights for relatively little computation, so its arithmetic intensity is low and memory bandwidth dominates. Prefill can reuse the same weights across many prompt tokens, turning the work into larger matrix operations with much higher intensity, so it can approach the compute roof instead.
Short definition: what is Roofline?