02.12 · Concept
Naming Your Bottleneck
Synthesise: given a profiler trace, compute arithmetic intensity, place the kernel on a roofline, and name whether it is compute, bandwidth, or overhead bound. Defend the answer with numbers.
A bottleneck claim should reduce a profiler region to FLOPs, bytes moved, arithmetic intensity, and a roofline comparison. If measured throughput sits near the bandwidth roof it is bandwidth-bound; near peak compute it is compute-bound; far below both, look for launch, scheduling, communication, or runtime overhead.
What this lesson answers
- how to identify compute or bandwidth bound kernels
- how arithmetic intensity maps onto a roofline
- when profiler traces indicate overhead bound inference
Notes
Naming the bottleneck means reducing a profiler trace to a falsifiable roofline claim: for a measured kernel or decode step, compute its arithmetic intensity , compare it to the machine balance , and bound achievable throughput by . If measured throughput is near , the kernel is bandwidth-bound; if it is near , compute-bound; if it is far below both, overhead-bound or latency-bound.
References
Common questions
- What does it mean to name a bottleneck from a profiler trace?
- It means making a numerical claim about the traced region, not guessing from the workload name. Count the FLOPs, count the relevant bytes moved, calculate arithmetic intensity, compare it with the hardware roofline, then check whether observed throughput is close to compute peak, memory bandwidth, or neither.
- Why is single-token LLM decode often bandwidth-bound?
- Decode commonly reuses little work per byte because each new token must read large model weights and often scan KV cache state. That gives low arithmetic intensity, so the bandwidth roof caps throughput long before tensor cores are saturated. The exact answer still depends on batch, context, precision, parallelism, and cache layout.
- When is roofline analysis the wrong explanation?
- Roofline analysis misses time spent outside steady compute and memory transfer. Small batches, short prompts, kernel launch gaps, scheduler delays, CUDA graph breaks, collective latency, page management, RPC, and queueing can leave both FLOP rate and bandwidth low. In that case the bottleneck is overhead or latency, not the nominal roof.