02.04 · Walkthrough
Arithmetic Intensity
Compute arithmetic intensity --- operations per byte --- for a kernel. Use it to predict whether a workload is compute bound or memory bound before running it.
Arithmetic intensity is operations divided by bytes moved, and it predicts the bottleneck before a kernel runs. Compare a kernel’s FLOP per byte with the hardware’s compute-to-bandwidth balance: below it, memory bandwidth caps throughput; above it, peak compute becomes the likely limit.
What this lesson answers
- how to calculate arithmetic intensity for a kernel
- is my GPU workload memory bound or compute bound
- why does batching improve LLM decode throughput
Notes
Arithmetic intensity is the ratio of work performed to data moved across the limiting memory interface: , usually in FLOP/byte for GPU kernels. The roofline test compares it with the machine balance ; if , attainable throughput is bounded by bandwidth as , and if it is bounded by compute as .
References
Common questions
- What does arithmetic intensity tell me?
- It tells you how much computation you get for each byte fetched across the limiting memory path. Low intensity means the processor waits on memory even if compute units are available. High intensity means data is reused enough that compute throughput, scheduling, occupancy or numerical format are more likely to set the ceiling.
- Why is single-token LLM decode usually memory bound?
- For a single generated token, dense model weights are streamed with little reuse. Each weight contributes a small amount of arithmetic, but it still has to be read from high-bandwidth memory. That gives roughly one operation per byte in common FP16 decode estimates, far below the balance point of modern accelerators.
- Does FlashAttention make decode attention compute bound?
- Not for ordinary single-token decode. FlashAttention improves cases where tiling avoids writing or rereading a large attention score matrix. In decode, the old keys and values in the KV cache are the data that must be read, so the dominant traffic remains the KV stream itself.
Short definition: what is Arithmetic Intensity?