Is arithmetic intensity the same as utilisation?

No. Arithmetic intensity is a property of the work and data movement you expect from a kernel. Utilisation is what the hardware actually achieves at runtime. Intensity helps predict the ceiling: bandwidth-bound or compute-bound. Utilisation then tells you how close the implementation came to that ceiling after overheads, scheduling, memory access patterns, and dependencies.

What bytes should I count?

Count bytes crossing the memory interface that limits the kernel, not every logical access in the source code. For a GPU kernel that is often HBM traffic, but for a tiled kernel it may be shared memory or another level. The honest answer is: count the bytes that cannot be served by reuse at a faster level.

Why does batching improve arithmetic intensity in model serving?

For weight-dominated decode, a larger batch can reuse the same streamed weights for multiple tokens. The weight bytes stay roughly fixed for the step, while the multiply-add work grows with the batch. That raises arithmetic intensity and can move the workload closer to the compute roof, although latency targets and KV memory usually limit how far you can push it.

Does a low arithmetic intensity mean the code is bad?

Not necessarily. Some algorithms inherently move a lot of data for little computation, such as streaming KV cache during single-token attention. Low intensity means bandwidth is the scarce resource, so optimisations should focus on reducing traffic, improving locality, compressing data, batching, or changing the algorithm, rather than expecting more compute units to help.

Arithmetic Intensity

Arithmetic intensity is the amount of computation a kernel performs per byte it moves across its limiting memory interface, usually expressed as FLOP per byte. It predicts whether performance is capped mainly by memory bandwidth or by compute capacity by comparing the kernel’s ratio with the hardware’s compute-to-bandwidth balance.

The need for arithmetic intensity appears when a processor has far more arithmetic capability than its memory system can feed. A GPU may have idle tensor cores while the kernel waits on HBM, or it may be fully compute-limited with bandwidth to spare. Looking only at peak FLOP/s is therefore misleading. You need a way to estimate, before benchmarking, whether the next optimisation should reduce bytes moved or improve arithmetic scheduling.

The calculation is deliberately simple: count the operations performed by the kernel, count the bytes transferred across the bottleneck memory interface, then divide operations by bytes. Compare that ratio with the machine balance, which is peak compute divided by memory bandwidth. If the kernel’s intensity is below that balance, the roofline says bandwidth limits throughput. If it is above, compute is the likely ceiling, assuming occupancy, instruction mix, and numerics do not intervene.

The trade-off is that arithmetic intensity is a model, not a profiler. The answer depends on which memory interface is limiting, what data is actually reused from cache or SRAM, whether reads are coalesced, and whether extra traffic appears from metadata, layouts, collectives, or spills. It also ignores many second-order limits. A high-intensity kernel can still run poorly because of scheduling, synchronisation, register pressure, or unfavourable tensor-core shapes.

Engineers meet arithmetic intensity when reasoning about GEMMs, attention, decoding, batching, and kernel fusion. Single-token transformer decode is commonly misunderstood: it looks compute-heavy, but much of it streams weights or KV cache with little reuse, so it is often bandwidth-bound. Batching changes the picture because the same weights can serve many tokens, increasing work faster than weight traffic. The roofline check is a quick sanity test before deeper measurement.

Go deeper

Inference EngineeringArithmetic IntensityCompute arithmetic intensity --- operations per byte --- for a kernel. Use it to predict whether a workload is compute bound or memory bound before running it.Open the lesson

Common questions

Is arithmetic intensity the same as utilisation?: No. Arithmetic intensity is a property of the work and data movement you expect from a kernel. Utilisation is what the hardware actually achieves at runtime. Intensity helps predict the ceiling: bandwidth-bound or compute-bound. Utilisation then tells you how close the implementation came to that ceiling after overheads, scheduling, memory access patterns, and dependencies.
What bytes should I count?: Count bytes crossing the memory interface that limits the kernel, not every logical access in the source code. For a GPU kernel that is often HBM traffic, but for a tiled kernel it may be shared memory or another level. The honest answer is: count the bytes that cannot be served by reuse at a faster level.
Why does batching improve arithmetic intensity in model serving?: For weight-dominated decode, a larger batch can reuse the same streamed weights for multiple tokens. The weight bytes stay roughly fixed for the step, while the multiply-add work grows with the batch. That raises arithmetic intensity and can move the workload closer to the compute roof, although latency targets and KV memory usually limit how far you can push it.
Does a low arithmetic intensity mean the code is bad?: Not necessarily. Some algorithms inherently move a lot of data for little computation, such as streaming KV cache during single-token attention. Low intensity means bandwidth is the scarce resource, so optimisations should focus on reducing traffic, improving locality, compressing data, batching, or changing the algorithm, rather than expecting more compute units to help.

Every term