What is the difference between FLOPs and FLOP/s?

FLOPs is a count of arithmetic work required by a computation. FLOP/s is a rate a machine may sustain while doing that work. Confusing them hides the key step: divide the required FLOPs by the achievable FLOP/s to get a compute-time lower bound, then compare it with the byte-traffic lower bound.

Why can decode be slow if its FLOP count is small?

Autoregressive decode usually processes only one new token per sequence, so the matmuls are small and weight reuse can be poor. The hardware may spend most of its time reading model weights and KV cache from memory rather than doing arithmetic. That makes bandwidth, not peak compute, the limiting resource.

Does FlashAttention reduce the FLOP count?

Not for the core attention arithmetic. Its main benefit is reducing memory traffic by tiling attention through fast on-chip memory and avoiding writes of large intermediate score or probability matrices. That can move attention closer to the hardware floor, especially in prefill, but it does not make the mathematical attention work disappear.

When does the FLOPs-and-bytes estimate stop being useful?

It stops being a good latency predictor when uncounted costs dominate: tiny batches, short prompts, launch overheads, host scheduling, sampling, communication, or awkward memory access patterns. It is still useful as a sanity check. If an optimisation cannot beat the accounting on compute or bytes, it cannot produce a real speedup.

FLOPs and Bytes

FLOPs and bytes are the two basic counts in performance accounting: how many floating-point operations a computation must perform, and how much data it must move. Comparing those counts with a device’s compute rate and memory bandwidth gives a lower bound on runtime and shows whether work is compute-bound or bandwidth-bound.

The problem is that model performance is easy to misdiagnose from symptoms alone. A slow transformer pass might be limited by matrix multiplication, memory bandwidth, cache layout, batching, kernel launch overhead, or communication. FLOP and byte accounting gives a first-principles baseline before tuning. If the required data movement already consumes the available bandwidth, a faster matmul kernel will not fix the bottleneck.

The mechanism is a roofline-style estimate. Count the mandatory arithmetic as FLOPs, count the tensor and parameter traffic as bytes, then divide each by the hardware’s peak compute and bandwidth. The larger time is the hardware floor. Arithmetic intensity, FLOPs per byte, tells you which side dominates. In transformer inference, this must be split by phase: prompt prefill reuses weights across many tokens, while token-by-token decode often rereads weights and streams KV cache.

The trade-off is that the floor is not a prediction of actual latency. It ignores scheduling overhead, kernel launch costs, imperfect utilisation, non-coalesced reads, paging overhead, communication, sampling, and host-side work. It also depends on batch size, sequence length, datatype, attention variant, and cache placement. A common misunderstanding is to treat FLOP count as the whole story; in decode, bytes often matter more.

Engineers meet FLOP and byte accounting when deciding whether to batch requests, quantise weights, use paged KV cache, separate prefill and decode workers, or adopt FlashAttention-style kernels. It explains why prefill can look compute-shaped while decode is bandwidth-shaped, and why long context changes the answer. Serving systems are best understood as attempts to move real execution closer to these phase-specific floors.

Go deeper

Inference EngineeringFLOPs and BytesCalculate the FLOP count and memory traffic for a single transformer forward pass. Understand why resource accounting is the first step in any performance investigation.Open the lesson

Common questions

What is the difference between FLOPs and FLOP/s?: FLOPs is a count of arithmetic work required by a computation. FLOP/s is a rate a machine may sustain while doing that work. Confusing them hides the key step: divide the required FLOPs by the achievable FLOP/s to get a compute-time lower bound, then compare it with the byte-traffic lower bound.
Why can decode be slow if its FLOP count is small?: Autoregressive decode usually processes only one new token per sequence, so the matmuls are small and weight reuse can be poor. The hardware may spend most of its time reading model weights and KV cache from memory rather than doing arithmetic. That makes bandwidth, not peak compute, the limiting resource.
Does FlashAttention reduce the FLOP count?: Not for the core attention arithmetic. Its main benefit is reducing memory traffic by tiling attention through fast on-chip memory and avoiding writes of large intermediate score or probability matrices. That can move attention closer to the hardware floor, especially in prefill, but it does not make the mathematical attention work disappear.
When does the FLOPs-and-bytes estimate stop being useful?: It stops being a good latency predictor when uncounted costs dominate: tiny batches, short prompts, launch overheads, host scheduling, sampling, communication, or awkward memory access patterns. It is still useful as a sanity check. If an optimisation cannot beat the accounting on compute or bytes, it cannot produce a real speedup.

Every term