FLOPs and Bytes

Notes

For one transformer forward pass, the hardware floor is the lower bound obtained by counting required floating-point operations and bytes moved, then comparing them with peak compute and bandwidth: $t\ge \max (F/P_{\mathrm{peak}},B/W_{\mathrm{peak}})$, with arithmetic intensity $I=F/B$ and ridge point $I^{\ast }=P_{\mathrm{peak}}/W_{\mathrm{peak}}$. For autoregressive inference this accounting must be per phase: prefill over $S$ prompt tokens has quadratic attention work, while decode for one new token has attention over the existing context and mostly streams weights. A dense decoder block with hidden size $d$, MLP expansion $d_{\mathrm{ff}}$, layers $L$, query heads $h_q$, KV heads $h_{kv}$, head dim $d_h$, and sequence length $S$ has approximate prefill FLOPs per layer $F_{\mathrm{prefill}}\approx 2S(4d^2+3d{d}_{\mathrm{ff}})+4S^2{h}_q{d}_h$, where the first term covers QKV/O and gated MLP matmuls and the second covers $Q{K}^{\top }$ and $PV$. For decode of one token, $F_{\mathrm{decode}}\approx 2(4d^2+3d{d}_{\mathrm{ff}})+4S{h}_q{d}_h$, but the bytes are often dominated by reading parameters, roughly $B_W\approx 2N_{\mathrm{params}}$ for FP16 weights if they cannot be reused across a large batch.

Take a Llama-2/3-style 70B dense model in FP16 on one 80GB H100 SXM, with $W_{\mathrm{HBM}}=3.35$ TB/s and FP16 tensor-core peak around $P=989$ TFLOP/s. The raw weight traffic for one decode token is about $B_W=70\times 10^9\times 2=140$ GB. Even before KV-cache traffic, the bandwidth floor is $t\ge 140/3350=0.0418$ s, or about $23.9$ tokens/s for batch 1 if every token rereads all weights from HBM. The FLOP count for the same token is about $F\approx 2N_{\mathrm{params}}=140$ GFLOP for the linear layers, plus attention over context. At $989$ TFLOP/s the compute floor is $0.000142$ s, so batch-1 decode has arithmetic intensity $I\approx 140\text{ GF}/140\text{ GB}=1$ FLOP/byte, far below the H100 ridge point $I^{\ast }=989/3.35\approx 295$ FLOP/byte. This is why decode throughput engineering starts by increasing effective batch, reusing weights across multiple tokens in flight, and making the KV cache addressable without copying it.

Now include concrete KV bytes, because the “weights dominate” statement stops being enough at long context and grouped-query attention changes the answer. With $L=80$, $h_{kv}=8$, $d_h=128$, FP16 KV entries, one cached token stores keys and values of size $B_{KV/token}=L\times 2\times h_{kv}\times d_h\times 2=80\times 2\times 8\times 128\times 2=327,680$ bytes, about $320$ KiB. Decoding at context $S=8192$ for one sequence reads roughly $S\times B_{KV/token}=8192\times 327,680=2.684$ GB of KV data if the attention kernel streams all keys and values once. Add that to $140$ GB of weights and the bandwidth floor becomes $(142.684\text{ GB})/3.35\text{ TB/s}=42.6$ ms, barely different. At $S=128,000$, however, KV traffic is $41.94$ GB per generated token, a 30% addition to weight traffic; at batch $B=32$, weight traffic is still $140$ GB per decode step but KV traffic scales with sequences to $32\times 41.94=1.34$ TB, making attention bandwidth the floor unless the kernel, paging layout, or cache placement changes.

Prefill is accounted differently because weights are reused across $S$ tokens and large GEMMs move toward the compute roof. For a 70B model and $S=4096$, the linear-layer FLOPs are approximately $2N_{\mathrm{params}}S=2\times 70\text{B}\times 4096=573$ TFLOP. The attention FLOPs are material but smaller with $h_q{d}_h=d=8192$: $4L{S}^{2}d=4\times 80\times 4096^2\times 8192\approx 44$ TFLOP, giving about $617$ TFLOP total. The compute floor on H100 is $617/989=0.624$ s if the whole pass sustained peak, while the one-time weight read floor is still only $41.8$ ms, so optimized prefill is compute-shaped rather than pure HBM-shaped. FlashAttention, from Dao et al. 2022 and FlashAttention-2 in 2023, changes the byte count of attention by tiling Q, K, and V through SRAM and avoiding materialization of the $S\times S$ score matrix; it does not change the $4S^2{h}_q{d}_h$ FLOP term, but it removes the otherwise catastrophic HBM traffic for attention probabilities and makes the above prefill floor plausible.

The named serving systems are best understood as attempts to move an implementation closer to these phase-specific floors, not as magic speedups outside the accounting. vLLM’s PagedAttention paper by Kwon et al. 2023 replaces contiguous KV allocation with page-table-managed blocks, reducing fragmentation and enabling high batch occupancy without copying long caches. SGLang’s runtime and RadixAttention, introduced in the 2024 SGLang paper, exploit prefix sharing so repeated prompts do not repay the same prefill FLOPs and KV writes. TensorRT-LLM’s in-flight batching and paged KV cache releases fuse decode kernels and keep the GEMMs large enough to reuse weights across requests. NVIDIA Dynamo, announced as a disaggregated inference serving stack in 2025, and llm-d, the Kubernetes-native distributed LLM serving project, push the same accounting across machines by separating prefill-heavy and decode-heavy workers and routing KV state explicitly. EAGLE, from Li et al. 2024, attacks the number of target-model decode passes using speculative feature-level drafting; its gain is bounded by the verifier FLOPs and acceptance rate, not by a different transformer cost model.

The floor calculation stops working as a predictor, and some optimizations make things worse, when the counted tensor traffic is no longer the limiting traffic or when batching changes latency constraints. For very small prompts and batch 1, launch overheads, sampling, host scheduling, tokenizer detours, and NCCL collectives can dominate a sub-millisecond kernel floor, so a roofline estimate overstates attainable tokens/s. PagedAttention can hurt when sequences are short and contiguous, because page indirection and non-coalesced KV reads add address-generation and TLB-like overhead without saving meaningful memory. FlashAttention-style kernels can lose to simpler attention for tiny $S$ or awkward head dimensions because tiling overhead exceeds the avoided HBM writes. Speculative decoding with EAGLE becomes negative when the acceptance rate is low or the draft/verifier synchronization serializes the path. Disaggregated systems such as Dynamo or llm-d stop helping when KV transfer over PCIe, NVLink, or the network exceeds the saved prefill/decode imbalance; moving a 40 GB long-context cache to avoid milliseconds of compute is bad arithmetic, not an implementation detail.