When One GPU Is Not Enough

Notes

Sharding an LLM request means partitioning either persistent parameters, per-request state, or per-token computation across devices because a single GPU violates one of three constraints: weights capacity, KV-cache capacity, or latency. The governing capacity test is $M_{\text{weights}}+M_{\text{KV}}+M_{\text{workspace}}\le M_{\text{GPU}}$, where for grouped-query attention the KV cache is $M_{\text{KV}}=2\cdot L\cdot B\cdot T\cdot H_{\text{kv}}\cdot d_{\text{head}}\cdot s$, with $L$ layers, batch size $B$, sequence length $T$, KV heads $H_{\text{kv}}$, head dimension $d_{\text{head}}$, and bytes per element $s$. The latency test is different: for decode on one GPU, a crude lower bound is $t_{\text{token}}\ge \max (W_{\text{bytes}}/\text{HBM BW},F_{\text{token}}/\text{FLOP/s})$, plus synchronization and kernel overheads. These three inequalities fail for different physical reasons, so “add GPUs” is not a single technique: tensor or pipeline parallelism addresses weight residency and per-token latency, while KV sharding or disaggregated serving addresses context residency and admission rate.

For weights, take a 70B model in FP16: $70\text{B}\cdot 2=140\text{GB}$ before scales, embeddings, allocator padding, and CUDA graph workspaces. An 80GB H100 cannot hold that as one replica, even though its 3.35TB/s HBM bandwidth is excellent, so the first sharding reason is simply that $140\text{GB}>80\text{GB}$. With tensor parallelism across two H100s, each stores roughly $70\text{GB}$ of weights, leaving only about $10\text{GB}$ for KV and runtime memory, so production systems usually use 4-way or 8-way TP for 70B FP16 if long contexts are expected. Megatron-LM’s tensor-parallel formulation, TensorRT-LLM’s tensor and pipeline parallel engines, vLLM’s distributed executor, SGLang’s multi-GPU serving, and NVIDIA Dynamo’s disaggregated runtime all implement variants of this idea. The trade is that every transformer layer now contains collectives, typically all-reduce or reduce-scatter/all-gather around attention and MLP projections, so insufficient NVLink bandwidth turns a capacity fix into a latency regression.

The second reason is KV cache capacity, and it is often invisible when people size only the checkpoint. For a 70B-class GQA model with $L=80$, $H_{\text{kv}}=8$, $d_{\text{head}}=128$, FP16 KV so $s=2$, and $B=64$ sequences at $T=8192$, the KV footprint is $2\cdot 80\cdot 64\cdot 8192\cdot 8\cdot 128\cdot 2$ bytes. That is $171,798,691,840$ bytes, or about $160\text{GiB}$, just for keys and values. Even if the weights are quantized to fit on one 80GB H100, this batch of long contexts cannot. PagedAttention from the vLLM paper “Efficient Memory Management for Large Language Model Serving with PagedAttention” attacks fragmentation and enables paging-style allocation, but it does not make the KV vanish; SGLang’s RadixAttention reuses prefix KV across requests, TensorRT-LLM supports paged KV cache, llm-d focuses on Kubernetes-native disaggregated inference, and Dynamo’s KV routing/cache management separates prefill-heavy and decode-heavy placement. This kind of sharding is about storing, reusing, and moving state, not splitting matrix multiplies.

The third reason is latency below what one GPU can reach, even when memory fits. Suppose a quantized 70B checkpoint occupies $45\text{GB}$ and a small active KV set fits on one H100. During batch-1 decode, each generated token still streams a large fraction of the weights, so the HBM lower bound is roughly $45\text{GB}/3.35\text{TB/s}=0.0134\text{s}$, or $13.4\text{ms}$ per token before attention, launch overhead, sampling, and framework costs. If the product target is $5\text{ms}$ inter-token latency for an interactive agent, one H100 cannot satisfy it by scheduling alone. Tensor parallelism can reduce per-device bytes to about $45/4=11.25\text{GB}$ on 4 H100s, giving a bandwidth lower bound near $3.36\text{ms}$, but now each layer pays NVLink collectives and the latency benefit depends on keeping those below the saved HBM time. FlashAttention, from “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” improves attention IO and is essential for prefill, but it does not eliminate the decode weight-streaming bottleneck that motivates latency sharding.

The technique stops working when the bottleneck shifts from local HBM or capacity to communication, imbalance, or underfilled kernels. Pipeline parallelism can fit a model by placing different layer ranges on different GPUs, but for a single decode stream it creates bubbles: with $p$ pipeline stages and microbatch count $m$, utilization is roughly $m/(m+p-1)$, so $m=1,p=4$ gives only $25\%$ stage utilization. Tensor parallelism past the NVLink/NVSwitch sweet spot can make each token slower because all-reduce latency is paid dozens of times per layer stack; crossing PCIe or Ethernet for intra-layer collectives is usually disastrous for decode. KV sharding also has a failure mode: if every decode step must fetch remote KV blocks for attention, the request becomes network-bound, so systems try to route continuations to where the KV already lives. Prefix caching can become counterproductive when prompts have low overlap, because the index, hashing, and eviction overheads remain while reuse disappears. Speculative methods such as EAGLE, from “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” reduce target-model decode steps, but they can hurt when acceptance rate is low or the draft path consumes scarce GPU memory and scheduler slots.

The named serving stacks differ mainly in which of the three failures they optimize first. vLLM popularized PagedAttention and continuous batching for KV-efficient high-throughput serving; SGLang added a programming/runtime layer with RadixAttention and aggressive prefix sharing for structured workloads; TensorRT-LLM packages fused kernels, quantization, tensor parallelism, pipeline parallelism, paged KV, and in-flight batching for NVIDIA GPUs; NVIDIA Dynamo, released as a distributed inference serving framework, emphasizes disaggregated prefill/decode, KV transfer, and fleet-wide routing; llm-d brings similar disaggregated LLM serving concerns into a Kubernetes-oriented open stack. FlashAttention is the kernel-level IO optimization underneath many prefill paths, not a fleet scheduler. EAGLE is a speculative decoding algorithm that trades extra draft computation for fewer expensive target passes. When diagnosing “one GPU is not enough,” the engineer should first identify which inequality failed: checkpoint bytes, KV bytes under the intended concurrency and context length, or per-token latency lower bound. Only then does the correct sharding axis become obvious.