Quantisation memory budget

Head dimension is 8192 / 64 = 128. This model uses grouped-query attention: 64 attention heads share 8 KV heads, so the cache is 8x smaller than the head count alone would suggest. Assuming one KV head per attention head is the commonest way to conclude a model does not fit when it does.

Llama-3.1 70B at Q4_K_M with a 8,192 token context needs 41.9 GiB, which fits on H100 80GB.

Fits on H100 80GB with 38.1 GiB to spare. The highest precision that fits here is fp8.

At Q4_K_M on H100 80GB, the longest context that still fits is 133,008 tokens at 1 concurrent sequence. That is solved by inverting the KV term rather than searched for: everything else is fixed and the cache is linear in context, so it is one division.

Q4_K_M is 4.5 bytes per 8, not 4. A block-quantised format stores a scale, and usually a zero-point, for every block of weights it packs, so the real cost is above the nominal bit width. The naive int4 row is there for contrast: it is what a calculator that multiplies by 0.5 tells you, and it understates the weights by 11%, which is exactly the margin a borderline fit turns on.

The KV column doubles when the context doubles - it is linear, and that is the whole shape of this table. Do not carry that intuition across to compute: attention cost grows with the SQUARE of the sequence length while the cache it reads grows linearly. Different quantities, and only the first one is on this page.

Real: the weights term, the KV cache term and the training terms are exact arithmetic on the architecture numbers shown, and you can redo any of them by hand. The layer, head and KV-head counts come from the models' published configuration files. The highest-precision-that-fits and longest-context-that-fits answers are solved, not approximated.

Standing in: activations and workspace are an allowance scaled off the weights and the batch, not a measurement - the real figure swings by gigabytes between one serving stack and another, and it is the reason this page reports headroom instead of a hard yes. The block-quantised rows use representative bits-per-parameter; GGUF, AWQ and GPTQ variants differ from each other. And a real deployment adds a question this page cannot answer: vLLM preallocates its KV pool from a fixed fraction of the card rather than growing it on demand, so "does it fit" there is partly a configuration setting.