Is weight-only quantisation the same as running the whole model in INT4?

No. In the common setup, only the stored weights are INT4 or INT8. Activations, residual streams, attention values, KV cache, and accumulators usually stay in FP16 or BF16. The kernel temporarily dequantises weight tiles as it multiplies, so the main saving is memory bandwidth and footprint, not an entirely integer model.

Why is it especially useful for decode?

During small-batch decode, each new token causes the model to reread most weight matrices while doing relatively little work per weight. That makes parameter bandwidth a natural bottleneck. Weight-only quantisation reduces those repeated reads. During prefill or large-batch decode, arithmetic intensity is higher, so the benefit can be smaller and depends more on the kernel and workload.

What is the difference between AWQ and GPTQ?

Both use calibration data instead of blindly rounding each weight. AWQ is activation-aware: it rescales or protects channels whose activations make their errors matter more, while keeping deployment close to ordinary groupwise INT4 GEMM. GPTQ is more explicitly second-order: it uses a Hessian proxy to decide rounding and to compensate unquantised weights after each column is quantised.

When should I avoid weight-only quantisation?

Avoid assuming it is free. It may be a poor trade if your batch is large enough to be compute-bound, your context length makes KV traffic dominate, your serving engine has slow unpacking kernels, or your model is sensitive to quantisation error. The honest answer is workload-specific: measure quality and throughput with your prompts, batch shape, and hardware.

Weight-Only Quantisation

Weight-only quantisation stores a model’s weights in lower precision, typically INT8 or INT4, while leaving activations and most intermediate values in FP16 or BF16. During inference, kernels unpack and rescale weight tiles just before multiplication, reducing memory bandwidth for decode without fully quantising the computation.

The reason this exists is that autoregressive decode repeatedly streams the same large weight matrices for each generated token. With small batches, the GPU often waits on HBM traffic rather than arithmetic. Compressing activations would complicate every token’s dynamic data path, but compressing weights attacks the static bytes reread at every layer. That is why INT4 weight-only quantisation is often the first practical optimisation for decode.

Concretely, each group of weights is stored as small integers plus scale, and often zero-point, metadata. In a linear layer, the kernel loads packed nibbles or bytes, reconstructs approximate FP16 or BF16 values for a tile, and feeds them into a fused dequantise-and-multiply path. AWQ chooses scales using calibration activations so important channels suffer less error. GPTQ uses a Hessian-style proxy from calibration data and compensates remaining columns as it quantises.

The trade-off is that fewer bytes are not automatically faster or equally accurate. Dequantisation instructions, packing layout, group metadata, and kernel quality all matter. Large batches can reuse weights enough that compute becomes the bottleneck. Long-context decode can shift pressure to KV-cache reads. Accuracy also depends on model size, outliers, calibration data, group size, and whether the rounding method preserves the layer’s behaviour on realistic inputs.

Engineers usually meet weight-only quantisation as an inference-engine option: INT4 or INT8 checkpoints, AWQ or GPTQ formats, and fused kernels in serving stacks. It is commonly misunderstood as making the whole model run in INT4. Usually it does not: activations, residuals, attention scores, KV cache, and accumulators often remain higher precision. The promise is narrower but powerful: reduce weight traffic during decode.

Go deeper

Inference EngineeringWeight-Only QuantisationExplain why INT4 weight-only quantisation is the natural first move for decode, since the weights are the bytes being read, and how AWQ and GPTQ decide what to round.Open the lesson

Common questions

Is weight-only quantisation the same as running the whole model in INT4?: No. In the common setup, only the stored weights are INT4 or INT8. Activations, residual streams, attention values, KV cache, and accumulators usually stay in FP16 or BF16. The kernel temporarily dequantises weight tiles as it multiplies, so the main saving is memory bandwidth and footprint, not an entirely integer model.
Why is it especially useful for decode?: During small-batch decode, each new token causes the model to reread most weight matrices while doing relatively little work per weight. That makes parameter bandwidth a natural bottleneck. Weight-only quantisation reduces those repeated reads. During prefill or large-batch decode, arithmetic intensity is higher, so the benefit can be smaller and depends more on the kernel and workload.
What is the difference between AWQ and GPTQ?: Both use calibration data instead of blindly rounding each weight. AWQ is activation-aware: it rescales or protects channels whose activations make their errors matter more, while keeping deployment close to ordinary groupwise INT4 GEMM. GPTQ is more explicitly second-order: it uses a Hessian proxy to decide rounding and to compensate unquantised weights after each column is quantised.
When should I avoid weight-only quantisation?: Avoid assuming it is free. It may be a poor trade if your batch is large enough to be compute-bound, your context length makes KV traffic dominate, your serving engine has slow unpacking kernels, or your model is sensitive to quantisation error. The honest answer is workload-specific: measure quality and throughput with your prompts, batch shape, and hardware.

Every term