Is a larger calibration set always better?

Not automatically. Coverage matters more than raw count once the sample is large enough to estimate useful ranges. A smaller set that includes the real system prompt, retrieval snippets, code, tool-call JSON, long prefill cases, and served languages is usually better than a larger generic text set that never exercises production activation patterns.

When should I prefer per-channel over per-tensor scaling?

Prefer per-channel scaling where different channels have very different magnitudes, especially linear-layer weights, attention projections, and MLP projections. Per-tensor scaling is simpler, but one outlier can force the whole tensor to use a wide range, reducing precision for normal values. It is commonly misunderstood as a universal switch; its value depends on the tensor and kernel support.

How should quality be reported after quantisation?

Report the delta from the exact unquantised model you would have served, using the exact quantised artefact and runtime configuration. Include held-out task results and distributional checks, not only an average score. For instruction systems, also test formatting, tool-call exactness, code execution, retrieval answers, long-context paths, and safety or refusal regressions.

Can calibration make a quantised model identical to the original?

No. Calibration chooses the least damaging integer representation for observed distributions; it does not remove quantisation error. If production inputs stay close to the calibrated distribution, the error may be small enough to ignore. If prompts hit unseen ranges or narrow logit margins, the same calibration can produce visible regressions.

Calibration and Quality

Calibration and quality is the practice of choosing representative data to set quantisation ranges, scales, and clipping behaviour, then measuring how much the served model changes. It connects compression to correctness: integer weights and activations are only useful if their reconstructed values preserve the logits, formats, and task behaviour that production depends on.

Quantisation needs calibration because tensors do not arrive with safe integer ranges attached. A single unusual activation channel, prompt format, or long-context path can stretch the observed range and make ordinary values use too few integer codes. If the calibration data looks unlike production traffic, the scales will protect the wrong cases. The result can be a model that benchmarks well on the calibration source but fails on code, tool calls, multilingual text, or long prompts.

Mechanically, post-training quantisation records statistics while running sample inputs through the model. Observers collect maxima, minima, outliers, or reconstruction signals, then choose scale metadata used to map floating values into integers and back. Per-tensor scaling gives one scale to a whole tensor. Per-channel scaling gives separate scales to channels, often for weights in linear layers, so an outlier column or row does not waste the useful range for everything else.

The trade-off is that better calibration and finer-grained scales reduce error but add complexity, metadata, and more cases to validate. Per-channel scales are often nearly free for large weight matrices, but they do not solve every activation outlier or every layer. Aggressive activation or KV-cache quantisation can accumulate error through attention and residual paths. The honest answer is workload-dependent: saved bytes help only when bandwidth or memory is the limiting factor.

Engineers meet calibration and quality when exporting AWQ, GPTQ, SmoothQuant, weight-only, INT8, INT4, or FP8 variants for real serving stacks. The test should use the deployed tokenizer, chat template, kernels, tensor parallel settings, sampling defaults, and KV-cache format. Compare against the unquantised model on held-out production-like data, not just a paper’s corpus. Look at task metrics, perplexity, token-level divergence, structured-output validity, refusal changes, and long-context behaviour.

Go deeper

Inference EngineeringCalibration and QualityChoose a calibration set, pick per-channel over per-tensor scaling where it matters, and measure the accuracy cost honestly rather than quoting the paper's number.Open the lesson

Common questions

Is a larger calibration set always better?: Not automatically. Coverage matters more than raw count once the sample is large enough to estimate useful ranges. A smaller set that includes the real system prompt, retrieval snippets, code, tool-call JSON, long prefill cases, and served languages is usually better than a larger generic text set that never exercises production activation patterns.
When should I prefer per-channel over per-tensor scaling?: Prefer per-channel scaling where different channels have very different magnitudes, especially linear-layer weights, attention projections, and MLP projections. Per-tensor scaling is simpler, but one outlier can force the whole tensor to use a wide range, reducing precision for normal values. It is commonly misunderstood as a universal switch; its value depends on the tensor and kernel support.
How should quality be reported after quantisation?: Report the delta from the exact unquantised model you would have served, using the exact quantised artefact and runtime configuration. Include held-out task results and distributional checks, not only an average score. For instruction systems, also test formatting, tool-call exactness, code execution, retrieval answers, long-context paths, and safety or refusal regressions.
Can calibration make a quantised model identical to the original?: No. Calibration chooses the least damaging integer representation for observed distributions; it does not remove quantisation error. If production inputs stay close to the calibrated distribution, the error may be small enough to ignore. If prompts hit unseen ranges or narrow logit margins, the same calibration can produce visible regressions.

Every term