05.05 · Walkthrough
Calibration and Quality
Choose a calibration set, pick per-channel over per-tensor scaling where it matters, and measure the accuracy cost honestly rather than quoting the paper's number.
Calibration is the release decision that decides whether quantisation preserves the model you meant to serve. The set must match real prompts, scaling should be chosen where tensor structure demands it, and quality must be measured on the deployed artefact, not borrowed from a paper or checkpoint report.
What this lesson answers
- how to choose a quantisation calibration set
- per channel versus per tensor quantisation scaling
- how to measure quantisation accuracy loss honestly
Notes
Post-training quantization replaces a floating tensor by an integer tensor plus scale metadata, usually and , with for affine per-tensor calibration or for symmetric per-channel weight calibration. The calibration set is the data used to estimate those ranges, clipping thresholds, activation outlier statistics, or learned reconstruction parameters before serving.
Common questions
- What should go into a calibration set for quantisation?
- Use samples that exercise the distributions your server will actually see: chat templates, tool calls, code, retrieval text, long prompts, multilingual input, and near-limit context lengths if those exist in production. A small but representative set is better than a larger generic corpus that misses the activation ranges created by your real traffic.
- When is per-channel scaling worth using instead of per-tensor scaling?
- Per-channel scaling is most valuable when channels have very different ranges, especially in linear weights, attention projections, and MLP projections. Per-tensor scaling lets one outlier force a coarse scale for the whole tensor. Per-channel scales let ordinary channels keep more useful integer resolution, usually for little metadata cost.
- How should quantisation quality be measured before deployment?
- Measure the exact artefact and serving path you plan to run: tokenizer, chat template, kernels, cache format, parallelism, and sampling defaults. Compare it with the unquantised model on held-out data, using task metrics plus distribution checks such as token agreement, KL movement, formatting validity, refusal behaviour, and long-context regressions.
Short definition: what is Calibration and Quality?