Quantization: Running Models With Fewer Bits

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. ONNX QuantizeLinear: quantization granularity

   For integer quantization, QuantizeLinear rounds x/scale to nearest-even, adds zero_point, and saturates to the output type's range. A scalar scale gives per-tensor quantization. A vector of scales along an axis gives per-axis quantization, commonly per-channel when that axis denotes channels. Blocked quantization shares parameters across blocks: an axis of length D with block size B has ceil(D/B) scale positions. Supplied zero points have the scale tensor's shape. Storage inference: N weights at b bits contribute approximately Nb/8 payload bytes, but scale tensors, optional zero points, tensor shape/type information, operator attributes, and container overhead prevent b alone from specifying serialized size.

2. Learning with Limited Numerical Precision Using the Cascade-Correlation Algorithm

   This Carnegie Mellon report investigated whether shorter fixed-point representations could accommodate more weights and processing elements in specialized hardware. Its experiments found that rounding small weight updates away could stall learning. Per-unit rescaling managed weight ranges, while probabilistic rounding allowed small updates to affect stored weights occasionally. Cascade-Correlation, the tested architecture, adds hidden units during training; additional units could sometimes compensate for reduced precision. The report is dated May 3, 1991, and explicitly places the experiments in fall 1990.

3. Deep Learning with Limited Numerical Precision

   IBM researchers' ICML 2015 study revisited fixed-point computation for deeper image-classification networks. Fixed point allocates bits between integer and fractional positions: fractional length F gives uniformly spaced values separated by 2^-F. At fixed total width, increasing fractional precision reduces available range. The experiments show why rounding policy matters: updates smaller than half a quantization step disappear under nearest rounding, whereas stochastic rounding sometimes preserves their effect. The computation uses wider intermediate accumulation before rounding or saturation, so a 16-bit representation does not mean every intermediate has 16 bits.

4. A Practical Guide to Efficient AI

   Weight quantization reduces the precision used to represent model weights, shrinking their storage and memory requirements.

5. TensorRT 10.x: Working with Quantized Types

   The documented INT4 path uses signed codes [-8,7], block scales with group sizes 64 or 128, and grouping along one of the last two dimensions. Two values are packed per byte. Weight-only GEMM reads these compressed weights and dequantizes them before a higher-precision dot product; lower storage precision therefore does not imply INT4 multiplication. Q/DQ graph nodes can fuse into quantized operators rather than execute as separate conversions. Unsupported low-precision implementations can cause engine-build failure. The same guide's dynamic block schemes calculate scales from runtime maxima; its NVFP4 recipe combines those with an offline-calibrated global scale.

6. MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

   MobileAIBench separates desktop task-quality evaluation from resource measurements in an iOS application using llama.cpp. Its four-bit TinyLlama 1B artifact occupies a reported 0.6 GB on disk, while its HotpotQA run on an iPhone 14 reports 3.34 GiB RAM usage. The same run reports 1.60 seconds to first token and 28.14 output tokens per second. These are different properties of the deployed configuration: compact weight storage does not describe the application's complete memory requirement.

7. Distilling the Knowledge in a Neural Network

   Distillation trains a receiving model using outputs from another model or ensemble as supervision. In this paper, predicted class distributions provide soft training targets. The mechanism changes what a student learns; it does not specify a lower-bit encoding of the original model's numerical values.

8. Quantizing for Minimum Distortion

   Joel Max's March 1960 paper, produced at MIT Lincoln Laboratory, studies how to allocate a fixed number of quantization levels to minimize expected signal distortion. Digital transmission maps an interval of input amplitudes to one code; reconstruction assigns that code one value, irreversibly losing distinctions within the interval. For squared error, neighboring reconstruction values determine midpoint decision boundaries, while each reconstruction value is the probability-weighted mean within its interval. Max numerically tabulates Gaussian-input solutions and separately examines equally spaced levels. Uniform spacing is therefore a design constraint, not generally the minimum-distortion allocation for every distribution.

9. BinaryConnect: Training Deep Neural Networks with binary weights during propagations

   Matthieu Courbariaux, Yoshua Bengio and Jean-Pierre David introduced BinaryConnect in 2015 to investigate computation suited to low-power hardware. Forward and backward propagation use weights restricted to minus one or plus one, but parameter updates accumulate in higher-precision weights so small updates are not immediately discarded. Deterministic binarization uses the sign; a stochastic alternative samples the binary value. The paper reports image-classification experiments on MNIST, CIFAR-10 and SVHN. Its central separation is between the representation used in propagation and the state used to learn that representation, rather than simply rounding a completed model.

10. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

    Benoit Jacob and colleagues at Google published the inspected preprint on December 15, 2017. Their motivation included evaluating already-efficient MobileNets on ordinary mobile CPUs: replacing multiplication with shifts was not automatically advantageous on hardware with efficient multiply-add instructions.

11. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — original preprint

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh introduced GPTQ in an October 31, 2022 preprint. Its problem was preparation scalability: retraining giant models was expensive, while more accurate one-shot quantizers were difficult to scale beyond smaller networks.

12. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

    Affine quantization represents a real value approximately as r=S(q-Z), where q is an integer, S>0 is scale, and Z represents real zero. Encoding rounds to a discrete level and clips outside the representable range. The paper uses eight-bit weights and activations with wider accumulators and integer rescaling, enabling integer matrix products instead of floating-point inference. Training simulates quantization while retaining floating-point optimization, and activation ranges are estimated during training. Accuracy depends on ranges, rounding, and quantization-aware adaptation; deployment speed depends on efficient kernels and hardware support for the resulting arithmetic.

13. OCP Microscaling Formats (MX) Specification, Version 1.0

    The September 2023 OCP MX specification, contributed by AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm, defines numerical encodings and basic operations for shared-scale blocks. Its concrete formats use 32 elements per block with an eight-bit E8M0 scale. MXFP4 combines that scale with four-bit E2M1 elements, giving 4.25 encoded bits per value before other storage costs. The specification deliberately leaves physical memory layout unspecified: scales may be stored beside elements or separately. Implementations may support only a subset of the defined formats. A shared numerical specification therefore does not establish identical packing or universal execution support.

14. FP8 Formats for Deep Learning

    The proposed FP8 encodings allocate one sign bit and either four exponent plus three fraction bits (E4M3), or five exponent plus two fraction bits (E5M2). Their maximum finite magnitudes are 448 and 57,344 respectively. Thus equal storage width does not imply equal range or resolution. The paper separates the interchange encoding from rounding and conversion policies. Its usage scheme scales higher-precision values before casting to FP8 and restores scale after conversion or a linear operation; arithmetic outputs can remain higher precision.

15. NVIDIA TensorRT: Accuracy Considerations

    FP32 means single-precision 32-bit floating point; FP16 is 16-bit half precision. BF16 is Brain Floating Point, a 16-bit format with a wider exponent field but fewer fraction bits than FP16. Format choice therefore changes representable range and precision. Overflow concerns results beyond a format's finite range; the guide describes FP16 overflow producing infinity and subsequent errors leading to NaN, meaning not-a-number. Storage format and the precision used for intermediate arithmetic can be selected separately rather than inferred from the input type.

16. LiteRT 8-bit quantization specification

    LiteRT specifies symmetric INT8 weights using codes [-127,127] and zero point zero, while activations use [-128,127] with potentially nonzero zero points. Weights are constant, whereas activations change with inference inputs. Setting the weight zero point to zero removes a correction term that otherwise depends on runtime activations. For CONV_2D, weight scales follow output-channel axis 0; bias is INT32 with zero point zero and scale equal to input scale times the corresponding weight scale. The output returns to per-tensor INT8. Some operators, including concatenation, require matching input and output quantization parameters.

17. gemmlowp: Building a quantization paradigm from first principles

    An accumulator holds the sum of products in a dot product. For affine-quantized operands, the integer sum is A=sum_i((q_xi-z_x)(q_wi-z_w)); its reconstructed value is s_x*s_w*A. Encoding that result with output parameters requires scaling A by s_x*s_w/s_y and adding z_y, with the output pipeline supplying integer rescaling. gemmlowp uses an INT32 accumulator and moves zero-point handling out of the innermost multiplication loop. Constructed example: centered operands [2,-1] and [3,4] give A=2; scales 0.5 and 0.25 reconstruct 0.25. Output scale 0.125 and zero point 0 give code 2.

18. Attention Is All You Need

    Learned embeddings map token identities to vectors; positional information distinguishes their places in a sequence. The original Transformer combines attention sublayers, position-wise feed-forward networks, residual connections, and normalization. Scaled dot-product attention computes QK-transpose divided by the square root of the key dimension, applies softmax, and uses the resulting weights to combine V. Multiple heads use different learned projections before their outputs are combined. A causal mask prevents decoder positions from using future tokens. A learned output projection and softmax convert decoder representations into next-token probabilities. These are numerical transformations, not lookups of verified facts. Section 3.3 specifies FFN(x)=max(0,xW1+b1)W2+b2: two affine transformations separated by ReLU, applied independently at each position with shared parameters across positions but different parameters across layers. Section 3.1 places LayerNorm after residual addition: LayerNorm(x+Sublayer(x)).

19. LiteRT: Post-training quantization

    Activations are outputs of intermediate layers and depend on model inputs. LiteRT's dynamic-range conversion quantizes constant weights without a representative calibration dataset; supported dynamic-range operators then quantize activations during inference and retain floating-point outputs. Full integer conversion instead runs representative inputs to estimate variable-tensor ranges. The documented representative-data interface supplies inputs without requiring labels. Conversion can permit floating-point operators where integer implementations are missing, or explicitly require integer operators and interfaces. The guide recommends first converting an unquantized model to separate conversion defects from quantization defects.

20. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

    SmoothQuant observes activation channels with much larger magnitudes than the bulk, which consume a shared quantization range and leave few levels for smaller channels. It transforms a linear layer using XW=(X diag(s)^-1)(diag(s)W). The unquantized product is algebraically unchanged, but the two operands acquire different ranges and therefore different quantization errors. Calibration estimates channel magnitudes; a migration parameter balances activation and weight difficulty. Compatible preceding operations absorb scaling offline. W8A8 denotes eight-bit weights and eight-bit activations. The paper also distinguishes per-token activation scales from per-output-channel weight scales and explains why scaling along the reduction dimension conflicts with its efficient INT8 GEMM path.

21. Hacking the Inference Pareto Frontier

    Disaggregation lets prefill and decode use different resource allocations and avoids competition between their scheduling needs.

22. From model weights to API endpoint with TensorRT-LLM

    The workshop configures weights and KV cache separately and argues that quantizing both can unlock faster computation on supported hardware.

23. Practical Quantization in PyTorch

    Quantization calibration selects clipping ranges. Observer modules gather value statistics and calculate quantization parameters; the guide discusses observed extrema, histogram methods, percentile ranges, and error-based criteria. Its published observer example produces different scales and zero points for the same input tensors. For sensitivity diagnosis, the guide supplies a loop that quantizes one named layer at a time, calibrates it, and evaluates the resulting model. It separately compares weights and intermediate outputs using numerical-error statistics. These comparisons locate numerical changes; the evaluation step tests their effect on the model.

24. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    The 2022 study found systematic large-magnitude activation features that defeated straightforward quantization in its larger transformers. LLM.int8() combines separate activation-row and weight-column scales with mixed-precision decomposition: selected outlier dimensions use FP16 multiplication, while the remainder uses INT8 with INT32 accumulation before rescaling and combination. Its BLOOM-176B benchmark separates capacity from speed. At batch size one, BF16 on eight A100 80GB GPUs took 239 milliseconds per generated token; LLM.int8() took 253 milliseconds on the same eight GPUs and 247 milliseconds on three. Fitting the model on fewer devices did not imply lower token latency.

25. Compression at the Edge

    Use mixed-precision quantization guided by sensitivity rather than compressing every layer equally.

26. Compression at the Edge

    Quantization sensitivity can involve combinations of layers and individual tensor values, making independent layer checks incomplete.

27. Compression at the Edge

    The panel recommends trying post-training quantization (PTQ) with selective precision for medium and large models before assuming retraining is necessary.

28. Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLMs

    The study varies calibration language for four-bit GPTQ and AWQ with group size 128, primarily using Llama 3.1 8B Instruct and Qwen 2.5 7B Instruct. It reports holding calibration token budgets constant and evaluates multilingual Wikipedia and C4 material separately. Language-matched and multilingual calibration often improve perplexity relative to English-only calibration, but exceptions depend on model and quantizer. Downstream tests also respond differently: language alignment helps some tasks, while diverse mixtures help others. Calibration language is therefore a conversion choice whose benefits need evaluation on the intended languages and tasks.

29. Quantize ONNX models — ONNX Runtime

    Linear quantization represents an approximate floating-point value as scale times an integer offset from a zero point. Fewer representable values reduce precision. Static activation quantization estimates parameters from calibration inputs and stores them for reuse; dynamic quantization calculates parameters during inference, adding work. Quantization-aware training retrains with quantization effects considered and is distinct from post-training conversion. The documentation recommends comparing original and quantized weights and activations to localize accuracy loss and keeping problematic tensors at higher precision when needed.

30. Special topics in Kernels, RL, Reward Hacking in Agents

    The speaker's dynamic quantization approach assigns different precisions to different layers instead of compressing every layer equally.

31. Cross-validation and held-out evaluation

    Testing on data used to fit or repeatedly tune a system can overestimate generalization. The documentation separates training, validation for selection, and an untouched final test set; cross-validation does not eliminate the need for final held-out evaluation. Applied to model judges, tune rubrics and examples on calibration data, freeze the judge, then measure agreement with independent human labels on unseen examples. Applied to context policies, select retrieval and compression settings on development tasks and evaluate the selected policy on held-out tasks. Repeatedly changing settings after seeing test results converts that test set into development data.

32. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    GPTQ minimizes layer-output reconstruction error ||WX-W_hat X||² using calibration-derived layer inputs X, rather than minimizing weight differences alone. With a fixed quantization grid, it rounds weights and adjusts remaining weights to compensate, using second-order information derived from XX-transpose. Shared column ordering, batched updates, and a damped Cholesky formulation make this preparation tractable. The reported setup uses 128 C4 segments of 2,048 tokens and processes one Transformer block at a time; subsequent blocks receive outputs from already quantized blocks. The original method quantizes weights, not activations, and its inference implementation gains from reduced memory loading rather than faster mixed INT4–FP16 multiplication.

33. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration

    AWQ is weight-only quantization informed by activation statistics. Its experiments found that protecting channels selected through activation magnitude helped more than selecting through weight magnitude alone. AWQ avoids storing those channels separately in FP16 by scaling weights and inversely scaling their input activations. It searches per-input-channel scaling choices against layer-output differences on cached calibration inputs and additionally selects weight clipping. Thus, its avoidance of backpropagation or GPTQ-style weight compensation does not mean it ignores output reconstruction error. Excessive scaling can enlarge a group's range and harm other weights. TinyChat integrates dequantization into computation and uses device-specific packing to turn compact weights into useful execution.

34. Data-Free Quantization Through Weight Equalization and Bias Correction

    Qualcomm's 2019 work addressed a deployment obstacle: hardware providers might receive trained models without access to customer training data or resources for fine-tuning. Its cross-layer equalization exploits ReLU's positive scaling property. Scaling one layer's output channels and inversely scaling the following layer's corresponding inputs preserves the unquantized computation while redistributing weight ranges. This helps channels use a shared quantization grid more effectively. The paper also demonstrates that weight quantization can introduce systematic shifts in layer outputs, rather than errors that necessarily cancel.

35. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

    SmoothQuant was developed by Guangxuan Xiao, Ji Lin and colleagues at MIT and NVIDIA. Its November 18, 2022 preprint addressed the implementation cost of separating outliers into higher-precision computation, seeking a training-free path to hardware-friendly INT8 execution.

36. Quantization-Aware Training for Large Language Models with PyTorch

    Fake quantization rounds and clamps values, then reconstructs them while retaining floating-point tensors. A straight-through estimator supplies approximate gradients through nondifferentiable quantization operations. TorchAO's described prepare step inserts this simulation; convert replaces it with actual quantization after training. The reported Llama3-8B deployment uses dynamic per-token INT8 activations and grouped INT4 weights, with additional INT4 embeddings. After lowering through ExecuTorch to XNNPACK, reported Wikitext word perplexity is 23.316 for PTQ versus 19.403 for QAT, with both artifacts at 3.881 GB. The broader fine-tuning experiments use 6–8 A100 GPUs for 5,000 steps.

37. Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or -1

    Courbariaux, Hubara, Soudry, El-Yaniv and Bengio's 2016 Binarized Neural Networks work applies binary representations to activations as well as weights, extending the propagation approach used in BinaryConnect. Binary dot products can use XNOR and population-count operations. Training retains real-valued update state. Because the sign function has zero derivative almost everywhere, the method substitutes a straight-through gradient that passes information within a bounded interval and cancels it outside. The paper demonstrates training on MNIST, CIFAR-10 and SVHN; its inference algorithm treats nonbinary first-layer inputs separately.

38. Quantization aware training comprehensive guide

    TensorFlow's guide separates supported deployment configurations from experimental quantization configurations. Its customizable QuantizeConfig controls weight, activation and output quantizers during forward computation, but the guide explicitly warns that experimental configurations and custom layers lack a supported deployment path. A configuration that can be simulated during training therefore does not, by itself, establish that conversion and backend kernels can execute it.

39. Compression at the Edge

    Quantization-aware distillation (QAD) can degrade a model when its training data does not match the capabilities produced by the original training process.

40. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference

    NVFP4 combines four-bit E2M1 values, an eight-bit E4M3 scale for each 16-value block, and a second-level FP32 scale per tensor. Its positive representable magnitudes include 0, 0.5, 1, 1.5, 2, 3, 4, and 6, illustrating nonuniform floating-point spacing. Block payload plus local scales costs 4+8/16=4.5 bits per value, before the per-tensor scale. This supplies a concrete example where the named four-bit format does not mean four effective storage bits per value.

41. vLLM GPU worker: profiling available KV capacity

    The GPU worker profiles a model forward pass and, when configured, estimates CUDA-graph memory separately. Its available KV-cache budget subtracts profiled non-KV consumption and the applied graph estimate from requested memory. Later warmup compares actual graph-pool use with its estimate and reserves additional headroom in suggested allocations. This connects weights, activations, runtime buffers and graph storage to the space left for request state. An explicit cache-byte override skips normal capacity inference and requires a suitable value.

42. Marlin: FP16×INT4 inference kernel

    Marlin reshuffles quantized weights and group scales offline into its kernel layout and schedules dequantization alongside Tensor Core computation. Its published A10 comparison uses group size 128 and a large, favorably partitioned matrix. The authors report that competing four-bit kernels lose much of their relative speedup as batch size increases, while Marlin remains near its approximately 3.87-fold weight-traffic limit through batches of roughly 16–32 tokens. The scale metadata adds 0.125 bits per weight, explaining why the idealized reduction from FP16 is below fourfold. The repository separately benchmarks smaller model-layer matrices and locked GPU clocks.

43. What every AI engineer needs to know about GPUs

    Evaluate arithmetic work relative to memory traffic, not operation count alone: GPUs favor substantial computation on data that has already been loaded.

44. From model weights to API endpoint with TensorRT-LLM

    Batch-size effects can have sharp latency changes and eventually encounter GPU-memory limits.

45. What every AI engineer needs to know about GPUs

    The talk describes prompt processing as a better fit for GPU computation than decoding, where repeatedly moving model weights can dominate.

46. Optimization and tuning — vLLM

    Prefill processes prompt tokens; decode generates continuations incrementally. Chunked prefill splits long prompt processing so a scheduler can mix it with decode work. In the documented scheduler, decode is prioritized and remaining token budget admits prefill chunks. Smaller batch-token budgets can improve inter-token latency, while larger ones can improve time to first token and throughput. Insufficient KV space can force preemption and recomputation.

47. NVIDIA Model Optimizer: PTQ recipes and schemes

    Current recipes separately specify model-body precision, scope and KV-cache precision. Examples include NVFP4 W4A4, weight-only W4A16, and expert-only schemes. Format and kernel support depend on the deployment target; listed recipes are not measured endpoint performance.

48. Metrics design — vLLM

    Serving telemetry distinguishes queue time, prefill, decode, time to first token, inter-token latency and end-to-end latency. Event boundaries and observation location matter: client network time differs from engine intervals. A successful finish reason does not establish task correctness. Request lengths, waiting/running requests and KV usage help explain a latency change.

49. PyTorch: torch.testing

    For finite real values, assert_close accepts an element when |actual−expected| ≤ atol + rtol·|expected|. Absolute tolerance therefore controls acceptance near a zero reference, while relative tolerance scales with reference magnitude. Infinities must match exactly; NaNs match only when explicitly enabled. Device, dtype, layout, and optionally strides have separate checks. A numerical closeness check therefore has both value and representation semantics.

50. Compression at the Edge

    Combine benchmark regression checks with execution in the intended application harness.

51. FP8 Quantization — TensorRT-LLM

    TensorRT-LLM's guide documents KV-cache precision as a separate configuration from model quantization. It also describes an FP8 GEMM-plus-SwiGLU fusion that can improve performance while slightly reducing accuracy because a quantization scaling factor is discarded. This is a concrete case where changing execution optimizations can change numerical behavior even without changing the headline precision label.

52. Accuracy is Not All You Need

    The study distinguishes capability metrics from differences relative to a baseline. Its flips metric counts answers changing between correct and incorrect in either direction; gains and regressions can offset in aggregate accuracy while affecting different cases. It defines perplexity as the exponential of average negative log likelihood over a dataset and separately evaluates free-form generation. Across its tested compressed models, the authors observe behavioral changes despite similar aggregate accuracy and propose reporting flips and distribution distances alongside capability scores.

53. Evaluating Quantized Large Language Models

    The study evaluates weight, activation and KV-cache quantization separately, including key-value retrieval up to 16K tokens and multi-document question answering up to 6K. Longer inputs expose additional degradation for some weight-only and KV-cache configurations; sensitivity differs between model families. In its 16K LongEval comparison, Vicuna-7B scores 57.80% with FP16, 56.40% with eight-bit weights and eight-bit KV cache, and 37.00% with eight-bit weights and four-bit KV cache. This isolates a consequential cache-precision change that a weight-only label would miss.

54. Compression at the Edge

    Choose the default precision based on usable behavior, retaining higher precision for small models when aggressive quantization harms the experience.

55. A Practical Guide to Efficient AI

    Evaluate task quality and trust and safety alongside device latency, hardware usage, and battery drain; Mobile AI Bench is presented as tooling for this work.

56. Hacking the Inference Pareto Frontier

    Set required quality and latency from the application experience, then minimize cost within those constraints.

57. 20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna AI

    Use Pareto plots with task-specific quality and an efficiency measure to identify multiple competitive models.

58. Taking Reinforcement Learning Cross Datacenter

    The talk's 'Adam absorption' or 'push versus floor' mechanism explains how small master-weight updates can disappear when projected into serving precision.

59. Compression at the Edge

    NVFP4 is described as combining four-bit floating-point values with shared micro-block scaling.

60. From model weights to API endpoint with TensorRT-LLM

    The speakers attribute improved KV-cache quantization behavior to FP8's nonuniform value spacing and dynamic range.

61. From model weights to API endpoint with TensorRT-LLM

    Consider calibration datasets resembling expected inputs because observed layer ranges can depend on the dataset.

62. From model weights to API endpoint with TensorRT-LLM

    Conversion normalizes model tensors and can also prepare quantized weights and tensor-parallel partitions.