Model compression and quantization

PTQ quantizes a fully trained model's weights/activations without further gradient updates, using a small calibration set to set scales; it is fast but can lose accuracy at low bit-widths. QAT inserts simulated (fake) quantization ops during training/fine-tuning so the model learns weights robust to quantization noise via straight-through estimators, typically recovering more accuracy at 4-bit or below at the cost of a training run.

The dequantized value is x ≈ s·(q − z), where s is a positive floating-point scale and z is an integer zero-point that maps the real zero exactly to an integer level. Asymmetric quantization uses both s and z to cover ranges that are not symmetric about zero (e.g., ReLU activations), whereas symmetric quantization forces z=0.

# Affine INT8 quantize / dequantize
qmin, qmax = -128, 127
s = (x.max() - x.min()) / (qmax - qmin)      # scale
z = round(qmin - x.min() / s)                # zero-point (int)
q = np.clip(np.round(x / s + z), qmin, qmax) # quantize
x_hat = s * (q - z)                          # dequantize

Activations contain large-magnitude outlier channels (especially after LayerNorm in big transformers) that blow up the dynamic range, making per-tensor activation quantization lossy, while weights are comparatively smooth. SmoothQuant migrates quantization difficulty from activations to weights by dividing activations by a per-channel smoothing factor and multiplying the corresponding weight rows by it, equalizing ranges so both can be quantized to INT8 with little loss.

GPTQ quantizes weights one column (or group) at a time and, after rounding each, updates the remaining unquantized weights to compensate for the introduced error, using second-order (approximate Hessian/inverse-Hessian) information derived from calibration activations. This error-compensation makes 3-4 bit weight quantization far more accurate than independent round-to-nearest.

AWQ observes that a small fraction of weight channels are salient because they multiply large-magnitude activations, and protecting these channels preserves most accuracy. Rather than keeping them in higher precision, AWQ scales salient weight channels up (and the corresponding activations down) before quantization so the important weights incur less relative rounding error, reducing loss without mixed-precision storage.

NF4 assumes weights are approximately zero-mean normally distributed, so its 16 quantization levels are placed at the quantiles of a normal distribution rather than uniformly. This makes each level carry roughly equal probability mass for Gaussian-like weights, minimizing expected quantization error (an information-theoretically motivated choice for normal data).

QLoRA freezes the base model stored in 4-bit NF4 and only trains small LoRA adapter matrices in higher precision (e.g., bf16); during the forward/backward pass the 4-bit weights are dequantized on the fly. It adds double quantization (quantizing the quantization constants) and paged optimizers to fit large models on a single GPU while gradients flow only through the adapters.

Double quantization quantizes the quantization metadata itself—specifically the per-block scaling constants (absmax values)—using a second, lower-precision quantization. Because block-wise quantization stores one FP32 scale per small block (e.g., 64 weights), these constants add overhead; double quantization compresses them (e.g., to 8-bit with a block size of 256), saving roughly 0.37 bits per parameter on average per the QLoRA paper.

A single per-tensor scale must cover the widest range across all channels, so channels with smaller ranges waste resolution and incur higher relative error. Per-channel/per-group quantization assigns each channel (or small group) its own scale, tightly fitting each range and substantially reducing error, at a modest cost of storing more scale values.

The rounding operation in quantization has zero gradient almost everywhere, which would block backpropagation. The STE approximates the gradient of the round/quantize function as the identity (passing the upstream gradient straight through, often only within the clipping range and zeroed outside it), allowing the network to be trained despite the non-differentiable quantizer.

A higher temperature T softens the teacher's output distribution, amplifying the relative probabilities of non-top classes (the 'dark knowledge') so the student learns inter-class similarities. The distillation loss on softened logits is typically scaled by T² to keep gradient magnitudes comparable to the hard-label cross-entropy term when combining the two losses.

Unstructured pruning zeroes individual weights, producing sparse matrices that need specialized sparse kernels/hardware to realize speedups (otherwise they only save storage). Structured pruning removes whole units—channels, heads, filters, or layers—shrinking dense tensor dimensions so standard dense hardware sees direct latency and memory gains, usually at a higher accuracy cost per unit of sparsity.

2:4 (fine-grained) sparsity requires that in every contiguous group of 4 weights, at most 2 are non-zero, giving 50% sparsity in a hardware-friendly pattern. Ampere+ Tensor Cores exploit this pattern to roughly double matmul throughput (up to ~2x) while a compressed format stores only the non-zeros plus 2-bit indices.

The Lottery Ticket Hypothesis claims a randomly-initialized dense network contains a sparse subnetwork ('winning ticket') that, when trained in isolation from the same initial weights, can match the full network's accuracy in a comparable number of iterations. Practically it motivates iterative magnitude pruning with weight rewinding to the original (or early) initialization rather than fine-tuning the pruned weights from their trained values.

Magnitude pruning keeps weights with large absolute values, but during transfer/fine-tuning the important weights are those that move away from zero, not necessarily those that started large. Movement pruning scores weights by how their magnitude changes during fine-tuning (using gradient-based importance), better identifying task-relevant weights and outperforming magnitude pruning in high-sparsity transfer settings.

LLM.int8() performs mixed-precision decomposition: it isolates the small number of outlier feature dimensions (those exceeding a threshold) and computes their matmul contribution in FP16, while the vast majority of non-outlier dimensions are computed in INT8 with vector-wise quantization. The two results are summed, preserving accuracy for large transformers that otherwise degrade catastrophically under plain INT8.

FP8's floating-point format devotes bits to an exponent, giving a much wider dynamic range than INT8's uniform spacing, so it represents both small and large (outlier) magnitudes with relatively low error without heavy per-tensor scale tuning. E4M3 (1 sign, 4 exponent, 3 mantissa) suits weights/activations needing precision, while E5M2 trades mantissa for range and suits gradients; modern accelerators have native FP8 Tensor Cores.

During autoregressive decoding, the key/value cache grows linearly with sequence length and batch size and often dominates memory for long contexts, becoming the main bottleneck. Quantizing the KV cache to INT8/INT4 (frequently per-token or per-channel) shrinks this memory and bandwidth, enabling longer contexts and larger batches; keys are typically harder to quantize than values due to outlier channels.

Symmetric quantization (zero-point = 0) makes the dequantized matmul a simple scale times an integer dot product, avoiding cross-terms. Asymmetric quantization's nonzero zero-point introduces extra correction terms (zero-point times sums) that must be precomputed or folded, adding overhead—hence weights are often quantized symmetrically while activations may use asymmetric to fit skewed ranges.

Calibration data is a small, representative sample run through the model to estimate activation ranges (min/max or percentiles) used to set quantization scales/clipping. If it is unrepresentative or too small, it can under- or over-estimate the true dynamic range—causing clipping of important values or wasted resolution—and yield activation distributions that don't match deployment, degrading accuracy.

GGUF k-quants use block-wise quantization with super-blocks and a secondary quantization of per-block scales/mins, and mixed bit-widths across tensor types (the 'M'/'S'/'L' variants assign more bits to more sensitive layers like attention and feed-forward). This gives better accuracy per average bit than uniform flat 4-bit by spending precision where it matters most.

INT8 weight quantization is typically near-lossless for LLMs; 4-bit (with good methods like GPTQ/AWQ and grouping) loses little perplexity and is the common deployment sweet spot; 2-3 bit causes noticeable degradation that requires advanced techniques (e.g., learned/vector quantization, fine-tuning) to remain usable. Returns diminish sharply below 4 bits as quantization noise approaches signal magnitude.

At inference, BatchNorm is an affine per-channel scale-and-shift that can be mathematically merged into the conv's weights and bias, removing a runtime op. If not folded, quantizing conv and BN separately introduces extra rounding points and the BN statistics interact badly with quantized weights; folding first ensures the quantized weights reflect the actual deployed (fused) operation.

Static quantization precomputes fixed activation scales offline via calibration, so inference uses constant scales and pure integer kernels—fastest, but sensitive to distribution shift. Dynamic quantization computes activation scales on the fly per input (e.g., per-tensor/per-token at runtime), adapting to the actual range at some compute overhead; weights are usually pre-quantized in both cases.

LoRA replaces a full d×k weight delta with the product of two low-rank matrices B (d×r) and A (r×k) where r ≪ min(d,k), so trainable parameters drop from d·k to r·(d+k). At inference the adapter can be merged by computing W' = W + (α/r)·BA once, adding zero latency, or kept separate to swap adapters dynamically.

# LoRA forward (B init 0, A random -> delta starts at 0)
class LoRALinear(nn.Module):
    def __init__(self, W, r, alpha):
        self.W = W                       # frozen d x k
        self.A = nn.Parameter(torch.randn(r, W.shape[1]) * 0.01)
        self.B = nn.Parameter(torch.zeros(W.shape[0], r))
        self.scale = alpha / r
    def forward(self, x):
        return x @ self.W.T + self.scale * (x @ self.A.T) @ self.B.T

Per-token quantization assigns a separate scale to each token's activation vector (each row of the activation matrix), computed dynamically at runtime. It is preferred for transformer activations because token magnitudes vary widely and a shared per-tensor scale would over-clip or waste range; combined with per-channel weight quantization it yields accurate W8A8 inference (e.g., SmoothQuant-style).

Autoregressive decoding with small batch size is memory-bandwidth bound: each token requires reading all weights from memory, so throughput is limited by how fast weights can be loaded. Quantizing weights to 4-bit cuts the bytes read per token by ~4x, directly improving decode latency even if matmuls are dequantized to FP16 for compute—the win comes from reduced weight traffic, not arithmetic.

Mixed-precision quantization assigns different bit-widths to different layers/tensors based on their sensitivity, spending more bits where accuracy loss is highest. Sensitivity is estimated via metrics like the Hessian trace/eigenvalues (e.g., HAWQ), per-layer perplexity impact, or quantization error, then a constrained optimization picks bit allocations meeting a model-size/latency budget.

Large models develop systematic, sparse outlier channels that targeted methods (LLM.int8, SmoothQuant) handle well, whereas small models lack redundancy so every weight carries more information and uniform quantization error is less easily absorbed. With fewer parameters there is less over-parameterization to compensate for rounding noise, so the same bit-width can cause proportionally larger degradation.

W4A16 quantizes only weights (to 4-bit) while keeping activations in FP16, targeting memory-bandwidth-bound decoding where reducing weight traffic gives the biggest latency win and accuracy stays high. W8A8 quantizes both weights and activations to INT8 to enable integer Tensor Core matmuls, maximizing compute throughput for prefill/large-batch (compute-bound) scenarios at the cost of harder activation quantization.

One-shot pruning removes a large fraction of weights at once, causing a big accuracy drop the network may not recover from. Iterative pruning removes a small percentage, fine-tunes to recover, and repeats over several cycles, letting the network gradually redistribute representational load; this gradual schedule reaches much higher sparsity for the same accuracy.

Clipping caps the represented range so outliers don't stretch the scale and starve normal values of resolution; the trade-off is clipping error (lost outlier info) vs rounding error (coarser steps). MSE calibration picks the threshold minimizing total quantization MSE, while percentile calibration clips at, say, the 99.9th percentile of observed activations—both sacrificing rare extremes to better quantize the bulk.

Quantization and pruning shrink and speed up a model but can leave an accuracy gap, while distillation transfers the original (teacher) model's soft knowledge to recover that gap by training the compressed student to match teacher outputs/features. The combination—e.g., distilling into a pruned or quantized student—often retains more accuracy than any single technique at a given compression ratio.

Vector/product quantization encodes groups of weights jointly by mapping each sub-vector to the nearest entry in a learned codebook, storing only the codebook index. Because it exploits correlations among grouped weights and places codewords optimally in multi-dimensional space, it achieves far lower error than independent scalar rounding at sub-2-bit effective rates, enabling extreme LLM compression like 2-bit AQLM.