Model compression and quantization quiz

Question 1

What is the fundamental difference between Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT)?

Accepted Answer

PTQ quantizes after training using calibration; QAT simulates quantization during training

Answer

PTQ retrains the model; QAT only calibrates scales

Answer

PTQ works only for activations; QAT only for weights

Answer

They are identical except QAT uses more calibration data

Question 2

In affine (asymmetric) integer quantization, how is a real value x reconstructed from its quantized integer q given scale s and zero-point z?

Accepted Answer

x ≈ s·(q − z)

Answer

x ≈ s·q + z

Answer

x ≈ (q − z)/s

Answer

x ≈ s·q · z

Question 3

Why are activations generally harder to quantize than weights in LLMs, and what does SmoothQuant do about it?

Accepted Answer

It shifts quantization difficulty from activations to weights via per-channel scaling

Answer

It prunes outlier channels entirely

Answer

It keeps activations in FP32 and quantizes only weights

Answer

It clips all activations to a fixed range

Question 4

What core idea distinguishes GPTQ from naive round-to-nearest weight quantization?

Accepted Answer

It compensates remaining weights using Hessian-based error feedback

Answer

It uses QAT-style gradient descent on all weights

Answer

It rounds each weight independently to the nearest level

Answer

It only quantizes the largest-magnitude weights

Question 5

What is the key insight behind AWQ (Activation-aware Weight Quantization)?

Accepted Answer

It identifies salient weight channels via activation magnitude and applies per-channel scaling before quantization

Answer

It keeps salient weights in FP16 and quantizes the rest

Answer

It quantizes weights and activations jointly with QAT

Answer

It uses random projections to compress weights

Question 6

In the NF4 (4-bit NormalFloat) data type used by QLoRA, what assumption makes its quantization levels optimal?

Accepted Answer

Weights follow a zero-mean normal distribution, so levels are normal quantiles

Answer

Weights are uniformly distributed in [-1,1]

Answer

Weights are sparse and mostly zero

Answer

Weights are log-normally distributed

Question 7

How does QLoRA enable fine-tuning a large model in 4-bit while still training effectively?

Accepted Answer

It freezes 4-bit base weights and trains higher-precision LoRA adapters with on-the-fly dequant

Answer

It trains all 4-bit weights directly with QAT

Answer

It converts the whole model to FP32 for training

Answer

It only fine-tunes the embedding layer

Question 8

What is 'double quantization' as introduced in QLoRA, and why is it used?

Accepted Answer

Quantizing the per-block scale constants to save memory

Answer

Quantizing the model twice for accuracy

Answer

Using two different bit-widths per layer

Answer

Running PTQ then QAT sequentially

Question 9

Why does per-channel (or per-group) quantization usually outperform per-tensor quantization for weights?

Accepted Answer

Per-channel fits each channel's range with its own scale, reducing wasted resolution

Answer

Per-tensor uses more memory than per-channel

Answer

Per-channel avoids the need for a zero-point

Answer

Per-tensor cannot represent negative weights

Question 10

What problem does the straight-through estimator (STE) solve in QAT?

Accepted Answer

It approximates the gradient of the non-differentiable rounding op as identity

Answer

It removes the need for a calibration set

Answer

It quantizes gradients to INT8

Answer

It selects the optimal bit-width automatically

Question 11

In knowledge distillation, what role does the temperature parameter T play in the softmax?

Accepted Answer

Higher T softens the distribution to expose inter-class similarity (dark knowledge)

Answer

It controls the learning rate of the student

Answer

It sets the number of teacher layers used

Answer

It scales the bit-width of the student

Question 12

What is the difference between structured and unstructured pruning, and why does it matter for real speedups?

Accepted Answer

Unstructured zeroes individual weights (needs sparse kernels); structured removes whole channels/heads for dense speedups

Answer

Structured prunes individual weights; unstructured prunes channels

Answer

They are identical except in naming

Answer

Structured pruning only applies to embeddings

Question 13

What does NVIDIA's 2:4 structured sparsity require, and what speedup does it target?

Accepted Answer

At most 2 non-zeros per group of 4 (50% sparsity), ~2x matmul speedup

Answer

Exactly 4 non-zeros per group of 2, for 4x speedup

Answer

Random 50% sparsity with no pattern

Answer

75% sparsity for 4x speedup

Question 14

State the Lottery Ticket Hypothesis and its practical implication for pruning.

Accepted Answer

A sparse subnetwork with its original init can match the dense net's accuracy

Answer

Any pruned subnetwork trains better than the dense one

Answer

Pruning always improves generalization

Answer

Random pruning is optimal at high sparsity

Question 15

Why is magnitude-based pruning often suboptimal for fine-tuned models, and what does movement pruning offer instead?

Accepted Answer

Magnitude pruning ignores fine-tuning dynamics; movement pruning scores by gradient-driven change

Answer

Movement pruning keeps the largest-magnitude weights

Answer

They produce identical masks

Answer

Movement pruning requires no training signal

Question 16

What is the LLM.int8() approach for handling activation outliers during INT8 inference?

Accepted Answer

It computes outlier dimensions in FP16 and the rest in INT8, then sums

Answer

It clips all outliers to the INT8 range

Answer

It uses 4-bit for outliers and 8-bit otherwise

Answer

It removes outlier channels by pruning

Question 17

What advantage does FP8 (e.g., E4M3) offer over INT8 for transformer quantization?

Accepted Answer

FP8's exponent gives wider dynamic range, handling outliers better than uniform INT8

Answer

FP8 always uses fewer bits than INT8

Answer

FP8 cannot represent negative numbers

Answer

INT8 has higher dynamic range than FP8

Question 18

Why is KV-cache quantization increasingly important for long-context LLM serving?

Accepted Answer

KV cache grows with sequence length and dominates long-context memory, so quantizing it saves memory/bandwidth

Answer

KV cache size is independent of sequence length

Answer

Only weights, not the KV cache, can be quantized

Answer

KV-cache quantization speeds up only the prefill phase

Question 19

What does 'symmetric' vs 'asymmetric' quantization imply for hardware matmul efficiency?

Accepted Answer

Symmetric (zero-point 0) avoids cross-terms, simplifying integer matmul

Answer

Asymmetric is always faster in hardware

Answer

Symmetric requires storing two scales per tensor

Answer

They have identical compute cost

Question 20

What is the purpose of a calibration dataset in PTQ, and what can go wrong if it is poorly chosen?

Accepted Answer

It estimates activation ranges for setting scales; poor choice mis-sets clipping ranges

Answer

It retrains the model weights

Answer

It is only needed for weight quantization

Answer

It permanently changes the model architecture

Question 21

How does GGUF k-quant (e.g., Q4_K_M) differ from a flat 4-bit quantization?

Accepted Answer

k-quants use block-wise super-blocks with mixed bit allocation across layers

Answer

k-quants use a single scale for the whole tensor

Answer

k-quants are identical to NF4

Answer

k-quants store weights in FP16

Question 22

What is the bit-width vs accuracy trade-off generally observed when quantizing LLM weights to 8, 4, and 2 bits?

Accepted Answer

8-bit is near-lossless, 4-bit is a sweet spot, 2-bit degrades sharply

Answer

2-bit is always as accurate as 8-bit

Answer

4-bit is unusable for LLMs

Answer

Lower bits always improve accuracy

Question 23

Why must BatchNorm typically be folded into the preceding convolution before quantizing a CNN?

Accepted Answer

BN is folded into the conv weights/bias so quantization matches the fused inference op

Answer

BN cannot be quantized at all

Answer

Folding BN increases the number of parameters

Answer

BN must stay in FP32 separately for accuracy

Question 24

What distinguishes static from dynamic activation quantization at inference time?

Accepted Answer

Static uses calibrated fixed scales; dynamic computes activation scales per input at runtime

Answer

Static computes scales at runtime; dynamic precomputes them

Answer

Both compute scales offline

Answer

Dynamic quantization never quantizes weights

Question 25

In LoRA, why does the low-rank update ΔW = BA reduce trainable parameters, and how is it merged at inference?

Accepted Answer

LoRA factors ΔW into B·A with rank r≪d, cutting params; merge as W+ (α/r)BA

Answer

LoRA adds a full-rank delta, increasing parameters

Answer

LoRA prunes the base weights

Answer

LoRA quantizes the base model to 4-bit by definition

Question 26

What is per-token (a.k.a. dynamic per-row) activation quantization and when is it preferred?

Accepted Answer

A separate dynamic scale per token row, handling variable token magnitudes

Answer

A single scale shared by all tokens

Answer

Quantizing only the first token

Answer

A fixed scale derived from weights

Question 27

How does quantization interact with the memory-bandwidth bottleneck in LLM decoding, and why does it speed up inference even when compute is dequantized to FP16?

Accepted Answer

Decoding is memory-bandwidth bound; fewer weight bytes per token speeds it up even with FP16 compute

Answer

Decoding is compute-bound, so quantization only saves storage

Answer

Quantization slows decoding due to dequant overhead always

Answer

Speedup comes only from INT arithmetic, never bandwidth

Question 28

What is mixed-precision quantization and how is the bit-width per layer typically chosen?

Accepted Answer

Bit-widths vary per layer by sensitivity (e.g., Hessian-based) under a size/latency budget

Answer

All layers get the same bit-width chosen randomly

Answer

Only the last layer is quantized

Answer

Bit-width is set by parameter count alone

Question 29

Why can naive 8-bit quantization of small models sometimes hurt accuracy more than for large models, contrary to the LLM outlier narrative?

Accepted Answer

Small models lack redundancy to absorb quantization noise, so error hurts more per parameter

Answer

Small models have more outliers than large ones

Answer

Large models are always harder to quantize

Answer

Bit-width has no relation to model size

Question 30

What is the difference between weight-only quantization (W4A16) and full quantization (W8A8) in terms of goals?

Accepted Answer

W4A16 (weights only) targets bandwidth-bound decode; W8A8 enables INT matmul for compute-bound cases

Answer

W4A16 quantizes activations only; W8A8 weights only

Answer

They are two names for the same thing

Answer

W8A8 keeps weights in FP16

Test yourself