Inference Optimization quiz

Question 1

What is the KV cache and why is it essential for autoregressive LLM inference?

Accepted Answer

KV cache stores past keys and values to avoid recomputing them at each generation step

Answer

KV cache stores model weights in GPU SRAM for faster access

Answer

KV cache reduces model parameter count during inference

Answer

KV cache is only needed for models with more than 32 attention heads

Question 2

What is continuous batching (iteration-level scheduling) and how does it improve GPU utilization?

Accepted Answer

Continuous batching fills batch slots immediately when sequences finish, keeping GPU continuously active

Answer

Continuous batching waits for all requests in a batch to complete before processing new ones

Answer

Continuous batching requires fixed sequence lengths for all requests in the batch

Answer

Continuous batching is only applicable to encoder-only models like BERT

Question 3

What is INT8 post-training quantization (PTQ) and what is the main challenge?

Accepted Answer

INT8 PTQ is challenging due to dynamic activation ranges; LLM.int8() handles outliers with mixed precision

Answer

INT8 PTQ is straightforward because LLM activations have uniform dynamic range

Answer

INT8 PTQ requires full retraining with INT8 arithmetic

Answer

INT8 quantization reduces compute by 8x compared to FP32

Question 4

What is AWQ (Activation-Aware Weight Quantization)?

Accepted Answer

AWQ scales salient weight channels (with large input activations) before quantization to preserve important information

Answer

AWQ randomly selects weights to quantize to INT4 for memory savings

Answer

AWQ requires full fine-tuning after quantization to recover quality

Answer

AWQ is only applicable to embedding layers, not linear layers

Question 5

What is tensor parallelism's role in reducing inference latency for large models?

Accepted Answer

Tensor parallelism distributes memory bandwidth across GPUs, reducing per-token latency for large models

Answer

Tensor parallelism only helps throughput for large batches, not single-request latency

Answer

Tensor parallelism is harmful for inference because all-reduce adds more latency than it saves

Answer

Tensor parallelism for inference requires the same TP degree as training

Question 6

What is TTFT and TPOT in LLM inference serving metrics?

Accepted Answer

TTFT is time to first token (prefill-dominated); TPOT is per-output-token time (bandwidth-dominated)

Answer

TTFT measures total response generation time; TPOT measures prefill latency

Answer

Both TTFT and TPOT are dominated by memory bandwidth at all batch sizes

Answer

TPOT is irrelevant for streaming applications; only TTFT matters

Question 7

What is prefix caching (prompt caching) and which inference engines support it?

Accepted Answer

Prefix caching reuses KV cache for shared prefixes, reducing TTFT to near-zero for cached content

Answer

Prefix caching recomputes KV cache for shared prefixes to ensure correctness

Answer

Prefix caching requires the full KV cache to be stored on CPU RAM

Answer

Prefix caching only works when all requests in a batch have identical prefixes

Question 8

What is quantization-aware training (QAT) and when is it preferred over PTQ?

Accepted Answer

QAT inserts fake quantization during training for better low-bit accuracy; preferred when PTQ quality is insufficient

Answer

QAT is always preferred over PTQ because it trains without quantization noise

Answer

QAT requires no calibration dataset and is faster than PTQ

Answer

QAT is only applicable to convolution layers, not transformer linear layers

Question 9

What is structured pruning vs unstructured pruning for inference?

Accepted Answer

Structured pruning removes whole units (heads, neurons, layers) giving dense models that speed up on standard hardware

Answer

Unstructured pruning provides immediate inference speedup on all hardware

Answer

Both pruning types require sparse CUDA kernels to achieve speedup

Answer

Unstructured pruning always maintains higher model quality than structured pruning

Question 10

What is TensorRT-LLM and what optimizations does it apply?

Accepted Answer

TensorRT-LLM applies INT4/INT8/FP8 quantization, fused kernels, continuous batching, and multi-GPU parallelism

Answer

TensorRT-LLM is a Python library with no CUDA kernel optimizations

Answer

TensorRT-LLM only supports encoder models like BERT

Answer

TensorRT-LLM requires retraining models from scratch for optimization

Question 11

What is EAGLE (speculative decoding with an auto-regressive draft model)?

Accepted Answer

EAGLE trains a draft model operating in the target LLM's hidden state space for higher-quality drafts

Answer

EAGLE uses the same draft heads as Medusa but with a different verification algorithm

Answer

EAGLE is a hardware-level optimization for NVIDIA GPU inference

Answer

EAGLE only works for summarization tasks, not general text generation

Question 12

What is key-value cache quantization and what precision is typically used?

Accepted Answer

KV cache quantization compresses K/V from FP16 to INT8/INT4 to double concurrent sequences or context length

Answer

KV cache quantization affects gradient computation during fine-tuning

Answer

KV cache quantization requires recomputing KV values from scratch every N steps

Answer

KV cache quantization is not possible below INT8 due to attention score sensitivity

Question 13

What is quantization granularity and what are the tradeoffs between per-tensor, per-channel, and per-group?

Accepted Answer

Per-group quantization (e.g., group=128) provides the best quality-compression tradeoff for INT4

Answer

Per-tensor quantization always produces the highest quality at INT4

Answer

Per-channel and per-group quantization produce identical results for INT4 models

Answer

Per-tensor quantization is required for FP8 on Hopper GPUs

Question 14

What is the MLA (Multi-head Latent Attention) in DeepSeek-V2/V3 and how does it reduce KV cache?

Accepted Answer

MLA projects KV to a low-dimensional latent space for caching, achieving ~93% KV cache reduction

Answer

MLA caches full KV vectors but compresses them with lossless encoding

Answer

MLA is equivalent to GQA with 1 KV head per 32 query heads

Answer

MLA requires retraining with frozen attention weights to function correctly

Question 15

What is weight sharing/recycling in speculative decoding (self-speculation)?

Accepted Answer

Self-speculation uses the target model itself for drafting via layer-skipping or Jacobi decoding, needing only one model

Answer

Self-speculation requires a separate draft model but reuses its weights

Answer

Self-speculation always produces lower quality than speculative decoding with a separate draft model

Answer

Self-speculation is only applicable to models with fewer than 7B parameters

Question 16

What is inference compute decomposition and why does generation become memory-bandwidth-bound?

Accepted Answer

LLM generation is memory-bandwidth-bound at small batch sizes because arithmetic intensity is ~1 FLOP/byte

Answer

LLM generation is always compute-bound regardless of batch size

Answer

Increasing model size makes generation compute-bound for all batch sizes

Answer

Memory bandwidth is only relevant for prefill, not generation

Question 17

What is sparse attention and what are its main variants for long-context inference?

Accepted Answer

Sparse attention variants (sliding window, dilated, global+local) reduce O(N^2) cost for long-context inference

Answer

Sparse attention always produces worse quality than dense attention regardless of task

Answer

Sparse attention requires special hardware not available on standard GPUs

Answer

Sliding window attention prevents models from attending to distant context entirely

Question 18

What is model sharding for inference and how does pipeline parallel inference work?

Accepted Answer

Pipeline parallelism splits layers across GPUs; has a bubble issue for single requests; TP preferred for latency

Answer

Pipeline parallelism is always faster than tensor parallelism for single-request inference

Answer

Pipeline parallelism requires NVLink and cannot work over InfiniBand

Answer

Pipeline parallel inference requires micro-batch size equal to the number of pipeline stages

Question 19

What is LLM.int8() specifically and how does it handle activation outliers?

Accepted Answer

LLM.int8() extracts outlier activation dimensions to FP16 and quantizes the rest to INT8

Answer

LLM.int8() quantizes all activation dimensions uniformly to INT8

Answer

LLM.int8() requires per-token activation calibration during inference

Answer

LLM.int8() only quantizes model weights, not activations

Question 20

What is Mamba and how does its selective state space model (SSM) approach reduce inference cost?

Accepted Answer

Mamba replaces attention with a selective SSM that runs in O(1) memory and compute per token at inference

Answer

Mamba uses a modified attention mechanism with sparse patterns for efficiency

Answer

Mamba requires the same KV cache as transformer models but accesses it differently

Answer

Mamba-2 eliminated the selective mechanism for better hardware utilization

Question 21

What is activation offloading for inference and when is it beneficial?

Accepted Answer

KV cache offloading to CPU allows more concurrent sequences at the cost of PCIe swap latency

Answer

Activation offloading for inference only works on systems with NVMe SSDs

Answer

Activation offloading during inference requires gradient computation to be enabled

Answer

CPU offloading improves inference throughput by reducing GPU memory pressure without any latency cost

Question 22

What is the purpose of top-p (nucleus) sampling and how does it interact with inference optimization?

Accepted Answer

Top-p samples from the smallest token set whose cumulative probability exceeds p; dynamic candidate count

Answer

Top-p sampling always samples from exactly p tokens

Answer

Top-p is identical to top-k with k set to p × vocab_size

Answer

Top-p=0.9 means exactly 10% of tokens are always excluded from sampling

Question 23

What is the difference between eager and compiled inference in PyTorch 2.x?

Accepted Answer

Compiled inference via torch.compile generates fused optimized kernels, providing 15-40% speedup over eager

Answer

Eager inference is faster than compiled inference for all model sizes

Answer

torch.compile requires FP8 precision to achieve inference speedups

Answer

Compiled inference cannot handle dynamic sequence lengths

Question 24

What is token healing and why is it needed in LLM completion APIs?

Accepted Answer

Token healing rolls back partial boundary tokens and regenerates from clean token boundaries

Answer

Token healing adds special healing tokens to corrupted model outputs

Answer

Token healing is only relevant for models with vocabulary sizes over 100k

Answer

Token healing increases TTFT by recomputing the entire prompt

Question 25

What is context length extrapolation via RoPE scaling and what are the main techniques?

Accepted Answer

Linear interpolation, YaRN, and LongRoPE extend context by scaling RoPE frequencies, followed by continued training

Answer

RoPE scaling requires retraining the full model from scratch for longer contexts

Answer

RoPE scaling only works for models with fewer than 32 attention heads

Answer

LongRoPE applies a uniform scaling factor across all RoPE frequency bands

Question 26

What is quantization calibration and what data is used for it?

Accepted Answer

Calibration uses 128-512 representative samples to determine scale factors; domain matters for activation quantization

Answer

Calibration requires millions of samples to produce accurate scale factors

Answer

Calibration is only needed for weight quantization, not activation quantization

Answer

Calibration data must exactly match the fine-tuning dataset for accurate results

Test yourself