LLM serving and inference optimization quiz

Question 1

In autoregressive LLM inference, what distinguishes the 'prefill' phase from the 'decode' phase, and why does this distinction matter for optimization?

Accepted Answer

Prefill is compute-bound and parallel; decode is memory-bandwidth-bound and sequential

Answer

Prefill is memory-bound; decode is compute-bound

Answer

Both phases are equally compute-bound and benefit identically from batching

Answer

Prefill generates tokens one at a time; decode processes the prompt in parallel

Question 2

What is the KV cache and why does it grow linearly with sequence length and batch size?

Accepted Answer

A cache of attention key/value tensors that grows linearly with tokens and batch size

Answer

A cache of model weights that is constant size regardless of sequence length

Answer

A cache of output logits that grows quadratically with sequence length

Answer

A compressed copy of the input embeddings reused across layers

Question 3

How does PagedAttention (vLLM) reduce KV cache memory waste compared to contiguous allocation?

Accepted Answer

By paging the KV cache into fixed-size non-contiguous blocks to eliminate fragmentation

Answer

By compressing KV tensors with quantization to fit more sequences

Answer

By recomputing KV values each step to avoid storing them at all

Answer

By moving the KV cache to CPU RAM during decode

Question 4

What is continuous (in-flight) batching and how does it improve throughput over static batching?

Accepted Answer

Iteration-level scheduling that swaps finished sequences for queued ones to keep GPU saturated

Answer

Batching requests only when they share identical prompt lengths

Answer

Increasing batch size dynamically by lowering precision

Answer

Running each request in its own CUDA stream without batching

Question 5

Explain speculative decoding and the role of the draft and target models.

Accepted Answer

A draft model proposes tokens that the target model verifies in parallel, preserving its distribution

Answer

The target model proposes tokens that a draft model approves to save memory

Answer

Two models vote and the majority token is emitted, changing the distribution

Answer

The draft model replaces the target model after warmup to reduce cost

Question 6

Why does speculative decoding preserve the exact output distribution of the target model when using the standard acceptance rule?

Accepted Answer

Rejection sampling with residual resampling makes emitted tokens equivalent to sampling from the target

Answer

It averages the draft and target logits, which happens to match the target

Answer

It only accepts tokens where draft and target agree exactly, guaranteeing identity

Answer

It re-runs the target on the whole sequence at the end to correct any drift

Question 7

What is the difference between latency-oriented metrics TTFT and TPOT (ITL) in LLM serving?

Accepted Answer

TTFT is time to first token (prefill+queue); TPOT is average inter-token decode latency

Answer

TTFT is total throughput; TPOT is the prompt processing time

Answer

TTFT measures decode speed; TPOT measures prefill speed

Answer

Both measure the same thing under different load conditions

Question 8

How does grouped-query attention (GQA) reduce KV cache size compared to multi-head attention (MHA)?

Accepted Answer

Multiple query heads share fewer K/V heads, shrinking the cache by num_heads/num_groups

Answer

It quantizes K/V to int4 while keeping all heads

Answer

It removes value vectors entirely, storing only keys

Answer

It caches K/V only for the most recent tokens, dropping older ones

Question 9

What problem does FlashAttention solve and how does it achieve speedups without changing the math?

Accepted Answer

Tiled, IO-aware attention with online softmax that avoids materializing the N x N matrix in HBM

Answer

An approximate attention that drops low scores to save compute

Answer

A quantized attention kernel that uses int8 scores

Answer

A sparse attention pattern that only attends to local windows

Question 10

Compare tensor parallelism and pipeline parallelism for serving a model too large for one GPU.

Accepted Answer

TP shards layers needing per-layer all-reduce (intra-node); PP splits layer stages with lighter comm (inter-node)

Answer

TP splits the batch; PP splits the sequence length

Answer

TP is communication-free; PP requires all-reduce every token

Answer

They are identical except TP works only on CPUs

Question 11

Why is decode-phase LLM inference typically memory-bandwidth-bound, and what does this imply for batching?

Accepted Answer

Each step reads all weights to emit one token, so batching amortizes weight reads and raises throughput

Answer

Decode is compute-bound, so batching cannot help

Answer

Memory bandwidth only matters in prefill, not decode

Answer

Batching reduces throughput because it increases KV cache reads per token

Question 12

What is the practical effect of weight-only INT4 quantization (e.g., GPTQ/AWQ) on LLM inference performance?

Accepted Answer

Cuts memory ~4x and speeds memory-bound decode by reducing weight-read traffic

Answer

Speeds up only the compute-bound prefill phase

Answer

Increases memory use but improves accuracy

Answer

Has no effect on latency, only on disk footprint

Question 13

What is prefix caching (prompt caching) and when does it provide the largest benefit?

Accepted Answer

Reusing KV for shared prompt prefixes to skip prefill; best with long common prefixes

Answer

Caching final responses keyed by exact prompt string

Answer

Caching model weights across requests

Answer

Storing decode tokens to skip generation on repeats

Question 14

In a roofline analysis of LLM inference, what determines whether increasing batch size will keep improving throughput?

Accepted Answer

Throughput grows until arithmetic intensity crosses the roofline ridge into compute-bound

Answer

Throughput grows indefinitely with batch size regardless of hardware

Answer

Only KV cache size determines the throughput ceiling, not FLOPs

Answer

Larger batches always reduce throughput due to scheduling overhead

Question 15

What is chunked prefill and what scheduling problem does it address?

Accepted Answer

Splitting prefill into chunks interleaved with decode to avoid stalling other requests

Answer

Caching prefill results to disk in chunks

Answer

Running prefill on CPU in chunks to free the GPU

Answer

Quantizing the prompt in chunks before prefill

Question 16

Why might prefill/decode disaggregation (separating the two phases onto different GPU pools) improve a serving cluster?

Accepted Answer

Each phase gets independently optimized hardware/batching, avoiding interference between TTFT and TPOT

Answer

It eliminates the need for a KV cache entirely

Answer

It allows running both phases on the same kernel for speed

Answer

It is only useful for models that fit on a single GPU

Question 17

What does it mean that LLM serving involves a throughput-vs-latency tradeoff governed by batch size?

Accepted Answer

Bigger batches raise throughput but increase per-request latency; serving balances the two

Answer

Bigger batches always reduce both latency and throughput

Answer

Latency and throughput are independent of batch size

Answer

Smaller batches maximize throughput and latency simultaneously

Question 18

How does the per-token KV cache size for a model scale with its hidden configuration? Give the formula.

Accepted Answer

2 x layers x kv_heads x head_dim x bytes per token

Answer

layers x heads x head_dim x bytes (no factor of 2)

Answer

2 x layers x vocab_size x bytes per token

Answer

heads x head_dim x seq_len^2 x bytes per token

Question 19

What is the difference between weight quantization, activation quantization, and KV cache quantization, and which one most directly relaxes the batch-size ceiling?

Accepted Answer

KV cache quantization, because the KV cache scales with batch/context and limits concurrency

Answer

Weight quantization, because weights grow with batch size

Answer

Activation quantization, because activations dominate serving memory

Answer

All three relax it equally

Question 20

What is the purpose of CUDA graphs in LLM decode serving and what constraint do they impose?

Accepted Answer

They replay captured kernel launches to cut CPU overhead but require static shapes

Answer

They compress weights to reduce memory bandwidth

Answer

They parallelize attention across multiple GPUs automatically

Answer

They allow dynamic shapes with zero overhead

Question 21

Why is FP8 inference attractive on modern GPUs (Hopper/Blackwell), and what is a key risk?

Accepted Answer

Doubles throughput and halves memory but risks accuracy loss from limited range without scaling

Answer

Has unlimited dynamic range so no calibration is ever needed

Answer

Only helps prefill and slows down decode

Answer

Increases memory use compared to FP16

Question 22

In continuous batching, what is a 'token budget' and how does it constrain scheduling per iteration?

Accepted Answer

A per-iteration cap on total processed tokens that bounds latency and gates admission

Answer

A limit on total tokens a user can request per day

Answer

The maximum context length the model supports

Answer

The number of tokens stored in the KV cache

Question 23

What is the acceptance rate in speculative decoding and how does it affect the realized speedup?

Accepted Answer

Fraction of drafted tokens accepted; higher rate and good draft alignment yield larger speedup

Answer

The fraction of requests admitted by the scheduler

Answer

A fixed constant independent of the draft model

Answer

The probability the target model rejects all tokens, maximizing speed

Question 24

How does Medusa (or EAGLE-style) self-speculative decoding differ from classic two-model speculative decoding?

Accepted Answer

They add extra heads to the target model to self-draft, avoiding a separate draft model

Answer

They use two identical large models voting on tokens

Answer

They quantize the draft model to int4 instead of using heads

Answer

They eliminate verification, trusting the drafted tokens

Question 25

What is the difference between data-parallel replicas and model-parallel sharding when scaling an LLM serving deployment?

Accepted Answer

Data parallelism replicates the whole model for QPS; model parallelism shards one model across GPUs

Answer

They are the same technique with different names

Answer

Data parallelism shards layers; model parallelism replicates the model

Answer

Model parallelism only increases throughput, never fits bigger models

Question 26

What is the effect of sampling parameters (temperature, top-p, top-k) on inference cost, and where is the cost actually incurred?

Accepted Answer

Sampling math is cheap; cost is the forward pass and total tokens (beam search multiplies cost)

Answer

Higher temperature directly multiplies GPU FLOPs per token

Answer

top-k is the single largest cost in inference

Answer

Sampling parameters have no effect on total generated tokens or cost

Question 27

How does request routing with a least-loaded or KV-cache-aware policy improve a multi-replica LLM cluster versus round-robin?

Accepted Answer

Prefix-affinity/least-loaded routing raises cache hits and balances load, cutting TTFT and tail latency

Answer

Round-robin always maximizes prefix cache reuse

Answer

Cache-aware routing increases memory use without latency benefit

Answer

Routing policy has no effect once continuous batching is enabled

Question 28

What is the role of an inference engine's 'max model len' / context budget in capacity planning?

Accepted Answer

It bounds per-request KV memory, determining how many sequences fit and thus max concurrency

Answer

It controls model weight precision

Answer

It only affects accuracy, not memory

Answer

It sets the GPU clock speed for inference

Question 29

Why is naive Python/HTTP-per-token streaming inadequate, and what serving pattern is used for token streaming at scale?

Accepted Answer

Use a single streaming connection (SSE/gRPC) fed by a continuously batched engine

Answer

Open a new HTTP request for every generated token

Answer

Generate the full response then send it once, never streaming

Answer

Stream only from a single-request, no-batch engine

Question 30

What is 'guided'/constrained decoding (e.g., grammar or JSON-schema constrained), and what is its main runtime cost?

Accepted Answer

Masking disallowed logits via a grammar FSM each step; cost is computing the allowed-token mask

Answer

Retraining the model on JSON each request

Answer

Running a second validation model after generation

Answer

It has no runtime cost and never affects logits

Test yourself