Latency, caching and serving optimization quiz

Question 1

In autoregressive LLM inference, what is the purpose of the KV cache and why does it dramatically reduce latency?

Accepted Answer

It stores past keys/values so prior tokens' attention need not be recomputed each step

Answer

It caches the model weights in GPU memory to avoid reloading per request

Answer

It caches the final logits to speed up sampling

Answer

It compresses the prompt embeddings to reduce VRAM

Question 2

LLM inference has two distinct phases. What distinguishes the prefill phase from the decode phase in terms of compute characteristics?

Accepted Answer

Prefill is compute-bound and parallel; decode is memory-bandwidth-bound and sequential

Answer

Prefill is memory-bound while decode is compute-bound

Answer

Both phases are equally compute-bound

Answer

Decode processes the whole prompt while prefill emits tokens

Question 3

What problem does PagedAttention (vLLM) solve, and how?

Accepted Answer

It pages KV cache into fixed blocks to eliminate fragmentation and enable sharing

Answer

It quantizes the KV cache to 4-bit to save memory

Answer

It offloads attention to CPU to free GPU memory

Answer

It replaces softmax with a linear-time approximation

Question 4

How does continuous (in-flight) batching differ from static batching, and why does it improve GPU throughput for LLM serving?

Accepted Answer

Continuous batching admits/evicts requests per iteration so finished ones free slots immediately

Answer

Continuous batching uses larger fixed batches than static batching

Answer

Continuous batching disables the KV cache to save memory

Answer

Static batching is always faster for variable-length requests

Question 5

Explain how speculative decoding reduces latency without changing the output distribution.

Accepted Answer

A draft model proposes tokens that the target verifies in one parallel pass, preserving the distribution

Answer

It trains a smaller model to replace the large one entirely

Answer

It caches common token sequences and replays them

Answer

It lowers precision of the target model during decode

Question 6

What do the metrics TTFT and TPOT (or ITL) measure, and which serving optimizations primarily target each?

Accepted Answer

TTFT is first-token latency (prefill-bound); TPOT is per-token latency (decode-bound)

Answer

TTFT measures throughput; TPOT measures memory usage

Answer

Both measure end-to-end request latency identically

Answer

TTFT is decode-bound and TPOT is prefill-bound

Question 7

What is prefix (prompt) caching and when does it yield the biggest latency win?

Accepted Answer

Reusing the KV state of a shared prompt prefix to skip recomputing its prefill

Answer

Caching the model's output tokens for identical full prompts only

Answer

Storing embeddings of all prompts in a vector DB

Answer

Caching attention scores between unrelated requests

Question 8

What is a semantic cache for LLM serving, and what is its main correctness risk compared to an exact-match cache?

Accepted Answer

It can return wrong answers via false similarity hits, unlike exact-match caching

Answer

It guarantees correctness because embeddings are deterministic

Answer

It is always slower than recomputing the LLM response

Answer

It only works for numeric queries

Question 9

Why is INT8/FP8 weight quantization an effective latency optimization specifically for the decode phase?

Accepted Answer

Decode is bandwidth-bound, so smaller weights cut HBM traffic per token

Answer

Decode is compute-bound so fewer FLOPs help

Answer

Quantization mainly speeds up prefill, not decode

Answer

It reduces network latency between nodes

Question 10

In a multi-GPU LLM deployment, what is the latency-vs-throughput trade-off between tensor parallelism and pipeline parallelism?

Accepted Answer

Tensor parallelism lowers per-request latency (heavy comm); pipeline parallelism favors throughput with bubble latency

Answer

Both reduce single-request latency equally

Answer

Pipeline parallelism eliminates all communication overhead

Answer

Tensor parallelism only helps throughput, never latency

Question 11

Why is p99 (tail) latency often more important than mean latency in production serving SLOs?

Accepted Answer

Tail latency dominates user experience because fan-out amplifies the chance of hitting slow responses

Answer

Mean latency captures fan-out amplification better than p99

Answer

p99 is irrelevant once mean latency is low

Answer

p99 only matters for batch jobs

Question 12

What is request hedging (speculative/backup requests) and what is its main cost?

Accepted Answer

It sends backup requests to cut tail latency at the cost of extra load

Answer

It batches requests to reduce compute, with no extra cost

Answer

It compresses payloads to reduce network latency

Answer

It caches responses to avoid recomputation

Question 13

What causes cold-start latency in serverless/autoscaled model serving, and name two mitigations.

Accepted Answer

Weight loading and init cause cold starts; mitigate with warm pools and snapshotting

Answer

Cold starts are caused only by network latency; fix with a CDN

Answer

Cold starts come from KV cache misses; fix by disabling the cache

Answer

They are caused by quantization and fixed by using FP32

Question 14

Why does increasing batch size improve throughput but eventually degrade per-request latency in GPU inference?

Accepted Answer

Up to a knee batching is near-free (bandwidth-bound), then it becomes compute/memory-bound and raises latency

Answer

Larger batches always reduce both latency and throughput

Answer

Batch size has no effect on GPU latency

Answer

Latency improves monotonically with batch size

Question 15

What does FlashAttention optimize, and is its benefit larger for prefill or decode?

Accepted Answer

It is an IO-aware exact attention; biggest gains are in prefill/long sequences

Answer

It approximates attention and mainly speeds up decode

Answer

It replaces attention with a linear layer

Answer

It only reduces parameter count, not latency

Question 16

When serving recommendation/search systems, why is precomputing and caching item embeddings the dominant latency optimization?

Accepted Answer

Precomputed item embeddings + ANN reduce online work to a query encode plus fast lookup

Answer

Item embeddings must be recomputed per request for correctness

Answer

Caching embeddings hurts latency due to memory pressure

Answer

Embeddings are cheap to compute so caching is unnecessary

Question 17

What is the difference between weight quantization and KV-cache quantization for serving long-context LLMs, and when does the latter matter most?

Accepted Answer

Weight quant shrinks static params; KV-cache quant shrinks per-token state that dominates at long context/large batch

Answer

They are identical optimizations

Answer

KV-cache quantization only helps the prefill phase

Answer

Weight quantization is needed only for short contexts

Question 18

How do TensorRT-LLM/ONNX Runtime achieve lower latency than an eager PyTorch model for the same weights?

Accepted Answer

They fuse ops and use precompiled kernels/CUDA graphs to cut launch and memory overhead

Answer

They retrain the model to be smaller

Answer

They run the model on CPU instead of GPU

Answer

They skip layers at inference time

Question 19

Why can CUDA Graphs reduce decode latency for small-batch LLM inference in particular?

Accepted Answer

They replay a captured launch sequence, removing per-step CPU kernel-launch overhead

Answer

They increase numerical precision

Answer

They compress weights to fit cache

Answer

They parallelize across multiple GPUs automatically

Question 20

What is chunked prefill and how does it help mixed prefill/decode batches?

Accepted Answer

It splits long prefills into chunks so decode requests aren't starved

Answer

It runs prefill entirely on CPU

Answer

It disables decode during prefill

Answer

It caches prefill results across users automatically

Question 21

What is prefill/decode disaggregation and why might it lower latency at scale?

Accepted Answer

It separates prefill and decode onto different GPU pools to avoid interference

Answer

It merges prefill and decode into one kernel

Answer

It removes the KV cache entirely

Answer

It only applies to CPU inference

Question 22

For a read-heavy feature store powering online inference, which cache write strategy best balances latency and consistency, and why?

Accepted Answer

Cache-aside reads with write-through/invalidation to balance latency and freshness

Answer

Write-back, because durability is guaranteed

Answer

No caching, to ensure strong consistency

Answer

Random eviction with no TTL

Question 23

What is a cache stampede (thundering herd) on a hot key, and how do you prevent it in a model-serving cache?

Accepted Answer

Use single-flight/request coalescing and jittered TTLs so only one recompute happens per key

Answer

Stampede is solved by lowering the TTL to zero

Answer

Disable the cache for hot keys

Answer

Increase batch size to absorb the load

Question 24

Why does knowledge distillation reduce serving latency, and what is the typical trade-off?

Accepted Answer

A smaller student mimics the teacher, cutting latency at some accuracy cost

Answer

It enlarges the model so it runs faster

Answer

It only speeds up training, not inference

Answer

It guarantees identical accuracy to the teacher

Question 25

In an online ML service, what is the latency benefit of dynamic (server-side) request batching with a max-delay window?

Accepted Answer

It amortizes per-call overhead across a batch, trading a bounded delay for higher throughput

Answer

It removes the GPU bottleneck by avoiding batching

Answer

It always reduces latency with no downside

Answer

It only works for CPU models

Question 26

Why is GPU memory (HBM) often the binding constraint that limits LLM serving concurrency, and how is it budgeted?

Accepted Answer

KV cache grows with concurrency so HBM caps batch size; budget = (VRAM − weights − overhead)/per-token KV

Answer

Compute, not memory, always limits concurrency

Answer

Memory is irrelevant because weights are streamed from disk

Answer

Concurrency is limited only by network bandwidth

Question 27

What latency-relevant problem does an ANN index (e.g., HNSW or IVF-PQ) solve compared to exact nearest-neighbor search?

Accepted Answer

ANN trades small recall loss for sublinear-time search vs. linear exact kNN

Answer

ANN is exact and slower than brute force

Answer

ANN only works on GPUs

Answer

Exact kNN is faster at billion scale

Question 28

Why is streaming tokens (server-sent events) a perceived-latency optimization even though total generation time is unchanged?

Accepted Answer

Streaming makes perceived latency ≈ TTFT by delivering tokens incrementally

Answer

Streaming reduces total compute time

Answer

Streaming requires recomputing tokens, adding latency

Answer

It only helps batch jobs, not interactive use

Question 29

What is the role of an admission-control / load-shedding policy in keeping serving latency within SLO under overload?

Accepted Answer

It bounds queue/in-flight load and sheds excess so accepted requests meet SLO

Answer

It increases queue depth to absorb all traffic

Answer

It disables autoscaling during overload

Answer

It improves accuracy under load

Question 30

When does grouped-query attention (GQA) / multi-query attention (MQA) help serving latency, and what is the mechanism?

Accepted Answer

They share K/V heads, shrinking the KV cache and cutting decode memory traffic

Answer

They add more KV heads to improve speed

Answer

They eliminate the need for a KV cache

Answer

They only affect training, not inference latency

Test yourself