Latency, caching and serving optimization

The KV cache stores the key and value tensors computed for all previously generated tokens so they need not be recomputed at each decoding step. Without it, generating token N would require re-running attention over all N-1 prior tokens (O(N^2) total work across a sequence); with it, each step only computes K/V for the new token and attends to cached entries, making per-step cost roughly O(N). This converts the decode phase from quadratic to linear work, sharply lowering time-per-output-token at the expense of memory.

Prefill processes the entire input prompt in parallel in a single forward pass, so it is compute-bound (high FLOPs, good GPU utilization) and determines TTFT. Decode generates one token at a time autoregressively, is memory-bandwidth-bound (low arithmetic intensity, dominated by reading weights and KV cache), and determines inter-token latency. This asymmetry motivates techniques like chunked prefill and prefill/decode disaggregation.

It addresses KV-cache memory fragmentation and over-reservation caused by allocating contiguous buffers sized to the max sequence length. PagedAttention borrows OS virtual-memory paging: the KV cache is split into fixed-size blocks allocated on demand and referenced via a block table, eliminating internal/external fragmentation and enabling near-zero-waste memory sharing (e.g., across beams or shared prefixes). This raises achievable batch size and throughput substantially.

Static batching waits to assemble a fixed batch and runs all requests to completion together, so short requests idle until the longest finishes (head-of-line/tail waste). Continuous batching schedules at the token/iteration level: completed sequences leave the batch and new requests join immediately, keeping the GPU saturated. This greatly increases throughput and lowers average latency under mixed-length, bursty traffic.

A small, fast draft model proposes several candidate tokens, and the large target model verifies them all in a single parallel forward pass; accepted tokens are kept and the first rejection triggers correction via the target's distribution. Because verification is one batched pass instead of K sequential passes, multiple tokens can be emitted per target step. With a properly designed acceptance/rejection step the final sampling distribution is provably identical to running the target model alone.

TTFT (time to first token) measures latency from request arrival to the first output token and is dominated by queuing plus prefill cost; TPOT (time per output token) or ITL (inter-token latency) measures the steady-state generation speed dominated by the decode phase. Prefix/prompt caching, chunked prefill, and faster prefill kernels reduce TTFT, while larger batches, quantization, and speculative decoding reduce TPOT. They trade off: aggressive batching can raise TTFT while lowering per-token cost.

Prefix caching stores the KV-cache state for a common leading portion of prompts (e.g., a long shared system prompt or few-shot examples) so the prefill for that prefix is computed once and reused across requests. It is most valuable in high-fan-out workloads with large shared prefixes and short user suffixes, where it can cut TTFT and prefill compute by an order of magnitude. The benefit shrinks when prompts are highly unique or the shared prefix is short.

A semantic cache embeds incoming queries and returns a stored response when a previous query is within an embedding-similarity threshold, catching paraphrases that an exact key cache would miss. Its main risk is false hits: two semantically close but materially different queries may map to a wrong cached answer, so the similarity threshold must be tuned and often guarded by additional checks. Exact-match caches never return semantically wrong answers but have far lower hit rates.

Decode is memory-bandwidth-bound, so the dominant cost is streaming model weights from HBM each step; halving or quartering weight bytes proportionally reduces memory traffic and thus per-token latency. FP8/INT8 also let larger models or batches fit in VRAM and exploit faster tensor-core paths on modern GPUs. Accuracy is preserved with calibration or quantization-aware methods, though outlier-heavy activations may need techniques like SmoothQuant.

Tensor parallelism splits each layer's matrices across GPUs, reducing per-layer latency but adding an all-reduce communication per layer, so it scales best within a fast NVLink node and lowers single-request latency. Pipeline parallelism splits layers into stages across devices, adding pipeline-fill bubble latency for a single request but achieving high throughput when many micro-batches keep all stages busy. Production stacks often combine TP within a node and PP/replication across nodes.

User-facing requests frequently fan out to many backends, so the probability that at least one component hits its tail grows with fan-out, making the slowest responses dominate perceived experience and SLO compliance. Mean latency hides bursts, GC pauses, queuing, and cold starts that p99/p99.9 expose. Optimizing tails (hedged requests, admission control, capacity headroom) usually matters more than shaving the average.

Hedging sends a duplicate request to a second replica after a short delay (often near the p95 latency) and uses whichever response returns first, cancelling the other; this clips tail latency caused by an unlucky slow server. The main cost is extra load: naive hedging can significantly increase total work, so it is gated by a tie-breaking delay and capped to a small fraction of traffic. It is most effective when slow responses are caused by transient per-replica conditions rather than global overload.

Cold starts come from spinning up a new instance: container pull, runtime init, loading large model weights into GPU/CPU memory, and CUDA/graph warmup, which can take seconds to minutes for big models. Mitigations include keeping a pool of warm/min replicas (provisioned concurrency), pre-baking weights into the image or using fast weight streaming, snapshotting initialized process state, and predictive/scheduled scaling ahead of demand. Smaller distilled models also start faster.

Because decode is memory-bandwidth-bound, small batches underutilize compute, so adding requests amortizes weight reads and raises tokens/sec with little latency cost (the 'free' region). Past the point where the kernel becomes compute-bound or KV-cache memory is exhausted, additional requests no longer hide latency and each step takes proportionally longer, increasing TPOT and queuing. The optimal batch size sits at the knee where throughput is high but the SLO on latency still holds.

FlashAttention computes attention in a tiled, IO-aware way that avoids materializing the full N×N score matrix in HBM, fusing the softmax and reducing memory reads/writes; it is exact, not approximate. Its biggest gains are in the prefill phase (and training) where long sequences make the quadratic attention IO costly, improving TTFT. During single-token decode the query length is 1, so memory traffic is dominated by weight/KV reads and FlashAttention helps less (though variants like FlashDecoding parallelize the KV dimension to help here).

Item corpora are large and relatively static, so encoding every item at request time would be prohibitively expensive; instead embeddings are computed offline/near-line and stored in an ANN index, leaving only the (small) query to encode online. At serving time the heavy work reduces to one query encode plus a fast approximate nearest-neighbor lookup, cutting tail latency by orders of magnitude. Freshness is handled with incremental/streaming updates to the index rather than full recompute.

Weight quantization shrinks the static model parameters, reducing load/bandwidth and VRAM, while KV-cache quantization compresses the per-token K/V state that grows with batch size and sequence length. For very long contexts and large batches the KV cache, not the weights, becomes the dominant memory consumer, so quantizing it (e.g., to INT8/FP8) is what unlocks larger batches and longer contexts. KV quantization is more accuracy-sensitive, so keys are often kept at higher precision than values.

They apply ahead-of-time graph optimizations such as operator fusion (e.g., fused attention, bias+GELU), constant folding, layout selection, and use of vendor-tuned/precompiled kernels and CUDA graphs to eliminate Python and launch overhead. This reduces kernel-launch count and memory round-trips, improving both TTFT and TPOT without altering the math (beyond chosen precision). The trade-off is a build/compilation step and less flexibility for dynamic shapes.

Each decode step launches many tiny kernels, and at small batch sizes per-kernel CPU launch overhead becomes a significant fraction of step time. CUDA Graphs capture the fixed sequence of kernel launches once and replay them as a single graph, eliminating repeated CPU launch latency and improving GPU occupancy. The constraint is that shapes/control flow must be static, which is why graphs are captured per bucketed batch size.

Chunked prefill splits a long prompt's prefill into smaller token chunks processed across several scheduler iterations, so a single huge prefill no longer monopolizes a step. This lets ongoing decode requests be interleaved with prefill work, preventing decode stalls (TPOT spikes) caused by a large incoming prompt while keeping the GPU busy. It improves latency fairness and overall utilization at a small cost to peak prefill efficiency.

Disaggregation runs the compute-bound prefill phase and the bandwidth-bound decode phase on separate pools of GPUs, transferring the KV cache between them. This prevents long prefills from interfering with steady decode (reducing TPOT jitter) and lets each pool be sized and tuned independently for its bottleneck, improving SLO attainment. The trade-off is KV-cache transfer overhead and added system complexity.

A cache-aside (lazy) read pattern combined with write-through or event-driven invalidation on updates is the common choice: reads hit the low-latency cache and fall back to the store on a miss, while writes keep the cache fresh to bound staleness. Write-back offers the lowest write latency but risks data loss and stronger inconsistency, so it is avoided for correctness-sensitive features. The TTL is tuned to the feature's freshness SLA.

When a popular cached entry (e.g., an expensive LLM response or embedding) expires, many concurrent requests miss simultaneously and all recompute it, overloading the backend. Prevention includes a single-flight lock / request coalescing so only one worker recomputes while others wait, probabilistic early recomputation (jittered TTLs), and serving slightly stale values while refreshing asynchronously. These cap the recompute fan-out to one per key.

Distillation trains a smaller student model to mimic a larger teacher's outputs (soft labels/logits), yielding far fewer parameters and FLOPs so inference is faster and cheaper to host. The trade-off is some loss of accuracy/capability, especially on rare or complex cases, that must be validated against SLOs. In practice distillation is often combined with quantization for compounding latency gains.

The server briefly buffers incoming requests (up to a small max wait or batch size) and runs them as one GPU batch, amortizing kernel launches and weight reads to raise throughput and lower cost per request. The max-delay window bounds the added queuing latency so individual requests still meet their SLO. It is a tunable trade: a larger window/batch boosts throughput but increases TTFT, so the window is set just below the latency budget.

VRAM must simultaneously hold model weights, activations, and the KV cache, and the KV cache grows linearly with concurrent sequences times their lengths, so memory—not compute—usually caps the achievable batch size. Operators budget memory as roughly (total VRAM − weights − overhead) / per-token KV size to derive a token/sequence limit, then use paging, quantization, or smaller context windows to fit more. Exceeding the budget forces preemption/swapping that spikes latency.

Exact kNN over millions/billions of vectors requires scanning the whole corpus per query (linear, latency-prohibitive at scale). ANN structures like HNSW (graph navigation) or IVF-PQ (coarse partitioning plus compressed codes) return near-optimal neighbors in sublinear time and reduced memory, trading a small, tunable recall loss for orders-of-magnitude lower query latency. The recall/latency trade is controlled by parameters like efSearch or nprobe.

Streaming begins delivering tokens as soon as the first is produced, so the user-perceived latency collapses to TTFT instead of the full end-to-end completion time. The total compute and wall-clock to finish are the same, but progressive rendering keeps the interface responsive and lets users start reading immediately. It also enables early cancellation, saving compute when the user stops.

When arrival rate exceeds capacity, queues grow unbounded and latency for everyone explodes; admission control caps the in-flight/queue length and rejects or sheds excess load early (often with priorities) so accepted requests still meet their SLO. This preserves goodput and tail latency rather than letting the whole system degrade. It is paired with autoscaling, which fixes the capacity shortfall on a slower timescale.

GQA/MQA share key/value heads across multiple query heads, shrinking the KV cache size per token by the head-sharing factor. Because decode is bandwidth-bound and KV-cache reads dominate at long context/large batch, a smaller KV cache reduces memory traffic and frees VRAM for larger batches, lowering TPOT and raising throughput. The trade-off is a modest quality reduction versus full multi-head attention, which GQA balances better than full MQA.

Training-serving skew from recomputing features differently online causes both correctness issues and added serving latency from heavy on-the-fly transforms. Precomputing expensive/slow-changing features into a low-latency online store (with the same logic that generated training data) removes per-request compute and guarantees parity. Only fast, request-specific features are computed at serve time, keeping the hot path minimal.

At least one full pass reads ~14 GB, so the floor is about 14 GB / 2000 GB/s ≈ 7 ms per token (~140 tokens/s), ignoring KV and overhead. This bandwidth bound implies that at small batch you cannot go faster without reducing bytes moved—hence quantization (fewer weight bytes) or amortizing the read across a larger batch are the only levers, not more FLOPs. It also explains why throughput scales with batch while single-stream latency is nearly fixed.

# Bandwidth-bound decode floor
bytes = 14e9          # FP16 weights
bw    = 2e12          # 2 TB/s
print(bytes / bw)     # ~0.007 s = 7 ms/token

CDN/edge caching stores responses geographically close to users keyed by the request URL/headers, cutting network RTT for cacheable, identical public requests; application-level caches (e.g., Redis) sit near the model and store computed results or embeddings keyed by semantic/normalized inputs. Edge caching applies to ML only when outputs are deterministic, non-personalized, and reused (e.g., a public embedding for a fixed string), whereas personalized or stochastic LLM outputs generally are not edge-cacheable. They are complementary layers, not substitutes.

With temperature > 0 and sampling, identical prompts produce different outputs, so a response cache would either return stale variability-killing results or have a near-zero correct hit rate for stochastic endpoints. Teams handle this by caching only when temperature is 0 / greedy (deterministic), by caching the expensive prefill KV state rather than the final text, or by fixing a seed for cacheable paths. Semantic caching is then reserved for use cases tolerant of a single representative answer.