Model Serving at Scale

REST uses HTTP/1.1 with JSON payloads — human-readable, universally supported by browsers and clients, but verbose and slower for high-frequency inference calls. gRPC uses HTTP/2 with Protocol Buffer binary serialization — compact, strongly typed, supports bidirectional streaming, and 3-10x faster for the same payload due to binary encoding and multiplexed connections. gRPC is preferred for microservice-to-microservice model calls where throughput and latency matter. REST is preferred for external-facing APIs and web clients. NVIDIA Triton supports both protocols on the same port. ONNX Runtime Serving and TensorFlow Serving offer gRPC and REST. Use gRPC for backend inference pipelines; REST for developer-facing model APIs.

# gRPC inference call (Triton client)
import tritonclient.grpc as grpcclient
client = grpcclient.InferenceServerClient('localhost:8001')
inputs = [grpcclient.InferInput('input', [1,224,224,3], 'FP32')]
result = client.infer('resnet50', inputs)

Triton Inference Server is an open-source production inference server from NVIDIA supporting multiple model frameworks (TensorFlow, PyTorch, ONNX, TensorRT, Python backend, OpenVINO) and deployment backends (GPU, CPU, Jetson). Key features: concurrent model execution (multiple model instances in parallel), dynamic batching (groups incoming requests into optimal batches automatically), model ensembles (chain models in a pipeline), streaming inference via gRPC, Shared Memory support (zero-copy between client and server on the same machine), model warmup, model versioning (serve multiple versions simultaneously), and GPU/CPU performance metrics. Triton exposes REST (/v2/) and gRPC APIs compatible with the KFServing V2 inference protocol, enabling drop-in replacement across serving frameworks.

# model_repository/resnet50/config.pbtxt
name: 'resnet50'
backend: 'onnxruntime'
max_batch_size: 32
dynamic_batching { preferred_batch_size: [8, 16] }

Dynamic batching (also called continuous batching for LLMs) aggregates multiple individual inference requests arriving at different times into a single larger batch for GPU processing. GPUs achieve maximum efficiency at large batch sizes because matrix operations (GEMMs in neural networks) scale super-linearly up to the hardware's maximum parallelism. Without batching, a single request uses a tiny fraction of GPU compute. With dynamic batching, the server waits for a configurable delay (e.g., 1-5 ms) or until a preferred batch size is reached before executing inference, significantly increasing GPU utilization. The trade-off is slightly increased per-request latency in exchange for dramatically higher throughput. Triton, TorchServe, and TF Serving all support configurable dynamic batching with max_queue_delay_microseconds.

# Triton dynamic batching config
dynamic_batching {
  preferred_batch_size: [ 4, 8, 16 ]
  max_queue_delay_microseconds: 5000  # wait up to 5ms
}

Quantization reduces model weight and/or activation precision from FP32/FP16 to lower-precision formats (INT8, INT4, FP8) to reduce memory footprint, increase throughput, and reduce latency. Main strategies: Post-Training Quantization (PTQ) calibrates scale factors from a representative dataset without retraining — fast but may cause 0.5-2% accuracy degradation. Quantization-Aware Training (QAT) simulates quantization during training, recovering accuracy. Weight-only quantization (GPTQ, AWQ, GGUF) quantizes weights to INT4 while keeping activations in FP16 — ideal for memory-bound LLM inference. Activation quantization adds activation scaling for additional compute speedup. INT8 on CPU uses oneDNN/OpenVINO; INT8 on GPU uses TensorRT or cuDNN. FP8 is available on Hopper H100 GPUs via Transformer Engine.

from transformers import BitsAndBytesConfig
config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3-8B', quantization_config=config)

Knowledge distillation trains a small student model to mimic a large teacher model's output distributions (soft labels) rather than hard ground-truth labels. The student learns from the teacher's confidence scores across all classes — richer signal than binary labels — enabling the student to achieve near-teacher accuracy at a fraction of the parameter count. Response distillation uses teacher output probabilities; feature distillation matches intermediate layer activations. For LLMs, distillation produces smaller models (e.g., DistilBERT — 40% smaller, 60% faster than BERT with 97% performance retention). At serving time, the smaller student model requires less GPU memory, lower latency, higher throughput, and lower compute cost. Distillation is particularly effective combined with quantization and pruning in a compression pipeline.

# Distillation loss
T = 4.0  # temperature
soft_teacher = F.softmax(teacher_logits / T, dim=-1)
soft_student = F.log_softmax(student_logits / T, dim=-1)
distill_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (T**2)

ONNX (Open Neural Network Exchange) is an open format for representing ML models as a computation graph with standardized operator definitions. Export models from PyTorch, TensorFlow, scikit-learn, or XGBoost to the .onnx format, then deploy with any ONNX Runtime-compatible inference engine (ONNX Runtime, TensorRT, OpenVINO, CoreML, WebAssembly). ONNX Runtime (ORT) provides cross-platform inference with execution providers (CUDA EP, TensorRT EP, CPU EP, DirectML EP for Windows). Benefits: decouple training framework from serving stack, enable hardware-specific optimizations without rewriting model code, and reduce deployment dependencies. Limitations: not all PyTorch ops have ONNX equivalents (custom ops require registration); dynamic shapes require explicit dynamic axis declarations; opset versioning can cause compatibility issues.

import torch
torch.onnx.export(
    model, dummy_input, 'model.onnx',
    input_names=['input'], output_names=['output'],
    dynamic_axes={'input': {0: 'batch_size'}},
    opset_version=17
)

TensorRT is NVIDIA's high-performance inference SDK that applies a pipeline of optimizations to a model graph. Layer fusion: consecutive operations (Conv + BN + ReLU) are merged into a single kernel, eliminating intermediate memory writes. Precision calibration: auto-converts layers to INT8 or FP16 while maintaining accuracy using calibration datasets. Kernel auto-tuning: selects the fastest cuDNN/cuBLAS algorithm for each layer given input shape via benchmarking. Tensor layout optimization: reorders data formats (NHWC vs. NCHW) for optimal memory access. Dynamic shape support allows variable batch sizes at inference. TensorRT engines are hardware-specific — build on the target GPU model. Typical speedup: 2-6x over FP32 PyTorch inference. Integrate via Torch-TensorRT (compile PyTorch models) or export to ONNX first.

import torch_tensorrt
trt_model = torch_tensorrt.compile(
    model,
    inputs=[torch_tensorrt.Input([1,3,224,224], dtype=torch.float16)],
    enabled_precisions={torch.float16}
)

vLLM is an open-source LLM inference and serving engine achieving state-of-the-art throughput through three key innovations. PagedAttention manages KV cache in fixed-size non-contiguous memory pages (analogous to OS virtual memory), eliminating memory fragmentation and enabling 24x higher throughput vs. naive implementations. Continuous batching (iteration-level scheduling) dynamically adds new requests mid-batch as others finish, maximizing GPU utilization. Speculative decoding uses a small draft model to generate multiple tokens per step verified by the large model. Additional optimizations: tensor parallelism, Flash Attention, quantized KV cache (INT8/FP8), prefix caching (reuse KV cache for repeated system prompts), and chunked prefill. vLLM achieves 10-24x higher throughput than HuggingFace Transformers naive inference for high-concurrency serving.

from vllm import LLM, SamplingParams
llm = LLM(model='meta-llama/Llama-3-8B-Instruct',
          max_model_len=8192,
          gpu_memory_utilization=0.90)
outputs = llm.generate(prompts, SamplingParams(temperature=0.7, max_tokens=512))

Static batching groups a fixed number of requests, runs the full generation for all, then starts the next batch. Requests with short outputs waste GPU time waiting for slow long-output requests to finish (padding problem at the sequence level). Continuous batching (iteration-level scheduling) processes each decoding step as a micro-batch: after each step, requests that have finished (generated EOS) are immediately replaced with new requests from the queue. The batch composition changes every iteration. This eliminates waiting and maximizes GPU utilization by keeping the batch full at all times. A GPU utilization increase from ~40% (static) to ~85-95% (continuous) is typical. All major high-throughput LLM serving frameworks (vLLM, TGI, TensorRT-LLM) use continuous batching as the default serving strategy.

Blue-green deployment maintains two identical serving environments (blue = current production, green = new model version). Traffic is routed entirely to blue while green is prepared, tested, and warmed up with real data. Cutover happens instantly by switching the load balancer to route all traffic to green. If the new version shows issues, rollback is immediate — switch back to blue. This eliminates downtime (no in-place upgrade) and provides a clear rollback path. For ML models, blue-green ensures zero inference gaps during model updates and prevents serving requests with a partially-loaded or warming model. The trade-off is double the infrastructure cost during the transition period. Implement with Kubernetes by updating the service selector, or using cloud load balancer target group swaps (AWS ALB weighted target groups at 0/100).

# Kubernetes blue-green: update service selector
kubectl patch svc model-service -p '{"spec":{"selector":{"version":"green"}}}'
# Rollback:
kubectl patch svc model-service -p '{"spec":{"selector":{"version":"blue"}}}'

Canary deployment gradually shifts a small percentage of production traffic to the new model version (e.g., 5% → 20% → 50% → 100% over hours or days) rather than doing an instantaneous full cutover. This limits the blast radius if the new model has degraded performance — only 5% of users are affected initially. Monitor metrics (latency, error rate, business KPIs, accuracy on observed labels) at each traffic split stage before increasing. Automated canary analysis tools (Argo Rollouts, Flagger) compare metrics between canary and baseline and automatically promote or roll back. For ML models, canary enables online A/B comparison of model quality under real production traffic with statistical confidence before full rollout. Requires more sophisticated traffic routing infrastructure than blue-green.

# Argo Rollouts canary strategy
strategy:
  canary:
    steps:
    - setWeight: 5
    - pause: {duration: 1h}
    - setWeight: 50
    - pause: {duration: 2h}
    - setWeight: 100

Shadow mode (also called shadow deployment or dark launch) runs a new model version in parallel with the current production model on the same live traffic, but only returns the production model's response to users — the shadow model's predictions are logged for offline analysis without affecting user experience. This enables risk-free evaluation of a new model on real production data distribution without any user impact. Compare shadow vs. production predictions to detect regression, measure latency overhead, and validate accuracy improvements on ground-truth labels as they arrive. Once confidence is established, promote the shadow to production via canary or blue-green. Requires infrastructure to duplicate requests to both models simultaneously (Envoy mirroring, Istio traffic mirroring, or application-level duplication).

# Istio traffic mirroring config
route:
- destination: {host: model-v1, port: {number: 80}}
  weight: 100
mirror: {host: model-v2}  # shadow receives mirrored traffic
mirrorPercentage: {value: 100}

KServe is an open-source Kubernetes-native model serving platform built on top of Knative and Istio, providing a standardized ModelServer CRD (Custom Resource Definition) for deploying ML models. It solves the problem of per-framework serving infrastructure: instead of maintaining separate serving deployments for TensorFlow, PyTorch, sklearn, and XGBoost models, KServe provides a unified interface with automatic scaling (including scale-to-zero), canary rollouts, traffic splitting, request batching, and monitoring. Supports multiple runtimes: Triton, MLServer, Hugging Face TGI, vLLM. Inference pipelines (InferenceGraph) chain multiple models. KServe implements the KFServing V2 inference protocol standard. Scale-to-zero on Knative enables cost savings for low-traffic models in shared Kubernetes clusters.

# KServe InferenceService manifest
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
spec:
  predictor:
    model:
      modelFormat: {name: sklearn}
      storageUri: s3://models/fraud/v3

BentoML is a Python-native ML serving framework that packages models, pre/post-processing code, dependencies, and serving logic into a self-contained deployable unit called a Bento. Build a Bento with @bentoml.service decorator on a class that defines runner (model) and API endpoints. BentoML handles: dependency packaging (pip, conda), multi-framework support (PyTorch, sklearn, XGBoost, Transformers, ONNX), async serving, adaptive batching, and containerization. Deploy Bentos to Kubernetes, AWS SageMaker, Google Cloud Run, or BentoCloud. BentoML's Runner abstraction separates model inference from API logic and enables efficient resource allocation (model inference on GPU, pre/post-processing on CPU). It is particularly popular for teams wanting a Pythonic, code-first serving approach without YAML-heavy Kubernetes configuration.

@bentoml.service(resources={'gpu': 1})
class FraudDetector:
    model = bentoml.sklearn.get('fraud_rf:latest')
    @bentoml.api
    def predict(self, features: np.ndarray) -> np.ndarray:
        return self.model.predict(features)

Ray Serve is a scalable model serving library built on Ray that enables deployment of Python functions and classes as HTTP endpoints. Key strengths: composition of multiple models into inference pipelines using the @serve.deployment decorator; each deployment runs as a Ray actor with independent scaling. Pipelines are built by chaining deployments where the output of one becomes the input of the next, all within the Ray object store (zero-copy where possible). Ray Serve integrates with MLflow, HuggingFace, and PyTorch natively. Horizontal scaling: add replicas with num_replicas; vertical scaling: configure GPU/CPU resources per replica. Ray Serve handles request batching, request routing, health checks, and gradual rollouts. Best for multi-step inference pipelines (embedding → retrieval → reranking → generation in RAG systems).

@serve.deployment(num_replicas=2, ray_actor_options={'num_gpus': 1})
class Embedding:
    def __call__(self, query: str) -> list:
        return embedder.encode(query).tolist()

app = Embedding.bind()
serve.run(app)

Pruning reduces model size by setting subsets of weights to zero (creating sparsity). Unstructured pruning zeros out individual weights based on magnitude (smallest weights are removed), creating irregular sparse patterns. While theoretically achieving high compression ratios (up to 90%+ sparsity), irregular sparsity requires specialized sparse GPU kernels to realize speedups — most hardware cannot exploit it efficiently without library support (cuSPARSELt). Structured pruning removes entire structures: filters (convolutional channels), attention heads, or entire neurons. Structured sparsity is hardware-friendly — removing a channel produces a smaller dense matrix that runs efficiently on standard GEMM hardware. 2:4 structured sparsity (2 of every 4 weights zero) is natively supported by NVIDIA Ampere+ Sparse Tensor Cores, doubling throughput. Most production pruning targets structured pruning for actual deployment benefits.

import torch.nn.utils.prune as prune
# Structured: prune 20% of filters in a conv layer
prune.ln_structured(conv_layer, name='weight', amount=0.2, n=2, dim=0)

Speculative decoding uses a small fast draft model (e.g., 7B) to speculatively generate multiple candidate tokens (typically 3-8), then verifies all candidates in a single parallel forward pass of the large target model. Verification is cheap because self-attention on K tokens costs roughly the same as generating 1 token when K is small. If all K draft tokens are accepted (their probabilities match the target's distribution), K tokens are produced for the price of one target forward pass. Rejected tokens cause rollback to the divergence point. Speedup depends on the draft-target acceptance rate (typically 2-4x for coherent prompts). Speculative decoding reduces time-to-first-token (TTFT) and per-token latency while maintaining the exact target model output distribution — mathematically equivalent to sampling from the target model directly.

# TensorRT-LLM speculative decoding
runtime_config = ModelRunnerCppConfig(
    use_draft_model=True,
    draft_model='llama-3-1b',
    max_draft_tokens=5
)

HPA automatically adjusts the number of model serving pod replicas based on observed metrics. Standard HPA scales on CPU and memory utilization. For ML workloads, custom metrics HPA scales on: GPU utilization (via DCGM Exporter + Prometheus Adapter), request queue depth, requests per second, or inference latency percentiles. KEDA (Kubernetes Event-Driven Autoscaling) extends HPA with 50+ scalers including Prometheus, Kafka consumer lag, and AWS SQS queue depth — useful for batched inference pipelines. Challenges for ML: pod startup time (pulling large model images + loading weights) can take 30-120 seconds — use pre-loaded model weights via PVC or model cache to reduce cold start. Scale-to-zero (KEDA or Knative) reduces idle cost but increases first-request latency after scale-down.

# HPA scaling on custom GPU utilization metric
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  metrics:
  - type: External
    external:
      metric: {name: gpu_utilization}
      target: {type: AverageValue, averageValue: 70}

Percentile latency metrics measure the distribution of request latencies rather than averages (which obscure tail behavior). P50 (median): 50% of requests complete within this time — representative of the typical user experience. P95: 95% of requests complete within this time — what most users experience at the tail. P99: 99% of requests complete within this time — the worst 1% of requests, critical for real-time systems. Averages hide tail latency: 100 fast requests + 1 slow outlier can produce a low average but terrible P99. SLA agreements for ML serving typically specify P99 < 200ms. Optimize tail latency with request timeouts, circuit breakers, hedged requests (send to two replicas, use first response), and pre-warming to eliminate cold-start spikes. Use Prometheus histograms (histogram_quantile function) to compute percentiles from request duration observations.

# Prometheus PromQL: P99 latency
histogram_quantile(0.99,
  rate(inference_request_duration_seconds_bucket[5m])
)

Throughput (requests per second) and latency (per-request response time) are inversely related in model serving. Increasing batch size increases GPU utilization and throughput but adds queuing delay (latency) while requests wait to fill a batch. Without batching, latency is minimal but GPU utilization is low and throughput suffers. The optimal operating point depends on use case: interactive APIs (chatbots, real-time recommendations) prioritize P99 latency < 100ms, accepting lower throughput. Batch scoring pipelines (overnight predictions, embedding generation) maximize throughput with large batches. Balance with dynamic batching: set max_queue_delay_microseconds to define acceptable latency budget for batching; requests arriving within the window are batched. Monitor both metrics simultaneously — throughput at a given latency percentile (e.g., max RPS at P99 < 200ms) is the key production SLA metric.

Model warming pre-runs inference with dummy inputs before the server accepts real traffic, ensuring all lazy initialization is complete: CUDA kernels are compiled and cached (first-run JIT compilation can add seconds); cuDNN algorithm selection benchmarks are stored; connection pools are established; JVM JIT compilation is complete (Java-based servers). Without warming, the first real requests experience 1-30 second latency spikes visible to users. Implement warming in readiness probes — Kubernetes will not route traffic until the probe passes, which only happens after warm-up completes. Send multiple warm-up requests with representative input shapes and batch sizes, especially if using torch.backends.cudnn.benchmark=True (which benchmarks per-shape). Triton's model warmup configuration runs warm-up automatically at server startup before marking the model ready.

# Triton warmup config in config.pbtxt
model_warmup [
  {
    name: 'warmup_batch8'
    batch_size: 8
    inputs: { key: 'input' value: { dims: [224,224,3] data_type: TYPE_FP32 }}
  }
]

Seldon Core is a Kubernetes-native ML serving platform that extends the serving use case beyond prediction to include outlier detection, concept drift monitoring, and model explainability as first-class components. Its graph-based inference pipeline (SeldonDeployment CRD) allows routing requests through multiple services: a pre-processor, a model, an outlier detector, and an explainer in a configurable graph. Built-in integration with SHAP (via Alibi Explain) produces Shapley value explanations alongside predictions. Seldon supports multiple runtime frameworks via pre-packed server images and custom Python components. Prometheus and Grafana integration provides automatic metrics. Seldon Alibi Detect integrates for data drift monitoring. This makes Seldon particularly suited for regulated industries requiring explainable, monitored ML decisions.

# SeldonDeployment with explainer
spec:
  predictors:
  - graph:
      name: fraud-model
      type: MODEL
    explainer:
      type: AnchorTabular
      modelUri: s3://models/fraud/explainer

Prefix caching (also called KV cache reuse or prompt caching) stores the computed Key-Value cache for common prompt prefixes — system prompts, few-shot examples, or shared context — so they do not need to be reprocessed for each request. When multiple requests share the same prefix, only the unique portion (the actual user query) requires KV cache computation; the shared prefix KV cache is loaded from the cache. This is transformative for high-traffic conversational AI or RAG applications where every request shares a long system prompt (500-2000 tokens). Speedups of 2-5x in time-to-first-token and significant cost reduction. vLLM, TensorRT-LLM, and Anthropic's API implement prefix caching. Cache hit rate depends on prefix length and request diversity — long fixed prefixes with diverse queries achieve near-100% cache hit rates.

# vLLM prefix caching (enabled by default in recent versions)
llm = LLM(model='Qwen2-72B', enable_prefix_caching=True)
# Requests sharing the same long system prompt reuse KV cache

Circuit breakers prevent cascading failures in ML serving systems. When a downstream model service is overwhelmed or slow, requests pile up, exhausting client threads and propagating failure upstream. A circuit breaker tracks failure rate and response time: in Closed state, all requests pass through; after the failure threshold is crossed (e.g., 50% errors in 10 seconds), the breaker Opens — requests immediately receive a fallback response (cached prediction, default value, or simpler fallback model) without hitting the failing service; after a timeout, the breaker enters Half-Open state to probe with a few requests before deciding to Close or remain Open. Implement with Resilience4j (Java), tenacity (Python), Envoy service mesh (Kubernetes-native), or Istio's outlier detection. Circuit breakers are essential in multi-model serving graphs where one model failure should not cascade.

# Python circuit breaker with pybreaker
import pybreaker
breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)
@breaker
def call_model(payload):
    return requests.post(MODEL_URL, json=payload).json()

Client-side request batching sends multiple inference inputs in one HTTP request payload (e.g., an array of 32 examples in one POST /predict call). This reduces HTTP round-trip overhead and enables the server to efficiently batch inputs for GPU processing. Dynamic batching is server-side: the server aggregates individually arriving single-example requests into a batch internally, combining requests from different clients within a time window. The two techniques complement each other: clients batch what they can (e.g., image processing pipelines sending 8 images per call), and the server further aggregates multiple client batches up to GPU-optimal size. For maximum throughput, align client batch size with GPU tensor core alignment requirements (multiples of 8 or 16 for FP16) and configure server max_batch_size accordingly.

# Client sends a batch of 16 examples in one request
import requests
payload = {'instances': [[...] for _ in range(16)]}
response = requests.post('http://model-server/v1/predict', json=payload)

Multi-version serving requires version-aware infrastructure at three levels. Framework level: Triton and TF Serving natively serve multiple model versions with version policies (latest_n, all, specific). Load balancer level: use weighted routing to split traffic between version-specific endpoints (e.g., 90% → v3, 10% → v4 for A/B testing). Kubernetes level: deploy multiple Deployments (one per version) with a shared Service and Ingress traffic weights (Nginx ingress weight annotations, Istio VirtualService weights). Use feature flags or request headers to force routing to a specific version for internal testing. The version served is logged with predictions for analysis. MLflow Registry + Seldon or KServe provide native version management with traffic split APIs. Always include version metadata in prediction responses for traceability.

# Istio VirtualService traffic split
http:
- route:
  - destination: {host: model-v3, port: {number: 80}}
    weight: 90
  - destination: {host: model-v4, port: {number: 80}}
    weight: 10

TorchServe is PyTorch's native model serving framework maintained by AWS and Meta. It packages PyTorch models into model archive (.mar) files using torch-model-archiver: a single file containing model weights, a handler class (pre/post-processing logic), and any additional files (tokenizers, config files). The handler implements initialize(), preprocess(), inference(), and postprocess() methods. Deploy .mar files to TorchServe's model store directory. TorchServe exposes REST (/predictions/) and gRPC APIs, management API for runtime model registration, metrics API (Prometheus format), and supports multi-model serving with per-model worker counts. Limitations vs. Triton: less framework-agnostic, no native ensemble support, Python GIL limits concurrency in multi-worker setup. Best for PyTorch-only shops wanting a simple serving solution.

# Package model
torch-model-archiver --model-name resnet50 \
  --version 1.0 \
  --model-file model.py \
  --serialized-file resnet50.pt \
  --handler image_classifier \
  --export-path model_store

Ollama is a lightweight tool for running LLMs locally with minimal setup. It manages model downloads (GGUF quantized format), provides a simple REST API compatible with the OpenAI API schema, and handles GPU/CPU inference routing automatically. Install via a single shell command; pull models with ollama pull llama3:8b; serve via ollama serve which exposes POST /api/generate and POST /api/chat at localhost:11434. Ollama uses llama.cpp under the hood for CPU inference and CUDA for GPU. Key use cases: developer workstations, air-gapped on-premise environments, RAG prototyping with LangChain/LlamaIndex (native Ollama integration), and edge deployments. Limitations: not designed for high-concurrency production serving (vLLM or TGI preferred for that); single request at a time on most configurations.

# Terminal
ollama pull llama3.1:8b
ollama serve

# API call
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Hello"}]
}'

SGLang (Structured Generation Language) is an LLM serving framework designed for complex generation programs with multiple LLM calls, branching, and structured outputs. It introduces RadixAttention — a KV cache data structure using a radix tree to share KV cache across requests that share common prefixes dynamically (including multi-call patterns). SGLang's runtime compiles generation programs into efficient batched execution, handling control flow (if/else based on model output), parallel sampling, and constrained decoding (JSON schema, regex). Benchmarks show 1.5-5x throughput improvement over vLLM for structured multi-call workflows (chain-of-thought, multi-agent, classification + extraction pipelines). Support for FP8, tensor parallelism, and speculative decoding. Key differentiator from vLLM: optimized for complex multi-call patterns rather than simple single-turn completion.

import sglang as sgl
@sgl.function
def multi_step(s, question):
    s += sgl.user(question)
    s += sgl.assistant(sgl.gen('thought', max_tokens=256))
    s += sgl.user('Now give a short answer.')
    s += sgl.assistant(sgl.gen('answer', max_tokens=50))

Kubernetes uses liveness probes (restart pod if unhealthy) and readiness probes (remove from load balancer if not ready) for model serving. For ML serving containers, implement: liveness at /health returning 200 OK when the server process is running and not deadlocked; readiness at /ready returning 200 only after model weights are loaded, GPU memory is allocated, and warm-up inference passes succeed. Use HTTP GET probes with appropriate initialDelaySeconds (60-120s for large models) to give models time to load before being killed. failureThreshold × periodSeconds should exceed the slowest expected startup time. Startup probes (separate from liveness) handle slow-starting models — they disable liveness checking during startup and kick in after the initial load phase completes.

livenessProbe:
  httpGet: {path: /health, port: 8080}
  periodSeconds: 30
readinessProbe:
  httpGet: {path: /ready, port: 8080}
  initialDelaySeconds: 120
  periodSeconds: 10
startupProbe:
  httpGet: {path: /ready, port: 8080}
  failureThreshold: 30
  periodSeconds: 10

LoRA (Low-Rank Adaptation) fine-tunes LLMs by training small adapter matrices (rank r << d) while keeping the base model frozen. For serving, multiple LoRA adapters can be loaded on top of a single shared base model, enabling multi-tenant serving where different customers or use cases have different fine-tuned behaviors without duplicating the expensive base model weights. Paged LoRA serving (implemented in vLLM and Punica) multiplexes requests from different adapters in the same batch: each request knows which adapter to apply, and the serving engine applies the appropriate LoRA delta matrices per request. Memory overhead is linear with the number of adapters (each ~10-100MB vs. base model's 10-40GB). This enables 100+ fine-tuned variants on one GPU cluster — a major cost advantage for multi-tenant ML platforms.

# vLLM LoRA serving
llm = LLM(model='base-model', enable_lora=True, max_loras=4)
request = {'prompt': '...', 'lora_request': LoRARequest('customer_A', 1, 'path/to/loraA')}

Cost efficiency for model serving is measured as cost per inference ($ per 1M requests) or cost per useful output ($/token for LLMs). Key optimization levers: GPU utilization (maximize with dynamic batching and continuous batching — idle GPUs still cost money); model optimization (quantization reduces GPU memory, enabling more model instances per GPU or smaller GPU instances); right-sizing (use GPU profiles that match model requirements — avoid V100 for a model that fits on a T4); spot/preemptible instances for batch scoring workloads (60-90% cheaper); scale-to-zero for low-traffic models; autoscaling to match capacity to demand curves. For LLMs, cost is dominated by KV cache memory: PagedAttention and prefix caching directly reduce cost by increasing effective throughput per GPU-hour. Track $/query as a primary metric alongside latency SLAs.

HuggingFace TGI is an open-source production-grade LLM serving toolkit. Key features: continuous batching for high-throughput generation; optimized attention kernels (Flash Attention 2, Paged Attention); tensor parallelism for multi-GPU serving via NCCL; quantization support (GPTQ, AWQ, EETQ, FP8 on H100); token streaming via Server-Sent Events; OpenAI API compatibility for easy client migration; Prometheus metrics; and support for HuggingFace Hub model loading. TGI is the backend for HuggingFace Inference Endpoints. It also supports vision-language models, speculative decoding, and guided generation (JSON schema outputs). Deploy via Docker: a single docker run command serves any supported HuggingFace model. TGI is particularly well-integrated with the HuggingFace ecosystem and is production-tested at scale by HuggingFace.

docker run --gpus all -p 8080:80 \
  -e MODEL_ID=meta-llama/Llama-3-8B-Instruct \
  ghcr.io/huggingface/text-generation-inference:latest

Rate limiting protects model serving from overload and ensures fair multi-tenant usage. Implementation options: API gateway (AWS API Gateway, Kong, NGINX Plus) enforce per-client rate limits (requests per second, concurrent connections) before requests reach the serving infrastructure. Token bucket and leaky bucket are the standard algorithms: token bucket allows bursts up to bucket capacity; leaky bucket smooths bursts into a steady rate. In Kubernetes, use Envoy/Istio's local rate limiting (per-pod) or global rate limiting (via Envoy's external rate limit service backed by Redis). Implement application-level rate limiting with Python libraries (slowapi for FastAPI, flask-limiter) for fine-grained per-user control. Return HTTP 429 (Too Many Requests) with Retry-After headers when limits are exceeded. Implement separate rate limits for synchronous vs. streaming requests.

# FastAPI rate limiting with slowapi
from slowapi import Limiter
from slowapi.util import get_remote_address
limiter = Limiter(key_func=get_remote_address)
@app.post('/predict')
@limiter.limit('60/minute')
async def predict(request: Request): ...

Many open-source LLM serving frameworks (vLLM, Ollama, LiteLLM, TGI, SGLang) implement the OpenAI REST API spec — specifically POST /v1/chat/completions, POST /v1/completions, and POST /v1/embeddings with identical request/response schemas. This compatibility means any application built for OpenAI's API (using the openai Python SDK or direct HTTP calls) can switch to a self-hosted model by changing only the base_url parameter — no code changes. This dramatically simplifies: vendor migration without rewriting application code, local development with Ollama instead of paying for API calls, model routing (LiteLLM proxies multiple backends behind one API), and testing with cheaper models. The OpenAI API has become the de facto inference protocol standard for LLM serving, similar to what S3-compatible APIs did for object storage.

from openai import OpenAI
# Point to self-hosted vLLM server instead of OpenAI
client = OpenAI(api_key='dummy', base_url='http://localhost:8000/v1')
response = client.chat.completions.create(model='llama3.1:8b', messages=[...])

LiteLLM is an open-source Python library and proxy server that provides a unified OpenAI-compatible interface across 100+ LLM providers (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Cohere, Ollama, vLLM, and more). As a library, litellm.completion() translates API calls to the correct provider SDK automatically. As a proxy, it exposes one OpenAI-compatible endpoint that routes to any configured provider. Key features: automatic provider failover (route to Anthropic if OpenAI times out), load balancing across providers, cost tracking per provider/model/user, rate limit handling with retry logic, prompt caching, and budget enforcement. LiteLLM solves the vendor lock-in problem in production LLM systems where application code must work with multiple models for cost, capability, and reliability reasons.

import litellm
response = litellm.completion(
    model='anthropic/claude-3-5-sonnet-20241022',
    messages=[{'role': 'user', 'content': 'Hello'}],
    fallbacks=['openai/gpt-4o', 'bedrock/claude-3']  # auto-failover
)

Cold starts occur when a model serving container starts from scratch — downloading the container image, loading model weights into GPU memory, and running warm-up inference can take 30-300 seconds for large models. Mitigation strategies: (1) Provisioned concurrency or minimum replicas (keep at least 1 warm replica always running — eliminates cold starts at cost of idle spend); (2) Pre-baked images with model weights included (avoids download time but increases image size); (3) Model weight caching on persistent volumes or node-local NVMe storage (warm weights available immediately on pod restart); (4) Progressive loading — serve smaller model immediately while large model loads, then switch; (5) Implement readiness probes that block traffic until warm; (6) Use scale-to-1 instead of scale-to-zero for latency-sensitive models; (7) CUDA context persistence to avoid re-initialization.

# KServe min replicas to prevent cold start
spec:
  predictor:
    minReplicas: 1   # always 1 warm replica
    maxReplicas: 10
    scaleTarget: 80  # scale at 80% GPU utilization

ONNX Runtime (ORT) uses Execution Providers (EPs) as pluggable hardware backends. Each EP is optimized for a specific hardware target. CUDAExecutionProvider: GPU inference with cuDNN and cuBLAS; fastest for NVIDIA GPU inference. TensorrtExecutionProvider: compiles to TensorRT for maximum GPU optimization; requires TRT engine build time but achieves highest throughput. CPUExecutionProvider: OpenMP-optimized CPU inference; default fallback. ROCmExecutionProvider: AMD GPU support. CoreMLExecutionProvider: Apple Silicon and iOS. DirectMLExecutionProvider: Windows GPU (NVIDIA/AMD). OpenVINOExecutionProvider: Intel CPU/GPU/VPU optimization. ORT falls back to CPUExecutionProvider for ops not supported by the selected EP. Configure EPs in priority order: ORT tries each EP in sequence; ops are split across EPs based on what each supports. Profile with ORT's performance tools to identify EP assignment.

import onnxruntime as ort
sess_options = ort.SessionOptions()
session = ort.InferenceSession(
    'model.onnx',
    providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'],
    sess_options=sess_options
)

A/B testing in ML serving compares model versions (A = control, B = treatment) by routing traffic randomly to each and measuring outcome metrics (click-through rate, conversion, accuracy on labeled samples, revenue). Statistical significance: use a two-sample t-test or z-test for continuous metrics; chi-squared test for conversion rates. Determine required sample size upfront using power analysis (balancing effect size, significance level α=0.05, power 1-β=0.80) to avoid underpowered tests. Run tests for at least one full business cycle (7 days minimum for weekly seasonality). Implement with feature flags in application code or traffic splitting at the API gateway level (not load balancer, as users should consistently hit the same variant per session). Log model version with each prediction for retrospective analysis. Avoid novelty effects by excluding users who saw model A before the experiment.

from scipy import stats
# Compare conversion rates between A and B
control_conv = control_conversions / control_requests
treat_conv = treatment_conversions / treatment_requests
_, p_value = stats.proportions_ztest([control_conversions, treatment_conversions],
                                      [control_requests, treatment_requests])
print(f'p-value: {p_value:.4f}, significant: {p_value < 0.05}')

Multi-model serving handles complex ML pipelines where multiple models process requests sequentially or in parallel. Triton's ensemble feature defines a pipeline in a config.pbtxt ensemble scheduler that specifies an input, a sequence of model steps (each step's input can be the previous step's output or the original request input), and final output. For example: preprocessing model → feature extraction model → classification model, chained with zero data copy between steps on the same GPU. Steps can run in parallel if their inputs are independent. The ensemble executes atomically from the client's perspective — one HTTP/gRPC call triggers the entire pipeline. This eliminates network round-trips between models, improves throughput, and simplifies client code. Use Triton BLS (Business Logic Scripting) for conditional routing within ensembles.

# Triton ensemble config.pbtxt
platform: 'ensemble'
ensemble_scheduling {
  step { model_name: 'preprocessor' model_version: 1 }
  step { model_name: 'classifier' model_version: 2 }
}

Model output caching stores predictions for recently seen or frequently repeated inputs, serving cached results without re-running inference. Effective for: search ranking (same query repeated millions of times), content moderation (identical social media posts), recommendation systems (hot items). Implement with Redis (key = hash of input, value = model output, TTL = cache validity period). Pitfalls: stale predictions after model updates (cache must be invalidated on model promotion — use model version in the cache key); cache poisoning if input hashing has collisions; inappropriate for personalized or time-sensitive predictions; false cache hits if input normalization is inconsistent (trailing spaces, casing). Semantic caching (used by GPTCache) embeds inputs and returns cached responses for semantically similar (not identical) inputs — reduces LLM cost but risks returning slightly off-topic cached responses.

import redis, hashlib, json
r = redis.Redis()
def predict_with_cache(features, model_version='v3'):
    key = f'{model_version}:{hashlib.md5(str(features).encode()).hexdigest()}'
    if cached := r.get(key):
        return json.loads(cached)
    result = model.predict(features)
    r.setex(key, 3600, json.dumps(result))
    return result

Graceful degradation ensures the system remains functional (at reduced quality) when the primary ML model fails or is overloaded. Implement a tiered fallback strategy: Tier 1 — primary ML model (best quality, highest cost); Tier 2 — simpler/smaller model (faster, lower quality); Tier 3 — rule-based heuristics or cached statistics (offline computed); Tier 4 — default response (genre recommendation → 'top 10 popular items'). Trigger fallbacks based on: primary model timeout exceeded, HTTP 5xx from serving endpoint, circuit breaker open, queue depth threshold exceeded, or feature store unavailability. Log all fallback events with reason codes for monitoring. Design fallback responses to be clearly labeled (e.g., response header X-Model: fallback-v2) for product and business analysis. Test fallback paths in staging by injecting faults.

async def predict_with_fallback(features):
    try:
        return await asyncio.wait_for(primary_model(features), timeout=0.1)
    except (TimeoutError, ServiceUnavailable):
        log_fallback()
        return await fallback_model(features)  # simpler model