Kubernetes and model serving

A Deployment manages a ReplicaSet that maintains a desired number of identical Pod replicas, providing self-healing (restarting failed Pods, rescheduling on node loss), declarative rolling updates, and easy rollbacks. A bare Pod has no controller, so if the node fails or the Pod crashes it is not rescheduled. For model serving this means the Deployment guarantees availability and lets you update model versions with zero-downtime rollouts.

A liveness probe determines whether the container is alive; if it fails, the kubelet restarts the container. A readiness probe determines whether the container can serve traffic; if it fails, the Pod is removed from Service endpoints but not restarted. During large model loading, you want the readiness probe to fail until weights are loaded (so no traffic is routed) while the liveness probe has a generous initialDelay/threshold (or a startup probe) so the loading Pod is not killed prematurely.

readinessProbe:
  httpGet: { path: /v1/health/ready, port: 8080 }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /v1/health/live, port: 8080 }
  initialDelaySeconds: 120
  failureThreshold: 6

The HPA periodically reads metrics (default: CPU utilization) and adjusts the replica count to keep the metric near a target. For GPU-bound LLM inference, CPU is often near-idle while the GPU and request queue are saturated, so CPU-based scaling under-provisions. Better signals are custom/external metrics such as GPU utilization, in-flight requests, queue depth, or tokens-per-second exposed via the Prometheus Adapter or KEDA.

KServe provides a higher-level InferenceService CRD that abstracts deployment, autoscaling (including scale-to-zero via Knative), canary rollouts, payload logging, and a standardized inference protocol (V2/Open Inference Protocol) across many runtimes (Triton, TorchServe, SKLearn, vLLM). Instead of hand-writing Deployments, Services, HPAs, and ingress, you declare one InferenceService and KServe orchestrates the rest, including transformer/predictor/explainer components.

You install the vendor device plugin (e.g., NVIDIA k8s-device-plugin), which advertises an extended resource like nvidia.com/gpu on each node. Pods request it under resources.limits; GPUs are non-overcommittable integer resources, so request equals limit and a whole GPU is allocated per request unit. The scheduler only places the Pod on nodes with enough free GPUs, and time-slicing or MIG is required to share a single GPU across Pods.

resources:
  limits:
    nvidia.com/gpu: 1

Scale-to-zero (e.g., via Knative/KServe) removes all replicas when there is no traffic, eliminating idle GPU cost. The tradeoff is cold-start latency: the next request must wait for a Pod to be scheduled, the container image pulled, and the (often multi-GB) model weights loaded into GPU memory, which can take tens of seconds to minutes. Mitigations include keeping a warm minScale of 1, node pre-warming, image/weight caching, and faster model loaders.

Triton supports multiple frameworks (TensorRT, ONNX, PyTorch, TensorFlow, Python) behind one server, with dynamic batching, concurrent model execution, model ensembles, GPU/CPU placement, and both HTTP and gRPC using the standard inference protocol. It maximizes GPU utilization by batching independent client requests and running multiple model instances per GPU, which is hard to replicate with a hand-rolled Flask server.

Dynamic batching collects individual incoming requests over a short window and runs them through the model as a single batch, amortizing kernel-launch and memory overhead to raise GPU throughput. The tradeoff is added queueing latency: a max_queue_delay (or batch timeout) controls how long requests wait to form a batch. Larger batches and longer windows increase throughput but also tail latency, so the window is tuned to an SLO.

A canary routes a small percentage of traffic (e.g., 10%) to the new model version while the rest stays on the stable version, letting you monitor quality and latency before full rollout. In KServe you set canaryTrafficPercent on the InferenceService, and the underlying network layer (Knative/Istio) splits traffic between the previous and latest revisions, allowing gradual promotion or instant rollback by reverting the percentage.

spec:
  predictor:
    canaryTrafficPercent: 10
    model:
      modelFormat: { name: pytorch }
      storageUri: s3://models/v2

An Init Container runs to completion before the app containers start and shares the Pod's volumes, making it ideal for setup tasks. A common use case is downloading or pulling model weights from object storage (S3/GCS/HF Hub) into a shared emptyDir or PVC so the main serving container starts with the model already present. This separates fetch logic from the runtime image and lets the readiness probe reflect only inference readiness.

A sidecar runs alongside the main container in the same Pod, sharing network and volumes, to add cross-cutting functionality without modifying the model server. Examples include an Istio Envoy proxy for mTLS and traffic routing, a log/metrics shipper, or a request/response logger that captures payloads for monitoring and drift detection. The sidecar can be updated independently of the model runtime.

A PDB limits how many Pods of an application can be voluntarily disrupted at once (e.g., during node drains, upgrades, or autoscaler scale-down) by specifying minAvailable or maxUnavailable. For inference it ensures that maintenance operations cannot take down so many replicas that the service drops below its capacity SLO. PDBs only constrain voluntary disruptions, not hardware failures.

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: model-server }

Requests are used by the scheduler to find a node with enough capacity and to set guaranteed resources; limits cap usage at runtime. If a Pod's memory usage exceeds its limit it is OOM-killed, and if requests are set far below actual usage the node can become overcommitted and evict Pods under pressure. For predictable inference, set memory request close to real usage and equal to the limit (Guaranteed QoS) to avoid eviction and OOM surprises.

MIG partitions an A100/H100 GPU into hardware-isolated instances each with dedicated compute SMs and memory, giving predictable performance and fault isolation between Pods. Time-slicing instead lets multiple Pods share the full GPU by interleaving in time with no memory or fault isolation, which is simpler and works on most GPUs but offers no QoS guarantees and risks contention. MIG suits multi-tenant SLO-bound serving; time-slicing suits bursty, best-effort workloads.

The predictor is the core component that loads the model and returns predictions. The transformer is an optional pre/post-processing step (e.g., tokenization, feature transforms, image decoding) that runs before/after the predictor, allowing the predictor to focus on raw tensor inference. The explainer is an optional component that produces model explanations (e.g., SHAP/LIME-style attributions). They are deployed as separate containers/services that KServe chains together.

Large weights are better stored in object storage (S3/GCS/Azure Blob) or a model registry (MLflow, Hugging Face Hub) and pulled at startup via an init container or the runtime's storageUri, with versioned paths or registry versions for reproducibility. Baking multi-GB weights into the image bloats it, slows image pulls and cold starts, couples model updates to image rebuilds, and wastes registry storage. Externalizing weights decouples model versioning from code/runtime versioning.

A ClusterIP Service provides a stable virtual IP and DNS name that load-balances across the healthy backing Pods selected by label, decoupling clients from individual Pod IPs that change on rescheduling. It only forwards traffic to Pods listed in its Endpoints, and a Pod is added to Endpoints only when its readiness probe passes. Thus readiness gating ensures the Service never routes requests to a Pod that is still loading its model.

gRPC uses HTTP/2 with multiplexed streams and compact Protobuf binary serialization, reducing payload size and per-request overhead versus JSON-over-HTTP/1.1, which lowers latency and raises throughput for tensor payloads. A key Kubernetes caveat is load balancing: HTTP/2 keeps long-lived connections, so an L4 (kube-proxy) load balancer pins a client connection to one backend Pod, causing uneven load — you need an L7/gRPC-aware proxy (Envoy/Linkerd/Istio) or client-side LB to spread requests.

The Cluster Autoscaler adds or removes nodes when Pods are unschedulable due to insufficient resources or when nodes are underutilized. It complements the HPA: when the HPA scales up inference replicas and there is no GPU node with capacity, the Pods stay Pending and the Cluster Autoscaler provisions a new GPU node (often from a dedicated node group). Conversely, when replicas scale down and GPU nodes go idle, it removes them to cut cost.

A taint on a GPU node repels Pods that do not explicitly tolerate it, preventing ordinary CPU workloads from landing on costly GPU hardware. Inference Pods add a matching toleration (and usually a nodeSelector/affinity for the GPU node label) so only they schedule there. Combined, taints+tolerations plus affinity reserve GPU nodes for GPU jobs and keep them from being wasted on non-GPU Pods.

tolerations:
- key: "nvidia.com/gpu"
  operator: "Exists"
  effect: "NoSchedule"

KEDA (Kubernetes Event-Driven Autoscaling) provides scalers for many external sources (Kafka, RabbitMQ, SQS, Prometheus, Redis, HTTP) and can scale a Deployment from zero based on event/queue depth, which the native HPA cannot do alone. Under the hood KEDA drives an HPA for the 1-to-N range and adds its own activation controller for the 0-to-1 transition, so a model consumer scales up when queue backlog grows and scales to zero when the queue is empty. This is ideal for asynchronous/batch inference pipelines.

vLLM uses PagedAttention to manage the KV cache in non-contiguous GPU memory pages, drastically reducing memory fragmentation and enabling far higher concurrency, while continuous (in-flight) batching adds/removes sequences from the running batch each step to keep the GPU busy. On Kubernetes you containerize vLLM, request GPUs, and scale replicas/HPA behind a Service or KServe, getting high token throughput per GPU with standard orchestration, autoscaling, and rollout tooling.

RollingUpdate incrementally replaces old Pods with new ones (governed by maxSurge/maxUnavailable), keeping the service available throughout, whereas Recreate terminates all old Pods before creating new ones, causing downtime. For stateless model servers, RollingUpdate is preferred because it gives zero-downtime upgrades and works with readiness probes; Recreate is mainly used when old and new versions cannot run simultaneously (e.g., incompatible exclusive resources).

A StatefulSet gives Pods stable, ordered identities (stable network names and per-Pod persistent volumes) and ordered, predictable creation/scaling, unlike a Deployment whose Pods are interchangeable. Most stateless model servers should use a Deployment, but a StatefulSet helps when serving requires sharded model state, stable peer addressing for tensor/pipeline-parallel multi-node inference, or large per-Pod local weight caches that must survive rescheduling. The tradeoff is slower, ordered rollouts.

Key metrics include request rate (RPS), error rate, p50/p95/p99 latency, queue/batch wait time, GPU utilization and memory, in-flight requests, and (for LLMs) time-to-first-token and tokens-per-second. Tail latency, especially p99/p95 and time-to-first-token, is most directly tied to user experience because users feel the slow requests, not the average. These are typically scraped by Prometheus and used to drive autoscaling and alerting.

ConfigMaps hold non-sensitive configuration (model paths, batch size, runtime flags, environment) and Secrets hold sensitive data (object-storage credentials, registry tokens, API keys), both injected as env vars or mounted files so config is decoupled from the image. The key caveat is that Kubernetes Secrets are only base64-encoded, not encrypted, by default; you should enable encryption-at-rest for etcd, restrict RBAC access, and prefer an external secrets manager (Vault, cloud KMS/CSI driver).

The Open Inference Protocol (V2, originated by KServe/Triton) is a standardized HTTP/REST and gRPC API for model inference, with common endpoints for server/model readiness, metadata, and inference request/response tensor formats. Standardizing the wire protocol lets clients and gateways work across runtimes (Triton, TorchServe, MLServer, vLLM adapters) and lets you swap serving backends without rewriting client code. This portability is central to KServe's multi-runtime model mesh.

ModelMesh loads many models into a shared pool of serving Pods and dynamically loads/unloads models in memory based on demand (an LRU-style cache), so thousands of models share a handful of Pods instead of dedicating a Pod per model. This dramatically improves resource density and cost for large numbers of small or infrequently used models. The tradeoff is cold-load latency when a requested model is not currently resident and must be loaded on demand.

An Ingress/Gateway exposes internal Services to external clients via a single entry point and provides L7 routing by host/path, TLS termination, and often rate limiting, authentication, and header-based or weighted traffic splitting. For model serving this lets you route /v2/models/... to the right backend, terminate HTTPS, enforce auth and quotas, and do canary traffic splits centrally instead of per-Service. The Gateway API is the newer, more expressive successor to Ingress.

Strategies include keeping a warm minimum replica (minScale/HPA minReplicas >= 1) to avoid scale-to-zero, pre-pulling images to nodes and slimming image size, caching weights on a fast PVC/local NVMe or via a shared cache (e.g., an init container or CSI), pre-provisioning/over-provisioning GPU nodes with low-priority pause Pods so capacity is ready, and using faster loaders (safetensors, lazy/streamed loading, GPUDirect/streaming weight loaders). Each addresses one of scheduling, image pull, or weight-load latency.

Node affinity attracts Pods to nodes with certain labels (e.g., GPU type/zone); pod affinity co-locates Pods near other specified Pods; pod anti-affinity repels Pods from each other. For availability, anti-affinity spreads inference replicas across different nodes/zones so a single node or zone failure cannot take down all replicas. Topology spread constraints are a more declarative way to achieve the same even distribution.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels: { app: model-server }
      topologyKey: kubernetes.io/hostname

A Job runs Pods to completion for finite work (and CronJob runs Jobs on a schedule), unlike a Deployment which keeps Pods running indefinitely for online serving. They suit batch/offline inference such as scoring a large dataset overnight, periodic embedding refreshes, or nightly re-ranking — work that starts, processes a bounded input, and exits. This avoids paying for idle long-running Pods between batch runs.

A ResourceQuota caps the aggregate resources (CPU, memory, GPUs, object counts) a namespace can consume, preventing one team/model from starving others, while a LimitRange sets default and min/max per-Pod/container requests and limits within a namespace. Together they enforce fair sharing of scarce GPUs, ensure every Pod has sensible requests/limits (avoiding overcommit), and provide guardrails for tenants. This is essential when many teams share an expensive GPU cluster.

A service mesh injects sidecar proxies (or runs an ambient/sidecarless data plane) that add mutual TLS encryption, fine-grained L7 traffic management (weighted routing for canary/A-B model tests, retries, timeouts, circuit breaking), and rich telemetry (latency, error rates, distributed tracing) without changing the model server code. For serving, this enables secure pod-to-pod communication, controlled progressive rollouts of model versions, and observability that drives SLO monitoring. KServe relies on such a mesh/Knative for its traffic splitting.