Kubernetes and model serving quiz

Question 1

What is the primary advantage of using a Kubernetes Deployment over a bare Pod for serving an ML model?

Accepted Answer

A Deployment manages replicas with self-healing, rolling updates, and rollbacks; a bare Pod has no controller and won't be rescheduled on failure

Answer

A Deployment runs Pods on GPUs while bare Pods only run on CPUs

Answer

A Deployment automatically encrypts model weights at rest

Answer

A Deployment is required to expose a model via a Service, but a Pod cannot be exposed

Question 2

In Kubernetes, what is the difference between a readiness probe and a liveness probe for a model server, and why does it matter during model loading?

Accepted Answer

Liveness restarts unhealthy containers; readiness gates traffic without restarting — keep readiness failing during load while liveness stays tolerant

Answer

They are identical; readiness is just the older name for liveness

Answer

Readiness restarts the container while liveness only logs a warning

Answer

Liveness gates Service traffic while readiness restarts the Pod

Question 3

How does the Horizontal Pod Autoscaler (HPA) scale an inference deployment, and why is CPU utilization often a poor metric for GPU-bound LLM serving?

Accepted Answer

HPA adjusts replicas toward a target metric; CPU is misleading for GPU work, so use custom metrics like queue depth or GPU/token throughput

Answer

HPA scales by changing the CPU/memory limits of existing Pods rather than replica count

Answer

CPU is the ideal metric because GPU load always tracks CPU load linearly

Answer

HPA can only scale on memory, never on custom metrics

Question 4

What problem does KServe (formerly KFServing) solve compared to manually building Deployments and Services for model serving?

Accepted Answer

It offers an InferenceService CRD abstracting autoscaling, canary, scale-to-zero, and a standard inference protocol across runtimes

Answer

It is only a GPU device plugin for scheduling

Answer

It replaces etcd with a model-optimized datastore

Answer

It is a CNI plugin that speeds up gRPC between Pods

Question 5

How do you expose GPUs to Pods in Kubernetes, and what scheduling behavior do GPU resource requests have?

Accepted Answer

A device plugin advertises nvidia.com/gpu; it's a non-overcommittable integer limit, so sharing needs MIG or time-slicing

Answer

You set a gpu: true flag in the Pod spec and Kubernetes auto-detects GPUs

Answer

GPUs are requested as fractional millicores like CPU

Answer

Any Pod can access the host GPU by default without configuration

Question 6

What is scale-to-zero in serverless model serving, and what is its main latency tradeoff?

Accepted Answer

It drops replicas to zero to save cost but introduces cold-start latency from scheduling, image pull, and weight loading

Answer

It guarantees zero latency by pre-loading all models into every node

Answer

It is a billing feature with no effect on request latency

Answer

It permanently deletes the model from the cluster when idle

Question 7

Why might you choose NVIDIA Triton Inference Server as the container runtime inside a model-serving Pod?

Accepted Answer

It serves many frameworks with dynamic batching, concurrent execution, ensembles, and gRPC/HTTP to maximize GPU utilization

Answer

It is a Kubernetes scheduler replacement specialized for ML

Answer

It only serves TensorRT models and nothing else

Answer

It is a CLI tool for converting models with no serving capability

Question 8

What is dynamic (server-side) batching in model serving and what latency-throughput tradeoff does it introduce?

Accepted Answer

It groups requests into one batch to raise throughput at the cost of added queueing latency controlled by a max delay window

Answer

It splits one request across multiple GPUs to reduce latency only

Answer

It caches responses so repeated requests are free

Answer

It compresses model weights to fit larger batches in memory

Question 9

How does a canary deployment for a new model version work in Kubernetes, and what does KServe provide to support it?

Accepted Answer

A small traffic share goes to the new version; KServe's canaryTrafficPercent splits traffic via Knative/Istio for safe promotion or rollback

Answer

Canary means deploying to a separate cluster with no traffic at all

Answer

KServe runs both versions and randomly returns whichever responds first

Answer

Canary requires manually editing iptables on each node

Question 10

What is the role of an Init Container in a model-serving Pod, and give a common use case.

Accepted Answer

It runs before app containers to prepare shared volumes — e.g., downloading model weights into a shared volume before serving starts

Answer

It runs alongside the app container as a continuous sidecar

Answer

It restarts the main container whenever the model updates

Answer

It is a privileged container that installs GPU drivers on the host

Question 11

Why is a sidecar container pattern useful in model serving, and name one concrete example.

Accepted Answer

A co-located helper container sharing network/volumes — e.g., an Envoy proxy or a payload logger for monitoring

Answer

A container that runs only after the main container exits

Answer

A separate Deployment that proxies to the model from another namespace

Answer

A tool that merges multiple Pods into one container at build time

Question 12

What is a PodDisruptionBudget (PDB) and why is it important for a highly available inference service?

Accepted Answer

It caps voluntary disruptions (e.g., node drains) via minAvailable/maxUnavailable so maintenance can't drop the service below capacity

Answer

It automatically reschedules Pods after any node hardware failure

Answer

It sets a budget in dollars for cluster autoscaling

Answer

It restarts Pods on a fixed schedule to refresh memory

Question 13

How do resource requests and limits affect inference Pod scheduling and stability, and what risk do mismatched memory request/limit values create?

Accepted Answer

Requests drive scheduling/guarantees and limits cap usage; under-set memory causes overcommit/eviction and exceeding the limit triggers OOM-kill

Answer

Limits are ignored for memory and only apply to CPU

Answer

Setting requests higher than limits gives the Pod priority access to GPUs

Answer

Requests and limits only affect logging verbosity

Question 14

What is the difference between MIG (Multi-Instance GPU) and time-slicing for sharing a GPU across inference Pods?

Accepted Answer

MIG gives hardware-isolated GPU partitions with dedicated memory/compute; time-slicing interleaves Pods on the full GPU with no isolation

Answer

MIG and time-slicing are two names for the same software feature

Answer

Time-slicing provides stronger memory isolation than MIG

Answer

MIG only works on CPUs while time-slicing is GPU-only

Question 15

In KServe, what are the predictor, transformer, and explainer components of an InferenceService?

Accepted Answer

Predictor does inference, transformer does pre/post-processing, explainer produces attributions — chained as separate components

Answer

They are three replicas of the same model for redundancy

Answer

Predictor handles HTTP, transformer handles gRPC, explainer handles logging

Answer

They correspond to dev, staging, and prod environments

Question 16

How should you store and version large model artifacts for Kubernetes serving, and why is baking weights into the container image often discouraged?

Accepted Answer

Store weights in object storage/registry and pull at startup; baking them in bloats images, slows pulls/cold starts, and couples model to image

Answer

Always bake weights into the image so Pods never need network access

Answer

Store weights in etcd for fast cluster-wide access

Answer

Keep weights only on a single node's local disk for speed

Question 17

What is the purpose of a Kubernetes Service of type ClusterIP in front of model-serving Pods, and how does it relate to readiness probes?

Accepted Answer

It gives a stable VIP/DNS load-balancing only across ready Pods; readiness gating keeps traffic off still-loading Pods

Answer

It exposes the model publicly to the internet by default

Answer

It assigns a static external IP per Pod

Answer

It bypasses load balancing and pins each client to one Pod

Question 18

Why is gRPC often preferred over REST/HTTP for high-throughput, low-latency model serving, and what is one operational caveat in Kubernetes?

Accepted Answer

gRPC's HTTP/2 + Protobuf cut overhead, but long-lived connections need L7/gRPC-aware load balancing to avoid pinning to one Pod

Answer

gRPC cannot be used inside Kubernetes clusters at all

Answer

gRPC automatically load-balances at L4 better than REST

Answer

gRPC requires no serialization, sending raw Python objects

Question 19

What does the Cluster Autoscaler do, and how does it complement the HPA for GPU inference workloads?

Accepted Answer

It adds/removes nodes for (un)schedulable Pods; with HPA it provisions GPU nodes when new replicas can't be placed and removes idle ones

Answer

It changes Pod replica counts based on CPU like the HPA

Answer

It only works for stateful workloads, not inference

Answer

It resizes the GPU memory of existing nodes on demand

Question 20

How do node taints and Pod tolerations help dedicate expensive GPU nodes to inference workloads?

Accepted Answer

A GPU node taint repels non-tolerating Pods; inference Pods add a toleration (plus affinity) so only they land on GPU nodes

Answer

Taints force all Pods onto GPU nodes regardless of need

Answer

Tolerations increase a Pod's GPU memory allocation

Answer

Taints are used to encrypt traffic between GPU nodes

Question 21

What is the inference autoscaling benefit of KEDA over the standard HPA for event/queue-driven model serving?

Accepted Answer

KEDA adds external/event scalers (Kafka, SQS, Prometheus) and scale-from-zero on queue depth, which native HPA can't do alone

Answer

KEDA replaces the kubelet for faster Pod startup

Answer

KEDA only scales based on node CPU, like the HPA

Answer

KEDA is a model registry for versioning artifacts

Question 22

For LLM serving, what does vLLM's PagedAttention and continuous batching contribute, and why deploy it on Kubernetes?

Accepted Answer

PagedAttention pages the KV cache to cut fragmentation and continuous batching keeps the GPU busy; K8s adds scaling/rollout around it

Answer

PagedAttention compresses model weights to 4-bit automatically

Answer

Continuous batching disables batching to lower latency to zero

Answer

vLLM only runs outside Kubernetes on bare metal

Question 23

What is the difference between RollingUpdate and Recreate deployment strategies, and which is safer for stateless model servers?

Accepted Answer

RollingUpdate replaces Pods gradually for zero downtime; Recreate kills all first (downtime). RollingUpdate is safer for stateless servers

Answer

Recreate is always zero-downtime; RollingUpdate causes outages

Answer

Both cause downtime; they only differ in logging

Answer

RollingUpdate requires StatefulSets and can't be used for Deployments

Question 24

How does a StatefulSet differ from a Deployment, and when would model serving need one?

Accepted Answer

StatefulSet gives stable identities/per-Pod storage and ordered rollout — useful for sharded or multi-node parallel inference; most serving uses Deployments

Answer

StatefulSet is just a faster Deployment with no identity guarantees

Answer

Deployments give stable Pod identities while StatefulSets do not

Answer

StatefulSets are required for all GPU workloads

Question 25

What are the key Kubernetes/serving metrics you should monitor for an inference SLO, and which is most directly tied to user experience?

Accepted Answer

RPS, errors, p95/p99 latency, queue time, GPU usage, TTFT/TPS — tail latency (p99/TTFT) most directly reflects user experience

Answer

Only average CPU utilization matters for user experience

Answer

Total disk space is the primary user-facing metric

Answer

Pod restart count is the only metric tied to latency

Question 26

Why are ConfigMaps and Secrets used in model serving, and what is a key security caveat about Secrets?

Accepted Answer

ConfigMaps for config, Secrets for credentials; Secrets are only base64 by default, so enable etcd encryption-at-rest, tight RBAC, or external managers

Answer

Secrets are AES-encrypted end-to-end by default with no extra setup

Answer

ConfigMaps are encrypted but Secrets are plaintext only

Answer

Secrets can only be used by StatefulSets

Question 27

What is the Open Inference Protocol (V2) and why does it matter for portable model serving on Kubernetes?

Accepted Answer

A standardized REST/gRPC inference API (readiness, metadata, predict) enabling runtime-agnostic clients and swappable backends

Answer

A proprietary protocol that only Triton can speak

Answer

A Kubernetes networking layer replacing CNI

Answer

A model compression format for smaller weights

Question 28

How does ModelMesh (multi-model serving) improve density compared to one Pod per model, and what is its main tradeoff?

Accepted Answer

ModelMesh shares Pods across many models with on-demand load/unload for high density; the tradeoff is cold-load latency for non-resident models

Answer

ModelMesh dedicates one GPU per model for maximum isolation

Answer

It reduces density by spinning up a Pod per request

Answer

It only works for a single model at a time

Question 29

What is an Ingress (or Gateway API) used for in front of model-serving Services, and what features does it typically add?

Accepted Answer

It is the L7 entry point: host/path routing, TLS termination, auth/rate-limiting, and weighted traffic splitting to backend Services

Answer

It is a device plugin that exposes GPUs to Pods

Answer

It replaces the kube-scheduler for inference Pods

Answer

It encrypts etcd data at rest

Question 30

How can you reduce cold-start time for large model Pods on Kubernetes?

Accepted Answer

Keep warm replicas, pre-pull/slim images, cache weights on fast storage, pre-provision GPU nodes, and use fast loaders like safetensors

Answer

Increase the Pod's CPU limit so weights load instantly

Answer

Disable readiness probes so traffic arrives sooner

Answer

Bake all models into etcd for instant access

Test yourself