Model Serving at Scale quiz

Question 1

What is the difference between REST and gRPC for model serving?

Accepted Answer

gRPC uses HTTP/2 + Protobuf for 3-10x speed; REST uses HTTP/1.1 + JSON for universal compatibility

Answer

REST is faster due to simpler protocol overhead

Answer

gRPC requires a GPU; REST runs on CPU only

Answer

They are identical; gRPC is just a REST wrapper

Question 2

What is dynamic batching in model serving and why does it improve GPU utilization?

Accepted Answer

Aggregates multiple requests into one GPU batch, maximizing matrix operation efficiency at the cost of slightly higher per-request latency

Answer

Splits one large batch into smaller sub-batches for parallel CPU processing

Answer

Dynamically adjusts model precision based on current load

Answer

A caching technique that reuses previous batch outputs for identical inputs

Question 3

What is ONNX and how does it enable cross-framework model deployment?

Accepted Answer

An open model format enabling training-framework-agnostic deployment with ONNX Runtime's cross-platform inference

Answer

A GPU-specific model format for NVIDIA hardware

Answer

A Python packaging format for ML model dependencies

Answer

A compression algorithm for reducing model file size

Question 4

How does continuous batching differ from static batching for LLM serving?

Accepted Answer

Continuous batching replaces finished requests mid-generation every decoding step; static batching waits for all to finish before accepting new requests

Answer

Continuous batching uses larger batch sizes than static batching

Answer

Static batching supports streaming; continuous batching does not

Answer

They differ only in the maximum batch size configuration

Question 5

What is a blue-green deployment strategy for ML models and what does it prevent?

Accepted Answer

Runs two full environments simultaneously; cutover switches all traffic instantly with immediate rollback capability

Answer

Gradually increases traffic to the new model version over hours

Answer

Deploys the new model alongside the old without load balancer changes

Answer

Uses shadow mode to test the new model before any traffic switch

Question 6

What is KServe (formerly KFServing) and what problem does it solve?

Accepted Answer

A Kubernetes-native serving platform with unified multi-framework support, auto-scaling, and canary rollouts via Knative/Istio

Answer

A Python library for serving scikit-learn models as REST endpoints

Answer

A Kubernetes operator that deploys TensorFlow models only

Answer

An alternative to Docker for containerizing ML models

Question 7

What is model pruning and what are structured vs. unstructured pruning?

Accepted Answer

Structured pruning removes entire channels/heads for hardware-friendly speedup; unstructured zeroes individual weights with irregular patterns

Answer

Unstructured pruning is always faster because it removes more weights

Answer

Structured pruning is for LLMs; unstructured is for CNNs

Answer

Both require custom GPU kernels to achieve speedups

Question 8

What is the throughput vs. latency trade-off in model serving and how do you balance them?

Accepted Answer

Larger batches increase throughput but add queuing delay; smaller batches minimize latency but under-utilize GPU

Answer

Higher throughput always reduces latency through amortization

Answer

Latency and throughput improve together with more GPU replicas only

Answer

Throughput is irrelevant for real-time serving systems

Question 9

What is prefix caching in LLM serving and how does it reduce compute cost?

Accepted Answer

Reuses pre-computed KV cache for shared prompt prefixes (system prompts), reducing TTFT by 2-5x for repeated-prefix workloads

Answer

Caches model outputs in a Redis layer for identical prompts

Answer

A quantization technique that compresses prompt tokens for faster tokenization

Answer

Batches multiple system prompts together to fill tensor core utilization

Question 10

What is the OpenAI API compatibility layer and why is it important for model serving?

Accepted Answer

OpenAI API compatibility allows any client to switch between providers by changing only base_url — no code changes

Answer

Compatibility requires rewriting models in OpenAI's proprietary format

Answer

Only GPT-4 models support the OpenAI API spec

Answer

OpenAI compatibility is required by law for commercial LLM serving

Test yourself