Cost optimization for ML in production quiz

Question 1

What is the difference between scale-to-zero and provisioned (always-on) serving, and when does scale-to-zero hurt cost-efficiency?

Accepted Answer

Scale-to-zero hurts when traffic is steady because frequent cold starts and headroom outweigh idle savings

Answer

Scale-to-zero is always cheaper because idle cost is zero

Answer

Provisioned serving has no cold starts because it never scales

Answer

Scale-to-zero only applies to training jobs, not inference

Question 2

Why are Spot/Preemptible instances attractive for ML training, and what design pattern makes them safe to use?

Accepted Answer

Spot offers large discounts and is safe with frequent checkpointing and resumable/elastic jobs

Answer

Spot is cheap but unsafe for any training

Answer

Spot guarantees the same uptime as on-demand

Answer

Spot should only be used for inference, never training

Question 3

How does quantization (e.g., INT8/FP8) reduce inference cost, and what is the main risk to monitor?

Accepted Answer

Quantization lowers memory/compute cost but risks accuracy loss that must be validated

Answer

Quantization always improves accuracy and speed for free

Answer

Quantization only reduces training cost, not inference

Answer

Quantization eliminates the need for GPUs entirely

Question 4

For an LLM API workload, how does prompt caching reduce cost, and what is the key constraint?

Accepted Answer

Prompt caching discounts repeated stable prefixes but requires exact prefix match within a TTL

Answer

Prompt caching compresses outputs to save tokens

Answer

Prompt caching works only on the dynamic part of the prompt

Answer

Prompt caching permanently stores results so cost is zero after first call

Question 5

What is the cost rationale for a model cascade (router) architecture in GenAI systems?

Accepted Answer

A cascade serves easy requests with a cheap model and escalates only hard ones to a costly model

Answer

Cascades always send every request to all models for voting

Answer

Cascades only reduce latency, never cost

Answer

Cascades require all models to be the same size

Question 6

Why can increasing inference batch size lower cost per request, and where is the limit?

Accepted Answer

Batching lowers cost/request by amortizing overhead but is limited by memory and latency SLO

Answer

Bigger batches always reduce both cost and latency

Answer

Batch size has no effect on GPU utilization

Answer

Only batch size 1 is valid in production

Question 7

What does the FinOps concept 'unit economics' mean for an ML product, and why track it?

Accepted Answer

Unit economics ties spend to a business unit (e.g., cost per inference/user) to assess margin and ROI

Answer

Unit economics is just the monthly total cloud bill

Answer

Unit economics only applies to training, not serving

Answer

Unit economics measures model accuracy per dollar of labeling

Question 8

How do Reserved Instances / Savings Plans / Committed Use Discounts reduce cost, and what is the central risk?

Accepted Answer

Commitments discount steady baseline usage but risk paying for unused capacity if usage drops

Answer

Commitments are risk-free and always optimal

Answer

Commitments only apply to storage, not compute

Answer

Commitments require no minimum usage

Question 9

What is knowledge distillation and how is it used to cut serving cost?

Accepted Answer

Distillation trains a small student to mimic a large teacher, cutting serving cost with modest quality loss

Answer

Distillation enlarges the model to improve accuracy

Answer

Distillation is only a training-data augmentation technique

Answer

Distillation requires retraining the teacher on every request

Question 10

Why is GPU utilization (not just GPU count) the key cost metric for ML serving, and what causes low utilization?

Accepted Answer

Low GPU utilization wastes paid capacity; batching/right-sizing/MIG improve cost per inference

Answer

GPU count alone determines cost-efficiency

Answer

Utilization is irrelevant since you pay per request

Answer

High utilization always means high cost

Question 11

What is MIG (Multi-Instance GPU) / GPU fractionalization and how does it save money?

Accepted Answer

MIG partitions one GPU into isolated slices so small workloads share a card, raising utilization

Answer

MIG combines many GPUs into one larger virtual GPU

Answer

MIG only works for training, not inference

Answer

MIG removes all memory limits on a GPU

Question 12

In RAG systems, name two cost levers that reduce LLM token spend without changing the model.

Accepted Answer

Reduce retrieved context via reranking/dedup and add prompt/response caching

Answer

Increase top-k retrieval and disable caching

Answer

Always send the full document corpus as context

Answer

Switch to a larger model to reduce tokens

Question 13

What is the cost trade-off between online (real-time) features and precomputed batch features in a feature store?

Accepted Answer

Precompute high-reuse expensive features; compute cheap/low-reuse or freshness-critical ones on demand

Answer

On-demand features are always cheaper than precomputed

Answer

Always precompute every feature regardless of reuse

Answer

Feature freshness has no cost implications

Question 14

How does setting max_tokens (output length cap) and using stop sequences affect LLM cost?

Accepted Answer

Capping max_tokens/stop sequences bounds expensive output generation and reduces cost

Answer

Output tokens are free, so max_tokens is irrelevant to cost

Answer

max_tokens only affects input billing

Answer

Stop sequences increase token usage

Question 15

What is autoscaling on the wrong metric, and why does CPU-based autoscaling often fail for GPU inference?

Accepted Answer

GPU inference should autoscale on GPU util/queue depth/concurrency, not CPU, to avoid waste and SLO breaches

Answer

CPU utilization is always the best autoscaling signal

Answer

Autoscaling metric choice does not affect cost

Answer

Only memory should drive autoscaling

Question 16

What is speculative decoding and how does it reduce LLM inference cost?

Accepted Answer

A small draft model proposes tokens that the large model verifies in parallel, raising throughput

Answer

Speculative decoding randomly skips tokens to save compute

Answer

It trains two models jointly to reduce data cost

Answer

It only works for image models

Question 17

Why can choosing a region or accelerator type be a major cost lever, and what is the catch?

Accepted Answer

Region/accelerator choice changes price-performance but adds egress, latency, compliance, and porting trade-offs

Answer

Region and accelerator choice never affect cost

Answer

Only the GPU brand matters, never the region

Answer

Data egress is always free between regions

Question 18

What is the difference between batch (offline) inference and real-time inference from a cost standpoint?

Accepted Answer

Batch inference is cheaper per prediction via high utilization and spot; real-time pays for idle readiness

Answer

Real-time inference is always cheaper than batch

Answer

Batch and real-time cost the same per prediction

Answer

Batch inference cannot use spot instances

Question 19

How does a semantic response cache differ from an exact-match cache for LLM endpoints, and what is its added risk?

Accepted Answer

Semantic caches match similar queries for higher hit rates but risk false hits needing threshold tuning

Answer

Semantic and exact caches behave identically

Answer

Exact-match caches catch paraphrases automatically

Answer

Semantic caching increases per-request token cost

Question 20

What is 'right-sizing' an inference deployment, and what data do you need to do it well?

Accepted Answer

Right-sizing matches resources to profiled load/SLO using throughput-latency curves and traffic data

Answer

Right-sizing means always picking the biggest GPU

Answer

Right-sizing can be done without any load testing

Answer

Right-sizing only applies to databases

Question 21

Why is data storage tiering (hot/warm/cold/archive) relevant to ML cost optimization?

Accepted Answer

Tiering rarely accessed data to cold/archive cuts storage cost, trading retrieval latency/cost

Answer

All ML data must stay in hot storage for correctness

Answer

Storage cost is negligible in ML so tiering is pointless

Answer

Archived data can never be retrieved

Question 22

What is the cost danger of redundant or 'shadow' model retraining, and how do you control it?

Accepted Answer

Use trigger-based retraining on drift/decay/new-data instead of blind schedules to cut compute

Answer

Always retrain daily to be safe

Answer

Never retrain models once deployed

Answer

Retraining cost is independent of schedule frequency

Question 23

How does LoRA / parameter-efficient fine-tuning (PEFT) reduce cost compared to full fine-tuning?

Accepted Answer

LoRA trains small adapters on a frozen base, cutting training memory and enabling shared multi-adapter serving

Answer

LoRA retrains all base weights, costing more

Answer

LoRA only reduces inference latency, not cost

Answer

LoRA requires a separate full model per task

Question 24

What is cost-aware model selection, and why is the cheapest-per-token model not always cheapest overall?

Accepted Answer

Cheapest per token may cost more per resolved task due to retries/longer prompts/quality fixes

Answer

The lowest price-per-token model is always cheapest overall

Answer

Model price never affects total cost

Answer

Only output token price matters for selection

Question 25

Why are idle/orphaned resources a top source of ML cloud waste, and how do you catch them?

Accepted Answer

Orphaned/idle GPUs and endpoints bill continuously; catch them with tagging, idle-detection, and TTL auto-shutdown

Answer

Idle resources cost nothing until used

Answer

Cloud providers auto-delete unused resources for free

Answer

Tagging has no role in cost control

Question 26

How does KV-cache memory pressure drive LLM serving cost, and what techniques mitigate it?

Accepted Answer

KV-cache memory limits concurrency; PagedAttention, KV quantization, GQA, and prefix sharing raise throughput/$

Answer

KV cache is irrelevant to serving cost

Answer

KV cache only matters during training

Answer

Longer context reduces KV-cache memory

Question 27

What is the role of an LLM gateway/proxy in cost optimization?

Accepted Answer

An LLM gateway centralizes caching, routing, budgets, and usage tracking to control and attribute spend

Answer

A gateway only adds latency with no cost benefit

Answer

A gateway can only call a single hardcoded model

Answer

Gateways are used only for security, never cost

Question 28

When is CPU inference more cost-effective than GPU inference?

Accepted Answer

For small models or low/spiky QPS, optimized CPU inference can be cheaper than idle GPUs

Answer

GPU is always cheaper for any inference

Answer

CPU inference is never viable in production

Answer

Only model size, not traffic, determines CPU vs GPU cost

Question 29

What is the cost impact of retries and timeouts in ML pipelines, and how do you tune them safely?

Accepted Answer

Bounded retries with backoff, circuit breakers, and SLO-based timeouts prevent runaway retry cost

Answer

Unlimited retries are free and improve reliability

Answer

Timeouts increase cost and should be removed

Answer

Retries never affect token or compute spend

Question 30

How does early stopping reduce training cost, and what guards against stopping too early?

Accepted Answer

Early stopping halts on validation plateau with patience, cutting wasted epochs and GPU cost

Answer

Early stopping increases training cost

Answer

Early stopping ignores validation metrics

Answer

Early stopping requires training for all epochs first

Test yourself