Cost Optimization quiz

Question 1

What are the three main cost drivers in LLM API usage and how is each measured?

Accepted Answer

Input tokens, output tokens, and request overhead are the three main LLM API cost drivers, with output tokens typically more expensive per token

Answer

GPU hours and memory bandwidth are the primary cost metrics for API users

Answer

Only output tokens are billed; input tokens are free on all major providers

Answer

Request count is the only cost driver, with unlimited tokens per request

Question 2

What is model cascading and how does it reduce LLM inference costs?

Accepted Answer

Model cascading routes easy queries to cheap small models and escalates difficult queries to large models, reducing cost 50-90%

Answer

Model cascading always uses the largest available model for best quality

Answer

Cascading requires retraining models at each tier from scratch

Answer

Cascading only applies to classification tasks, not generation

Question 3

How does the output length control strategy reduce costs in LLM deployments?

Accepted Answer

Output tokens cost 3-5x more than input tokens; controlling length via max_tokens, stop sequences, and format constraints directly reduces cost

Answer

Output tokens are always cheaper than input tokens because they don't require attention computation

Answer

max_tokens parameter is only available for OpenAI models

Answer

Output length control degrades quality proportionally for all tasks

Question 4

What is FlashAttention and how does it reduce memory and compute costs in transformers?

Accepted Answer

FlashAttention tiles attention computation in SRAM to reduce HBM memory access from O(N²) to O(N), enabling 2-4x speedup

Answer

FlashAttention reduces model parameter count by fusing attention heads

Answer

FlashAttention only applies to training; inference uses standard attention

Answer

FlashAttention increases memory usage but reduces computation

Question 5

How does LoRA reduce fine-tuning compute and memory costs compared to full fine-tuning?

Accepted Answer

LoRA trains only small rank-decomposition matrices, reducing trainable parameters 100x and enabling fine-tuning in 16-24GB vs 80+ GB VRAM

Answer

LoRA requires the same compute as full fine-tuning but reduces inference cost

Answer

LoRA can only be applied to embedding layers, not attention layers

Answer

QLoRA and LoRA are identical; 'Q' stands for Quick

Question 6

What is the cost impact of using long-context models vs. RAG for knowledge retrieval?

Accepted Answer

RAG retrieves ~2K tokens vs. 100K+ for long context, making RAG 50-100x cheaper per query for most use cases

Answer

Long-context models are always cheaper because they avoid embedding and retrieval overhead

Answer

Long context and RAG cost the same because both process equivalent information

Answer

RAG is only cheaper when using open-source models

Question 7

What is PagedAttention and how does it improve GPU memory efficiency during inference?

Accepted Answer

PagedAttention uses virtual memory paging for KV-cache, eliminating pre-allocation fragmentation and enabling 2-4x better memory utilization

Answer

PagedAttention replaces self-attention with paged convolutions for faster inference

Answer

PagedAttention is a model compression technique reducing weight size

Answer

PagedAttention only works with beam search, not greedy or sampling decoding

Question 8

What is the LiteLLM library and how does it enable cost optimization through provider routing?

Accepted Answer

LiteLLM provides unified API access to 100+ providers with automatic cost-optimized routing, fallback, and centralized cost tracking

Answer

LiteLLM only works with OpenAI models

Answer

LiteLLM requires separate authentication flows for each provider

Answer

LiteLLM increases costs by adding a proxy layer to each request

Question 9

How do streaming responses reduce perceived latency cost in user-facing applications?

Accepted Answer

Streaming reduces perceived latency and can reduce effective output cost when users interrupt early; compute cost per token is identical

Answer

Streaming doubles inference cost because tokens must be sent twice

Answer

Streaming is only available for models with fewer than 7B parameters

Answer

Streaming increases output token count because the model generates more tokens when streaming

Question 10

What is the Amdahl's Law implication for LLM system optimization and where should effort be focused?

Accepted Answer

Amdahl's Law means optimization effort should focus on the dominant bottleneck; for LLM latency this is usually decode throughput and KV-cache

Answer

Amdahl's Law implies that all LLM components should be optimized equally

Answer

Amdahl's Law only applies to CPU-bound systems, not GPU-accelerated LLMs

Answer

Improving prefill speed always provides the largest latency reduction for chat applications

Test yourself