Natural language processing quiz

Question 1

In standard scaled dot-product self-attention, why is the dot product divided by the square root of the key dimension d_k?

Accepted Answer

To normalize the logit variance to ~1 and prevent softmax gradient vanishing

Answer

To make the attention matrix symmetric

Answer

To reduce the time complexity from O(n^2) to O(n)

Answer

To enforce that attention weights sum to one

Question 2

What is the time and memory complexity of vanilla self-attention with respect to sequence length n, and why does it matter for long contexts?

Accepted Answer

O(n^2) in both time and memory

Answer

O(n log n) time, O(n) memory

Answer

O(n) time and memory

Answer

O(n^3) time, O(n^2) memory

Question 3

How do Rotary Position Embeddings (RoPE) encode positional information differently from learned absolute embeddings?

Accepted Answer

It rotates Q/K so attention depends on relative position

Answer

It adds a learned vector per absolute position

Answer

It concatenates sinusoids to the token embedding

Answer

It masks future positions during decoding

Question 4

What is the key algorithmic difference between Byte-Pair Encoding (BPE) and the Unigram language model tokenizer used by SentencePiece?

Accepted Answer

BPE greedily merges frequent pairs; Unigram prunes a vocab by likelihood

Answer

Both are identical, only the name differs

Answer

BPE is probabilistic; Unigram is greedy merging

Answer

Unigram works on characters, BPE on whole words only

Question 5

Why do modern LLMs predominantly use Pre-LayerNorm (Pre-LN) rather than Post-LayerNorm (Post-LN) in transformer blocks?

Accepted Answer

Pre-LN gives a clean residual path and more stable deep-network gradients

Answer

Post-LN uses fewer parameters

Answer

Pre-LN removes the need for residual connections

Answer

Post-LN cannot be used with attention

Question 6

What is the purpose of the KV cache during autoregressive decoding, and what is its memory cost?

Accepted Answer

It caches past keys/values to avoid recomputation; memory grows linearly with length

Answer

It caches gradients to speed up backprop

Answer

It stores the full attention matrix to reuse across batches

Answer

It compresses weights to int8 at runtime

Question 7

How does Grouped-Query Attention (GQA) reduce inference cost compared to standard Multi-Head Attention (MHA)?

Accepted Answer

Multiple query heads share KV heads, shrinking the KV cache

Answer

It removes the value projection entirely

Answer

It doubles the number of attention heads

Answer

It replaces softmax with a linear kernel

Question 8

What is exposure bias in sequence-to-sequence models trained with teacher forcing?

Accepted Answer

Train-time uses gold prefixes but test-time uses model outputs, compounding errors

Answer

The model is exposed to too much training data

Answer

Positional embeddings leak future tokens

Answer

The decoder cannot see the encoder output

Question 9

How do nucleus (top-p) sampling and top-k sampling differ when controlling generation diversity?

Accepted Answer

Top-k uses a fixed count; top-p uses an adaptive cumulative-probability cutoff

Answer

Top-p uses a fixed count; top-k uses cumulative probability

Answer

Both pick exactly one token deterministically

Answer

Top-k requires beam search while top-p does not

Question 10

Why is perplexity equal to the exponential of the average cross-entropy loss, and what does it intuitively measure?

Accepted Answer

exp of mean per-token cross-entropy; the effective branching factor

Answer

The variance of the loss across tokens

Answer

The number of model parameters per token

Answer

The KL divergence between train and test sets

Question 11

What does BERTScore measure that BLEU and ROUGE do not capture well?

Accepted Answer

Semantic similarity via contextual embeddings rather than exact n-gram overlap

Answer

Exact string match precision

Answer

Sentence length penalties only

Answer

Grammaticality via a parser

Question 12

What is the core idea of Direct Preference Optimization (DPO) and how does it differ from RLHF with PPO?

Accepted Answer

DPO optimizes a pairwise classification loss directly on the policy, no reward model or RL loop

Answer

DPO trains a larger reward model than PPO

Answer

DPO requires an online environment simulator

Answer

DPO removes the reference model entirely

Question 13

In LoRA fine-tuning, what is actually being trained and why does it save so much memory?

Accepted Answer

Two low-rank matrices approximating the weight update, with base weights frozen

Answer

The full weight matrix at lower precision

Answer

Only the embedding layer

Answer

A separate copy of all attention heads

Question 14

What problem does FlashAttention solve, and is it an approximation of attention?

Accepted Answer

Exact attention computed IO-efficiently via tiling/online softmax, avoiding materializing the full matrix

Answer

An approximate low-rank attention

Answer

A sparse attention that drops tokens

Answer

A quantized 4-bit attention

Question 15

In a Retrieval-Augmented Generation (RAG) pipeline, why is chunking strategy critical to answer quality?

Accepted Answer

Chunk granularity controls retrieval recall and how self-contained the grounded context is

Answer

Chunking only affects database storage size

Answer

Chunking determines the model's parameter count

Answer

Chunking replaces the need for embeddings

Question 16

Why is cosine similarity (rather than raw dot product) often preferred for comparing sentence embeddings, and when is dot product better?

Accepted Answer

Cosine ignores magnitude (good when norms vary); dot product when magnitude is meaningful

Answer

Cosine is always strictly better in every case

Answer

Dot product cannot be computed on embeddings

Answer

They always produce identical rankings

Question 17

What is the role of in-batch negatives in contrastive training of dense retrieval / sentence embedding models?

Accepted Answer

Reuse other batch items as negatives, giving many cheap negatives per step

Answer

Duplicate the positive to stabilize loss

Answer

Remove negatives to speed training

Answer

Use only the hardest single negative globally

Question 18

Why can chain-of-thought (CoT) prompting improve reasoning, and what is a key reliability caveat?

Accepted Answer

It adds intermediate compute/decomposition, but the chain may be unfaithful

Answer

It guarantees the model's reasoning is faithful

Answer

It reduces inference cost

Answer

It only works in models under 1B parameters

Question 19

Which technique most directly reduces factual hallucination by constraining the model to external evidence?

Accepted Answer

Retrieval-augmented grounding in external evidence

Answer

Increasing sampling temperature

Answer

Removing the system prompt

Answer

Reducing the vocabulary size

Question 20

How does a Mixture-of-Experts (MoE) layer increase model capacity without proportionally increasing inference FLOPs?

Accepted Answer

Only top-k of many experts are activated per token, so FLOPs stay sparse

Answer

All experts run for every token in parallel

Answer

It compresses weights to 4 bits

Answer

It shares one FFN across all layers

Question 21

What is speculative decoding and what guarantee does it provide about output quality?

Accepted Answer

A draft model proposes tokens the target verifies, preserving the target's exact distribution

Answer

It approximates the target output to gain speed, changing the distribution

Answer

It quantizes the target model

Answer

It caches outputs across requests

Question 22

What is the difference between pretraining, instruction tuning, and preference optimization in the modern LLM training stack?

Accepted Answer

Pretrain (knowledge) -> SFT (follow instructions) -> preference opt (align to humans)

Answer

All three are the same objective at different learning rates

Answer

Preference opt comes before pretraining

Answer

Instruction tuning uses unlabeled web text

Question 23

What is catastrophic forgetting in the context of fine-tuning LLMs, and name one mitigation.

Accepted Answer

Loss of prior abilities during narrow fine-tuning; mitigated by LoRA/replay

Answer

Memory overflow on the GPU during training

Answer

The tokenizer dropping rare words

Answer

Overfitting only to the validation set

Question 24

How do GPTQ/AWQ post-training quantization methods preserve accuracy at 4-bit, beyond naive rounding?

Accepted Answer

They use calibration/activation-awareness to protect salient weights, not independent rounding

Answer

They retrain the model from scratch in INT4

Answer

They only quantize the embedding layer

Answer

They store weights in FP32 but compute in INT4

Question 25

How do subword tokenizers eliminate the out-of-vocabulary (OOV) problem that plagued word-level models?

Accepted Answer

Unknown words are split into known subword/byte units, so nothing is truly OOV

Answer

They include every possible word in the vocabulary

Answer

They map OOV words to a single UNK token

Answer

They skip unknown words during inference

Question 26

In sequence labeling (e.g. NER), why might a CRF layer on top of a neural encoder outperform independent per-token softmax classification?

Accepted Answer

The CRF models label-transition constraints for globally consistent tag sequences

Answer

The CRF adds more hidden units to the encoder

Answer

The CRF removes the need for word embeddings

Answer

The CRF speeds up inference by skipping decoding

Question 27

What distinguishes the skip-gram and CBOW training objectives in word2vec, and which suits rare words better?

Accepted Answer

Skip-gram predicts context from the center word and handles rare words better

Answer

CBOW predicts context from the center word and handles rare words better

Answer

Both predict the next sentence

Answer

Skip-gram averages context to predict the center word

Question 28

Why do transformers handle long-range dependencies better than RNNs despite RNNs having theoretically unbounded memory?

Accepted Answer

Attention gives constant-length paths between any tokens, avoiding RNN gradient decay

Answer

RNNs cannot represent more than 100 tokens

Answer

Transformers use larger hidden states

Answer

RNNs lack nonlinearities

Question 29

What is knowledge distillation and how does using soft targets help the student model?

Accepted Answer

Student mimics the teacher's softened logits, learning inter-class similarity info

Answer

Student copies the teacher's weights directly

Answer

Soft targets are random labels for regularization

Answer

It increases the model size for accuracy

Question 30

What is prompt injection and why is it fundamentally hard to fully eliminate in LLM applications?

Accepted Answer

Untrusted text issues instructions the model obeys; no hard data/instruction boundary exists

Answer

A bug that crashes the tokenizer

Answer

Injecting extra training data into the model

Answer

A method to speed up prompting

Test yourself