Learning to rank quiz

Question 1

What are the three main paradigms of Learning to Rank (LtR)?

Accepted Answer

Pointwise, Pairwise, Listwise

Answer

Supervised, Semi-supervised, Unsupervised

Answer

Feature-based, Neural, Hybrid

Answer

Greedy, Exhaustive, Probabilistic

Question 2

How is NDCG@k computed, and what property of the metric makes it suitable for graded relevance?

Accepted Answer

It sums discounted gains normalized by the ideal ranking's DCG, supporting graded relevance

Answer

It measures average precision at each recall level, normalized by query length

Answer

It measures the rank of the first relevant document across queries

Answer

It counts the fraction of relevant documents retrieved in the top-k results

Question 3

What is the key insight behind RankNet's pairwise loss function?

Accepted Answer

It models pairwise preference probability via a sigmoid of the score difference and minimizes cross-entropy

Answer

It directly optimizes NDCG using a smooth surrogate approximation

Answer

It uses contrastive loss to push relevant documents further from irrelevant ones

Answer

It applies softmax over all documents and minimizes KL divergence to the label distribution

Question 4

How does LambdaRank improve upon RankNet, and what is the 'lambda' it defines?

Accepted Answer

LambdaRank scales RankNet pairwise gradients by the absolute NDCG change from swapping each pair

Answer

LambdaRank replaces the sigmoid with a linear activation for faster convergence

Answer

LambdaRank uses a listwise softmax loss instead of pairwise comparisons

Answer

LambdaRank adds L2 regularization to the RankNet loss to prevent overfitting to rare queries

Question 5

Describe LambdaMART and explain how it combines gradient boosting with LambdaRank.

Accepted Answer

LambdaMART fits gradient boosted regression trees to LambdaRank pseudo-gradients at each iteration

Answer

LambdaMART is a neural network that uses NDCG as its closed-form loss function

Answer

LambdaMART uses random forests with a pairwise ranking objective

Answer

LambdaMART is a linear model regularized with the lambda loss

Question 6

What is the fundamental difference between MAP and NDCG as ranking metrics?

Accepted Answer

NDCG supports graded relevance with log discounting; MAP assumes binary relevance and averages precision at recall points

Answer

MAP supports graded relevance while NDCG only supports binary relevance

Answer

MAP penalizes documents at lower ranks exponentially more than NDCG does

Answer

They are equivalent when relevance is binary and each query has exactly one relevant document

Question 7

What is Mean Reciprocal Rank (MRR) and for which retrieval tasks is it most appropriate?

Accepted Answer

MRR is the mean of the reciprocal rank of the first relevant document per query

Answer

MRR averages precision at every relevant document position across all rank positions

Answer

MRR sums graded relevance scores discounted by the log of the rank position

Answer

MRR measures the fraction of queries with at least one relevant document in the top 10

Question 8

What is position bias in click-based training data for LtR and why is it problematic?

Accepted Answer

Position bias is the tendency of users to click higher-ranked items regardless of true relevance, creating training signal bias

Answer

Position bias means the model assigns too much weight to query features versus document features

Answer

Position bias refers to a model's overfit to the most frequent query terms in the training corpus

Answer

Position bias occurs when documents from authoritative domains are systematically promoted by the annotators

Question 9

Explain Inverse Propensity Scoring (IPS) for unbiased learning to rank.

Accepted Answer

IPS reweights clicks by the inverse probability that the position was examined, debiasing training

Answer

IPS reweights clicks by the probability that the document is relevant given all features

Answer

IPS normalizes click signals by the CTR of the top-ranked document in each query session

Answer

IPS applies a log transformation to click counts to reduce the influence of popular queries

Question 10

What is ListNet and how does it define a listwise loss?

Accepted Answer

ListNet minimizes KL divergence between predicted and ground-truth top-one permutation probabilities via softmax

Answer

ListNet minimizes pairwise hinge loss summed over all document pairs in the list

Answer

ListNet uses a softmax loss on the explicit position index of each relevant document

Answer

ListNet directly optimizes NDCG by approximating it with a differentiable step function

Question 11

What is the LambdaLoss framework and how does it give LambdaRank a theoretical justification?

Accepted Answer

LambdaLoss is a framework where metric-specific weight functions on permutation log-likelihoods unify LtR losses

Answer

LambdaLoss is a neural architecture that directly optimizes NDCG end-to-end via backpropagation

Answer

LambdaLoss introduces a listwise surrogate for MAP that is both convex and differentiable

Answer

LambdaLoss combines gradient boosting with a listwise softmax for improved NDCG on LETOR benchmarks

Question 12

How are features typically categorized in a production Learning to Rank system?

Accepted Answer

Query features, Document features, and Query-document interaction features

Answer

Static, Dynamic, and Contextual features

Answer

Lexical, Semantic, and Behavioral features

Answer

Sparse features, Dense features, and Graph-based features

Question 13

What is Expected Reciprocal Rank (ERR) and how does it differ from NDCG?

Accepted Answer

ERR models a cascade where high-relevance top results reduce the probability of examining lower results

Answer

ERR replaces the log discount in DCG with a linear discount for mathematical convenience

Answer

ERR is identical to NDCG but computed at the expected rank rather than a fixed cutoff k

Answer

ERR measures the expected rank at which a user finds a satisfying result using ordinal regression

Question 14

What is counterfactual learning to rank and how does it differ from online LtR?

Accepted Answer

Counterfactual LtR learns from historical logs with off-policy corrections; online LtR explores live with real-time feedback

Answer

Counterfactual LtR uses synthetic negative examples, while online LtR uses only positive clicks

Answer

Counterfactual LtR requires a reward model, while online LtR requires a propensity model

Answer

They are equivalent; the difference is only in whether GPUs or CPUs are used for training

Question 15

How does BM25 fit into a modern two-stage LtR pipeline?

Accepted Answer

BM25 serves as first-stage retriever and its score is a feature in the downstream LtR re-ranker

Answer

BM25 replaces LtR re-rankers in production because it is differentiable and fine-tunable

Answer

BM25 is a listwise ranking model that optimizes MAP directly using term frequencies

Answer

BM25 is deprecated in modern systems and replaced entirely by dense retrieval models

Question 16

What is the difference between a cross-encoder and a bi-encoder for neural ranking?

Accepted Answer

Bi-encoders encode query and doc independently enabling fast ANN retrieval; cross-encoders encode them jointly for accurate re-ranking

Answer

Cross-encoders are faster than bi-encoders because they share a single transformer encoder

Answer

Cross-encoders produce fixed document embeddings that can be precomputed and cached offline

Answer

Bi-encoders attend over the full query-document pair, making them more accurate at the cost of speed

Question 17

What is ColBERT's late interaction mechanism and why is it significant for neural ranking?

Accepted Answer

ColBERT independently encodes tokens, then scores via MaxSim: sum of per-query-token max dot products

Answer

ColBERT concatenates all query and document tokens and uses a [CLS] token for scoring

Answer

ColBERT uses average pooling of BERT embeddings and ranks by cosine similarity like a standard bi-encoder

Answer

ColBERT trains separate encoders using a contrastive triplet loss only without any token-level matching

Question 18

How does Dueling Bandit Gradient Descent (DBGD) work for online learning to rank?

Accepted Answer

DBGD perturbs the model, interleaves current and candidate rankings, and updates toward the click-preferred ranker

Answer

DBGD is a gradient boosting variant that uses bandit feedback to set regression tree leaf values

Answer

DBGD applies an epsilon-greedy policy over a discrete set of pre-trained ranking models

Answer

DBGD minimizes a dueling softmax loss where two rankers compete to minimize click entropy

Question 19

What is interleaved evaluation and why is it statistically more efficient than A/B testing for ranker comparison?

Accepted Answer

Interleaved evaluation merges two rankers' results and infers preference from clicks on the merged list, eliminating query-level variance

Answer

Interleaved evaluation splits traffic 50/50 between two rankers and compares NDCG on held-out labels

Answer

Interleaved evaluation uses offline click logs to simulate user preferences without any live traffic

Answer

Interleaved evaluation trains a reward model on clicks and scores each ranker's output independently

Question 20

How is XGBoost configured for learning to rank and what objectives does it support?

Accepted Answer

XGBoost uses pairwise or LambdaRank objectives within query groups specified by the group parameter

Answer

XGBoost ranks by minimizing a listwise softmax loss over all document permutations

Answer

XGBoost converts ranking to regression by treating relevance labels as continuous regression targets globally

Answer

XGBoost uses a multi-class objective where each rank position is treated as a separate class label

Question 21

What is query normalization in LtR training and why is it important?

Accepted Answer

Query normalization averages loss or gradients per query to prevent high-doc-count queries from dominating training

Answer

Query normalization scales all feature values to zero mean and unit variance within each query group

Answer

Query normalization removes queries with fewer than 10 documents from the training set

Answer

Query normalization applies weight decay only to query-level features to prevent overfitting

Question 22

What is SPLADE and how does it bridge lexical and semantic retrieval?

Accepted Answer

SPLADE uses a BERT MLM head to produce sparse vocabulary-sized vectors enabling inverted-index retrieval with semantic expansion

Answer

SPLADE is a dense bi-encoder that uses product quantization for efficient nearest-neighbor search

Answer

SPLADE reweights BM25 term statistics using a cross-encoder relevance score during retrieval

Answer

SPLADE is a listwise neural ranker that fine-tunes BERT on NDCG loss with soft relevance labels

Question 23

What is Kendall's tau and how is it used to evaluate ranking systems?

Accepted Answer

Kendall's tau measures rank correlation via the normalized difference between concordant and discordant pairs

Answer

Kendall's tau measures the ratio of relevant documents to total documents in the top-k results

Answer

Kendall's tau is an information-theoretic divergence measure between two ranked distributions

Answer

Kendall's tau is the recall-weighted precision averaged over all rank positions

Question 24

What is ApproxNDCG and why is a differentiable rank approximation needed?

Accepted Answer

ApproxNDCG replaces the non-differentiable rank with a sigmoid-based soft rank, enabling direct gradient-based NDCG optimization

Answer

ApproxNDCG clips NDCG gradients at extreme score differences to stabilize training

Answer

ApproxNDCG replaces the log discount with an exponential discount that has a closed-form gradient

Answer

ApproxNDCG is a Monte Carlo estimator of NDCG computed over sampled document permutations

Question 25

What is a two-tower model in ranking/recommendation and what are its fundamental limitations?

Accepted Answer

A two-tower model encodes users and items independently; the dot-product bottleneck prevents rich cross-feature interactions

Answer

A two-tower model uses cascaded training: first a pointwise stage then a pairwise stage

Answer

A two-tower model processes queries and documents in a single transformer with cross-attention and concatenated outputs

Answer

A two-tower model maintains two ranking functions and selects the higher-scoring result per query

Question 26

How does the Plackett-Luce model underpin listwise LtR methods such as ListNet and ListMLE?

Accepted Answer

The Plackett-Luce model defines a distribution over permutations via sequential softmax selection of remaining documents

Answer

The Plackett-Luce model is a Gaussian graphical model over pairwise document preferences

Answer

The Plackett-Luce model is restricted to binary relevance and cannot handle graded labels

Answer

The Plackett-Luce model is a copula that combines marginal relevance scores into a joint ranking distribution

Question 27

What are click models and how are they used to generate training signals for LtR?

Accepted Answer

Click models are probabilistic models that decompose clicks into relevance and examination components for debiased LtR labels

Answer

Click models are neural networks that predict CTR directly from query and document features

Answer

Click models convert ranking into a multi-armed bandit problem using Thompson sampling over document slots

Answer

Click models aggregate clicks across sessions using exponential moving averages as relevance proxies

Question 28

How does SoftRank differ from ApproxNDCG in its approach to differentiating NDCG?

Accepted Answer

SoftRank computes expected NDCG by treating document ranks as random variables under Gaussian score noise

Answer

SoftRank is a pairwise method that applies a soft-margin hinge loss between ranked document pairs

Answer

SoftRank uses softmax normalization over scores as a proxy for the permutation probability distribution

Answer

SoftRank replaces NDCG with rank-weighted binary cross-entropy for numerical stability

Question 29

How is diversity handled in ranking, and what are MMR and DPPs?

Accepted Answer

MMR greedily balances relevance and novelty; DPPs model set probability via a kernel determinant encoding diversity

Answer

Diversity is handled by training separate rankers for each result category and round-robining their outputs

Answer

Diversity in ranking converts all sub-objectives into a single NDCG variant with a weighted relevance grade

Answer

Diversity requires online RL since the Pareto front between relevance and diversity cannot be estimated offline

Question 30

What is the key difference between offline and online evaluation for LtR models, and what metrics are used in each?

Accepted Answer

Offline evaluation measures NDCG/MAP from ground-truth labels; online evaluation measures user metrics via A/B tests or interleaving

Answer

Offline evaluation always uses implicit signals (clicks), while online evaluation always uses explicit labels

Answer

Offline evaluation runs the ranker on the production index; online evaluation uses a sampled document subset

Answer

Offline evaluation is only feasible for document collections under 1 million items due to annotation cost

Test yourself