Recommender systems at scale

Scoring every item with a heavy model for every request is computationally infeasible at catalog sizes of millions to billions. Retrieval uses a cheap, sublinear method (e.g., ANN over embeddings) to narrow billions of items down to hundreds of candidates, then an expensive, feature-rich ranker scores only those candidates. This decouples recall (don't miss good items) from precision (order them well) and keeps latency within tight serving budgets.

The two-tower design forces user and item representations to interact only through a final dot product (or cosine), so item embeddings have no dependence on the query. This lets you precompute and index all item vectors offline and serve retrieval as an approximate nearest-neighbor / maximum-inner-product search, which is what makes sublinear retrieval over millions of items possible. Allowing cross-features between towers would break this and require scoring every item online.

With in-batch negatives, popular items appear as negatives far more often than rare ones, biasing the model to under-score popular items. The logQ correction subtracts log(sampling probability) of each item from its logit, debiasing the sampled softmax so it approximates the full softmax over the catalog. Without it, retrieval quality degrades, especially for head items.

logits = (user @ item.T) / temp
logits -= tf.math.log(item_sampling_prob)  # logQ correction
loss = softmax_cross_entropy(labels=eye, logits=logits)

In-batch negatives reuse the positive items of other examples in the same batch as negatives, giving many negatives almost for free and amortizing item-tower compute. Their limitation is that they are sampled by popularity (frequency in the data) and tend to be 'easy' negatives, so models often add explicitly mined hard negatives and a logQ correction to improve discrimination near the decision boundary.

ALS alternately fixes the item factors and solves a regularized least-squares problem for each user's factors in closed form, then fixes users and solves for items, repeating to convergence. For implicit feedback it weights observed interactions by a confidence term and treats unobserved entries as low-confidence zeros. Each subproblem is independent per user/item, so ALS is embarrassingly parallel and maps well to Spark/distributed compute.

BPR is a pairwise objective that maximizes the probability that an observed (positive) item is ranked above an unobserved (negative) item for the same user, optimizing the AUC-like ordering rather than predicting absolute scores. It trains on triples (user, positive, negative) with a sigmoid of the score difference. This pairwise formulation aligns better with ranking metrics than pointwise regression/classification on individual interactions.

loss = -log(sigmoid(score(u, i_pos) - score(u, i_neg)))

NDCG (Normalized Discounted Cumulative Gain) applies a position-based logarithmic discount so relevant items ranked higher contribute more, and it supports graded relevance, not just binary hits. Precision@k ignores the order within the top-k and treats all hits equally. Normalization by the ideal DCG makes NDCG comparable across queries with different numbers of relevant items.

Recall@K is the fraction of the relevant (later engaged) items that appear within the K candidates returned by retrieval. Retrieval's job is to not lose good items before ranking, so high recall@K (with K = candidate set size) ensures the ranker has the right items to choose from. Precision matters less here because the downstream ranker re-orders and filters the candidates.

HNSW builds a multi-layer navigable small-world graph where upper layers have long-range links for coarse navigation and lower layers refine locally, so a greedy search reaches the neighborhood in roughly logarithmic hops. The tradeoff is high memory (storing graph edges per vector) and index build cost, plus it is approximate, so you tune efSearch/M to trade recall against latency. It does not natively support fast deletes/updates, complicating freshness.

PQ splits each vector into sub-vectors and quantizes each sub-vector to the nearest centroid from a small learned codebook, storing only compact codes (e.g., 8 bits per sub-vector) instead of full floats. Distances are computed via precomputed lookup tables over codes (asymmetric distance), giving large memory savings and faster scans. The cost is quantization error, which reduces recall, so PQ is often combined with an IVF coarse index and a re-ranking step on exact vectors.

Inner product is not a metric (it lacks the triangle inequality and self-similarity), so standard NN indices don't apply directly. A common trick augments each item vector with extra dimensions so that Euclidean nearest-neighbor (or cosine) search on the transformed vectors is equivalent to MIPS on the originals. Alternatively, if all item norms are equal, MIPS coincides with cosine/NN search, which is why many systems L2-normalize embeddings.

Direct CVR (conversion given click) models suffer from sample selection bias (trained only on clicked impressions) and data sparsity. ESMM models CTR and CTCVR (click-and-conversion) over the entire impression space and derives CVR = CTCVR / CTR, sharing embeddings across tasks. Training on all impressions removes the selection bias and the shared embeddings alleviate sparsity.

MMoE shares a set of expert sub-networks across tasks but gives each task its own gating network that learns a task-specific weighted combination of experts. This lets tasks share representation where it helps but specialize where objectives conflict, reducing the negative transfer seen in hard parameter sharing. It is widely used in feed/video ranking where many engagement signals must be predicted jointly.

The wide (linear) component with cross-product feature transformations provides memorization of frequent, specific co-occurrence patterns, while the deep component (embeddings + MLP) provides generalization to unseen or sparse feature combinations. Joint training lets the model both memorize exceptions and generalize, addressing the recall-vs-novelty tradeoff. The wide part is cheap and handles known rules; the deep part smooths over sparsity.

The cross network explicitly models bounded-degree feature interactions: each cross layer multiplies the original input with the current representation, so stacking L layers captures interactions up to order L+1 in a parameter-efficient, deterministic way. A plain MLP can in principle learn interactions but does so implicitly and inefficiently, often needing many parameters/data to approximate even simple multiplicative crosses. DCNv2 uses low-rank weight matrices to make the cross layers scalable.

DeepFM puts an FM component and a deep MLP side by side that share the same input embeddings, with FM capturing pairwise (second-order) feature interactions automatically and the MLP capturing higher-order interactions. Unlike Wide & Deep, it needs no manual feature-cross engineering for the wide part because the FM learns all second-order crosses directly from embeddings. End-to-end joint training over shared embeddings improves both efficiency and accuracy.

DLRM's memory is dominated by huge categorical embedding tables (often hundreds of GB to TB), far exceeding a single accelerator's memory, while the MLP compute is comparatively small. The standard solution is hybrid parallelism: model-parallel sharding of embedding tables across many devices/hosts plus data-parallel MLPs, with an all-to-all to exchange embedding lookups. Techniques like quantization, hashing, and frequency-based pruning further shrink the tables.

SASRec uses a unidirectional (causal) self-attention transformer over a user's interaction history to predict the next item, capturing both short- and long-range dependencies and adaptively attending to relevant past items. BERT4Rec instead uses bidirectional attention with a masked-item (cloze) objective, predicting randomly masked items from both left and right context. Bidirectionality can give richer representations but requires the masking trick since true next-item prediction would leak future information.

LightGCN removes the feature transformation matrices and nonlinear activations from standard GCN layers, keeping only neighborhood aggregation (weighted sum) over the user-item bipartite graph. For collaborative filtering with no rich node features, these components mainly add noise and overfitting; pure linear propagation that smooths embeddings over the graph works better and is cheaper. The final embedding is a weighted combination of all layers' outputs.

PinSage uses localized random-walk-based neighborhood sampling and importance pooling so each node's convolution only touches a bounded sampled neighborhood, avoiding full-graph operations. It performs on-the-fly minibatch computation (producer-consumer with CPU sampling and GPU compute), MapReduce inference to precompute all embeddings, and curriculum hard-negative sampling. These let it scale to billions of nodes/edges for Pinterest-scale retrieval.

Position bias means items shown higher get more clicks regardless of relevance, because users examine top positions more. A ranker trained on raw clicks treats this propensity as relevance, learns to favor whatever was already ranked high, and creates a feedback loop that entrenches incumbents. Correcting it requires modeling examination probability (e.g., position-aware features dropped at serving, or inverse-propensity weighting).

IPS reweights each logged outcome by the inverse probability that the logging (behavior) policy took the observed action, so the estimator approximates what the new (target) policy would have earned as if data were collected under it. This corrects for the fact that logs are biased toward actions the old policy preferred. Its main weakness is high variance when target and logging policies diverge (small propensities), motivating clipping/self-normalization or doubly-robust estimators.

V_hat = mean( (pi_target(a|x)/pi_log(a|x)) * reward )

Easy random negatives are trivially separated, so the model never learns fine-grained distinctions near the decision boundary; mining semantically similar (hard) negatives sharpens embeddings and improves top-K precision. The risk is false negatives: very hard negatives are often actually relevant unlabeled items, so training to push them away teaches the model the wrong thing. Practitioners mix hard and random negatives and sometimes filter likely false negatives.

Training uses offline (batch) feature pipelines while serving uses online (low-latency) pipelines; if their computation logic or point-in-time semantics differ, the model sees a different feature distribution at inference than at training. This train/serve skew silently degrades production accuracy even though offline metrics look fine. Feature stores enforce shared definitions and point-in-time-correct joins (no label leakage) so the same feature value is reproduced in both paths.

The system recommends popular items, which get more exposure and thus more interactions, which makes them look even more popular to the next model, progressively narrowing diversity and starving the long tail. Mitigations include exploration (bandits/randomized injections), exposure/IPS debiasing in training, popularity-discounted scoring, and diversity-promoting re-ranking. Without intervention, offline metrics can improve while real catalog coverage and long-term engagement decline.

Thompson sampling maintains a posterior over each arm's (item/policy) reward, draws a sample from each posterior, and picks the arm with the highest sampled value, so uncertain arms are occasionally explored in proportion to the probability they are best. As evidence accumulates posteriors sharpen and the policy naturally shifts toward exploitation. Contextual bandits extend this with features so exploration is personalized rather than global.

In ads and many ranking systems the predicted probability feeds downstream value computations (e.g., expected revenue = pCTR * bid), so the absolute probability, not just the relative order, must be correct. A model can rank perfectly yet be miscalibrated (e.g., systematically too high), distorting auctions/blending. Calibration is checked with reliability diagrams/calibration plots and fixed post-hoc via Platt scaling, isotonic regression, or temperature scaling.

MMR greedily selects items that maximize a weighted tradeoff between relevance to the user and dissimilarity from items already chosen, controlled by a lambda parameter. It is used at re-ranking to inject diversity and reduce redundancy in the slate (e.g., not showing five near-duplicate items) which improves perceived quality and long-term engagement. Pure relevance ranking tends to produce homogeneous, redundant lists.

score(i) = lambda*rel(i,u) - (1-lambda)*max_{j in S} sim(i,j)

New items have no interaction history, so their collaborative (ID) embeddings are untrained; the standard fix is to rely on content/side features (text, image, category, metadata) fed through the item tower so a usable embedding exists from day one. Hybrid models blend content and collaborative signals, and exploration (bandits) gives new items initial exposure to gather interactions. Over time the collaborative component takes over as data accrues.

For features with millions/billions of values (user IDs, item IDs, n-grams), maintaining an explicit vocabulary and table is memory-prohibitive and can't handle unseen values; hashing maps each value to one of a fixed number of buckets, bounding the embedding table size and gracefully absorbing new IDs. The downside is hash collisions, where distinct values share an embedding and interfere; this is mitigated with larger tables, multiple hash functions (e.g., QR/compositional embeddings), or hashing only the long tail.

Recommendations for trending, news, or just-uploaded content degrade quickly if new/changed item embeddings aren't searchable, but immutable ANN indexes (like HNSW) are costly to rebuild and don't delete cheaply. Systems handle this with incremental/streaming index updates, a small fresh 'real-time' index merged with a large periodically-rebuilt batch index, and TTL-based eviction. The architecture trades off rebuild cost, recall, and time-to-availability of new items.

A large, expensive teacher model (or an ensemble, or a model with privileged features unavailable at serving) is trained, then a smaller student is trained to match the teacher's soft scores in addition to the ground-truth labels. The student inherits much of the teacher's accuracy while meeting strict latency/cost budgets, and distilling privileged-feature teachers lets the student benefit from signals it cannot see online. It is a common way to ship high-quality rankers under tight serving constraints.

Offline data is logged under the old policy, so it suffers exposure/position bias, missing-not-at-random feedback, and cannot observe how users react to genuinely new recommendations (the counterfactual). A model that fits logged clicks may just exploit existing biases or feedback loops rather than improve true utility. Counterfactual/off-policy estimators (IPS, doubly-robust), interleaving experiments, and ultimately randomized online A/B tests with long-term guardrail metrics guard against shipping regressions.