How recommendations work

A recommendation system is a computer program that suggests items — like movies, songs, or products — that a user is likely to enjoy, based on data about their past behaviour or preferences. It learns patterns from many users and uses those patterns to predict what any individual user might want next. Amazon, Spotify, and Netflix all depend on recommendation systems to keep users engaged.

Netflix uses its recommendation system to suggest movies and TV shows that a specific viewer is likely to enjoy, based on what they have watched before and how users with similar tastes have behaved. This reduces the time users spend searching and keeps them watching longer. Roughly 80% of viewing on Netflix comes directly from recommendations rather than manual searching.

Collaborative filtering is a technique where the recommendation system looks at the behaviour of many different users, finds people who have similar taste to you, and then recommends items that those similar people liked. The core idea is: 'If two users agreed on many things in the past, they will probably agree in the future.' Amazon's 'Customers who bought this also bought' feature is a classic example of collaborative filtering.

Explicit feedback is information a user directly provides to express their opinion, such as giving a film a five-star rating, clicking a 'like' button, or writing a review. It is very clear and straightforward for the algorithm to use because the user is stating their preference openly. However, not every user bothers to rate or review items, so explicit feedback can be rare.

Implicit feedback is behavioural data collected from a user's actions without them directly stating an opinion — for example, how long they watched a video, which items they clicked, or what they searched for. It is easier to collect in large amounts because users generate it automatically as they use the app. However, it can be noisy: clicking on something does not always mean you liked it.

A user profile is a stored collection of data the system maintains about an individual — including their past ratings, purchase history, search queries, and watch history — that helps the system understand their preferences. The richer and more complete this profile, the more accurately the system can personalise recommendations. Spotify's 'Discover Weekly' playlist is essentially built by analysing each user's profile.

Content-based filtering recommends items by comparing the features of items a user has already liked to the features of new items available. For example, if you enjoyed fast-paced action movies, the system looks for other movies with similar attributes — like fast pace and action genre — and suggests those. Unlike collaborative filtering, it does not need to know what other users think; it only analyses the items themselves.

The cold start problem occurs when a recommendation system has very little or no data about a brand-new user or a brand-new item, making it very difficult to produce useful suggestions. For a new user, the system doesn't know their preferences yet; for a new item, nobody has rated it yet. A common workaround is to ask new users a few quick questions about their interests when they first sign up.

A rating matrix is a table where each row represents a user and each column represents an item, with the cells holding the rating that user gave that item (or left blank if they haven't rated it). Most cells are empty because no single user has watched, bought, or rated every available item — this property is called 'sparsity.' Many recommendation algorithms work by predicting the values of those missing cells.

Cosine similarity measures how similar two lists of numbers are by calculating the angle between them when treated as arrows (vectors) in mathematical space; a score of 1 means the two arrows point in exactly the same direction (very similar) and 0 means they are completely different. A recommendation system represents each user as a vector of their item ratings and then computes cosine similarity between users. The users with the highest similarity scores have the most compatible tastes and are the best candidates to borrow recommendations from.

A filter bubble happens when a recommendation system keeps showing a user only content that matches their existing beliefs or tastes, so they rarely encounter different ideas or viewpoints. The algorithm is just trying to maximise engagement by showing what you click on most, but the side effect is that your information world becomes narrower over time. For example, a news recommendation engine might only ever show articles that already agree with your current opinions.

Popularity bias is the tendency of a recommendation system to suggest only the most popular items — those with the most ratings or sales — while largely ignoring niche or lesser-known items. Popular items have more ratings so the algorithm is more confident about them, but this means new or obscure creators rarely get discovered. It can also be unfair to users with unusual tastes, who never receive relevant personalised suggestions.

User-based collaborative filtering finds other users who are most similar to you and recommends items those similar users liked, essentially asking 'Who is like me?' Item-based collaborative filtering instead finds items most similar to ones you already liked and recommends those, asking 'What is like what I enjoyed?' Item-based is often preferred in practice because item similarities stay stable over time, whereas user similarities change as users' tastes evolve.

A hybrid recommendation system combines two or more recommendation techniques — for example, blending collaborative filtering with content-based filtering — to get better results than either approach alone. Each technique has weaknesses that the other can cover: content-based filtering can handle new items with no ratings (solving part of the cold start problem), while collaborative filtering catches cross-genre discoveries that item features alone would miss. Netflix is a well-known example of a platform that uses a hybrid approach.

This happens because the model may not have been designed to filter out items the user already owns, or it relies on collaborative filtering where many similar users frequently re-purchase the same item. Models trained on clicks or views rather than actual purchase records are particularly prone to this mistake. Filtering out already-purchased items is a practical engineering step that is sometimes overlooked when building recommendation pipelines.

Serendipity in recommendations refers to the quality of occasionally suggesting something unexpected that the user ends up genuinely loving — a pleasant surprise rather than just more of what they already know they like. A system that only recommends perfectly predictable items can feel boring and trap users in a narrow set of experiences. Adding a small amount of diversity or controlled randomness to recommendations helps users discover new interests they never knew they had.

The long tail refers to the large number of niche or obscure items that each individually have very few ratings or sales but together make up the majority of a platform's catalogue. Recommendation systems tend to ignore these items because there is not enough data about them to make confident predictions. However, some users specifically love rare or unusual items, so a good system should surface long-tail content to the right audience rather than only promoting well-known titles.

K-Nearest Neighbours (KNN) is an algorithm that finds the K users (or items) most similar to a target, and then uses their ratings or data to predict what the target user would rate unseen items. For example, with K=5, the system finds the 5 users most similar to you and averages their ratings to suggest new things. KNN is simple and easy to understand, but it can become very slow when the dataset has millions of users and items.

Matrix factorization is a technique that decomposes the large sparse user-item rating matrix into two smaller matrices — one describing users and one describing items — each containing hidden numbers called latent factors. Multiplying these two smaller matrices back together reconstructs predicted ratings for all the missing entries. It is powerful because the algorithm automatically discovers meaningful hidden patterns, such as genre preferences, without anyone manually labelling them.

A feedback loop happens when a recommendation system promotes certain items, those items receive more user interactions, which causes the algorithm to recommend them even more — creating a self-reinforcing cycle that amplifies a narrow set of content while suppressing everything else. Over time this can trap users in a very limited information diet and make it nearly impossible for diverse or new items to break through. Breaking feedback loops often requires deliberately injecting diversity or exploration into the recommendation process.

Latent factors are hidden numerical values that a recommendation model learns automatically from user-item interaction data — they represent underlying patterns like 'this user enjoys fast-paced stories' or 'this item has a melancholic mood' without anyone explicitly labelling those patterns. Each user and each item is described by a short list of these latent factor numbers, and the system recommends items whose factor numbers closely match a user's own numbers. They are called 'latent' because they are inferred from data rather than directly observed or named.

When a news recommendation system is optimised for clicks or watch time, it can end up promoting sensational, misleading, or emotionally extreme content because such content triggers stronger reactions and keeps people engaged for longer. This can contribute to the spread of misinformation and deepen political polarisation by consistently showing people only content that confirms what they already believe. Ethical design requires balancing engagement metrics with signals for accuracy, source credibility, and diversity of viewpoints.

The explore-exploit trade-off is the tension between 'exploiting' things the system already knows work well for a user (recommending proven favourites) versus 'exploring' new or unknown items that might be even better. If the system only exploits, users get bored and the system may miss better recommendations; if it only explores, users see too many irrelevant or unfamiliar suggestions. Most production systems balance both, for example showing roughly 90% proven recommendations and 10% exploratory ones.

Deep learning models can automatically learn complex, non-linear patterns directly from raw data such as images, audio, and text — for example, they can understand what a movie's visual style looks like or how a song sounds — and use this to build much richer representations of users and items. This means a system can recommend a song because of how it actually sounds, not just its metadata tags. Neural networks also capture subtle and complicated patterns in user behaviour that simple algorithms like KNN or basic matrix factorization would miss.

Cross-domain recommendation uses data from one area of a user's activity (e.g., their music listening history) to improve recommendations in a completely different area (e.g., movies or books), because a person's interests often overlap across categories in predictable ways. For example, knowing a user loves jazz music might help a system infer they would enjoy certain drama films. It is especially valuable when one domain has very little data about a user — a classic cold start scenario.

A/B testing is a controlled experiment where two versions of a feature — version A and version B — are shown simultaneously to two different random groups of users, and their real behaviour (clicks, ratings, time spent) is compared to see which version performs better. Recommendation engineers run A/B tests constantly to prove that a new algorithm genuinely improves the user experience before rolling it out to everyone. This scientific approach prevents teams from making costly changes based on guesses.

A personalised recommendation is one tailored specifically to an individual user based on their unique history, preferences, and context — rather than showing everyone the same list of popular items. Two users on the same platform may see completely different home screens because the algorithm treats each person as unique. Personalisation improves user satisfaction because suggestions feel relevant to that particular person rather than generic.

Spotify uses a rich combination of data including your listening history, which songs you replayed or skipped, which tracks you added to playlists, and how users with similar taste behave. It also analyses the actual audio features of songs — such as tempo, energy level, danceability, and musical key — to identify music that sounds similar to what you enjoy. All of this is blended together to power personalised playlists like 'Discover Weekly' and 'Daily Mix.'

Real-time recommendations are generated on the fly in milliseconds as you browse, instantly adapting to your most recent actions such as what you just clicked or searched. Batch recommendations are computed in advance — for example, overnight — and stored so they can be quickly served to users without any live computation. Real-time is more relevant and responsive but requires powerful infrastructure, while batch is cheaper and simpler but may feel slightly out of date.

If the training data contains historical biases — for example, if certain demographic groups were historically shown fewer choices or had less of their data collected — the algorithm learns and reproduces those biases, giving those users worse or less relevant recommendations. A system trained predominantly on data from one demographic group may simply fail to understand the preferences of underrepresented groups. Checking recommendation quality separately across different demographic groups is an important fairness and ethics step.

Click-through rate (CTR) is the percentage of recommended items that a user actually clicks on — for example, if 8 out of 100 recommended items are clicked, the CTR is 8%. A high CTR sounds great but can be misleading: a sensational thumbnail or headline might attract clicks even when the underlying content disappoints the user. A more complete picture of recommendation quality combines CTR with signals like watch completion rate, user satisfaction ratings, and return visits.

This code represents each user as a vector of their item ratings and then uses cosine similarity to measure how closely User A's rating pattern aligns with every other user's pattern. A score close to 1 means those users have very similar tastes and are good candidates to borrow recommendations from, while a score close to 0 means their tastes are very different. The system would then recommend items that the most similar user enjoyed but User A has not yet rated.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Rows = users, Columns = items (0 = not yet rated)
ratings = np.array([
    [5, 3, 0, 1],   # User A
    [4, 0, 4, 1],   # User B
    [1, 1, 0, 5],   # User C
])

# Compare User A (index 0) with all users
sims = cosine_similarity([ratings[0]], ratings)[0]
print(sims)  # Closer to 1 = more similar taste

Recommendation systems collect deeply personal data — viewing habits, purchase history, location, and even the time of day you use an app — which together can reveal sensitive information about a person's health, beliefs, finances, or lifestyle. If this data is misused, sold to third parties without consent, or exposed in a security breach, real harm can follow. Good recommendation systems comply with privacy laws such as GDPR, minimise the data they collect, and give users meaningful control over what is stored and how it is used.

Optimising for an objective means the algorithm is mathematically trained to maximise one specific measure — such as total watch time, click count, or revenue — so every decision it makes is aimed at increasing that number. The chosen objective powerfully shapes what content gets recommended: optimising purely for watch time might lead the system to recommend addictive or extreme content because it keeps people glued to the screen the longest. Choosing a responsible objective — one that also considers user wellbeing, information diversity, and truthfulness — is one of the most critical ethical decisions in designing a recommendation system.