What is collaborative filtering, and how does it recommend?

In this article

1. Before you start
2. Why interviewers ask this
3. Collaborative filtering from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Collaborative filtering is one of the core ideas of classical machine learning and the starting point of most recommender systems.

Why interviewers ask this

Collaborative filtering is one of the two strategies recommender systems are built on, with content filtering (Koren, Bell and Volinsky, 2009), so the question tests whether you can explain a ranking system from its data and not only from a model name. The usual forms are "content-based versus collaborative filtering", "what are the trade-offs", and "how would you recommend items to a user".

A weak answer says "find similar users and recommend what they liked". A strong answer says what the matrix contains, how blanks are treated, how a neighbour's rating becomes a prediction, why teams moved to latent factors, and which cases break the method, such as a new user. The usual follow-ons are the cold start problem and the two-tower model.

Collaborative filtering from first principles

In plain words

Collaborative filtering

Collaborative filtering is a recommendation method that predicts a user's interest in an item from the recorded behaviour of many users, such as ratings, purchases or clicks, without using any description of the items themselves.

Latent factor

A latent factor is a hidden number, learned from the ratings, that scores one aspect of a user or an item. Multiply a user's and an item's numbers factor by factor and add the products, and the total approximates the rating.

Think of a video-store clerk who remembers what every customer rented. A new customer says they loved three films. The clerk thinks of regulars who loved the same three. Then the clerk suggests what those regulars liked next. The analogy breaks in one place. The clerk chats with customers, but the method has only the overlap of ratings. A film nobody rated tells it nothing, not even a dislike.

In one precise sentence, collaborative filtering fills the blanks in a user-by-item matrix using similar rows, similar columns, or a low-rank model of both. The authors of the 2009 paper name two families: neighbourhood methods and latent factor models. Content-based filtering is the other approach. It describes each item with a profile, such as genre or cast, and needs that outside information.

Step 1: Collect behaviour into a matrix

The first step puts every recorded interaction into a matrix with one row per user and one column per item. The input is the event log; the output is a sparse matrix, which means most cells are blank because any single user has rated only a small percentage of the items, as Koren, Bell and Volinsky (2009) note. A blank means unknown, not zero.

The scale explains why this matters. The paper reports more than 100 million ratings of more than 17,000 movies by about half a million anonymous people in the Netflix Prize data. From those rounded figures, about 1.2 percent of the cells are filled, which is 200 ratings per person on average, and a dense table of all cells would hold 8.5 billion entries.¹ Ratings you choose to give are explicit feedback; purchases and clicks are implicit feedback and usually fill a denser matrix. What you now know: the input is a mostly empty matrix.

Step 2: Measure similarity on what both have rated

The second step scores how alike two users, or two items, are using only the cells both have filled. The input is the matrix; the output is a similarity between 0 and 1, or between minus 1 and 1 when scores are centred on each person's average. Sarwar et al. (2001) describe cosine, correlation and adjusted cosine similarity for this job.

The error is to turn blanks into zeros and compute cosine on the result. In the worked example below, that gives user A a similarity of 0.80, 0.82, 0.74 and 0.79 with users B to E, so user E, whose tastes oppose A's on the items both rated, looks as close as the best match. Computed on co-rated items with each person's ratings centred on their average over those items (the Pearson correlation), E scores minus 0.63 and B scores 0.85.¹ Koren, Bell and Volinsky also warn that inaccurate imputation can distort the data considerably. What you now know: similarity must ignore blanks and discount each person's own scale.

Step 3: Predict with a weighted sum of neighbours

The third step turns neighbours' ratings into a number for the blank cell. The input is the similarities and the neighbours' ratings; the output is a predicted rating. A common user-based form adds a similarity-weighted average of the neighbours' deviations to the user's own average:

${\hat{r}}_{u,i}={\bar{r}}_u+\frac{{\sum }_v{s}_{uv}(r_{vi}-{\bar{r}}_v)}{{\sum }_v|s_{uv}|}$

Here ${\bar{r}}_u$ is user u's average rating, $s_{uv}$ the similarity of users u and v, and $r_{vi}$ the rating v gave item i. Sarwar et al. (2001) call the item-based counterpart a weighted sum too: it averages the user's own ratings of the items most similar to the target. Keep only neighbours with positive similarity, and in real data keep only a small set of the most similar neighbours. In the example, user A's predicted rating for item I5 is 4.54 from users B and C, and 4.63 from items I1 and I4.¹ What you now know: a prediction is a weighted vote of neighbours.

Step 4: Learn latent factors instead of neighbours

The fourth step replaces neighbours with hidden numbers learned from the observed ratings. The input is the known ratings; the output is one short vector per user and per item whose dot product approximates the rating. Koren, Bell and Volinsky (2009) describe minimising the squared error on known ratings plus a penalty on the vectors' size, which they call regularisation, to avoid overfitting the few known entries.

The method scales well: with 50 factors, the Netflix-scale model above needs about 25.9 million numbers, or about 103 MB at four bytes each, against 8.5 billion cells for the dense table.¹ In the toy example, a two-factor model fitted without one known rating predicted 2.80 for A's rating of I6, whose true value is 2, an error of 0.80 against 1.23 for always guessing the average.¹ What you now know: factors compress the matrix and generalise from its observed cells only.

Step 5: Serve the neighbours you built ahead of time

The fifth step makes the request cheap by building the expensive part ahead of time. Linden, Smith and York (2003) describe item-to-item collaborative filtering at Amazon: a similar-items table is built in advance, and at request time the system looks up items similar to the user's purchases and ratings, so its online cost depends only on how many items the user has bought or rated, not on catalogue size or customer count.

They put the build cost at O(N²M) in the worst case, with N items and M customers, and closer to O(NM) in practice because most customers have few purchases. The paper also says the method works with as few as two or three items from a user, which eases the new-user case, and lists, as a common requirement for e-commerce, results in no more than half a second. What you now know: precompute the pairs, look them up at request time.

1 · Matrix2 · Similarity3 · Predict4 · Factors5 · ServeReplay

Step-by-step description

1. Collect behaviour into a user-item matrix.
2. Compare only what both have rated.
3. Weight the neighbours' ratings.
4. Learn hidden numbers instead.
5. Look up the table built in advance.

• Collaborative filtering starts from a sparse user-item matrix.
• Collaborative filtering fills the blanks in a user-by-item matrix using similar rows, similar columns, or a low-rank model of both.

Worked example

The example uses an invented matrix of five users and six items. It has 27 of 30 cells filled, far denser than real data, so every number is illustrative and every result comes from the script.¹ The aim is to predict user A's rating of item I5.

Only the centred column sees that E rates the way A does not.

Using B and C, the user-based prediction is 3.60 + (0.853 × (3 − 2.40) + 0.721 × (5 − 3.667)) / (0.853 + 0.721) = 3.60 + 1.473 / 1.574 = 4.54. Using the zero-filled similarities as weights in a plain average gives 3.26. Most of that gap is the missing centring and the kept opposite-taste neighbour E, not the zeros alone: the same plain average with co-rated cosine gives 3.35. The item-based method, with adjusted cosine similarity to I1 of 0.84 and to I4 of 0.50 and A's own ratings of 5 and 4 on those items, predicts 4.63. The two-factor model gives 5.48, which is clipped to the top of the scale, 5.0.¹

Text description of the diagram

• The zero-filled cosine average predicts 3.26 for user A on item I5.
• The user-based method predicts 4.54, the item-based method 4.63 and the two-factor model 5.0 after clipping.

Trade-offs and edge cases

Collaborative filtering trades the need for item descriptions against the need for behaviour data, which is the heart of the "content-based versus collaborative" question. Content-based filtering builds a profile for each item or user, such as genre and cast, and needs outside information that may not exist; collaborative filtering needs nothing but past behaviour. Koren, Bell and Volinsky (2009) say collaborative filtering suffers from the cold start problem, because it cannot handle new products and new users, and that content filtering is superior there.

• A new user or item. With no ratings there is no row or column to compare. Fall back to content profiles or popular items; this is the cold start problem.
• Sparsity. When users share few rated items, similarities rest on one or two cells and are unreliable. Require a minimum overlap, or move to factors.
• Implicit feedback. A click is not a rating: the absence of a click is not a dislike, so treat the data as confidence-weighted presence, as Koren, Bell and Volinsky describe.
• Scale. User-based lookups slow as users grow; Amazon's answer was a similar-items table built ahead of time, whose request-time cost does not grow with the number of customers (Linden, Smith and York, 2003). Factor models answer it with short vectors, and the 2009 authors report they were more accurate than classic nearest-neighbour techniques on the Netflix Prize data.

Text description of the diagram

• User-based methods compare users, item-based methods compare items and latent-factor methods learn hidden numbers.
• Item-based and latent-factor methods precompute the expensive part; user-based lookups slow as users grow.

Many current systems learn user and item embeddings, as in the two-tower model, and retrieve candidates by embeddings and vector search in a vector space, instead of using an explicit matrix.

How to answer it in an interview

The spoken structure is: define it from behaviour, say how blanks and similarity work, name the scalable version, then give the new-user limit and the fix.

Common mistakes

1. Blanks as zeros. A zero reads as a rating below the scale that nobody gave. (wrong) "I fill missing ratings with zero and take cosine." (right) Compute similarity on co-rated items and centre each person's ratings, or learn from observed cells only.
2. Calling it content-based. (wrong) "It finds items with similar attributes." (right) Collaborative filtering uses only behaviour; attribute similarity is content-based filtering.
3. Ignoring the new-user case. (wrong) "It works for everyone." (right) A new user has no row; use popular or content-based items first.
4. Treating a click as a rating. (wrong) "No click means the user disliked it." (right) Implicit feedback shows presence only, so weight it by confidence.
5. Predictions outside the scale. Ratings stop at 5, so 5.48 is not a rating. (wrong) "The model said 5.48 stars." (right) Clip to the rating range and rank by predicted score.

Follow-up questions

Content based vs collaborative filtering?

Content-based filtering recommends items whose descriptions match a user's profile, and needs item information. Collaborative filtering uses only behaviour, so it finds surprising items but fails for new users and items. Many systems combine both.

What are the trade-offs between collaborative filtering vs content-based approaches?

Content-based filtering handles new items and explains itself, but stays near what the user already likes. Collaborative filtering finds cross-genre taste, but needs enough interactions and suffers from sparsity. The hybrid is the usual production answer.

How do you handle a new user or a new item?

Fall back to popularity or content features until interactions arrive, and ask a few explicit questions at sign-up; this is the cold start problem.

Why does matrix factorization beat nearest neighbours?

It models all observed ratings jointly and compresses them into short vectors, which scales and generalises. The 2009 authors report it was superior to classic nearest-neighbour techniques on the Netflix Prize data.

What if each user has rated only 1 percent of the items?

Most pairs of users then share few or no rated items, so user-to-user similarities rest on one or two cells. Require a minimum overlap, or move to item-based or factor models, which learn from every observed cell; the Netflix Prize matrix was about 1.2 percent filled.

How does implicit feedback change the method?

There are no negative examples, only presence of events, so the model treats each event as a confidence-weighted preference and samples or weights the unobserved cells. Koren, Bell and Volinsky describe confidence levels for this.

Sources

1. Matrix factorization techniques for recommender systems — IEEE Computer 42(8), Koren, Bell and Volinsky (August 2009), 1 Aug 2009
2. Amazon.com recommendations: item-to-item collaborative filtering — IEEE Internet Computing, Linden, Smith and York (January to February 2003), 1 Jan 2003
3. Item-based collaborative filtering recommendation algorithms — WWW10 (Tenth International World Wide Web Conference), Sarwar, Karypis, Konstan and Riedl (2001), 1 May 2001