In this article
Before you start
This page is one step of the machine learning algorithms learning path, and it uses averages from the statistics guide.
Why interviewers ask this
A retrieval or recommendation system graded with the wrong metric gets tuned in the wrong direction, so interviewers use these metrics to test whether a candidate can match a score to a product goal. Candidates report prompts such as implementing precision at k and normalized discounted cumulative gain (NDCG) at k from scratch, hand-computing a score for a list with hits at positions 2, 4, 6 and 8, and choosing metrics to evaluate the chunks a retrieval-augmented generation (RAG) system fetches before the model answers.
A weak answer lists names: "precision, recall, NDCG." A strong answer says what each metric ignores, picks one for the stated goal, and does the arithmetic aloud. A Staff-level answer adds the limits: relevance labels are incomplete, results lists are truncated, and a better score on a labeled set may not move a live experiment.
Ranking and retrieval metrics from first principles
- Ranking metric
- A ranking metric is a score that compares an ordered list of results with relevance judgments and summarizes, in one number per query, how many relevant items the list holds and how early it places them.
- Normalized discounted cumulative gain
- Normalized discounted cumulative gain (NDCG) is a ranking metric that adds up the graded relevance of results with a discount that shrinks with rank, then divides by the best possible sum so the score for one query lies between 0 and 1.
In plain words
Picture a shop assistant who brings you a tray of ten books you might like. You could ask how many on the tray you liked, whether your favorite was even on it, how near the top your first pick sat, or how well the whole tray was ordered from best to worst. Each question is a different metric. The analogy breaks in one place. A real system returns many candidates, so metrics look only at the first k, a cutoff you choose.
In one precise sentence: a ranking metric turns a ranked list and relevance judgments into one number. Metrics differ in what may change that number. It can be the count of hits, the position of the first hit, or the position of every hit. The list used through this page has ten results with relevant ones at positions 2, 4, 6 and 8, and exactly four relevant items exist in total (illustrative).1
Step 1: Precision at k and recall at k count hits in the top k
Precision at k (written precision@k) is the number of relevant results in the top k divided by k. Recall at k (written recall@k) is the same count divided by the total number of relevant items that exist. In the book by Manning, Raghavan and Schütze, these are set-based measures, so they treat the top k as an unordered set.
On the list above, precision at 10 is 0.4 and recall at 10 is 1.0.1 Swap the hits to positions 1 to 4 and both numbers stay the same, which is the point: neither metric reads order inside the top k. Recall never falls as k grows, because adding results cannot remove hits. Precision can rise or fall. The same book notes that precision at k does not need an estimate of how many relevant items exist, but averages poorly across queries because that total strongly affects it. Next you will see a metric that does read the position of a hit.
Step 2: Reciprocal rank rewards only the first hit
The reciprocal rank of a query is one divided by the rank of its first relevant result: 1 if the first result is right, 0.5 if the second is, 0.3333 if the third is. Averaged over many queries it is the mean reciprocal rank (MRR). On the list above the first hit is at rank 2, so the reciprocal rank is 0.5.1
For three illustrative queries with reciprocal ranks 1, 0.3333 and 0.5, the MRR is 0.6111.1 MRR suits questions with one right answer, such as a lookup or a "which document answers this" task, because the first hit is all the user needs. It ignores every hit after the first, so a list with one early hit and nothing else useful scores the same as a list full of good results. For that you need a metric that sees all hits.
Step 3: Average precision rewards early hits across the list
Average precision for one query is the mean of the precision values measured at each rank where a relevant result appears, and a relevant item that is never retrieved counts as precision 0, as the Stanford text defines it. Mean average precision (MAP) is the mean of average precision over queries, weighting each query equally.
On the list above the precision at the four hit ranks is 1/2, 2/4, 3/6 and 4/8, all 0.5, so average precision is 0.5.1 Move the hits to positions 7 to 10 and it falls to 0.2815; move them to 1 to 4 and it rises to 1.0, while precision at 10 stays 0.4.1 Hits near the top lift every later precision value. Average precision still treats relevance as yes or no, which brings you to the last step.
Step 4: NDCG rewards graded relevance and early position
Normalized discounted cumulative gain at k (written NDCG@k) handles results that are more or less relevant, such as grades 0 to 3. The Stanford text describes NDCG as designed for non-binary relevance, evaluated over the top k, with a normalization factor that makes a perfect ranking score 1.
The measure was introduced by Järvelin and Kekäläinen (2002). A common form works in three moves. Discounted cumulative gain (DCG) adds each result's gain divided by log2 of its rank plus 1, so rank 1 counts fully and later ranks count less. The ideal DCG is the DCG of the same items sorted best first. NDCG is DCG divided by ideal DCG. Because both sums use the same logarithm base, changing the base leaves NDCG unchanged.
Step-by-step description
- Precision and recall at k count hits and ignore order.
- Reciprocal rank reads only the first hit.
- Average precision uses every hit with yes-or-no labels.
- NDCG adds graded relevance and a rank discount.
- Precision at 10 is 0.4 and recall at 10 is 1.0 for the list with hits at ranks 2, 4, 6 and 8.
- Reciprocal rank is 0.5 because the first hit is at rank 2.
- Average precision is 0.5 and NDCG at 10 is 0.6766.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take the ten-result list with relevant items at positions 2, 4, 6 and 8, four relevant items in total, binary relevance. Candidates report this "positions 2, 4, 6, 8" case in interviews as a question about context precision@10, a rank-weighted metric from RAG toolkits. It equals 0.5 here, against 0.4 for plain precision at 10; the follow-up below shows the formula. The numbers are computed, not looked up.1
The discount at rank r is 1 divided by log2(r + 1): 0.6309 at rank 2, 0.4307 at rank 4, 0.3562 at rank 6 and 0.3155 at rank 8. Their sum is the DCG, 1.7333. The best possible order puts the four hits at ranks 1 to 4, giving discounts 1, 0.6309, 0.5 and 0.4307, an ideal DCG of 2.5616. So NDCG at 10 is 1.7333 / 2.5616 = 0.6766.1
| Metric at k = 10 | Value for hits at ranks 2, 4, 6, 8 |
|---|---|
| Precision at 10 | 0.4 |
| Recall at 10 | 1.0 |
| Reciprocal rank | 0.5 |
| Average precision | 0.5 |
| DCG at 10 | 1.7333 |
| Ideal DCG at 10 | 2.5616 |
| NDCG at 10 | 0.6766 |
Text description of the diagram
- The DCG of the list with hits at ranks 2, 4, 6 and 8 is 1.7333.
- The ideal DCG is 2.5616, so the NDCG at 10 is 0.6766.
Now keep the four hits and move them. At ranks 1 to 4 the precision at 10 is 0.4, average precision 1.0 and NDCG 1.0. At ranks 7 to 10 precision at 10 is still 0.4, while average precision is 0.2815, reciprocal rank 0.1429 and NDCG 0.4836.1 That is the page's central result: precision at k is blind to the order that the other metrics punish.
Text description of the diagram
- Precision at 10 is 0.4 for all three lists.
- Average precision is 1.0, 0.5 and 0.2815, and NDCG is 1.0, 0.6766 and 0.4836, as the hits move down.
The setup of an exercise in the Stanford text makes the same point. Two systems each return four relevant items in their top ten, one with hits at ranks 1, 3, 9, 10 and the other at 2, 5, 6, 7. Both have precision at 10 of 0.4 and recall 1.0, but average precision is 0.6 against 0.4929 and reciprocal rank is 1.0 against 0.5.1
For graded relevance, grades 3, 2, 3, 0, 1, 2 in ranked order (illustrative) give an NDCG of 0.9608 with the raw grade as gain and 0.9488 when the gain is 2 to the grade minus 1.1 The gain formula changes the number, so always state it. A short function covers the two interview coding tasks:
import math
def precision_at_k(ranked_ids, relevant_ids, k):
return sum(1 for i in ranked_ids[:k] if i in relevant_ids) / k
def ndcg_at_k(ranked_gains, all_gains, k):
def dcg(gains):
return sum(g / math.log2(rank + 1)
for rank, g in enumerate(gains[:k], start=1))
ideal = dcg(sorted(all_gains, reverse=True)) # best possible order
return dcg(ranked_gains) / ideal if ideal > 0 else 0.0On the ten-result list it returns 0.4 and 0.6766, matching the table.1 Computing it for many queries is a loop over the same function, and the ideal DCG can be cached per query because it does not depend on the system.
Trade-offs and edge cases
Each metric trades a blind spot for simplicity, and the right choice follows the product.
| Metric | Sees order? | Handles grades? | Best when |
|---|---|---|---|
| Precision at k | No | No | the first page is all users see and order inside it matters little |
| Recall at k | No | No | a later stage reranks the list, so missing an item is the costly error |
| MRR | First hit only | No | there is one right answer |
| Mean average precision | Every hit | No | many right answers, yes-or-no labels |
| NDCG at k | Every hit | Yes | graded labels, top positions matter most |
Edge cases are where answers separate. A query with no relevant items has an ideal DCG of 0, so NDCG is undefined; state a convention, such as skipping the query or scoring it 0, and keep it fixed. Ties need a rule too: when two items share a model score, their order is arbitrary, so either break ties deterministically or average over the tied orders. The scikit-learn documentation lists an ndcg_score function in version 1.9.1 as of 2026-10-26; read its tie option before comparing numbers across libraries.
Labels are incomplete. The Stanford text notes that a known-relevant set may be incomplete when judgments cover only the pooled top results of some systems, so an unjudged item counts as non-relevant. That means a good new system that returns unjudged items can be penalized, which is our inference from the source, not its wording. The same source says MAP scores vary widely between queries for one system, so a test set must be large and varied.
As of October 2026, RAG evaluation toolkits such as Ragas can fill the missing labels with a language model: its context precision documentation describes judging each retrieved context against a reference answer. That scales, but the judge adds its own error.
Pre-deployment gains may not carry into a live experiment, because the labeled set is a sample of past queries and users; the distribution shift page covers why. Treat the score as a filter that removes bad candidates, then confirm with a live test. For the related curve view, see how to compute a receiver operating characteristic (ROC) curve in scikit-learn, and for metrics of generated text rather than ranked lists, see the bilingual evaluation understudy (BLEU) and recall-oriented understudy for gisting evaluation (ROUGE) comparison in BLEU vs ROUGE. Recommenders such as those built with collaborative filtering are graded the same way.
How to answer it in an interview
How would you evaluate the chunks a RAG system retrieves, and which metrics would you use?
I would build a labeled set of questions with their relevant chunks, then run the retriever at a fixed k. I would start with recall at k, because the generator cannot use a chunk that was never retrieved, and add precision at k to see how much noise reaches the prompt. Neither sees order, so I would add mean reciprocal rank or NDCG, depending on whether one chunk or several matter. I would average over many questions, then check final answer quality separately, since retrieval scores alone do not settle it.
The spoken structure: name the labeled set and the cutoff, say which metric covers missing items, which covers noise and which covers order, then add the limit.
Common mistakes
- "Precision at k tells me how well the system ranks." (wrong) It ignores order inside the top k. (right) "Precision at k counts hits; for ranking quality I use average precision or NDCG."
- "NDCG is always between 0 and 1, so I can compare it directly across queries." (wrong) The 0 to 1 range holds only with non-negative gains, and NDCG is undefined for a query with no relevant items. (right) "NDCG is normalized per query, and I state how I treat queries with no relevant items."
- "Recall at k drops as k grows." (wrong) It cannot fall. (right) "Recall at k never decreases with k; precision at k can go either way."
- "MRR and MAP are the same." (wrong) MRR reads only the first relevant result. (right) "MRR scores the first hit; MAP averages precision at every hit."
- "A higher NDCG means a better live product." (wrong) It measures agreement with labels. (right) "It is a pre-deployment filter, and I confirm in a live experiment."
Follow-up questions
How will you evaluate the retrieved chunks before generation, and which metrics would you use?
Report each metric as a mean over all test questions together with its spread across questions, at the k you feed the generator. Then judge the generated answers separately, because a retrieval score is only a proxy for answer quality.
What is context precision@10 when the relevant chunks sit at positions 2, 4, 6 and 8?
Plain precision at 10 is 0.4, because four of ten results are relevant.1 The Ragas context precision sums precision@k at each relevant rank and divides by the number of relevant items in the top K: (1/2 + 2/4 + 3/6 + 4/8) / 4 = 0.5.1 The 0.5 says every hit sits behind as many misses as hits, so the retriever is not placing relevant chunks first; the same four hits at ranks 1 to 4 would score 1.0. Toolkits differ, so state the formula you use.
When do you choose precision at k over NDCG at k?
Choose precision at k when labels are yes or no and only the count of good results in a visible page matters. Choose NDCG at k when relevance is graded or when the top positions matter much more than the rest.
How do you handle ties in NDCG?
Fix a rule before comparing systems: break ties deterministically, for example by item id, or average the score over all orders of the tied items. Report which rule you used.
What if a better NDCG does not show up in a live experiment?
The labeled set may differ from live traffic, labels may be incomplete, or the live goal may differ from relevance, such as diversity or engagement. Check the data for distribution shift, add metrics that match the goal, and run the live test before deciding.
Why are some context metrics reference-free and others reference-dependent?
A reference-dependent metric needs labeled relevant items or a reference answer to compare against. A reference-free one scores relevance without them, usually with a model acting as judge, which scales better but adds the judge's own error. Prefer reference-free scoring for fast monitoring and reference-dependent scoring for a trusted benchmark.
- A ranking metric turns a ranked list and relevance labels into one score per query.
- Precision and recall at k count hits in the top k and ignore order.
- Reciprocal rank scores the first hit; average precision uses every hit with yes-or-no labels.
- NDCG handles graded relevance by dividing discounted gain by the best possible gain.
- State k, the gain formula, tie rule and the treatment of queries with no relevant items.
Sources
- Introduction to Information Retrieval, Evaluation of ranked retrieval results — Manning, Raghavan and Schütze, Cambridge University Press (online edition), 1 Jan 2009
- Cumulated gain-based evaluation of IR techniques — ACM Transactions on Information Systems (Järvelin and Kekäläinen), 1 Oct 2002
- Context Precision, Ragas documentation — Ragas (open-source RAG evaluation library), 26 Oct 2026
- ndcg_score, scikit-learn 1.9.1 documentation — scikit-learn, 26 Oct 2026
