Evaluation metrics from scratch in Python, step by step: F1, AUC and NDCG

In this article

1. Before you start
2. Why interviewers ask this
3. Evaluation metrics from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Writing these metrics without a library is a staple of the machine learning (ML) coding round.

Why interviewers ask this

Metric code is short, and a wrong metric hides a bad model. Coding prompts reported by candidates at large tech companies and AI labs include writing precision and recall without a library and extending them to several classes, explaining and implementing AUC, and writing NDCG@K with ties and graded relevance.

A weak answer recites formulas and divides by zero on an empty class. A strong answer states what each number means, counts before dividing, and says which metric fits the data. A Staff-level answer adds the traps: imbalance, averaging choices, tie rules and per-example errors in a harness. Models such as those in logistic and linear regression from scratch produce the scores that AUC ranks.

Evaluation metrics from first principles

Confusion matrix

A confusion matrix is a table that counts, for every true class, how many examples a classifier assigned to each class, so the diagonal holds correct calls and every other cell holds one specific kind of mistake.

AUC

AUC is the area under the receiver operating characteristic curve of a scoring model, and it equals the probability that a randomly chosen positive example gets a higher score than a randomly chosen negative one, counting ties as one half.

In plain words

Think of a fishing net. Precision asks how much of what is in the net is fish rather than boots. Recall asks how many of the fish in the lake ended up in the net. A bigger net raises recall and usually lowers precision. The analogy breaks for ranking: a net has no order, but a search result list does, and ranking metrics reward putting the best items first.

In one precise sentence, a metric is a function from predictions and true answers to a number, and every metric here is built from counts of outcomes, counts of correctly ordered pairs, or position-weighted gains.

Step 1: Count four outcomes, because accuracy alone can mislead

For one positive class, each example falls into one of four cells: true positive (positive, called positive), false positive (negative, called positive), false negative (positive, called negative) and true negative (negative, called negative). Accuracy is correct calls over all calls. Now the trap: take 1,000 examples of which 10 are positive and a model that always answers negative. Its accuracy is 0.99 and its recall is 0.¹ The headline number looks excellent while the model finds nothing.

Precision and recall avoid that. In a ten-example case with 4 real positives, a model finds 3, wrongly flags 2 negatives and misses 1: precision is 0.6 and recall is 0.75, while accuracy is 0.7.¹ The F1 score is the harmonic mean of precision and recall, 2 × precision × recall divided by their sum, and here it is 0.6667.¹ In symbols:

You now have the base: count first, divide second, and guard every division against a zero denominator.

Step 2: Extend to many classes with macro and micro averages

For several classes, treat each class in turn as "positive" and everything else as "negative", which gives a precision and recall per class read off the confusion matrix. A macro average is the plain mean of the per-class scores, so every class counts equally. A micro average is the score computed after pooling the true positive, false positive and false negative counts of all classes and then dividing, so every example counts equally.

On the illustrative three-class matrix in the worked example, macro precision is 0.6556 and micro precision is 0.6667.¹ With one label per example, micro precision, micro recall and accuracy are the same number, 0.6667 here, because every mistake is one class's false positive and another's false negative.¹ Macro F1, in the convention scikit-learn uses, is the mean of per-class F1 scores rather than the F1 of the two macro averages; the literature has both versions, so say which you mean.

Step 3: AUC counts correctly ordered pairs

A model that outputs scores can be judged without choosing a threshold. AUC asks: take one random positive and one random negative; how often does the positive get the higher score? Ties count one half. That probability is the physical meaning of AUC. The receiver operating characteristic (ROC) curve plots the true positive rate (recall) against the false positive rate, which is false positives divided by false positives plus true negatives, as the score threshold moves from high to low. Fawcett, 2006, §7 states that the area under that curve equals the probability that a random positive is ranked above a random negative.

For eight scored examples, four positive and four negative, there are 16 positive-negative pairs. The positive wins 13, so AUC is 13 / 16 = 0.8125.¹ Counting 16 pairs is quadratic, so interviews ask for the faster version. Sort all scores, rank them from 1 (average ranks over ties), add up the ranks of the positives, subtract n(n+1)/2 for n positives and divide by positives times negatives. That takes time proportional to n log n, and it gives 0.8125 as well.¹

Here $R_{\text{pos}}$ is the sum of the positives' ranks, and $n_{\text{pos}}$ and $n_{\text{neg}}$ are the numbers of positives and negatives.

Step 4: NDCG@K rewards putting the best items first

Normalized discounted cumulative gain at K (NDCG@K) is the discounted cumulative gain (DCG) of the top K results divided by the DCG of the best possible ordering of the same results, so it lies between 0 and 1. DCG, introduced by Järvelin and Kekäläinen (2002), adds each result's relevance divided by a discount that grows with position, so later positions count less. This page uses the common modern discount, log base 2 of the position plus one; the 2002 paper uses a log-base-b discount applied from position b, so state which form you use.

Take relevance grades [3, 2, 3, 0, 1, 2] in the order the system returned them. The discounts for positions 1 to 6 are 1, 0.6309, 0.5, 0.4307, 0.3869 and 0.3562, so DCG is 6.8611 against 7.141 for the ideal order [3, 3, 2, 2, 1, 0], and NDCG is 0.9608.¹ At K = 3 the value is 0.9778.¹ Using 2 to the power relevance minus 1 as the gain gives 0.9488 instead, so state which gain you use.¹

1 · Count outcomes2 · Average classes3 · Rank pairs4 · Discount gainsReplay

Step-by-step description

1. Precision and recall are ratios of counts.
2. Macro weights classes, micro weights examples.
3. AUC counts ordered positive-negative pairs.
4. NDCG rewards the best items first.

• Precision and recall are ratios of counted outcomes.
• Macro averaging weights classes equally and micro averaging weights examples equally.
• AUC counts correctly ordered pairs of a positive and a negative.
• NDCG divides the discounted gain of the returned order by the ideal order.

Worked example

The inputs are small enough to redo with a calculator. The binary case has labels [1,1,1,1,0,0,0,0,0,0] and calls [1,1,1,0,1,1,0,0,0,0]. The three-class case has a confusion matrix whose rows are true classes: class 0 is [4,1,1], class 1 is [1,2,0] and class 2 is [0,1,2]. The AUC case has scores [0.9,0.8,0.7,0.6,0.55,0.54,0.4,0.3] with labels [1,1,0,1,0,1,0,0]. All of these are illustrative.¹

Micro precision weights each class by how many predictions it made: class 0 made 5 of the 12 predictions at precision 0.8, so micro gives it 5/12 of the weight against a third under macro, which lifts micro precision (0.6667) above macro (0.6556). Recall is 0.6667 for every class, so macro and micro recall agree.¹

Text description of the diagram

• Per-class precision is 0.8, 0.5 and 0.6667.
• Macro precision is 0.6556 and micro precision is 0.6667.

The code below computes all of it with no library. The prf function returns zero instead of dividing by zero, a choice you should say out loud.

import math

def counts(y_true, y_pred, label):
    tp = fp = fn = 0
    for t, p in zip(y_true, y_pred):
        if p == label and t == label: tp += 1
        elif p == label: fp += 1
        elif t == label: fn += 1
    return tp, fp, fn

def prf(tp, fp, fn):
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    return precision, recall, f1

def multiclass(y_true, y_pred):
    labels = sorted(set(y_true) | set(y_pred))
    per = {l: prf(*counts(y_true, y_pred, l)) for l in labels}
    macro = tuple(sum(per[l][i] for l in labels) / len(labels) for i in range(3))
    pooled = [sum(counts(y_true, y_pred, l)[i] for l in labels) for i in range(3)]
    return per, macro, prf(*pooled)

def auc(scores, labels):
    order = sorted(range(len(scores)), key=lambda i: scores[i])
    ranks, i = [0.0] * len(scores), 0
    while i < len(order):
        j = i
        while j + 1 < len(order) and scores[order[j + 1]] == scores[order[i]]:
            j += 1
        for t in range(i, j + 1):
            ranks[order[t]] = (i + j) / 2 + 1   # average rank over ties
        i = j + 1
    n_pos = sum(labels)
    n_neg = len(labels) - n_pos
    rank_sum = sum(r for r, l in zip(ranks, labels) if l)
    return (rank_sum - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)

def dcg(rels, k):
    return sum(r / math.log2(i + 2) for i, r in enumerate(rels[:k]))

def ndcg_at_k(rels, k):
    ideal = dcg(sorted(rels, reverse=True), k)
    return dcg(rels, k) / ideal if ideal else 0.0

Two things to say about this code. The auc function sorts once, so its time is proportional to n log n. The ndcg_at_k function needs the relevance of every candidate, not only the top K returned, because the ideal ordering comes from all of them.

A ranking system usually returns scores, not a list already in order, so the variant below sorts by score first. When scores tie, it gives every tied position the average relevance of the tied group, so the result does not depend on input order. Ties in relevance do not change the ideal DCG, because sorting the grades gives the same list either way; ties in predicted scores do change the DCG, which is why the tie rule matters. Binary relevance is simply grades in {0, 1}, so the same code serves both.¹

def dcg_from_scores(scores, rels, k):
    order = sorted(range(len(scores)), key=lambda i: -scores[i])
    total, i = 0.0, 0
    while i < len(order):
        j = i
        while j + 1 < len(order) and scores[order[j + 1]] == scores[order[i]]:
            j += 1
        avg = sum(rels[order[t]] for t in range(i, j + 1)) / (j - i + 1)
        for pos in range(i, min(j + 1, k)):
            total += avg / math.log2(pos + 2)   # tied items share the average gain
        i = j + 1
    return total

def ndcg_from_scores(scores, rels, k):
    ideal = dcg(sorted(rels, reverse=True), k)
    return dcg_from_scores(scores, rels, k) / ideal if ideal else 0.0

For mean NDCG over many queries, compute each query's ideal DCG once, then average the per-query values. A query whose ideal DCG is zero has no relevant item; the code above returns zero for it, and skipping the query is the other common choice, so state which one you use.

Trade-offs and edge cases

Each metric answers one question and ignores others. Precision ignores positives that were missed, recall ignores false alarms, AUC ignores which threshold you will deploy, and NDCG ignores how many relevant items exist beyond K.

Text description of the diagram

• Accuracy can reach 0.99 while recall is 0 on imbalanced data.
• AUC needs scores, and NDCG needs graded relevance and a ranking.

Edge cases to name in an interview:

• Empty denominators. A class never predicted has no precision, and a class with no examples has no recall. Return zero, return not-a-number or raise, but state the rule.
• Ties in scores. AUC counts a tied pair as one half; for NDCG, decide whether tied items keep input order or share an average gain, and say so.
• Gain formula. Linear and exponential gain give different NDCG values, as the worked example shows (0.9608 against 0.9488).¹
• Multiple-choice benchmarks. HellaSwag asks a model to select the most likely follow-up to an event description, so accuracy over examples is the natural headline. Its authors report that humans exceed 95% while the 2019 models they tested scored below 48%. How a harness scores each candidate ending, for example raw or length-normalized likelihood from a numerically stable softmax, is a design choice to state, because the abstract does not specify it.
• Metric versus loss. A training loss must be differentiable, as in gradient descent; precision, recall and NDCG are not, so they are optimized only through differentiable surrogates (ranking losses such as LambdaRank are one family) and otherwise just reported.

A harness that survives bad examples

An evaluation harness runs a model over a dataset and computes several metrics without letting one bad example stop the run. The loop below catches the error per example, logs the failed index, computes the metrics on the successes and reports the error count with the scores.

def evaluate(model, dataset, metrics):
    golds, preds, failed = [], [], []
    for idx, (x, y) in enumerate(dataset):
        try:
            preds.append(model(x))
            golds.append(y)
        except Exception as err:          # one bad example must not stop the run
            failed.append((idx, repr(err)))
    results = {name: fn(golds, preds) for name, fn in metrics.items()}
    results["n_errors"] = len(failed)
    return results, failed

Metrics on the successes can be biased if the failures are not random, which is why the error count and the failed indices are returned and never dropped.

How to answer it in an interview

The spoken structure: define the two ratios from the counts, extend by class, name the averaging choice, then name the trap that accuracy hides.

Common mistakes

• "The model has 99% accuracy, so it is good." (wrong) On imbalanced data a model that never finds a positive can score that. (right) "I check recall and precision per class, not accuracy alone."
• "Macro and micro averages are the same thing." (wrong) They differ when per-class scores differ and the class weights are unequal: predicted counts for precision, support for recall. (right) "Macro treats classes equally; micro treats examples equally; I say which one I report."
• "AUC is just the area under a curve." (wrong) That hides the meaning. (right) "AUC is the probability a random positive scores above a random negative, ties counting half."
• "NDCG@K is the DCG of the top K." (wrong) Without dividing by the ideal DCG the value is not comparable across queries. (right) "I normalize by the best possible ordering of the same items."

Follow-up questions

What are evaluation metrics?

Evaluation metrics are functions that compare a model's outputs with known answers and return a number. Classification uses counts-based metrics such as precision and recall, scoring models use AUC, and ranking systems use NDCG@K.

How is recall defined in evaluation metrics?

Recall is true positives divided by true positives plus false negatives: the share of the real positives that the model found. In the binary example it is 3 out of 4, or 0.75.¹

How would you build an evaluation harness that survives bad examples?

Wrap the model call for each example in its own error handler, record the example index and the error, and continue. Compute every metric on the examples that succeeded, and report the error count next to the scores so failures are never silently dropped. Writing it like production code, with small tested functions, is what ML coding rounds reward, and the habits overlap with debugging training loops and pipelines.

How would you evaluate a model on HellaSwag?

Score every candidate ending with the model, pick the highest-scoring one, and report accuracy over the examples. State the scoring rule, since the HellaSwag paper frames the task as selecting the most likely follow-up and the choice between raw and length-normalized scores changes the result. A language model built as in build generative pre-trained transformer (GPT) from scratch can be evaluated this way.

How do you handle ties when computing NDCG@K?

Pick one rule and state it: keep the input order of tied items, or average the gain over the tied positions. The two rules can give different values, so the rule belongs next to the number you report.

What does AUC mean in plain words?

AUC is the chance that a random positive example outranks a random negative one. A value of 0.5 means the scores order pairs no better than chance, and 1 means every positive outranks every negative; the worked example gives 0.8125.¹

Sources

1. HellaSwag: Can a Machine Really Finish Your Sentence? (arXiv 1905.07830) — arXiv (Zellers, Holtzman, Bisk, Farhadi, Choi), 19 May 2019
2. An introduction to ROC analysis — Pattern Recognition Letters (Fawcett), 1 Jun 2006
3. Cumulated gain-based evaluation of IR techniques — ACM Transactions on Information Systems (Jarvelin and Kekalainen), 1 Oct 2002