What are LLM benchmarks and perplexity, and how do you read them?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM benchmarks and perplexity from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the LLM evaluation learning path. It also builds on the transformer architecture, the model family behind most LLMs.

Why interviewers ask this

A candidate who quotes a leaderboard number without asking how it was produced is a risk to a team that has to choose or ship a model. Questions on this topic are reported by candidates in interviews at AI labs and large tech companies, in forms such as "explain perplexity and what other benchmarks we use", "what are MMLU, HumanEval and GSM8K", and "how do you detect benchmark contamination".

A weak answer lists benchmark names and says "lower perplexity is better." A strong answer says what each score measures, what setup it depends on, and what it cannot see. A Staff-level answer adds the failure that matters most in practice: a public test set that leaked into training data, so the score measures memory, not skill. The neighbouring questions on LLM-as-a-judge and on evaluating retrieval systems with Ragas build on the same idea: a metric is only as good as what it measures.

LLM benchmarks and perplexity from first principles

Perplexity

Perplexity is a score for an autoregressive language model that is the exponential of the average negative log-likelihood the model assigns to each token of a text. It is the exponential of the cross-entropy loss, which is the average negative log-likelihood between the data and the model's predictions, and lower means the model was less surprised by the text.

Benchmark contamination

Benchmark contamination is the presence of a benchmark's test questions, or their answers, in a model's training data, so that the score reflects memorised material and not general ability. It inflates scores in a way the final number does not reveal.

pass@k

pass@k is the probability that at least one of k code samples written by a model for a problem passes that problem's unit tests. It is estimated from n samples, of which c pass, with an unbiased formula and not by simple repetition.

In plain words

Imagine two ways to test a student. In one, you read a page of a book aloud, stop before each word and ask the student to guess it. A student who is rarely surprised knows the language well. In the other, you give an exam and mark the answers. The first is perplexity. The second is a benchmark.

A benchmark is a fixed set of tasks with an agreed scoring rule, so different models can be compared on the same test. The analogy breaks in one place. A student who has seen the exam beforehand scores high without knowing the subject, and a model can do the same.

In one precise sentence, perplexity scores how well a model's probabilities match real text, and a benchmark scores whether a model's answers pass a fixed test.

Step 1: Perplexity is the exponential of the average negative log-likelihood

A language model gives each real token a probability given the tokens before it. The negative log-likelihood of a token is minus the natural logarithm of that probability: a likely token costs little, an unlikely token costs a lot. The Hugging Face documentation states that perplexity is the exponentiated average negative log-likelihood of a sequence, and that this equals the exponentiation of the cross-entropy between the data and the model's predictions.

Here t is the number of tokens. Input: the probability the model gave each real token. Output: one number that you can read as an effective number of equally likely choices per token. The same page notes that the metric applies to autoregressive (causal) language models and is not well defined for masked models such as Bidirectional Encoder Representations from Transformers (BERT). You now have the measure; next comes what changes it.

Step 2: The tokenizer and context window change the number

Perplexity is computed per token, so anything that changes how text is cut into tokens changes the score. A tokenizer is the component that splits text into the tokens a model reads. A tokenizer that makes long tokens produces fewer, harder-to-guess tokens, so its perplexity per token comes out higher even when the model is no worse.

The context window matters too. The Hugging Face page notes that the largest generative pre-trained transformer (GPT)-2 model has a fixed length of 1024 tokens, so a longer text has to be cut into segments, and how you cut it affects the result. The page also says the tokenization procedure has a direct impact on a model's perplexity, which should be considered when comparing models. Segmentation matters separately: each segment sees a different amount of preceding text, so cutting the same text differently gives a different score.

That is why models are compared only on the same text, tokenization and segmentation. To compare across tokenizers, convert to a per-byte measure called bits per byte. A nat is the unit of surprise measured with the natural logarithm, and dividing by ln 2 converts nats to bits, so bits per byte is the total surprise in nats divided by ln 2 times the number of bytes; the worked example below does the arithmetic.

Text description of the diagram

• The same text gets perplexity 7.3891 under tokenizer A and 2.7183 under tokenizer B.
• The model's total surprise is identical in both cases.

Step 3: Task benchmarks grade answers, not probabilities

A task benchmark asks questions and scores the answers. Massive Multitask Language Understanding (MMLU), introduced by Hendrycks et al. (2020), covers 57 tasks including elementary mathematics, US history, computer science and law. Its questions are multiple choice with four options, so random chance is 25%.¹ When it appeared, the very largest GPT-3 model improved over random chance by almost 20 percentage points on average. The same paper reports that models had lopsided performance, with near-random accuracy on some subjects such as morality and law.

Grade School Math 8K (GSM8K), from Cobbe et al. (2021), is a dataset of 8.5K grade-school math word problems built to diagnose multi-step reasoning failures. The paper also proposes training verifiers to judge whether a model's completion is correct. Input: a question. Output: a graded answer, so the headline number is accuracy. An aggregate score can hide which subjects a model fails, so look at the per-task breakdown.

Step 4: Code benchmarks run unit tests and report pass@k

HumanEval, introduced with Codex by Chen et al. (2021), is a set of hand-written programming problems. A generated solution is judged by running unit tests, not by comparing text with a reference; the paper gives evidence that the bilingual evaluation understudy (BLEU) score may not be a reliable indicator of functional correctness.

Because a model can write many different solutions, the paper reports pass@k. It generates n samples per task (the paper uses n = 200 and k at most 100), counts the number c that pass the unit tests, and applies an unbiased estimator: one minus the number of ways to choose k failing samples from the n minus c failures, divided by the number of ways to choose any k samples from n. Input: n samples and c passes. Output: the chance that k tries include a pass. Estimating this from a larger sample is steadier than running k tries once.

Step 5: Contamination makes a benchmark score unreliable

Benchmark contamination is the presence of test items in training data. It happens easily because models train on large crawls of the web and benchmarks are published on the web. A contaminated score measures memory, so a model that looks strong on a public benchmark can be weaker on fresh questions of the same kind.

Three checks are common, and each has gaps. An n-gram overlap check searches the training text for long word sequences copied from the test set; Brown et al. (2020) ran a 13-gram overlap analysis of their benchmarks against GPT-3's training data. It misses paraphrases. A canary is a unique marker string placed in a benchmark's files, so that a model able to reproduce it reveals that the files were in training; the Beyond the Imitation Game benchmark paper (Srivastava et al., 2022) puts a canary string in its task files.

A perturbation test rewrites questions, for example by changing numbers or option order, and a large score drop suggests memorisation. Mirzadeh et al. (2024) report that model accuracy declines when the numbers in grade-school math questions are changed, and Zhang et al. (2024) report accuracy drops for several model families on a freshly written set of such problems. Prevention means keeping test sets private or refreshed and reporting results on held-out items. If an agent browses the web during evaluation it may read the answers directly; see AI agent security for the wider risk of untrusted content.

1 · Perplexity2 · Setup effects3 · Task benchmarks4 · Code tests5 · ContaminationReplay

Step-by-step description

1. Average the surprise per token, then exponentiate.
2. Tokenizer and context window change the number.
3. MMLU and GSM8K grade answers.
4. HumanEval runs unit tests and reports pass@k.
5. Leaked test items inflate scores.

• Perplexity is the exponential of the average negative log-likelihood per token.
• The tokenizer and context window change perplexity.
• Benchmark contamination means test items leaked into training data.

Worked example

Take a text of four tokens to which a model gave the probabilities 0.5, 0.25, 0.125 and 0.5 (illustrative). The negative log-likelihoods are 0.6931, 1.3863, 2.0794 and 0.6931, which sum to 4.852 and average 1.213. The exponential of 1.213 is 3.3636, so the perplexity is 3.3636.¹ Two sanity checks follow from the formula: a model that gives every token of a 50,000-token vocabulary (illustrative) the same probability has perplexity 50,000, and a model that is always certain and right has perplexity 1.¹

Now the tokenizer effect. Suppose a 24-byte text has a total negative log-likelihood of 12.0 nats (illustrative). Tokenizer A cuts it into 6 tokens and tokenizer B into 12 tokens. The perplexity is 7.3891 under tokenizer A and 2.7183 under tokenizer B, yet the model's total surprise is identical. Bits per byte is the total negative log-likelihood divided by (ln 2 times the number of bytes): 12.0 ÷ (0.6931 × 24) = 0.7213, in both cases.¹

Last, pass@k with n = 200 samples of which c = 20 pass (illustrative). Then pass@1 is 0.1 and pass@10 is 0.6602. The naive formula one minus 0.9 to the power 10 gives 0.6513, close but not the estimator the paper uses.¹

Trade-offs and edge cases

Perplexity is cheap and needs no labelled answers, but it scores only the probability of real text. It does not measure factual accuracy, instruction following or safety, and a good perplexity does not show that a model's answers are correct or helpful. Task benchmarks measure behaviour that people care about, but they need careful setup: prompt wording, number of examples shown and answer extraction all move the score.

The three scores answer different questions. Perplexity scores token probabilities, MMLU-style tests score choice accuracy, and HumanEval-style tests score unit tests passed. No one number describes a model.

Text description of the diagram

• Perplexity scores token probabilities, MMLU-style tests score choice accuracy, and HumanEval-style tests score unit tests passed.
• No one number describes a model.

Edge cases: a benchmark score from a small test set is noisy. With 100 questions and an accuracy of 0.5 (illustrative), the standard error is √(p(1−p)/n) = √(0.25/100) = 0.05, so a two-point gap between models means little.¹ A leaderboard can also be inflated when test items leak into training, when a model is tuned toward a public test, or when only the best run is reported; a strong answer says it checks for these before trusting a rank. Public benchmarks are a starting filter, so also evaluate on your own task with golden sets and regression tests, and use a judge model with care, as the LLM-as-a-judge page explains.

The state of play moved after these tests were written. As of October 2026, Wang et al. (2024) report that performance on MMLU has plateaued and introduce MMLU-Pro, a harder test that they say reduces prompt sensitivity, and Jain et al. (2024) built LiveCodeBench, a code benchmark that keeps collecting new problems to limit contamination. A strong answer names MMLU and HumanEval as the classics and adds that newer tests exist for these reasons.

How to answer it in an interview

The spoken structure: define perplexity, state its condition for comparison, name its blind spot, then give the task benchmarks and the contamination check.

Common mistakes

• "Lower perplexity always means a better model." (wrong) The number depends on tokenizer, context window and text. (right) "Perplexity is comparable only under an identical setup, and it does not measure factuality."
• "Perplexity works for any language model." (wrong) It is not well defined for masked models such as BERT. (right) "It fits autoregressive models."
• "A high MMLU score means expert knowledge." (wrong) The paper reports lopsided performance across subjects. (right) "I read the per-subject breakdown."
• "HumanEval matches the reference solution." (wrong) It runs unit tests. (right) "It checks functional correctness with tests."
• "pass@k is just one minus the miss rate to the power k." (wrong) The paper uses an unbiased estimator from n samples. (right) "I sample n, count c passes, and apply the estimator."

Follow-up questions

What are MMLU, HumanEval, and GSM8K, and how do you interpret LLM benchmark scores?

MMLU tests knowledge across 57 tasks, GSM8K tests grade-school math word problems, and HumanEval tests code with unit tests. Read a score together with the setup, the per-task breakdown and the contamination risk, not as one ranking number.

What happens to perplexity if you double the context window or switch tokenizer?

Switching to a tokenizer that cuts the same text into fewer tokens raises the per-token number even though the model is no worse, as the worked example shows, so compare bits per byte. A longer context gives each token more preceding text, which typically lowers perplexity, but the score is then no longer comparable with one measured at the old length.

What is perplexity, and what are its limits as a language-model metric?

Perplexity is the exponential of the average negative log-likelihood per token. Its limits are the dependence on tokenizer and context, the restriction to autoregressive models, and its silence about factuality and helpfulness.

How do you detect and prevent benchmark contamination (n-gram overlap, canaries, perturbation tests), and why are public LLM leaderboards often inflated?

Detect it with an overlap search of the training text, a canary check and rewritten test items, and combine them because no single check is conclusive; prevent it with private or refreshed test sets and held-out evaluation. Leaderboards inflate when test items leak, when models are tuned toward public tests, or when only the best run is shown.

Why can't you just rely on MMLU and public benchmarks?

MMLU and other public benchmarks are fixed, public and general, so they may leak and may not match your task. Add an evaluation built from your own data and track quality over time; the LLM evaluation guide covers the full picture.

Sources

1. Perplexity of fixed-length models — Hugging Face Transformers documentation (undated page; accessed 2026-10-26), 26 Oct 2026
2. Measuring Massive Multitask Language Understanding (arXiv 2009.03300) — arXiv (Hendrycks et al.), 7 Sep 2020
3. Evaluating Large Language Models Trained on Code (arXiv 2107.03374) — arXiv (Chen et al.), 7 Jul 2021
4. Training Verifiers to Solve Math Word Problems (arXiv 2110.14168) — arXiv (Cobbe et al.), 27 Oct 2021
5. Language Models are Few-Shot Learners (arXiv 2005.14165) — arXiv (Brown et al.), 28 May 2020
6. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (arXiv 2206.04615) — arXiv (Srivastava et al., BIG-bench), 9 Jun 2022
7. A Careful Examination of Large Language Model Performance on Grade School Arithmetic (arXiv 2405.00332) — arXiv (Zhang et al., GSM1k), 2 May 2024
8. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (arXiv 2410.05229) — arXiv (Mirzadeh et al.), 7 Oct 2024
9. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (arXiv 2406.01574) — arXiv (Wang et al.), 3 Jun 2024
10. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (arXiv 2403.07974) — arXiv (Jain et al.), 12 Mar 2024