In this article
Before you start
Long-context evaluation builds on the transformer architecture and how neural networks work.
Why interviewers ask this
A model card that advertises a long window says what the model accepts, not what it uses well, and a team that picks a model on that number can ship a product that quietly ignores half its input. Interviewers ask this question to see whether a candidate treats the advertised length as a claim to be tested.
A weak answer says "run the needle-in-a-haystack test and look at the heatmap." A strong answer explains what that test shows, why a green heatmap is not enough, and what to add: harder tasks, position sweeps and a test on the real workload. A Staff-level answer also covers noise and cost, so the result is a length you can defend with numbers rather than a color. The question is reported in interviews for large language model (LLM) roles; no company is named here because no public report is cited.
Evaluating a long-context model from first principles
Step-by-step description
- Grid of lengths and depths.
- Find the effective length.
- Report the weakest depth.
- Your documents decide.
- The needle grid plants a fact at set lengths and depths.
- A harder task at the same lengths gives the effective length.
- Position and needle style are varied, and the real workload is tested last.
- Needle-in-a-haystack test
- A needle-in-a-haystack test is a retrieval check that hides one short fact (the needle) at a chosen depth inside long filler text (the haystack) of a chosen length, asks the model a question only that fact answers, and records whether the answer is right for each length and depth.
In plain words
Think of testing how well a person can use a very long report. You could hide one sentence on some page and ask for it back. That shows they can find a sentence they were asked about. It does not show they can compare page 3 with page 90. The analogy breaks in one place: a model does not skim or tire. It reads every token through attention, yet its accuracy can still fall as the input grows.
In one precise sentence, evaluating a long-context model means measuring a task score as a function of input length (and the position of the relevant text), then reporting the length where the score is no longer acceptable. A pass rate is the share of runs of a test in which the model gives the right answer.
Step 1: Plant a needle at set lengths and depths
Build a grid. Choose several context lengths, several depths (where in the text the needle sits, from the start to the end), and run each cell several times. The input is the filler plus one needle and a question; the output is a pass or a fail per run.
The toy grid used on this page (illustrative) has five lengths from 8,192 to 131,072 tokens, five depths and five trials per cell: 25 cells and 125 runs.1 Averaged over depths, each length has 25 runs. The grid then becomes a heatmap with length on one axis and depth on the other.
You now know what the standard test measures: whether the model can find a clearly marked sentence anywhere in a long input. The next step shows why that is a low bar.
Step 2: Find the effective length with a harder task
A green needle grid does not mean the whole window is usable. The benchmark from Hsieh et al. (2024) adds harder tasks, including multi-hop tracing and aggregation. The authors report that near-perfect needle accuracy hides "large performance drops as the context length increases" on harder tasks, and that only half of the models claiming 32K tokens or more keep satisfactory performance at 32K.
Run a harder task at the same lengths and set a threshold. In the illustrative run, the rule is 85% of the harder task's score at the shortest length, which is 81.6%.1 The harder task passes 96%, 88%, 76%, 52% and 32% of runs at the five lengths, so the last length above the threshold is 16,384 tokens: that is the effective context length, against a tested maximum of 131,072.1
| Context length (tokens) | Needle test pass rate | Harder task pass rate |
|---|---|---|
| 8,192 | 100% | 96% |
| 16,384 | 100% | 88% |
| 32,768 | 100% | 76% |
| 65,536 | 100% | 52% |
| 131,072 | 96% | 32% |
Scores are noisy. With 25 runs per length, 19 passes (76%) has a 95% confidence interval of roughly 57% to 89%, so a single point near the threshold is weak evidence; add trials before you rely on it.1
Text description of the diagram
- The needle test pass rate stays between 96% and 100% at every length.
- The harder task falls from 96% at 8,192 tokens to 32% at 131,072 tokens.
Step 3: Vary position and needle style
The position effect is the finding that a model's accuracy depends on where the relevant text sits in the input. Liu et al. (2023) studied multi-document question answering and key-value retrieval and found that performance is often highest when the relevant information is at the beginning or end of the input and degrades significantly when it is in the middle, even for explicitly long-context models.
That is why the depth axis exists, and why one number averaged over depths can hide a weak middle. Report the worst depth as well as the mean.
The needle itself matters too. A strong suite does more than repeat a sentence the question quotes word for word. The NoLiMa benchmark (Modarressi et al., 2025) is built so that needles share little wording with the question, because literal matching otherwise gives the model a shortcut. Prior extensions of the needle test, which the NoLiMa abstract describes as increasing distractors and fact chaining, make the task harder but still allow literal-match shortcuts, so a low-overlap needle moves the test closer to what real documents demand.
Step 4: Test the real workload and score it reliably
Synthetic tasks are cheap to build, but they do not stand in for your product. Yen et al. (2024) introduce the benchmark HELMET (How to Evaluate Long-context Models Effectively and Thoroughly) and compare 59 long-context models on seven application-centric categories, and report that synthetic tasks like the needle test do not reliably predict downstream performance. They also report that the categories show distinct trends with low correlations with each other, and that most models reach perfect needle-test scores while open-source models lag closed ones on tasks needing full-context reasoning.
So finish with your own task at your own lengths: your documents, your questions, your answer format. The same authors add model-based evaluation for reliable metrics; string-overlap metrics can mark a correct but reworded answer as wrong. They also recommend retrieval-augmented generation (RAG) tasks for fast model development, since those are easy to run and predict other downstream performance better. Report one number per length, with trials and an interval.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How do you evaluate a long-context model?
I evaluate it in four parts. First, a needle-in-a-haystack grid: plant a fact at several depths and lengths and check retrieval, which only shows the model can find a clearly marked sentence. Second, harder tasks at the same lengths, such as low-overlap and multi-fact needles. I report the effective length: the longest length where the harder task still scores above a threshold I set in advance, not the advertised window. Third, a depth sweep, reporting the weakest depth, since models often miss the middle. Fourth, my real workload, scored with a metric I checked by hand. Scores are noisy, so I run several trials per cell and report intervals.
What makes it Staff-level: it names what the needle test cannot show, defines the output as a length with a threshold, and treats noise and the real workload as part of the method.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Describes the needle test only | Describes the needle test and its limits | Defines effective length with a stated threshold |
| Depth | Reads the heatmap by color | Adds harder tasks and position sweeps | Explains why each variant removes a shortcut |
| Trade-offs | Ignores cost and noise | Mentions trials per cell | Reports intervals and weighs run cost against coverage |
| Communication | Lists benchmark names | Explains steps in order | Ends with the length you would ship and why |
| Follow-up readiness | Stuck on "why not trust the model card?" | Cites the advertised-versus-usable gap | Proposes testing on the team's own documents |
Three variants of this question
How do you find a model's real context length?
Run a demanding task at increasing lengths, pick a threshold in advance, and report the longest length that stays above it. The advertised window is a claim; the effective length is a measurement.
Why does a model miss a fact in the middle of a long input?
Measured accuracy often depends on position: the Lost in the Middle study found it highest at the start and end of the input and lowest in the middle. Sweep the depth axis and report the weakest depth, not only the mean.
How would you evaluate long context for our product's documents?
Build test cases from your real documents and questions at the lengths you will serve, score them with a metric you checked against human judgment, and compare candidate models at each length. Use public benchmarks only to shortlist.
Common mistakes
- "The heatmap is all green, so the model handles the full window." (wrong) The needle test is the simplest retrieval check. (right) "It finds a marked sentence at every length; I still need harder tasks to find the usable length."
- "The model card says it supports a long window, so that is its context length." (wrong) A window size says what is accepted, and Hsieh et al. report that only half of the models they tested held up at 32K despite larger claims. (right) "The advertised window is the claim; I measure the effective length."
- "One run per cell is enough." (wrong) A single pass or fail is a coin you cannot repeat. (right) "I run several trials per cell and report an interval."
- "The needle benchmark ranks models for my use case." (wrong) Yen et al. report that synthetic tasks like the needle test do not reliably predict downstream performance. (right) "I use public benchmarks to shortlist and my own task to decide."
Follow-up questions
What does a needle-in-a-haystack heatmap actually show?
It shows, for each length and depth, how often the model retrieved one planted fact. It does not show reasoning over the whole input, multi-fact questions or performance on your documents.
Why not trust the context length on the model card?
The card states what the model accepts. Hsieh et al. report that only half of the models they tested kept satisfactory performance at 32K although all claimed 32K or more, so verify with your own threshold. The context window page explains what the window costs.
How do you score the answers?
Use exact match only when the answer has one canonical form. Otherwise use a model-based judge, as Yen et al. do for reliable metrics, and check a sample by hand.
Why not just measure perplexity on long documents?
Low perplexity on long text shows the model predicts the next token well, not that it uses distant tokens to solve a task. A model can score well on perplexity and still miss a fact planted far back, so task-based tests are needed to find the usable length.
How much does a long-context evaluation cost to run?
Cost grows with length, because every run reads the whole input. In the toy grid, one task for one model reads 6,348,800 input tokens (illustrative), and the longest length alone is 51.6% of them.1 The longest length is where cutting trials saves the most, but it widens the interval there. The reason long inputs are costly is covered in why attention is quadratic and how to make it cheaper, and more questions of this kind are in the LLM interview questions.
- A needle-in-a-haystack grid shows whether a model can find one planted fact, not whether it uses the whole window.
- Report an effective context length: the longest length where a demanding task stays above a stated threshold.
- Sweep position as well as length, and report the weakest depth.
- Finish on your own workload; synthetic scores do not reliably predict it.
- Run several trials per cell and report intervals, since single cells are noisy.
Sources
- RULER: What's the Real Context Size of Your Long-Context Language Models? (arXiv 2404.06654) — arXiv (Hsieh et al., NVIDIA), 9 Apr 2024
- Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — arXiv (Liu et al.), 6 Jul 2023
- HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (arXiv 2410.02694) — arXiv (Yen et al., Princeton University and Intel), 3 Oct 2024
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (arXiv 2502.05167, v1 7 Feb 2025, revised v3 9 Jul 2025) — arXiv (Modarressi et al.), 7 Feb 2025
