LLM Evaluation

Why is annotation hard?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

Why is annotation hard? Because a label is a judgment, not a lookup: trained people disagree, raw agreement flatters the result because chance also produces matches, and the errors that remain end up in the test sets that rank models. The one number most candidates quote, percent agreement, can look excellent while the chance-corrected score says almost nothing.

In this article

Before you start
Why interviewers ask this
Annotation from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is one step of the large language model (LLM) evaluation learning path.

Why interviewers ask this

A model is only as good as the labels it is trained and scored on, so a candidate who treats labels as ground truth reports numbers that cannot be trusted. Candidates report questions like this in interviews at AI labs and large tech companies, and it separates people who have only used datasets from people who have built them.

A weak answer says humans are slow and expensive. A strong answer says people disagree for real reasons, names a measure that corrects for chance, and explains what bad labels do to an evaluation. A Staff-level answer turns that into decisions: how to write guidelines, how many annotators to use, how to treat disagreement, and where to spend a limited review budget. The same reasoning applies when a model acts as the judge, which the page on LLM-as-a-judge covers.

Annotation from first principles

Annotation: Annotation is the work of attaching labels, such as a category, a score or a preference, to examples of data, done by people who follow written instructions, so that a model can be trained or tested against those labels. The people who do it are called annotators or labelers.
Inter-annotator agreement: Inter-annotator agreement is a measure of how often two or more annotators give the same label to the same item, reported either as a raw percentage of matches or as a chance-corrected coefficient, and used as a check that the instructions and the task are clear enough to produce consistent labels.
Cohen's kappa: Cohen's kappa is a coefficient for two annotators that subtracts the agreement expected by chance from the observed agreement and divides by the most that could be gained over chance, so it is one for perfect agreement and zero when annotators agree no more than chance.

In plain words

Picture two film critics. Each sorts a pile of films into "good" and "bad" using the same sheet of rules. They will not match everywhere. The sheet does not say what to do with a good film that has a bad ending. If nearly every film is "good", they also match by luck.

The analogy breaks in one place. You can rewrite the sheet and run the sort again, which is the main lever an annotation team has.

In one precise sentence: annotation is hard because a label is a measurement made by a person, so it carries disagreement, chance agreement and leftover errors that a single accuracy number never shows.

Five separate problems stack up, and a single accuracy number hides all of them.CC BY 4.0 — source: Why is annotation hard?

Step-by-step description

Raw agreement includes matches by luck; kappa removes them.
A dominant label gives high agreement but a low kappa.
Trained labelers still disagree on many items.
Telling them apart needs more than two annotators.
Label errors can change which model looks better.

Cohen's kappa subtracts the agreement expected by chance from raw agreement.
Skewed labels can give high raw agreement and a low kappa.
Trained labelers still disagree on many items.
Telling hard items from careless errors needs more than two annotators.
Label errors in test sets can change which model looks better.

Step 1: Raw agreement overstates quality, so use kappa

Raw percent agreement overstates label quality because two annotators match by luck some of the time. Cohen's kappa subtracts the agreement expected by chance from the observed agreement. James et al. (2026) describe it as one for perfect agreement, zero for no better than chance, and negative for systematic disagreement.

κ = \frac{P_{o} - P_{e}}{1 - P_{e}}

Here $P_{o}$ is the observed agreement and $P_{e}$ the chance agreement, computed from each annotator's own label mix. Take two annotators who label 100 items as toxic or not (illustrative). Both say toxic on 1 item and not toxic on 91, and they split on 8, four each way. Observed agreement is 0.92. Each annotator calls 5 items toxic, so chance agreement is 0.905 and kappa is only 0.16.¹ Two coin-flippers who each say "toxic" 5% of the time match 90.5% of the time with no skill at all.¹

Step 2: Skewed labels: when low kappa is right and when it misleads

Skewed labels can give high raw agreement and a low kappa, and that low kappa is sometimes right and sometimes misleading. The same 0.92 raw agreement with balanced labels, half toxic and half not, gives a chance agreement of 0.5 and a kappa of 0.84.¹ Raw agreement is 0.92 in both cases. Kappa is 0.84 with balanced labels and 0.16 with skewed labels.

In the skewed example of Step 1 the low kappa is right. Of the 9 items either annotator called toxic, they agree on only 1. Positive-class agreement, twice the items both called toxic divided by all the toxic calls the two annotators made, is 2 × 1 / (2 × 1 + 4 + 4) = 0.20, against 0.92 in the balanced case.¹ The kappa paradox is the other case: agreement on the rare class is genuinely good, yet prevalence alone pulls kappa down. Positive-class agreement tells the two apart: low means real disagreement, as here; high beside a low kappa means the paradox.

James et al. (2026) describe Gwet's alternative coefficients as chance-corrected measures proposed for high-agreement or highly skewed cases, the "kappa paradox". An applied annotation-validity study reports over 90% agreement with a kappa it calls paradoxically moderate.

At the same raw agreement of 0.92, skewed labels give a kappa of 0.16 and balanced labels give 0.84.
Case (illustrative)	Raw agreement	Chance agreement	Kappa	Band (as used by the applied study)
5% toxic, skewed	0.92	0.905	0.16	none to slight
50% toxic, balanced	0.92	0.5	0.84	almost perfect

The bands in the last column are the rule of thumb the applied study uses, which lists 0.81 to 1.00 as almost perfect and 0.01 to 0.20 as none to slight. Artstein and Poesio (2008) add that weighted, alpha-like coefficients may suit many corpus tasks but make the value harder to interpret.

The same raw agreement can mean excellent or almost no agreement once chance is removed.CC BY 4.0 — source: Why is annotation hard?

Text description of the diagram

Raw agreement is 0.92 in both cases.
Kappa is 0.84 with balanced labels and 0.16 with skewed labels.

Step 3: Trained labelers still disagree

Trained labelers still disagree on many items, and no hiring fix removes that. In the InstructGPT study, Ouyang et al. (2022) report that "training labelers agree with each-other 72.6 ± 1.5% of the time, while for held-out labelers this number is 77.3 ± 1.3%." The same paper gives researcher-to-researcher agreement of 73 ± 4% in earlier summarization work. These are raw agreement rates, not kappa, so they cannot be compared with the kappa figures in Step 2.

The authors also write the labeling instructions themselves and answer edge-case questions in a shared chat room, and they note that some tasks rest on value judgments. Disagreement is therefore partly legitimate: the task has no single right answer, and the instructions decide what "correct" means.

Step 4: One score cannot separate ambiguity from error

An agreement score cannot say why annotators disagree. Artstein and Poesio (2008) note that telling hard items from careless errors needs more than two annotators, and that some ambiguous items are found only through systematic disagreement between annotators.

Instructions add a second source of bias. Parmar et al. (2023) study annotators who write examples for a dataset, not labelers who assign categories, and show that they pick up on patterns in the crowdsourcing instructions, write many similar examples, and so over-represent them in the collected data, which they study in 14 recent language-understanding benchmarks. A practical team reads the items where annotators disagree before it blames the annotators, because the fix is often a clearer instruction or a new label option.

Step 5: Label errors reach the test sets

Label errors in test sets can change which model looks better. Northcutt et al. (2021) identify label errors in the test sets of 10 widely used vision, language and audio datasets, estimating an average of at least 3.3% errors and at least 6% of the ImageNet validation set. On an illustrative test set of 1,000 items, 3.3% is at least 33 wrong labels.¹

Finding them takes people too: algorithms flagged candidate errors, and crowdsourced review confirmed only 51% as truly mislabeled. On ImageNet with corrected labels, ResNet-18 beats ResNet-50 once the prevalence of originally mislabeled test examples rises by just 6%. The authors' practical advice is to give test labels priority when the labeling budget is limited.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.