Classical ML

How do you get annotated training data?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 11 min read

How do you get annotated training data? You buy or build labels by one of four routes: in-house experts, crowd or vendor workers, programmatic labeling rules, or a model that pre-labels and picks what a human labels next. Annotated training data is a set of examples that each carry a label the model learns from. The error that costs the point is trusting raw percent agreement between annotators: 95% agreement can mean a kappa under 0.5.¹

In this article

Before you start
Why interviewers ask this
Annotated training data from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

Why interviewers ask this

A model learns only what its labels teach it, so interviewers use this question to see whether you treat labeling as an engineering problem with cost, quality and risk. Labeling is a real bottleneck: Ratner et al. (2017) write that "Labeling training data is increasingly the largest bottleneck in deploying machine learning systems."

The question is reported by candidates in machine learning interviews, usually as a short opener before loss functions or evaluation. A weak answer says "we hire people to label it." A strong answer names the routes and says how it would check label quality. A Staff-level answer ties the route to the task's ambiguity and budget, measures agreement beyond chance, and keeps the evaluation set clean of the shortcuts used to build the training set.

Annotated training data from first principles

Labeling function: A labeling function is a small piece of code that looks at one unlabeled example and returns a label or abstains, so a person can write a rule of thumb once and apply it to the whole dataset.

In plain words

Think of a teacher who needs an answer key for a thousand exam questions. She can write it herself. She can ask several students and take the vote. She can write a rule that grades the easy questions. Or she can let a draft grader point out the hardest ones. The analogy breaks in one place. A real answer key is correct by definition, while labels are only someone's best judgment, so they can be wrong and must be checked.

In one precise sentence: a labeling route produces the labels, and an agreement statistic checks them. They are then merged into one label per example. Rules, and a model that asks humans where it is unsure, stretch them further.

Step 1: Pick a route by the task, not by habit

Four routes produce labels: experts, crowd workers, labeling functions and model-assisted labeling. In-house experts give the best labels for ambiguous or specialist tasks and cost the most per label. A crowd or vendor workforce gives volume; one cloud provider calls its public workforce "large and economical but as with any set of diverse workers, can be prone to errors", so it needs a written guideline and several annotators per item. Labeling functions replace hand labels with rules, covered in the fourth step. In model-assisted labeling, a model, including a large language model, pre-labels and humans review; Label Studio documents this as a standard use case.

Check first for labels you already have, such as logged clicks, purchases or chargebacks; they are cheap but biased by what the product showed, delayed and noisy.

Input: unlabeled data and a budget. Output: a chosen mix of routes. Real projects combine them, for example experts for a small gold set and a crowd for the bulk.

Step 2: Measure label quality with kappa, not raw agreement

Inter-annotator agreement is how often independent annotators give the same label to the same item. Raw percent agreement misleads because annotators also agree by chance. Cohen's kappa corrects agreement for chance before labels are trusted:

κ = \frac{P_{o} - P_{e}}{1 - P_{e}}

Here $P_{o}$ is observed agreement and $P_{e}$ is the agreement expected from each annotator's label frequencies. Cohen's kappa compares two annotators; with three or more, use Fleiss' kappa or Krippendorff's alpha (Fleiss, 1971). Take two annotators on 100 items with a 50/50 label split who agree on 90 (illustrative): $P_{o}$ is 0.90, $P_{e}$ is 0.50 and κ is 0.80.¹ A rule of thumb after Landis and Koch (1977), reproduced in a 2026 paper, calls 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect.

Take 1,000 items, 95% in one class, annotators agree on 950 (illustrative). Raw agreement is 0.95 on skewed labels but kappa falls to 0.4737, because $P_{e}$ is 0.905.¹ The kappa paradox, described in an analysis of kappa under imbalanced ratings (2023), is that kappa can be low despite high raw agreement when ratings are imbalanced. Raw agreement is 0.90 on balanced labels with kappa 0.80, so report both.¹

Raw agreement rises on skewed labels while kappa falls, so report both.CC BY 4.0 — source: How do you get annotated training data?

Text description of the diagram

Raw agreement is 0.90 on balanced labels with kappa 0.80.
Raw agreement is 0.95 on skewed labels but kappa falls to 0.4737.

Step 3: Consolidate several annotations into one label

An annotation is the result of one worker's labeling task. Annotation consolidation is the step that combines the annotations of two or more workers into one label for each item. Majority vote is the simplest rule. The SageMaker documentation says its multi-class consolidation uses a variant of expectation maximization that estimates each worker's reliability and applies Bayesian inference to estimate the true class. The same page says more workers can raise label accuracy but also raise cost.

If each of three workers is independently right with probability 0.8 (illustrative, and independence is an assumption), the majority is right with probability 0.896; five workers give 0.9421.¹ On 10,000 items, three workers per item means 30,000 annotations and five means 50,000.¹ Amazon Web Services (AWS) says it does not plan to add new features to this service (as of October 2026), so treat it as a vendor example, not a recommendation.

Step 4: Write labeling functions instead of labeling by hand

Weak supervision makes the training set from rules. In Snorkel, users write labeling functions that "express arbitrary heuristics, which can have unknown accuracies and correlations". A model then estimates how accurate each function is from where they agree and disagree, with no ground truth, and outputs one probabilistic label per example. A cross-entropy loss can train a downstream model on such confidence-weighted labels.

A toy case with four support messages (illustrative) and three functions, one voting billing on the word refund, one billing on invoice and one account on password. The message about a refund for an invoice gets two billing votes, the password reset gets one account vote, "refund after password reset" splits one against one, and "hello" gets no vote.¹

So two of four messages get a clear label, one is a conflict and one is uncovered: the functions cover three of four messages but agree on only two.¹ In the Snorkel user study (Ratner et al., 2017), subject matter experts built models 2.8x faster and with 45.5% higher average predictive performance than with seven hours of hand labeling. Humans are still needed: experts write the functions.

Step 5: Send the hardest items to humans with active learning

Active learning sends the items the model is least sure about to humans. In the pool-based setting Settles (2010) describes, queries are chosen from a large pool of unlabeled examples. Uncertainty sampling is the query strategy that selects the example about which the model is least certain how to label. The loop: train on a small labeled set, score the pool, label the least certain items, add them and repeat.

In one toy text task, after 30 new labels uncertainty sampling reached 81% accuracy while a random baseline reached 73% (Settles, 2010). That is a single illustrative example, not a general rule. Label Studio wires the loop with a webhook, through which the machine learning (ML) backend can retrain the model on new annotations. Because the model chooses its own data, keep a random labeled slice for evaluation so you notice distribution shift.

Labels come from a chosen route, are checked for agreement, then stretched with rules and with questions to humans.CC BY 4.0 — source: How do you get annotated training data?

Step-by-step description

Choose experts, crowd, rules or model-assisted labeling.
Kappa corrects agreement for chance.
Several annotations become one label.
Labeling functions label the bulk.
Humans label the items the model doubts.

Four routes produce labels: experts, crowd workers, labeling functions and model-assisted labeling.
Cohen's kappa corrects agreement for chance before labels are trusted.
Active learning sends the items the model is least sure about to humans.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.