What data strategies facilitate model learning?

In this article

1. Before you start
2. Why interviewers ask this
3. Data strategies for model learning from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the how large language models are trained learning path and assumes the basics of the transformer architecture.

Why interviewers ask this

Answering "collect more data" says almost nothing, because the next question is which data, how much, in what form and how you would know it helped. Questions of this kind are reported by candidates in machine learning and large language model (LLM) interviews, and they separate people who treat data as a pile from people who treat it as a budget with failure modes.

A weak answer lists words: more data, clean data, augmentation. A strong answer splits the problem by stage, pretraining, supervised fine-tuning and preference tuning, and names one concrete risk for each. A Staff-level answer ties every strategy to something you can measure: a duplicate rate, a repeat count per source, a gap between validation and test scores. It also says what you would verify before trusting a gain.

Data strategies for model learning from first principles

Data strategy

A data strategy is a set of choices about how much data to use, which data to keep, how to mix and order it, and how to hold some of it back for testing, made so that a model learns the target skill within a fixed compute and labeling budget.

Deduplication

Deduplication is a cleaning step that finds identical or near-identical documents or passages in a training set and keeps one copy, so the model does not spend capacity memorizing repeats and test text is less likely to appear in training.

Data mixture

A data mixture is the set of proportions in which different sources, such as web text, code and books, are sampled during training, so the share of each source in what the model sees is a deliberate choice rather than whatever the raw collection contains.

In plain words

Think of studying for an exam with a stack of past papers. More papers help, but only if they cover the right topics. Ten photocopies of one paper teach little. And you must not practice on the exact paper that will be the real test.

The analogy breaks in two places. A model can memorize text word for word, which a student rarely does. And a model learns through millions of small weight updates, so the share of each source it sees changes what it becomes.

In one precise sentence, a data strategy decides how many tokens, which tokens, in what proportions and order, and which examples stay unseen, so that measured performance reflects real learning.

1 · Scale tokens2 · Clean the data3 · Choose the mix4 · Curate tuning data5 · Split and protectReplay

Step-by-step description

1. Grow training tokens with model size.
2. Remove duplicates and junk.
3. Set source shares and order.
4. Small, diverse, checked examples.
5. Keep the test set untouched.

• Parameters and training tokens should be scaled in equal proportion.
• Deduplication and filtering clean the data before training.
• Splits keep test examples out of training.

Step 1: Scale tokens with parameters

Parameters and training tokens should be scaled in equal proportion for a fixed compute budget. Hoffmann et al. (2022) trained over 400 language models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens, and found that current large models were significantly undertrained because earlier work scaled parameters while keeping the data roughly constant.

Their model Chinchilla used the same compute budget as the 280-billion-parameter Gopher, but 70 billion parameters and four times more data, and it outperformed Gopher and several other larger models on a wide range of downstream tasks. Compute grows roughly with parameters times tokens, so one budget buys either a big model on little data or a smaller model on more data.¹

Gopher's 300 billion training tokens against Chinchilla's 1.4 trillion is about 4.7 times, which the abstract rounds to four.¹ The compute-optimal point that Hoffmann et al. fit works out to about 20 tokens per parameter: 1.4 trillion tokens divided by 70 billion parameters is 20.¹ Later models are often trained well past that point, because a smaller model is cheaper to run once trained; the Llama 3 model card lists over 15 trillion tokens for the 8 billion parameter model, about 1,875 tokens per parameter.¹ You now know that volume must match model size. The next steps ask what is inside that volume.

Step 2: Clean the data: remove duplicates and junk

Deduplication and filtering clean the data before training. Lee et al. (2021) report that deduplicating training data makes language models better, and that models trained on deduplicated data emit memorized text ten times less frequently. Repeats make the model spend capacity on text it has already seen, and they raise the chance that test text sits inside training data.

Take a toy corpus of 12 documents (illustrative). After lowercasing and collapsing spaces, only 6 are distinct, so exact deduplication removes 6 documents, or 50%.¹ Real systems also catch near-duplicates, where a few words differ, but the idea is the same: keep one copy.

Filtering drops documents that fail quality rules, such as wrong language, boilerplate, very short text or a low score from a quality classifier. Each rule is a judgment call: a filter that is too strict removes rare but valuable text. Input: raw documents. Output: a smaller, cleaner set. Always measure how much each rule removes.

Step 3: Choose the mix and the order

A data mixture sets how much of each source the model sees. Suppose a pool of 800 million web, 150 million code and 50 million book tokens (illustrative) and a budget of 1,000 million tokens. A target mix of 60% web, 25% code and 15% books needs 600, 250 and 150 million tokens, so web is used for 0.75 of a pass, code for about 1.67 passes and books for 3.0 passes.¹ The books source is repeated three times to reach its target share, while the web source is used for less than one pass.

Text description of the diagram

• The books source is repeated three times to reach its target share.
• The web source is used for less than one pass.

Repeating data costs little at first and then stops helping. Muennighoff et al. (2023) report that training with up to 4 epochs of repeated data yields negligible changes to loss compared with unique data, and that with more repetition the value of added compute eventually decays to zero. The books source at 3 passes sits inside that range. Raising books to a 30% share would need 300 million tokens from a pool of 50 million, which is 6 passes, and that would not.¹ Order matters too. Presenting easier or cleaner material first is called curriculum learning, introduced by Bengio et al. (2009), and whether a given schedule helps depends on the task, so treat order as something to test.

Step 4: Curate fine-tuning and preference data

For fine-tuning, quality and diversity beat volume. Supervised fine-tuning (SFT) is training on pairs of a prompt and an ideal response. Zhou et al. (2023) argue in "Less Is More for Alignment" (LIMA) that a small, carefully curated set of about 1,000 examples goes a long way toward a strong assistant.

Gunasekar et al. (2023) make a related point for pretraining: a small model trained on textbook-quality and synthetic data can do well on code. Synthetic data, written by another model, is cheap and controllable, but it copies the generator's errors, and Shumailov et al. (2024) report that models trained on recursively generated data lose the tails of the original distribution. Check a sample by hand.

Preference data is a different shape: people rank several answers to the same prompt, and the rankings are turned into comparisons that train a reward model, as in Ouyang et al. (2022). The explainer on reinforcement learning from human feedback (RLHF) and the comparison with direct preference optimization (DPO) show how that data is used. Preference tuning can cost general ability; the alignment tax page covers that. Classic machine learning tools such as data augmentation (new examples made by transforming existing ones) and class rebalancing (resampling so rare classes are not drowned out) mostly apply to labeled data like this, not to raw pretraining text.

Step 5: Split the data and guard the test set

Splits keep test examples out of training. A training set fits the model, a validation set guides choices such as stopping, and a test set is touched once at the end. When test text leaks into training, scores rise without any real learning; that is called data leakage.

Order is the trap. In the toy corpus (illustrative), splitting the 12 rows as 8 train, 2 validation and 2 test leaves 2 of the 4 evaluation rows with an identical copy in training. Deduplicating first and then splitting the 6 distinct documents as 4, 1 and 1 leaves 0 of 2 leaked.¹ Split by the unit that must stay apart, such as document, source, user or time, after deduplication.

The Staff-level answer

What makes it Staff-level: each strategy is tied to a failure you can measure, and the answer ends with how you would verify the gain.

The rubric

Three variants of this question

How would you fix a model that memorizes its training data?

Deduplicate the training set first, since repeated text is memorized most, then check how often any source repeats in the mix. Lee et al. (2021) report that deduplication reduces memorization.

Does more data always make a model better?

No. Duplicates, junk and leaked test text add volume without learning, and a source repeated many times stops adding information. At fixed compute, more data also means a smaller model, as the Chinchilla comparison shows.

How do you know the test set is not contaminated?

Run an n-gram overlap check between the training data and the test set, split by document or source rather than by row, and keep test data out of every tuning decision. Lee et al. (2021) report that such overlap "affects over 4% of the validation set of standard datasets". A gap between validation and test scores is only a weak signal on its own.

Common mistakes

• "A bigger model is always better." (wrong) At fixed compute a smaller model on more data can win. (right) "I size the model and the token count together."
• "More data fixes everything." (wrong) Repeats and noise add cost, not learning. (right) "I deduplicate and filter first, then add data."
• "I split first, then clean." (wrong) Duplicates then land on both sides of the split. (right) "I deduplicate, then split by document."
• "Fine-tuning needs a huge dataset." (wrong) Curation can matter more than volume. (right) "I start with a small, diverse, hand-checked set and grow it only if evaluation says so."

Follow-up questions

How much data does a model of a given size need?

About 20 tokens per parameter at the compute-optimal point that Hoffmann et al. (2022) fit, which is 1.4 trillion tokens for a 70 billion parameter model. That ratio is the fitted law's output, not a number to recite, and recent models are often trained well past it because a smaller model is cheaper to run. For small models, see small language models.

What changes when the data is for alignment instead of pretraining?

Alignment data comes in two shapes: prompt and ideal-response pairs for supervised fine-tuning, and comparisons of model answers for preference tuning. Zhou et al. (2023) used fine-tuning on only 1,000 carefully curated prompts and responses for a 65B model and report strong instruction following. Preference comparisons train a reward model, as in Ouyang et al. (2022); the RLHF page shows how they are used.

How does LoRA change the data you need?

Low-rank adaptation (LoRA) changes how many weights are trained, not what the data must be, so data quality and clean splits matter just as much.

Why is synthetic data risky?

A model that writes the data passes its mistakes and blind spots to the model trained on it. Sample and check it by hand, and keep real data in the test set.

Sources

1. Training Compute-Optimal Large Language Models (arXiv 2203.15556) — arXiv (Hoffmann et al.), 29 Mar 2022
2. Deduplicating Training Data Makes Language Models Better (arXiv 2107.06499) — arXiv (Lee et al.), 14 Jul 2021
3. LIMA: Less Is More for Alignment (arXiv 2305.11206) — arXiv (Zhou et al.), 18 May 2023
4. Textbooks Are All You Need (arXiv 2306.11644) — arXiv (Gunasekar et al.), 20 Jun 2023
5. Scaling Data-Constrained Language Models (arXiv 2305.16264) — arXiv (Muennighoff et al.), 25 May 2023
6. Training language models to follow instructions with human feedback (arXiv 2203.02155) — arXiv (Ouyang et al.), 4 Mar 2022
7. Llama 3 model card — Meta (GitHub), 18 Apr 2024
8. Curriculum learning — ICML 2009 (Bengio et al.), 14 Jun 2009
9. AI models collapse when trained on recursively generated data — Nature (Shumailov et al.), 24 Jul 2024