What are LLM scaling laws, and what does Chinchilla say?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM scaling laws from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

It builds on the transformer architecture and assumes the basics from how neural networks work.

Why interviewers ask this

Scaling laws decide how labs spend very large training budgets, so a candidate who can explain them shows they understand why models are the size they are. The question is reported by candidates in several forms, including one that bundles it with transformer internals and reinforcement learning basics, in interviews at AI labs and large tech companies.

A weak answer says "bigger models are better." A strong answer says what is measured (loss), what it is plotted against (parameters, tokens, compute), and that the useful output is a rule for splitting a fixed budget. A Staff-level answer adds the caveats: the fits hold over a tested range, "compute-optimal" counts only training cost, and the rule of thumb of about twenty tokens per parameter is a summary, not a law of nature. The LLM interview questions page shows where this question sits among its neighbors.

LLM scaling laws from first principles

Scaling law

A scaling law is an empirical formula, fitted to many training runs, that predicts a model's loss from a quantity chosen before training, such as parameters, tokens or compute, so that results at small scale can be extended to larger scale within the range that was tested.

Power law

A power law is a relationship in which one quantity equals a constant times another quantity raised to a fixed power, so on a log-log chart it is a straight line and every tenfold change in the input multiplies the output by the same factor.

Compute-optimal training

Compute-optimal training is the choice of model size and number of training tokens that reaches the lowest loss for a fixed training compute budget under the fitted law, so no other split of the same budget between parameters and tokens does better.

In plain words

Think of baking bread for a fixed amount of oven time. You can bake one huge loaf or many small ones, and each choice uses the same oven hours. A scaling law is the recipe book. From past bakes, it says which mix of loaf size and loaf count gives the best bread. The analogy breaks in one place. The recipe book is fitted from earlier bakes. It is only trustworthy for oven sizes close to those it saw.

In one precise statement, a scaling law says that loss falls as a power law as parameters, tokens or compute grow. That lets you choose the best split of a fixed compute budget before you train.

Step 1: What are scaling laws in LLM training? Loss falls as a power law

Kaplan et al. (2020) report that language model loss scales as a power law with model size, dataset size and training compute, with some trends spanning more than seven orders of magnitude. Their models range from 768 to 1.5 billion non-embedding parameters (§2). They also report that architectural details such as width or depth have minimal effects within a wide range.

Input: a bigger model, more tokens or more compute. Output: a lower loss, along a smooth curve. For intuition, suppose the loss above its floor follows $L\propto N^{-\alpha }$. With an exponent of 0.1 (illustrative), each tenfold increase in parameters multiplies that loss by 0.7943, and a hundredfold increase by 0.631.¹ Kaplan et al. (§6) fit exponents of about 0.076 for parameters and 0.095 for data, which give factors of 0.8395 and 0.8035 per tenfold.¹ You now know the basic object: a straight line on a log-log chart that can be extended.

Step 2: A fixed compute budget must be split between model size and tokens

Training compute is the total number of floating-point operations (FLOPs), the additions and multiplications, spent on a training run. Kaplan et al. (2020), §2.1 derive the forward pass as about 2N operations per token and training, with the backward pass, as about 6N per token, so $C\approx 6ND$, where $N$ is the number of parameters and $D$ is the number of training tokens.

The consequence is a trade. With $C$ fixed, doubling $N$ halves $D$. A bigger model sees fewer tokens, a smaller model sees more, and both cost the same. Input: a budget $C$. Output: a choice of $(N,D)$ on the line where $C=6ND$ holds. The scaling law's job is to say which point on that line gives the lowest loss, which is the compute-optimal point. The budget is the same, so the question is only how to divide it.

Step 3: Chinchilla changed the split: scale model size and tokens equally

Kaplan et al. conclude that optimally compute-efficient training involves very large models on a relatively modest amount of data, stopping significantly before convergence. Hoffmann et al. (2022) trained over 400 language models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. They find that current large models are significantly undertrained and that model size and training tokens should be scaled equally: every doubling of model size should come with a doubling of tokens.

They tested this by training Chinchilla, which uses the same compute budget as Gopher but has 70B parameters and four times more data. Hoffmann et al. (2022) report that Chinchilla outperformed Gopher (280B) and other larger models on a large range of downstream tasks, and reached 67.5% average accuracy on the Massive Multitask Language Understanding (MMLU) benchmark, more than 7 percentage points above Gopher. Their Table 1 lists Chinchilla at 70B parameters and 1.4 trillion tokens, which is 20 tokens per parameter, against Gopher's 280B parameters and 300B tokens.¹

The gap with Kaplan et al. has a cause: Hoffmann et al. (§2) note that Kaplan et al. used one fixed token count and learning-rate schedule for all models.

Step 4: Real limits come from inference cost and the tested range

A fit is only as good as the runs behind it. The Chinchilla fits cover models up to over 16 billion parameters, so a rule applied to a far larger model is an extrapolation. The papers also optimize training compute only. The Chinchilla abstract notes that a smaller model uses substantially less compute for fine-tuning and inference, which is a reminder that the cost of serving a model is a separate part of the bill.

Input: a fitted law and a target budget. Output: a prediction with an error bar nobody has measured beyond the tested range. You now have the whole picture: a power-law loss curve, a budget split, equal scaling, and limits.

1 · Power law2 · Budget split3 · Equal scaling4 · Real limitsReplay

Step-by-step description

1. Loss falls smoothly as scale grows.
2. A fixed budget trades size against tokens.
3. Chinchilla grows size and tokens together.
4. Inference cost and tested range apply.

• Loss falls as a power law in size, data and compute.
• A fixed budget must be split between model size and tokens.
• Chinchilla scales model size and tokens equally.
• Real limits come from inference cost and the tested range.

Worked example

Take four training budgets, in FLOPs, and apply two relations: $C\approx 6ND$ (Kaplan et al., §2.1) and the 20 tokens per parameter of Chinchilla (Hoffmann et al., Table 1), so $D=20N$. Then $N=\sqrt{C/120}$ and $D=20N$. At a budget of 1e23 FLOPs, $N$ is 28.87 billion parameters and $D$ is 577.4 billion tokens; check: $C=6\times N\times D$ returns 1e23.¹ A tenfold bigger budget multiplies both $N$ and $D$ by $\sqrt{10}$, about 3.162, because each grows with the square root of $C$.¹

Text description of the diagram

• A tenfold bigger budget multiplies both parameters and tokens by about 3.162.
• At a budget of 1e23 FLOPs the model has 28.87 billion parameters and 577.4 billion tokens.

The strongest check uses Chinchilla's own numbers. With 70 billion parameters and 1.4 trillion tokens, six times the product is 5.88e23 FLOPs, and the square root of that budget divided by 120 returns 70.0 billion parameters, so the two relations reproduce the published model.¹ A second check compares sizes. Chinchilla has 70B parameters against Gopher's 280B, a quarter of the size, and four times the data. Since $C\approx 6ND$ grows with the product of the two, the factors cancel (0.25 times 4 is 1), which matches the paper's statement that the compute budget is the same.¹

Trade-offs and edge cases

The two papers differ in how they split a fixed budget, as the figure shows. Kaplan et al. fitted the optimal model size as growing with compute to the power 0.73 and tokens to the power 0.27 (Kaplan et al., §6), while Hoffmann et al. fitted both exponents at about 0.50 (approach 1; the other two approaches give 0.49 to 0.54) (Hoffmann et al., Table 2). Hoffmann et al. (§3.3) also fit a parametric form of the loss:

Here $E$ is the loss floor that no size or data removes, and the two other terms shrink as parameters and tokens grow. The fitted values (Hoffmann et al., §3.3) are E = 1.69, A = 406.4, B = 410.7, α = 0.34 and β = 0.28. This is the "loss above its floor" of Step 1.

Text description of the diagram

• Kaplan et al. advise very large models on relatively modest data.
• Hoffmann et al. say model size and tokens should be scaled equally.
• Hoffmann et al. trained over 400 models and then trained Chinchilla.

Hoffmann et al. (§1) say that recent large models followed Kaplan et al.'s recommendation, and they list GPT-3, a generative pre-trained transformer (GPT) model with 175 billion parameters trained on 300 billion tokens (Hoffmann et al., Table 1; model size in Brown et al., 2020). Edge cases follow from the definitions. First, compute-optimal means lowest loss for a training budget, not the best model to deploy: a model that will serve heavy traffic may be better smaller and trained on more tokens, because the serving cost is not in the fit. Second, the fits describe loss on a language-modeling objective; downstream skill and behavior after later training steps are separate measurements.

Third, Kaplan et al. find width and depth matter little within a wide range, which is a statement about the architectures they tried, not about every design. Fourth, deployed models are often trained far past the compute-optimal point: Meta's Llama 3 model card (April 2024) reports pretraining on over 15T tokens, which for the 8B model is about 1,875 tokens per parameter against Chinchilla's 20.¹ Fifth, efficiency work such as FlashAttention keeps the FLOPs roughly the same and raises hardware throughput, which lowers the time and money per FLOP and moves what a budget buys, not the shape of the split.

How to answer it in an interview

The spoken structure: define the law, say what it is used for, give the Chinchilla result, then name the caveat.

Common mistakes

• "Chinchilla proves exactly twenty to one." (wrong) The paper states equal scaling of size and tokens, and twenty is the ratio of Chinchilla's own run (Table 1), a point on a fitted trend. (right) "Equal scaling keeps the ratio roughly constant; Chinchilla sat at about twenty."
• "Scaling laws are laws of nature." (wrong) They are fits over the model sizes and data that were tried. (right) "They are empirical, and extrapolating past the tested range is a risk."
• "Compute-optimal means the best model to ship." (wrong) It means the lowest loss for a training budget and ignores serving cost. (right) "If inference demand is high, a smaller model trained longer can be the better choice."
• "A bigger model always wins at the same compute." (wrong) The Chinchilla result shows a smaller model with more data beating a larger one at the same budget. (right) "At fixed compute, size and data trade against each other."

Follow-up questions

If the budget grows 100 times, how much bigger should the model be?

Ten times bigger, and trained on ten times more tokens. Under equal scaling both grow with the square root of the budget, and the square root of 100 is 10.¹ The transformer guide covers the models being scaled.

How do you fit a scaling law from small runs before a big one?

Train many small models across sizes and token counts, as Hoffmann et al. did with over 400 runs, and fit the loss curve to them. Then predict a larger run and check it by training that run, which is what Chinchilla was for Hoffmann et al.

Do scaling laws hold for reasoning models?

Pretraining fits do not cover a second axis: compute spent at answer time. Snell et al. (2024) study scaling test-time compute and report that, on some problems, it can beat a much larger model, with the gain depending on how hard the prompt is. A good answer says that reasoning models need their own scaling measurements.

What happens to the ratio at much larger budgets?

Under equal scaling it stays roughly constant, because tokens and parameters both grow with the square root of the budget. Whether the fits hold far beyond the tested range is untested, so treat a very large budget as an extrapolation.

Why would a team train past the compute-optimal point?

Because the fit counts training cost only. A smaller model needs less compute to serve, so when it will answer many requests, spending extra training tokens on it can lower total cost.

Sources

1. Scaling Laws for Neural Language Models (arXiv 2001.08361) — arXiv (Kaplan et al.), 23 Jan 2020
2. Training Compute-Optimal Large Language Models (arXiv 2203.15556) — arXiv (Hoffmann et al.), 29 Mar 2022
3. Language Models are Few-Shot Learners (arXiv 2005.14165) — arXiv (Brown et al.), 28 May 2020
4. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (arXiv 2408.03314) — arXiv (Snell et al.), 6 Aug 2024
5. Llama 3 model card — Meta, 18 Apr 2024