What are small language models and how are they built?

In this article

1. Before you start
2. Why interviewers ask this
3. Small language models from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Small language models are asked about because they force you to do the arithmetic that large-model talk hides. An interviewer wants to know whether you can turn a device limit into a parameter budget, explain why a small model is trained on far more data than a large one, and name the tools that shrink a model after training. A weak answer says "fewer parameters, so cheaper". A strong answer gives the memory formula, the data trade-off and the compression steps, and says what a small model cannot do.

The topic is reported in interviews at AI labs and large tech companies, mostly in design rounds on on-device assistants, cost-controlled serving and agent systems. The fine-tuning hub places it in the wider training path.

Small language models from first principles

Small language model

A small language model is a language model that can fit on a common consumer electronic device and run inference with latency low enough to serve one user; as of 2025, most models below ten billion parameters are treated as small.

Overtrained model

An overtrained model is a model trained on many more tokens than the compute-optimal ratio of about 20 tokens per parameter, a deliberate choice that spends extra training compute to get a smaller model that is cheaper to run.

In plain words

A small language model is like a pocket dictionary. An editor wrote it after reading a whole library. It is light enough to carry everywhere. It is not a library, so it covers less. The analogy breaks in one place. A dictionary only drops entries, while a small model keeps general skills and loses depth.

Stated precisely, the size limit is a hardware limit, not a fixed number. A 2025 position paper defines a small language model as one that "can fit onto a common consumer electronic device and perform inference with latency sufficiently low to be practical" for one user. Its authors treat most models below ten billion parameters as small (Belcak et al., 2025). A large language model (LLM) is simply one that is not small.

Step 1: Fix the memory budget and the device

The first step turns a device into a parameter count. The input is the memory you can spare; the output is the largest model that fits. The weights of a model take its parameter count times the bytes per parameter: 2 bytes in a 16-bit format such as bfloat16, 1 byte at 8 bits, half a byte at 4 bits. A 1.7-billion-parameter model is therefore 3.4 GB in bfloat16 and 0.85 GB at 4 bits.¹ These are weights only. The attention cache and the working memory of a run need more, as the KV cache page explains. What you now know: the device sets the parameter budget before any training.

Step 2: Curate the data

The second step is choosing what the model reads, and for small models quality matters more than volume. The input is a pool of text; the output is a smaller, cleaner training set. The phi-1 paper trained a 1.3-billion-parameter model on about 6 billion tokens of filtered "textbook quality" web data plus about 1 billion tokens of synthetic textbooks and exercises, and reports 50.6 percent pass@1 accuracy on the HumanEval coding benchmark (Gunasekar et al., 2023).

SmolLM2 pairs web text with specialised math, code and instruction data, and its authors built new datasets "at stages where we found existing datasets to be problematically small or low-quality" (Allal et al., 2025). What you now know: a small model has no spare capacity for noisy data.

Step 3: Train past the compute-optimal token count

The third step sets how many tokens to train on, and small models are deliberately overtrained. The input is the parameter count; the output is a token budget far above the compute-optimal rule. That rule comes from Hoffmann et al.: their 70-billion-parameter Chinchilla saw 1.4 trillion tokens, 20 per parameter, and they found that model size and training tokens "should be scaled equally" (Hoffmann et al., 2022). SmolLM2 1.7B saw 11 trillion tokens, about 320 times that ratio.¹

It pays because serving costs about 2N operations per generated token (Kaplan et al., 2020). Extrapolating Hoffmann et al.'s fitted loss formula, a compute-optimal model with SmolLM2's predicted loss has about 8 billion parameters, so each served token costs SmolLM2 about a fifth as much, and the extra training is repaid after roughly 7 trillion served tokens.¹ What you now know: small models trade a one-off training cost for a cheaper cost on every request.

Step 4: Inherit from a bigger model

The fourth step shortens the road by borrowing from a larger model. The input is a trained large model; the output is a small model that starts ahead. Knowledge distillation is the main tool: Hinton et al. showed how to distil the knowledge of an ensemble into a single model that is much easier to deploy (Hinton et al., 2015).

Meta describes making its Llama 3.2 1B and 3B models by "structured pruning in a single shot manner" from the 8-billion-parameter Llama 3.1, then adding "logits from the Llama 3.1 8B and 70B models" (the raw scores a model gives each possible next token) to pre-training to recover quality (Meta AI, September 2024). What you now know: a small model rarely learns alone.

Step 5: Post-train and compress

The fifth step makes the model useful and small enough to ship. The input is the pretrained small model; the output is a chat or task model in a compact number format. Post-training teaches instructions: Meta used several rounds of supervised fine-tuning, rejection sampling and direct preference optimization for the Llama 3.2 chat models, and released the weights in bfloat16 while exploring quantised variants (Meta AI, September 2024). Quantization stores weights in fewer bits, as the LLM quantization page explains; preference-tuning methods are compared in direct preference optimization (DPO) versus reinforcement learning from human feedback (RLHF). What you now know: the shipped file is smaller than the trained one.

1 · Budget2 · Data3 · Tokens4 · Inherit5 · CompressReplay

Step-by-step description

1. The device sets the parameter count.
2. Curate clean, useful text.
3. Train far past 20 tokens per parameter.
4. Distil or prune from a larger model.
5. Post-train, then quantise.

• The device sets the parameter budget before any training.
• Small models are trained past the compute-optimal token count.
• Knowledge distillation and pruning let a small model inherit from a larger one.

Worked example

Three published models show how far small models sit from the compute-optimal rule. The inputs are parameters and pretraining tokens from the papers and the SmolLM2 model card, which reports 11 trillion tokens, a transformer decoder, bfloat16 and 256 H100 graphics processing units (GPUs) (model card, read October 2026). Training compute uses the estimate of about 6N floating-point operations per training token, where N is the parameter count (Kaplan et al., 2020).

For SmolLM2 1.7B, 6 times 1.7 billion times 11 trillion is about 1.1 times 10 to the power 23 operations. Chinchilla's 70 billion parameters over 1.4 trillion tokens come to about 5.9 times 10 to the power 23. So the small model used roughly a fifth of Chinchilla's compute, on a model about 41 times smaller, and its weights fit in 3.4 GB instead of 140 GB.¹ The compute-optimal token count for 1.7 billion parameters would be about 34 billion tokens; training on 11 trillion is the deliberate overtraining of Step 3.

Text description of the diagram

• Chinchilla trained on 20 tokens per parameter.
• Phi-3-mini trained on about 870 tokens per parameter.
• SmolLM2 1.7B trained on about 6,470 tokens per parameter.

Compression then divides the weight memory: 3.4 GB at 16 bits, 1.7 GB at 8 bits and 0.85 GB at 4 bits for SmolLM2 1.7B, and 7.6, 3.8 and 1.9 GB for Phi-3-mini.¹

Text description of the diagram

• Weights of a 1.7 billion parameter model take 3.4 GB at 16 bits and 0.85 GB at 4 bits.
• Weights of a 3.8 billion parameter model take 7.6 GB at 16 bits and 1.9 GB at 4 bits.

Trade-offs and edge cases

A small model is not a drop-in replacement for a large one. It stores less knowledge and handles hard reasoning worse, so the best use is a narrow job. The authors of the position paper argue that small models are "sufficiently powerful, inherently more suitable, and necessarily more economical" for many repetitive subtasks of agent systems, and also note that not all language models in an agent system are replaceable by small ones (Belcak et al., 2025). Treat that as a position, not a measurement.

A second case is cost. Large models are overtrained too, only less: Meta trained Llama 3 70B on over 15 trillion tokens, about 214 per parameter (Meta AI, April 2024).¹ Overtraining spends more training compute than the compute-optimal run for the same loss, and it only pays when the model is served enough.

Hoffmann et al. make the serving side of the argument: a smaller, better-trained model "reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware" (Hoffmann et al., 2022). A third case is the quality claim. Microsoft reports that the 3.8-billion-parameter Phi-3-mini performs in a range that "rivals that of models such as Mixtral 8x7B", and that it is small enough to deploy on a phone (Phi-3 report, 2024); that is the authors' benchmark result, so check it on your own task.

In short, a small model runs on one consumer device, needs a few GB for 16-bit weights and up to about 20 GB at ten billion parameters, and suits narrow, repeated tasks; a large model needs one or more data-centre GPUs and tens to hundreds of GB, and suits broad, open-ended tasks.

Text description of the diagram

• Small models run on one consumer device and suit narrow repeated tasks.
• Large models run on one or more data-centre GPUs and suit broad open-ended tasks.

How to answer it in an interview

The spoken structure is: budget first, then data and tokens, then inherit and compress, then name the limit.

Common mistakes

1. Treating "small" as a fixed size. (wrong) "Small means under one billion parameters." (right) "There is no standard cut-off; a common working line is the device the model must fit on, and most models below ten billion parameters are treated as small."
2. Using the compute-optimal token count. (wrong) "Train a 1.7-billion-parameter model on 34 billion tokens." (right) "Train it far longer, because training is paid once and serving is paid every time."
3. Counting only the weights. (wrong) "3.4 GB model, so it needs 3.4 GB." (right) "Weights plus the attention cache and working memory."
4. Saying distillation copies the weights. (wrong) "The small model is a slice of the large one." (right) "It is trained to imitate the large model's outputs; pruning is the method that removes weights."
5. Promising large-model quality. (wrong) "A small model matches a large one everywhere." (right) "It can match on a narrow task; check it on yours."

Follow-up questions

How to train a small language model?

Train it as you would any language model: a transformer decoder, curated data, a token budget far above 20 per parameter, then post-training. For most teams the practical route is not pretraining from scratch but fine-tuning an open small model, for example with low-rank adaptation (LoRA), covered in LoRA fine-tuning.

How to build a small language model?

Set the memory budget, pick a size that fits, curate the data, pretrain or distil from a larger model, post-train, then compress. The five steps above are the whole recipe; the cheapest start is an existing open model of the right size, such as the SmolLM2 family, which comes in 135 million, 360 million and 1.7 billion parameter sizes (SmolLM2 model card, read October 2026).

Why train a small model on more tokens than the compute-optimal rule says?

Because the rule minimises training cost for a given loss, not serving cost. A smaller model trained longer reaches a loss that would otherwise need a bigger model, and then costs less at every request (Hoffmann et al., 2022). Sardana et al. conclude that teams expecting large inference demand, around a billion requests, "should train models smaller and longer than Chinchilla-optimal" (Sardana et al., 2023).

Should I fine-tune a small model or prompt a large one?

Fine-tune a small model when the task is narrow, repeated and cost or privacy matters; prompt a large model when the task is varied or needs broad knowledge. The trade-off is covered in fine-tuning versus prompt engineering.

How much memory does a small language model need?

Multiply the parameter count by the bytes per parameter for the weights, then add the attention cache and working memory. A 1.7-billion-parameter model needs 3.4 GB for 16-bit weights and 0.85 GB at 4 bits before those extras.

Sources

1. Small Language Models are the Future of Agentic AI — arXiv (Belcak et al.), working definition in section 2.1, 2 Jun 2025
2. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone — arXiv (Microsoft), v4 revised 2024-08-30, 22 Apr 2024
3. Textbooks Are All You Need — arXiv (Gunasekar et al.), v2 revised 2023-10-02, 20 Jun 2023
4. SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model — arXiv (Hugging Face), 4 Feb 2025
5. SmolLM2-1.7B model card — Hugging Face (date = read date), 24 Oct 2026
6. Training Compute-Optimal Large Language Models — arXiv (Hoffmann et al.), 29 Mar 2022
7. Scaling Laws for Neural Language Models — arXiv (Kaplan et al.), 23 Jan 2020
8. Introducing Meta Llama 3: The most capable openly available LLM to date — Meta AI, 18 Apr 2024
9. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — arXiv (Sardana et al.), v3 revised 2025-04-14, 31 Dec 2023
10. Distilling the Knowledge in a Neural Network — arXiv (Hinton, Vinyals and Dean), 9 Mar 2015
11. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models — Meta AI, 25 Sep 2024