What is knowledge distillation, and how does it work?

In this article

1. Before you start
2. Why interviewers ask this
3. Knowledge distillation from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

See how large language models are trained for pretraining, fine-tuning and alignment.

Why interviewers ask this

A model that is too large to serve is a daily problem in production, so knowledge distillation comes up in machine learning design rounds as one of three ways to cut inference cost. It has been reported by candidates in senior large language model (LLM) interviews, often as a comparison with quantization and pruning, as a coding task ("implement it"), or as a cost follow-up in a design round.

A weak answer says "a small model copies a big model." A strong answer says what the student copies (a probability distribution, not a label), writes the loss as a mix of two terms, and explains the temperature. A Staff-level answer adds when distillation is the wrong tool, how it stacks with the other compression methods, and what it does to quality.

Knowledge distillation from first principles

Knowledge distillation

Knowledge distillation is a training method in which a small student model is trained to match the output probabilities of a large teacher model or ensemble (an ensemble is several models whose predictions are averaged), so that the student gives similar answers at a much lower cost to deploy.

Soft target

A soft target is the full probability distribution a teacher gives over all possible answers, such as 0.54, 0.26 and 0.20 for three classes, used as the training label for the student instead of a single correct answer.

Temperature

The temperature is a number T that divides the teacher's scores before the softmax; a T above 1 flattens the distribution so that small probabilities get larger and the student can see how plausible each wrong answer is.

In plain words

Think of a master chef teaching an apprentice. Marking only "right" or "wrong" on each dish teaches little. Saying "this one was close to a pasta sauce, a bit like a stew, nothing like a dessert" teaches far more. The teacher model is the chef, the student model is the apprentice, and the soft target is the richer feedback. The analogy breaks in one place: the apprentice here is a smaller network that cannot hold everything the chef knows, so it learns what it can fit.

In one precise sentence, the student minimizes a loss that mixes two terms. One is cross-entropy against the true label. The other is a divergence from the teacher's softened distribution. The idea builds on work by Caruana and collaborators. They showed that the knowledge in an ensemble can be compressed into a single model that is easier to deploy (Hinton et al., 2015). The method is not the same as fine-tuning an LLM or low-rank adaptation (LoRA) adaptation, which adapt a model of the same size.

Step 1: The teacher turns its scores into soft targets with a temperature

The teacher is the large trained model that supplies the targets, and it is frozen during distillation. For each input it outputs raw scores, called logits. Softmax at temperature 1 usually puts almost all probability on one answer. Dividing the logits by T before softmax flattens the result: per the PyTorch tutorial, larger T gives smoother distributions, so smaller probabilities get a larger boost. Hinton et al. (2015), section 2 use a high temperature for this reason when they produce soft targets.

Input: the teacher's logits. Output: a soft target. The teacher's weights do not change, so its forward pass runs without storing gradients. You now know what the student will copy: a smoothed distribution, with the wrong answers' ranking intact.

Step 2: The student produces its own distribution

The student is the small model being trained, with fewer layers or smaller width than the teacher. It sees the same input and outputs its own logits. The same temperature divides them before softmax, so the two distributions are compared on the same scale. At the same time, the student's logits at temperature 1 are compared with the true label.

Input: the same example as the teacher saw. Output: the student's softened distribution and its normal prediction. The student is trained on a transfer set, a dataset of inputs that the teacher also scores; this set can be the original training data or other unlabeled inputs, because the teacher supplies the targets. You now have two distributions to compare, which is the next step.

Step 3: A mixed loss pulls the student toward the teacher and the label

The loss has two parts. One is cross-entropy between the student and the true label. The other is the Kullback-Leibler (KL) divergence, a measure of how far one probability distribution is from another, between the teacher's and the student's softened distributions. Practical Insights writes the mix with a weight α: the label term weighted by 1 minus α and the teacher term by α. The PyTorch tutorial notes that tuning these weights pushes the network toward one objective or the other.

Soft-target gradients shrink as T grows. Hinton et al. (2015), section 2 state that they scale as 1 over T squared, so many implementations multiply the KL term by T squared; the worked example below measures that shrinkage. For language models, Gemma 2 uses the same idea per token. It replaces the one-hot next-token target with the teacher's distribution. One-hot is the name for a label that puts probability 1 on the correct answer and 0 on all others, and a one-hot label is called one-hot because only one position is switched on. Gemma 2 then minimizes the negative log-likelihood between teacher and student probabilities.

Step 4: The student trains alone and is deployed alone

The optimizer updates only the student's weights. After training, the teacher is discarded from the serving path, so the deployed system has only the cheap model. This is the point of the whole method: the teacher's cost is paid once, during training, and the student's lower cost is paid on every request.

Distillation can happen at several stages. DistilBERT used it during pretraining, where the paper reports a triple loss combining language modeling, distillation and cosine-distance terms. Gemma 2 trains its smaller models with distillation instead of plain next-token prediction. Post-training can distill too: Gemma 2 reports behavioral cloning on responses mostly generated by the teacher. A smaller model built this way is one kind of small language model.

1 · Soft targets2 · Student output3 · Mixed loss4 · DeployReplay

Step-by-step description

1. The frozen teacher outputs a soft target.
2. The student outputs its own distribution.
3. One loss mixes teacher and label terms.
4. The student ships without the teacher.

• The teacher is frozen and gives soft targets.
• The student gives its own softened distribution.
• A mixed loss compares both with the label.
• The trained student is deployed alone.

Worked example

The numbers below use illustrative logits for a three-class problem: teacher logits 5, 2 and 1, and student logits 3, 2.5 and 1.¹ At temperature 1 the teacher gives probabilities 0.9362, 0.0466 and 0.0171, so the wrong answers get very little probability. At temperature 4 it gives 0.5434, 0.2567 and 0.1999, so the student can see that class B is a more plausible wrong answer than class C by a margin the hard label hides.¹

The gradient shrinks too. The size of the gradient on the student's logits is 0.4752 at temperature 1 and 0.0444 at temperature 4, a drop by a factor of 10.7; multiplying by T squared (16) brings it back to the same order of size (0.71).¹ With the weight α set to 0.5 (illustrative), the label cross-entropy is 0.555, the scaled KL term is 0.6658, and the mixed loss is 0.6104.¹

Text description of the diagram

• At temperature 1 the teacher gives probabilities 0.9362, 0.0466 and 0.0171.
• At temperature 4 the teacher gives probabilities 0.5434, 0.2567 and 0.1999.

The speech result is the cleanest picture of the idea: Hinton et al. (2015) report frame accuracy of 58.9% for the baseline, 61.1% for the ensemble and 60.8% for the distilled model, with word error rate (WER) of 10.9%, 10.7% and 10.7%. The distilled single model keeps about 86% of the ensemble's gain while running as one model.¹

A minimal implementation of the loss, in PyTorch, follows the tutorial's structure:

import torch
import torch.nn.functional as F

def distill_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.5):
    soft_targets = F.softmax(teacher_logits / T, dim=-1)
    soft_prob = F.log_softmax(student_logits / T, dim=-1)
    kd = F.kl_div(soft_prob, soft_targets, reduction="batchmean") * (T * T)
    ce = F.cross_entropy(student_logits, labels)
    return (1 - alpha) * ce + alpha * kd

In the training loop, the teacher is put in eval() mode and its logits are computed inside torch.no_grad(), because the teacher's weights do not change. Only the student's parameters go to the optimizer.

Trade-offs and edge cases

Distillation trades some quality for a lower serving cost. DistilBERT keeps 97% of BERT's language understanding, not all of it, so the student does not always match the teacher. The gain is largest when the teacher is much better than the student could become alone: the Gemma 2 report says distillation gives smaller models richer gradients than a one-hot label. A teacher can also be too large: Mirzadeh et al. (2019) report that a student can learn less from a much larger teacher than from a mid-size one.

Three cases change the plan. First, a student cannot exceed what its architecture can represent, so a very small student plateaus. Second, white-box distillation uses the teacher's logits, while black-box distillation trains only on text the teacher generated; with a teacher reached through an application programming interface (API), you mostly get the black-box form (some APIs expose top-k log-probabilities), and the provider's terms of use decide whether the output may be used for training. Third, the method pairs with other compression: the table below separates them.

Text description of the diagram

• Distillation trains a new smaller model.
• Quantization lowers number precision of an existing model.
• Pruning removes weights from an existing model.

Quantization is a technique that stores a trained model's numbers at lower precision, such as 8 bits instead of 16. Pruning is a technique that removes weights or whole structures from a trained model. Both start from an existing model; distillation trains a new one. The three can be combined: a distilled student can be quantized afterward, and a pruned network can be retrained with distillation to recover quality.

To decide between a small model and a large one per request, a routing system sends easy requests to the student and hard ones to the teacher; the choice depends on the quality gap and the traffic mix. Where task-specific behavior is the goal, compare with fine-tuning versus prompt engineering and parameter-efficient fine-tuning (PEFT) methods.

How does model distillation work for LLMs?

Model distillation for a large language model usually changes two choices of the classic recipe: which divergence is minimized, and whose text the student trains on. The loss on this page is forward KL, from teacher to student. MiniLLM (Gu et al., 2023) argues that reverse KL suits generative models better, because it stops the student from spreading probability over answers it cannot model well.

GKD (Agarwal et al., 2023) trains the student on sequences it generates itself, with the teacher scoring them, which reduces the mismatch between training text and the text the student produces when serving. For reasoning, DeepSeek-AI (January 2025) report fine-tuning smaller Qwen and Llama models on about 800,000 samples curated with DeepSeek-R1, and report that this beat running reinforcement learning directly on the small models. Treat these as reported results, not settled rules; the best divergence still depends on the task.

How to answer it in an interview

The spoken structure: define the student and teacher, give the soft target and the loss, say the teacher is dropped at serving time, then contrast the other two methods and name the cost.

Common mistakes

• "The student only learns the right answer." (wrong) The signal is the teacher's whole distribution. (right) "The student learns how likely every answer is, including the wrong ones."
• "Distillation is the same as quantization." (wrong) Quantization changes number precision of an existing model. (right) "Distillation trains a new smaller model; I can quantize it afterward."
• "The distilled model matches the teacher." (wrong) DistilBERT retains 97%, not all. (right) "It keeps most of the quality at lower cost, and I measure the gap."
• "Distillation is only for compression after training." (wrong) Gemma 2 uses it as the pretraining objective. (right) "It can run during pretraining, post-training or for a single task."
• "I need the teacher's weights." (wrong) Black-box distillation uses generated text. (right) "With only an API I train on its outputs, subject to its terms."

Follow-up questions

What is the difference between distillation, quantization and pruning?

Distillation trains a new, smaller student on a teacher's outputs, quantization lowers the numeric precision of an existing model, and pruning removes weights from an existing model. The three can be stacked, since a distilled student can still be quantized.

When do you route to a small distilled model instead of a large LLM?

Route to the small model when a request is simple and the measured quality gap on that traffic is acceptable, and keep the large model for hard requests. A router needs an evaluation set per request type and a fallback path when the small model is unsure.

How do you implement knowledge distillation?

Freeze the teacher, compute its logits without gradients, divide both sets of logits by a temperature, take the KL divergence of the softened distributions scaled by the temperature squared, and mix it with the label cross-entropy using a weight. The code block above does exactly that.

How does distillation work with speculative decoding?

In speculative decoding, a small draft model proposes tokens and the large model checks them, so the draft's agreement with the large model sets the speedup. DistillSpec (Zhou et al., 2023) distills the draft model from the large model to raise that agreement and so the speedup.

What happens if the temperature is set much higher?

As the temperature grows, the teacher's distribution approaches uniform, so the soft target carries less information about which answers are better. Hinton et al. (2015), section 2 show that at a high enough temperature, matching soft targets becomes matching the logits. A moderate value, tuned on validation data, is the usual choice.

Can you distill from a model you only reach through an API?

Yes, in the black-box form: collect the teacher's generated responses for many prompts and train the student on them as ordinary text. The student then learns from sampled text rather than full probabilities, so the signal is weaker, and the provider's terms of use may limit the practice.

Sources

1. Distilling the Knowledge in a Neural Network (arXiv 1503.02531) — arXiv (Hinton, Vinyals, Dean), 9 Mar 2015
2. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (arXiv 1910.01108) — arXiv (Sanh et al.), 2 Oct 2019
3. Gemma 2: Improving Open Language Models at a Practical Size (arXiv 2408.00118) — arXiv (Google DeepMind), 31 Jul 2024
4. Knowledge Distillation Tutorial — PyTorch Tutorials (no last-updated date shown; accessed 2026-10-26), 26 Oct 2026
5. MiniLLM: Knowledge Distillation of Large Language Models (arXiv 2306.08543) — arXiv (Gu et al.), 14 Jun 2023
6. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (arXiv 2306.13649) — arXiv (Agarwal et al.), 23 Jun 2023
7. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948) — arXiv (DeepSeek-AI), 22 Jan 2025
8. DistillSpec: Improving Speculative Decoding via Knowledge Distillation (arXiv 2310.08461) — arXiv (Zhou et al.), 12 Oct 2023
9. Improved Knowledge Distillation via Teacher Assistant (arXiv 1902.03393) — arXiv (Mirzadeh et al.), 9 Feb 2019
10. Practical Insights into Knowledge Distillation for Pre-Trained Models (arXiv 2402.14922) — arXiv, 1 Feb 2024