What are entropy, cross-entropy and KL divergence?

In this article

1. Before you start
2. Why interviewers ask this
3. Entropy, cross-entropy and KL divergence from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A candidate who can write the cross-entropy formula but cannot say where it comes from has memorised a line of a loss function. These three quantities appear in machine learning theory rounds as "write the cross-entropy formula and derive it", "how does cross-entropy relate to KL divergence", "why is KL not symmetric", and as a bridge into perplexity for language models. They usually come as a chain, each question building on the last.

A weak answer states the formulas. A strong answer builds all three from one idea, surprise, and connects them to maximum likelihood.

Entropy, cross-entropy and KL divergence from first principles

In plain words

Entropy

Entropy is the expected surprise of an outcome drawn from a probability distribution, where the surprise of an outcome is minus the logarithm of its probability. It is high when outcomes are hard to predict and zero when one outcome is certain.

Cross-entropy

Cross-entropy is a score, written H(P, Q), that rates outcomes drawn from the true distribution P using the probabilities of a model Q; it is minus the sum over outcomes of P times the log of Q. It is never smaller than the entropy of P.

KL divergence

KL divergence (Kullback-Leibler divergence), also called relative entropy, is the cross-entropy H(P, Q) minus the entropy of P, written D(P‖Q) with the true or data distribution first. It measures the extra surprise caused by using Q when the truth is P, and it is zero only when the two distributions are equal.

Picture a weather station sending one-line messages about tomorrow's weather. Rare weather deserves a long message and common weather a short one, which saves effort overall. Entropy is the best possible average message length for the real weather. If the station instead designs its messages for a different climate, the average length goes up. That increase is the KL divergence.

The analogy breaks at fractions. Real messages need whole symbols, while the information measures allow fractional lengths.

In one precise sentence, these three numbers measure the best possible, the actual and the wasted average surprise.

Step 1: Entropy is the average surprise

The surprise of an outcome with probability $p$ is $-\log p$, and entropy is the average of that over all outcomes: $H(P)=-{\sum }_i{p}_i\log p_i$. The SciPy 1.18.0 reference gives this formula and says the logarithm base defaults to $e$, giving units called nats, while base 2 gives bits. A bit is the information in one fair coin flip and a nat is the information measured with the natural logarithm.

A fair coin has an entropy of exactly 1 bit. A coin that lands heads 90 percent of the time has 0.469 bits, and one that lands heads 99 percent of the time has 0.081 bits, and a certain outcome has 0¹. Four equally likely outcomes have 2 bits, which is the number of yes-or-no questions needed to name one.

The input is a distribution. The output is a single number, its average unpredictability.

Step 2: Cross-entropy is the surprise under a wrong model

Cross-entropy scores outcomes from the true distribution P using the probabilities of a model Q: $H(P,Q)=-{\sum }_i{p}_i\log q_i$. When $Q$ equals $P$ it reduces to the entropy, and otherwise it is larger.

Take four outcomes with true probabilities 0.5, 0.25, 0.125 and 0.125. Their entropy is 1.75 bits. If a model wrongly treats all four as equally likely, each gets $q=0.25$ and the cross-entropy is exactly 2 bits¹. The model pays a quarter of a bit more per outcome than the best possible description. A softmax from the activation functions page is how a classifier produces the model distribution Q. The cross-entropy loss page uses this quantity with the true distribution replaced by one-hot labels.

The input is a true distribution and a model distribution. The output is the average surprise the model suffers, never below the entropy.

Step 3: KL divergence is not symmetric

KL divergence is the cross-entropy minus the entropy, $D(P\Vert Q)={\sum }_i{p}_i\log (p_i/q_i)$, read as the divergence of P from Q, and it is not symmetric: $D(P\Vert Q)$ and $D(Q\Vert P)$ are generally different. That is why it is a measure of mismatch rather than a distance. The SciPy reference describes it as the extra surprise incurred when you assume the distribution is Q but it is actually P.

For the four outcomes above, the extra is $H(P,Q)-H(P)=2-1.75=0.25$ bits, and the identity entropy plus KL equals cross-entropy holds exactly¹. For two coins, P heads 90 percent and Q a fair coin, $D(P\Vert Q)=0.3681$ nats while $D(Q\Vert P)=0.5108$ nats¹. If Q gives zero probability to an outcome that P can produce, $D(P\Vert Q)$ is infinite.

The PyTorch 2.14 KLDivLoss reference shows a practical trap: the first argument is the model's log-probabilities and the second the observed data, the reverse of the usual $D(P\Vert Q)$ order, and batchmean is the reduction that gives the true KL value. Mixing up the order silently computes the other direction.

The input is two distributions in a chosen order. The output is the extra surprise, zero only if they match.

Step 4: Training on cross-entropy is maximum likelihood and minimizes KL

Minimizing cross-entropy loss against the data is the same as maximum likelihood, and it is the same as minimizing the data-first KL divergence $D(\hat{P}\Vert Q)$, where $\hat{P}$ is the data distribution and Q the model. Maximum likelihood is the principle of choosing the model parameters that make the observed data most probable. Because the entropy of the data does not depend on the model, minimizing cross-entropy and minimizing KL differ by a constant.

Ten coin flips with 7 heads give an empirical distribution of 0.7 and 0.3. The average negative log-likelihood per flip is 0.6931 nats for a guess of 0.5, 0.6109 for a guess of 0.7 and 0.7645 for a guess of 0.9. The smallest value, at the guess that matches the data, equals the entropy of the data, 0.6109 nats, so the KL there is zero¹.

The derivation is short. With $k$ heads in $N$ flips the likelihood of a guess $q$ is $q^k(1-q{)}^{N-k}$, so for 7 heads in 10 flips it is $q^7(1-q{)}^3$. Dividing the log of it by $N$ gives the average log-likelihood, and grouping the flips by outcome turns that into the negative cross-entropy with the data distribution:

So maximizing the likelihood minimizes the cross-entropy. Setting the derivative $\frac{0.7}{q}-\frac{0.3}{1-q}$ to zero gives $q=0.7$, the guess that matches the data¹. In general, outcome $k$ appears $N{\hat{p}}_k$ times, so $\frac{1}{N}{\sum }_i-\ln q(x_i)=-{\sum }_k{\hat{p}}_k\ln q_k$.

The input is a data set and a model family. The output is the model whose distribution is closest to the data in the KL sense.

Perplexity is the exponential of cross-entropy per token

Perplexity is the exponential of the average cross-entropy per token, and it is how language models usually report their loss. The Hugging Face guide defines it as the exponentiated average negative log-likelihood and notes it equals the exponentiation of the cross-entropy between data and model predictions. A cross-entropy of 3.0 nats is a perplexity of 20.086, and a uniform guess over a 50,000-token vocabulary has a cross-entropy of 10.8198 nats and a perplexity of 50,000¹. Perplexity also depends on the tokenization, the test text and the context length, so compare only models that share all three; see byte-pair encoding.

1 · Entropy2 · Cross-entropy3 · KL divergence4 · LikelihoodReplay

Step-by-step description

1. Average surprise of the true distribution.
2. Average surprise when scoring with a model.
3. The extra surprise; not symmetric.
4. Maximum likelihood minimizes cross-entropy, so it minimizes KL.

• With true probabilities 0.5, 0.25, 0.125 and 0.125 and a uniform model, entropy is 1.75 bits, cross-entropy is 2 bits and KL divergence is 0.25 bits.
• The identity entropy plus KL equals cross-entropy holds exactly.

Worked example

A reader can redo it: the surprises of the four outcomes under P are 1, 2, 3 and 3 bits, so the average is

Under the uniform model every outcome costs 2 bits, so the average is 2¹.

Text description of the diagram

• A fair coin has an entropy of exactly 1 bit.
• A coin that lands heads 90 percent of the time has 0.469 bits, and one at 99 percent has 0.081 bits.
• A certain outcome has no entropy.

The same identity checks the distributions on the cross-entropy loss page: for $P=(0.7,0.2,0.1)$ and $Q=(0.5,0.3,0.2)$ the entropy is 0.8018 nats, the cross-entropy 0.8869 and the KL divergence 0.0851¹.

Trade-offs and edge cases

The three quantities are exact for discrete distributions, but their use in practice has sharp edges. Direction, zeros and units cause most errors.

Text description of the diagram

• For a 90 percent coin P and a fair coin Q, D(P‖Q) is 0.3681 nats while D(Q‖P) is 0.5108 nats.
• If Q gives zero probability to an outcome that P can produce, D(P‖Q) is infinite.

• Direction. $D(P\Vert Q)$ with P as the data is infinite for a model that gives zero to something that happens, while $D(Q\Vert P)$ is a different number for the same two distributions. State which direction you minimize.
• Zeros. A model probability of zero where the data has mass makes KL infinite, which is why implementations use log-probabilities and clip or smooth.
• Units. Bits and nats differ by a factor of $\ln 2$, about 0.693; report the unit, and note that perplexity is the same whether you use bits with base 2 or nats with base $e$.
• Continuous variables. For densities the sum becomes an integral, and the entropy can be negative; the Gaussian distribution page covers the density these integrals use.
• Not a metric. KL is neither symmetric nor satisfies the triangle inequality, so do not call it a distance.

How to answer it in an interview

The spoken structure is: define surprise, give the three quantities in order, state the identity, then say what training minimizes and the asymmetry.

Common mistakes

• "KL divergence is the distance between two distributions" (wrong). It is not symmetric and does not obey the triangle inequality (right: call it a divergence or a measure of mismatch).
• "Cross-entropy and entropy are the same" (wrong). Cross-entropy uses a second distribution and is at least the entropy (right: they are equal only when the model equals the truth).
• "Use the natural log for bits" (wrong). Bits use base 2 and nats use base $e$ (right: state the base).
• "Pass probabilities to the KL loss" (wrong). The PyTorch loss expects the model output as log-probabilities (right: use log-softmax and check the argument order).
• "A lower perplexity always means a better model" (wrong). It depends on the tokenization, the test text and the context length (right: compare with the same tokenizer, test text and context length).

Follow-up questions

How do you write the cross-entropy formula and derive it?

Start from the average negative log-likelihood of $N$ samples, $\frac{1}{N}{\sum }_i-\ln q(x_i)$. Outcome $k$ appears $N{\hat{p}}_k$ times, so it equals $-{\sum }_k{\hat{p}}_k\ln q_k=H(\hat{P},Q)$, the cross-entropy, which splits into entropy plus KL divergence.

What is the relationship between cross-entropy loss and KL divergence?

Cross-entropy loss against a data distribution equals the data's entropy plus the data-first KL divergence $D(\hat{P}\Vert Q)$, with the data $\hat{P}$ first and the model Q second. The entropy is constant during training, so minimizing one minimizes the other; see classical machine learning for the models trained this way.

Why is KL divergence not symmetric, and is it a distance?

It averages the log ratio under the first distribution only, so swapping the arguments changes the weights and the result, for example $D(P\Vert Q)=0.3681$ against $D(Q\Vert P)=0.5108$ nats for the coins above. It is therefore not a distance.

What is perplexity and how does it relate to cross-entropy?

Perplexity is the exponential of the average cross-entropy per token, so a cross-entropy of 3.0 nats is a perplexity of 20.086. It reads as an effective number of equally likely next tokens, as in a transformer language model.

Why does maximum likelihood training minimize cross-entropy?

Maximizing the likelihood of the data is the same as minimizing the average negative log-likelihood, which is the cross-entropy between the empirical distribution and the model. The deep learning loss layers apply exactly this.

What happens to KL divergence if the model gives an outcome zero probability?

If Q gives zero probability to an outcome that P can produce, $D(P\Vert Q)$ is infinite, because the model is infinitely surprised by something that happens. Implementations avoid it with log-probabilities and by clipping or smoothing Q, and the reverse direction $D(Q\Vert P)$ is a different number.

Sources

1. scipy.stats.entropy, SciPy v1.18.0 manual — SciPy (page shows version only; read 2026-10-24), 24 Oct 2026
2. torch.nn.KLDivLoss, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
3. Perplexity of fixed-length models, Hugging Face Transformers documentation — Hugging Face (page undated; read 2026-10-24), 24 Oct 2026