What is cross-entropy loss?

In this article

1. Before you start
2. Why interviewers ask this
3. Cross-entropy loss from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the classical machine learning guide, which places the loss between the model and the optimiser.

Why interviewers ask this

A candidate who can only recite the formula cannot debug a classifier that will not train, and loss functions are where most such bugs live. Cross-entropy appears in machine learning screens in three forms: "which loss would you use for this classification problem", "does your framework's loss apply softmax", and "what should the loss be on the first step". Each of them can come up inside a modelling discussion, not only as a standalone definition.

A weak answer names the loss. A strong answer explains why the logarithm is there, why the gradient is simple, what the framework expects as input, and which numerical check catches a wrong setup.

Cross-entropy loss from first principles

In plain words

Cross-entropy loss

Cross-entropy loss is a loss function for classification that equals the negative natural logarithm of the probability a model assigns to the true class, averaged over examples. It is zero only when the model is certain and right.

Binary cross-entropy

Binary cross-entropy is a form of cross-entropy loss that scores two outcomes, written as minus the sum of the label times the log of the predicted probability and one minus the label times the log of one minus that probability.

Label smoothing

Label smoothing is a training option that replaces a one-hot target with a mixture of that target and a uniform distribution over classes, which stops the model from being pushed toward certainty.

Picture a weather forecaster who must give a probability for rain every day. You score her by the probability she gave to what actually happened. Saying 90 percent and being right is cheap. Saying 1 percent and being wrong is very expensive. Over a year the forecaster who is honest about her uncertainty scores best.

The analogy breaks at multiple classes. A forecaster gives one number, but a classifier must spread a total of 1 across many classes.

In one precise sentence, cross-entropy loss is the average of minus the log of the probability on the correct class. That one sentence is all the formula you need to remember; the steps below explain where the probabilities come from and why PyTorch asks for logits.

Step 1: Softmax turns raw scores into probabilities

Softmax converts a vector of logits into probabilities by exponentiating each logit and dividing by the sum, so every output is positive and the outputs add to 1. The softmax section of the activation functions page covers it in detail. Adding the same number to every logit leaves the probabilities unchanged, because it multiplies the top and the bottom of every fraction by the same factor, and that fact is what makes the stable version in Step 3 possible.

The input is the logit vector. The output is a probability for each class. With illustrative logits of 2.0, 1.0 and 0.1, the probabilities are 0.659, 0.242 and 0.099¹. The largest logit gets the largest probability, and a gap of one logit is a factor of $e$, about 2.7, in probability.

Step 2: The loss is minus the log of the right answer's probability

For one example the loss is $L=-\ln p_c$, where $p_c$ is the probability given to the true class $c$. For a batch, the loss is the average of these values. scikit-learn writes the binary case as $-(y\ln p+(1-y)\ln (1-p))$ and averages over samples by default.

If the true class is the first one, the loss on our example is $-\ln 0.659=0.417$. If the true class is the third one, the loss is $-\ln 0.099=2.317$¹. The loss is never negative for a valid probability, because the natural log of a probability between 0 and 1 is at most zero.

The input is the probability vector and the true label. The output is a single non-negative number that is smaller when the model is right and sure.

Step 3: Why the loss layer takes logits, not probabilities

PyTorch computes cross-entropy from logits, not from probabilities, because fusing softmax and the logarithm is more stable and avoids applying softmax twice. Keras is the opposite by default: its cross-entropy losses assume probabilities unless you set from_logits=True (Keras losses reference, read 2026-10-24). The PyTorch 2.14 reference says the loss expects raw, unnormalised logits, that softmax is applied internally, and that the class-index case equals log-softmax followed by negative log-likelihood, so passing softmax outputs would apply softmax twice.

Stability is the practical reason. With logits of 1000, 999 and 998, a naive softmax overflows and returns not-a-number for every class, while the fused log-softmax gives the correct values −0.408, −1.408 and −2.408¹. Passing probabilities to a loss that applies softmax again gives a flatter, wrong distribution and a loss that stalls.

The input is raw logits and a class index. The output is the same loss as Step 2, computed without ever forming overflow-prone probabilities.

Step 4: The gradient is simply prediction minus label

The gradient of softmax cross-entropy with respect to the logits is the probability vector minus the one-hot label (a vector that is 1 for the true class and 0 elsewhere). For our example with the first class true, that is −0.341, 0.242 and 0.099: the true class is pushed up, the others down, in proportion to their probabilities¹.

This simplicity is why cross-entropy trains well. Take a single sigmoid output (a function that squeezes any number into the range 0 to 1) with a label of 1 and a pre-activation (the number the sigmoid receives) of −6, so the model is confidently wrong at a probability of 0.0025. Squared error, taken as half the squared difference, then gives a gradient of only −0.0025 with respect to the pre-activation, because the sigmoid is flat there. Cross-entropy gives −0.9975¹. The loss is large exactly when the model is most wrong, and the gradient is too.

1 · Softmax2 · Minus log3 · Pass logits4 · The gradientReplay

Step-by-step description

1. Raw logits become probabilities that add to one.
2. The loss is minus the log of the true class probability.
3. A fused loss, as in PyTorch, takes logits and applies softmax itself.
4. The gradient with respect to logits is probabilities minus the label.

• Softmax turns the logits 2.0, 1.0 and 0.1 into probabilities 0.659, 0.242 and 0.099.
• The loss when the first class is true is 0.417.
• The fused loss layer takes raw logits and applies softmax internally, giving the same loss 0.417.
• The gradient with respect to the logits is the probabilities minus the one-hot label.

Worked example

A nat is the unit of information when the logarithm is natural. The numbers are the binary case with label 1, computed from $-\ln p$¹.

Text description of the diagram

• A probability of 0.9 on the correct class costs 0.105 nats.
• A probability of 0.01 on the correct class costs 4.605 nats.
• The loss is minus the natural log of the probability on the correct class.

The uniform-guess rows give a sanity check. A fresh classifier with 1,000 classes should start near 6.908. If the first loss is far higher, the initial logits are too large, so the outputs are over-confident, or the targets are malformed, such as multi-hot rows passed as class probabilities, which PyTorch does not validate. Shuffled labels do not raise the first loss, because an untrained model's outputs do not depend on which class is marked true. The scikit-learn row reproduces the documented value of 0.21616 from the four-example input in its reference¹.

The link to information theory is an identity: cross-entropy equals entropy plus Kullback-Leibler (KL) divergence. For the illustrative distributions $P=(0.7,0.2,0.1)$ and $Q=(0.5,0.3,0.2)$ the entropy is 0.8018 nats, the cross-entropy is 0.8869 and the KL divergence is 0.0851, and 0.8018 plus 0.0851 gives 0.8869¹. Training with the true labels fixed can only shrink the KL term, because the entropy of the labels is constant. The entropy, cross-entropy and KL divergence page derives it, and chapter 3 of Goodfellow, Bengio and Courville (2016) is the textbook treatment under probability and information theory.

Text description of the diagram

• Squared error gives a gradient of only 0.0025 in size when the sigmoid is saturated.
• Cross-entropy gives a gradient of 0.9975 in size in the same case.
• The model output in this case is a probability of 0.0025 for a true label of 1.

Trade-offs and edge cases

Cross-entropy is the right loss when the output is a probability over classes, and the wrong one when the target is a real number. Binary and multi-label classification use binary cross-entropy on a sigmoid per label, multiclass classification uses cross-entropy on softmax logits, and regression uses squared error, which is maximum likelihood under Gaussian noise as the Gaussian distribution page shows.

Text description of the diagram

• Binary and multi-label classification use binary cross-entropy on a sigmoid per label.
• Multiclass classification uses cross-entropy on softmax logits.
• Regression uses squared error, which is maximum likelihood under Gaussian noise.

Cross-entropy has three known weak points. First, class imbalance: PyTorch exposes a per-class weight argument that rescales each class's loss, which is one lever for skewed data. Second, over-confidence: PyTorch's label_smoothing option mixes the target with a uniform distribution, and with a smoothing of 0.1 over three classes the target becomes 0.933, 0.033 and 0.033 under the common mixture of the label scaled by 0.9 plus 0.1 spread evenly (the PyTorch page says only that the target is a mixture)¹. Third, label noise: because $-\ln p$ grows without limit as $p$ approaches zero, one mislabelled example can dominate a batch, so audit labels before swapping the loss.

scikit-learn guards against $\ln 0$ by clipping probabilities to a small range around 0 and 1 set from the machine precision of the input type, which shows why probabilities are fragile near 0 and 1; a loss that takes logits avoids the problem.

How to answer it in an interview

The spoken structure is: name the loss, say what it computes, give the gradient or the stability point, then the sanity check.

Common mistakes

• "I apply softmax in the model, then call PyTorch's cross-entropy loss" (wrong). The loss applies softmax itself, so the double softmax flattens the output and stalls learning (right: pass logits; in Keras, drop the softmax and set from_logits=True).
• "Cross-entropy and log loss are different things" (wrong). For classification, log loss is cross-entropy with natural logs (right).
• "Squared error is fine for classification" (wrong). With a saturated sigmoid or softmax it gives a vanishing gradient where the model is most wrong (right: use cross-entropy).
• "A good loss value is below some universal threshold" (wrong). The scale depends on the number of classes; compare against the log of the class count (right).
• "Accuracy and loss always move together" (wrong). Loss can rise while accuracy holds, because it also measures confidence (right).

Follow-up questions

Does cross-entropy loss apply softmax?

In PyTorch the loss layer applies softmax internally and expects logits, so you must not add a softmax before it. Other libraries differ: Keras assumes probabilities unless from_logits=True, and scikit-learn's log loss takes probabilities, which is why you must read each function's input contract.

Can cross-entropy loss be negative?

Not for discrete classes with valid probabilities, because $-\ln p$ is at least zero when $p$ is between 0 and 1. A negative value points to invalid inputs, such as probabilities above 1 or a continuous density used as the predicted value.

How do I choose a loss function?

Start from the output type: probabilities over classes get cross-entropy, a real number gets squared error, and a real number with outliers gets a loss that grows more slowly such as absolute error. Then check that the loss rewards what the product needs, and see classical machine learning for the models these losses train.

How do I tie the loss to precision and recall?

Cross-entropy scores probability quality, not precision or recall directly. Train with cross-entropy, optionally with class weights, then choose a decision threshold on the probabilities to hit the recall or precision target read from the confusion matrix.

What should the loss be at the start of training for a classifier with 1,000 classes?

About 6.908, the natural log of 1,000, if the initial outputs are close to uniform. A start far above that, such as 20, points to large initial logits or to target rows that do not sum to 1, which is a bug to fix before tuning anything in a deep learning model.

Sources

1. Probabilistic losses, Keras 3 API documentation — Keras (page shows no date; read 2026-10-24), 24 Oct 2026
2. torch.nn.CrossEntropyLoss, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
3. sklearn.metrics.log_loss, scikit-learn 1.9 documentation — scikit-learn developers (page shows version only; read 2026-10-24), 24 Oct 2026
4. Deep Learning, Part I and Part II contents (chapters 3 and 6 cover information theory and feedforward networks) — MIT Press (Goodfellow, Bengio, Courville; year only), 1 Jan 2016