What loss function did you employ?

In this article

1. Before you start
2. Why interviewers ask this
3. Loss function from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

It builds on machine learning algorithms and uses a little probability from the statistics interview guide.

Why interviewers ask this

A wrong loss gives a model that trains well and answers the wrong question, so the interviewer uses this question to test whether your modelling choices had reasons. The question usually follows a project walkthrough, and it is easy to answer with one word, "cross-entropy", and then stall at the follow-up.

A weak answer names a loss and stops. A strong answer ties the loss to the type of target and the shape of the errors, and says what it did to the model's behaviour. A Staff-level answer also separates the loss from the metric reported to the business, and says what would make it change the loss.

Interviewers can only test this with follow-ups, so expect "why not the alternative?" and "how did you know it worked?" The rubric below shows what each level sounds like.

Loss function from first principles

Loss function

A loss function is a rule that scores one prediction against its true target as a single number, where zero or a small value means a good prediction, so that training can lower the average score over the data by changing the model's weights.

Huber loss

Huber loss is a regression loss that grows with the square of the error when the error is small and only linearly when the error is large, so it behaves like squared error on typical points and like absolute error on outliers.

In plain words

Think of a teacher's marking scheme. Each wrong answer loses marks, and the scheme decides whether one huge blunder costs more than several small slips. The loss function is that scheme for a model, and training is the student adjusting until the lost marks are as few as possible. The analogy breaks in one place. A marking scheme can be any rule, but a loss must give gradient descent a usable slope almost everywhere, so it can see which way to move. Accuracy fails that test, while absolute error and hinge loss pass it despite a kink.

In one precise sentence, you pick the loss that matches the type of target and the kind of errors you expect.

Step 1: Training minimises a loss, and you report a metric

The loss is what training minimises, and the metric is what you report, and the two are often different. Accuracy is a poor training signal because it only changes when a prediction crosses the decision threshold, so its slope is zero almost everywhere. A smooth loss changes with every small improvement.

Take four labelled examples, two positive and two negative. Model A outputs probabilities 0.9, 0.6, 0.4 and 0.1 for the positive class, and model B outputs 0.51, 0.51, 0.49 and 0.49. Both reach an accuracy of 1.0, yet the log loss is 0.3081 for A and 0.6733 for B (illustrative probabilities).¹ The loss sees confidence that accuracy hides. For reporting you choose a metric such as the area under the receiver operating characteristic (ROC) curve (AUC); the scikit-learn ROC curve page shows how to compute it.

You now know the split: train on the loss, report the metric, and check that they move together.

Step 2: Regression starts with squared error or absolute error

Mean squared error (MSE) is the average of the squared differences between predictions and targets, and mean absolute error (MAE) is the average of their absolute values. Squaring makes large errors count much more. An outlier is a target value far from the others, and it dominates a squared-error loss.

Take five residuals (prediction minus target) that are all 1. Both losses equal 1. Change one residual to 10 (illustrative) and MSE becomes 20.8 while MAE becomes only 2.8.¹

The two losses also fit different summaries. For targets 10, 11, 12, 13 and 50, a single constant minimises MSE at the mean, 19.2, and minimises MAE at the median, 12.¹ Goodfellow, Bengio and Courville (2016) show that squared error is what maximum likelihood gives when the noise is Gaussian, which is why the linear regression assumptions matter for this choice.

Step 3: Huber loss is the compromise when outliers exist

Huber loss follows squared error for errors up to a threshold called delta and absolute error beyond it. For an error $r$ with $|r|\le \delta$ it is $\frac{1}{2}{r}^2$, and above that it is $\delta (|r|-\frac{1}{2}\delta )$. Huber (1964) introduced it for robust estimation, and PyTorch ships it as undefined.

With delta set to 1 (illustrative), the same five residuals give 0.5 when all are 1 and 2.3 when one becomes 10.¹ Huber loss with delta 1 moves from 0.5 to 2.3. Huber is therefore far calmer than squared error on the outlier and still smooth around zero, which absolute error is not.

Text description of the diagram

• With five residuals of 1, mean squared error and mean absolute error are both 1.
• When one residual becomes 10, mean squared error becomes 20.8 and mean absolute error becomes 2.8.
• Huber loss with delta 1 moves from 0.5 to 2.3.

Delta is in the units of the target, so set it near the size of a typical error you are willing to treat as normal. You now have the three regression options and the one question that separates them: how heavy are the outliers?

Step 4: Classification uses cross-entropy

Cross-entropy loss for a classifier is the negative log of the probability the model gave to the true class, and it is the standard classification loss; the cross-entropy loss page derives it. It is the negative log-likelihood of the labels, the quantity maximum likelihood minimises (Goodfellow et al., 2016). In scikit-learn the same quantity is undefined.

For a true label of 1, a probability of 0.9 costs 0.1054, 0.5 costs 0.6931 and 0.1 costs 2.3026.¹ A confident wrong answer is punished hard: at a probability of 0.01 the log loss is 4.6052, while squared error on the probability is only 0.9801.¹ The gradient with respect to the logit (the raw score before it becomes a probability) is -0.99 for cross-entropy, against -0.0196 for squared error, so cross-entropy keeps learning where squared error stalls.¹

Use binary cross-entropy for two classes and categorical cross-entropy for many. Check what your loss expects: PyTorch's undefined and BCEWithLogitsLoss take raw logits and apply the softmax or sigmoid inside, which is more numerically stable, while scikit-learn's log_loss takes probabilities.

Step 5: Adjust the loss for the data you have

After the base choice, adapt the loss to the data. If one class is rare, weight its errors more, so that missing it costs more than a false alarm. Class weights shift the predicted probabilities, so recalibrate or move the threshold on a validation set. A support vector machine (SVM) is trained with hinge loss, which penalises examples that fall inside the margin or on the wrong side, and is zero beyond it, instead of using probabilities.

Whatever you choose, check it. Compare training and validation loss, as in the bias versus variance page, and confirm the metric improves when the loss does. Regularisation adds a penalty on the weights to the loss; it is a separate choice, covered in L1 versus L2 regularisation.

1 · Loss and metric2 · Regression3 · Outliers4 · Classes5 · AdjustReplay

Step-by-step description

1. Train on a loss, report a metric.
2. Squared or absolute error.
3. Huber loss is the compromise.
4. Cross-entropy for class labels.
5. Weight rare classes.

• Training minimises a loss and you report a metric.
• Regression starts with squared error or absolute error.
• Huber loss is the compromise when outliers exist.
• Classification uses cross-entropy.
• Adjust the loss with class weights for the data you have.

The Staff-level answer

What makes it Staff-level is the last three sentences: it separates loss from metric, shows how you verified the choice, and says what would change it.

The rubric

Three variants of this question

Why did you choose that loss instead of another?

Say what the target is and what the errors look like. Squared error suits clean, symmetric noise, absolute error or Huber suits outliers, and cross-entropy suits class labels, because it scores the probability given to the true class.

What loss would you use if the data had heavy outliers?

Use Huber loss or absolute error. In the worked example one residual of 10 raised MSE from 1 to 20.8 but Huber loss only from 0.5 to 2.3, so the outlier no longer decides the fit.¹

What loss would you use for imbalanced classes?

Keep cross-entropy but weight the rare class, then choose the threshold on a validation set. Report a metric that sees the rare class, such as recall at a fixed precision, not accuracy.

Common mistakes

• "I used accuracy as the loss." (wrong) Accuracy is not smooth, so it gives gradient descent almost nothing. (right) "I trained on cross-entropy and reported accuracy and AUC."
• "I used L2." (wrong) That is ambiguous: it could be squared error or a weight penalty. (right) "I used squared error as the loss and an L2 penalty on the weights."
• "I applied softmax and then cross-entropy loss." (wrong) PyTorch's CrossEntropyLoss already applies log-softmax, so softmax is applied twice and the scores are flattened. (right) "I passed raw logits to the loss."
• "Mean squared error is the default for any regression." (wrong) It is dominated by outliers. (right) "I checked the residuals for outliers and used Huber loss because they were heavy."
• "Lower loss means a better product." (wrong) The loss need not match the business metric. (right) "I tracked the loss for training and the metric for the decision."

Follow-up questions

Is a loss function the same as a cost function?

Usage varies. Many texts use loss for the error on one example and cost or objective for the average over the dataset, possibly with a regularisation term added.

Should I apply softmax before cross-entropy loss in PyTorch?

No. Pass the raw logits to the loss: the PyTorch documentation says the input is expected to contain the unnormalized logits for each class, and applying softmax first applies it twice and flattens the scores.

How do you choose the delta in Huber loss?

Choose a delta close to the size of an ordinary error in the target's units, so that typical points are squared and only the extreme ones are treated linearly. Tune it on a validation set.

Does regularisation change the loss?

It adds a penalty on the weights to the training objective, so the total being minimised changes, but the data loss stays the same. See L1 versus L2 regularisation.

How do you know the loss you picked is working?

Watch training and validation loss together, and confirm that the reported metric improves on held-out data when the loss falls. A falling training loss with a flat validation metric points to overfitting.

Sources

1. Robust Estimation of a Location Parameter — Annals of Mathematical Statistics (Huber), 1 Mar 1964
2. Deep Learning, chapter 5: Machine Learning Basics — MIT Press (Goodfellow, Bengio and Courville), 18 Nov 2016
3. torch.nn.HuberLoss — PyTorch documentation (accessed 2026-10-26), 26 Oct 2026
4. sklearn.metrics.log_loss — scikit-learn documentation (accessed 2026-10-26), 26 Oct 2026
5. torch.nn.CrossEntropyLoss — PyTorch documentation (accessed 2026-10-26), 26 Oct 2026