Debugging a model that will not learn: what to check first

In this article

1. Before you start
2. Why interviewers ask this
3. Debugging a model that will not learn from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the deep learning learning path, and it uses a little probability and statistics.

Why interviewers ask this

A stuck loss curve is the most ordinary problem in machine learning work, so the question shows how a candidate thinks when nothing tells them what is wrong. Candidates report versions of it in machine learning interviews: the validation loss stopped decreasing, the model is not converging, the training loss falls but the validation loss stays flat.

A weak answer lists remedies at random: a bigger model, more layers, a new optimizer. A strong answer orders the checks from cheapest to most expensive and says what each one rules out. A Staff-level answer separates two situations that look alike on a plot: a model that cannot fit even its training data (a bug or a setting) and a model that fits the training data but not new data (a generalization problem). The deep learning interview questions page shows where this sits among its neighbours.

Debugging a model that will not learn from first principles

Tiny-batch overfit test

The tiny-batch overfit test is a check that trains a model on a handful of examples with regularization switched off and expects the training loss to reach about zero, so a failure points to a bug or a setting rather than to model capacity.

Generalization gap

The generalization gap is the difference between the loss, or accuracy, on training data and on held-out validation data, and it grows when a model memorizes details of the training set that do not carry over to new examples.

In plain words

Think of a cake that comes out wrong. You do not buy a new oven first. You check the ingredients, then the measurements, then the oven temperature, and only then the recipe itself. The analogy breaks in one place: a kitchen lets you taste the cake at once, while training gives you one number per step and says nothing about why it is high.

In one precise sentence, a model that will not learn has a training loss that does not fall. The causes sort into three families: wrong data or labels, wrong code, and wrong settings such as the learning rate. Karpathy (2019) explains why this needs a method: in ordinary code a mistake will often raise an exception, but neural network training fails silently. You can read all five steps below on one picture at the end of this section.

Step 1: Look at the data before touching the model

The first check is to read the data itself, with no model code involved. Karpathy (2019) puts it first in his recipe: begin by thoroughly inspecting the data. Print a few raw examples with their labels, count how often each label appears, and look at the exact tensors that enter the network, after any resizing, scaling or tokenizing.

Input: the dataset. Output: a belief that inputs and labels match. A label column shifted by one row, inputs scaled twice, or an empty field filled with a constant all produce a run that trains without any error and learns nothing. You now know the data is sane, so the next question is whether the model starts where it should.

Step 2: Check the loss at the very first step

The loss at step zero of a correct classifier has a known value. Karpathy (2019) says to verify the loss at initialization: for a softmax it should be $-\ln (1/k)$ for k classes, which is $\ln k$. CS231n gives the same check: with ten classes the expected starting loss is 2.302 (ln 10 is about 2.3026), because each class gets a probability of 0.1.

Input: one forward pass on one batch. Output: one number to compare with $\ln k$. If the printed loss is far above it, the final layer outputs scores that are too large at initialization, the loss is summed over the batch instead of averaged, or k is wrong. If it is far below, the loss itself is computed wrongly, for example divided twice or masked to almost nothing. A wrong step-zero loss points to a loss, initialization or class-count bug. Leaked labels do not lower the starting loss; they show up later as a loss that falls suspiciously fast. Regularization should be off for this check. A wrong start shows a bug in seconds.

Step 3: Overfit one tiny batch

Take a handful of examples, turn regularization off, and train on them until the loss is about zero. CS231n recommends exactly this on a tiny portion of the data, such as 20 examples, and says that unless this passes it is not worth moving to the full dataset.

Input: a few examples and the full training loop. Output: a loss close to zero. A model with enough weights can memorize a few examples, so failure cannot mean a weak model. It means the gradient is not reaching the weights, the inputs reach the model as constants or zeros, or the optimizer is misconfigured. Typical code causes are gradients not cleared between steps, a missing optimizer call, and layers frozen by accident. This one test splits bugs from capacity problems, which is the point of the whole method.

Step 4: Tune the learning rate

The learning rate is the number that scales each weight update: the weights change by the gradient times the learning rate. Too small and the loss barely moves; too large and the loss jumps around or blows up. CS231n gives a rough rule: the size of an update divided by the size of the weights should be around one thousandth (1e-3), with lower values hinting at a rate that is too low and higher values at one that is too high.

Input: a few short runs at rates a factor of ten apart. Output: the rate whose loss falls fastest without wobbling. The exploding gradients page covers the blow-up case, and Pascanu et al. (2012) is a paper that proposes gradient-norm clipping for it. The optimizer states page explains the extra numbers an optimizer carries.

Step 5: Read the gap between training and validation

If the training loss falls but the validation loss stays flat, the model learns something that does not carry over; this is a generalization gap, not a bug. CS231n reads the gap between training and validation accuracy as the amount of overfitting and suggests stronger regularization when it is large.

High bias means the model underfits: the training loss itself stays high. High variance means it fits the training data but not new data: training loss is low while validation loss stays high. A stuck training loss therefore means high bias or a bug (Steps 1 to 4), and a gap means variance (this step).

Input: both curves on one plot. Output: a verdict. Train loss down and validation flat points to more data, regularization, a smaller model or earlier stopping. Both flat points back to Steps 1 to 4. If validation was never a fair sample of the training distribution, the gap can come from the split itself. Fix the cheap causes first, then change the model.

1 · Data2 · Step-zero loss3 · Tiny batch4 · Learning rate5 · Validation gapReplay

Step-by-step description

1. Read inputs and labels first.
2. Compare with the chance value.
3. Overfit a handful of examples.
4. Sweep it by factors of ten.
5. A gap means generalization.

• Look at the data before touching the model.
• Compare the step-zero loss with the chance-level value.
• If the model cannot overfit a tiny batch, the cause is a bug or a setting.
• Read the gap between training and validation loss last.

Worked example

The numbers below come from a small illustrative experiment: 300 synthetic points in three well-separated groups with two features each, and a linear softmax classifier trained for 200 full-batch steps.¹ It is a toy, so it shows the shape of each check, not the size of real effects.

Step 2 first: the chance-level loss is $\ln k$, which is 0.693 for two classes, 1.099 for three, 2.303 for ten and 6.908 for a thousand.¹ The toy model, with three classes, starts at 1.114, close to 1.099, so the start is sane.¹

Step 4 next. At a learning rate of 0.0001 the loss falls only from 1.114 to 1.046; at 0.01 it falls to 0.134 and at 1 to 0.014.¹ Notice that at 0.0001 the training accuracy is already 0.943 while the loss has barely moved, so accuracy alone can hide a stalled run.¹

Text description of the diagram

• At a learning rate of 0.0001 the loss falls only from 1.114 to 1.046.
• At 0.01 the loss falls to 0.134 and at 1 to 0.014.

The last row is a one-weight loss $\frac{1}{2}·4·w^2$. Plain gradient descent diverges when the step size exceeds 2 divided by the curvature, here 0.5; at 0.4 the weight flips sign each step but the loss still shrinks to about zero.¹ The zero-input row follows the input-independent baseline of Karpathy (2019): the real-input run must beat it, and here it does (0.035 against 1.099).¹

Trade-offs and edge cases

The checks are cheap, but each can mislead. The tiny-batch overfit test can pass while a bug that only matters on new data survives, such as a model left in training mode during evaluation, so it rules out the training loop, not everything. Dropout and data augmentation make memorizing a handful of examples slower, so switch them off for the test.

The starting-loss check changes for fine-tuning. A pretrained language model does not start at $\ln k$ over its vocabulary, because it already knows language; the start value is the loss of the pretrained model on your data, so measure it before training and compare. Common causes of a fine-tuning loss that does not fall are labels shifted twice, adapter or low-rank adaptation (LoRA) layers that are not trainable, and a wrong chat template; also check that padding and prompt tokens are masked out of the loss. A learning rate that is too small can stall the loss as well.

Numerical format matters too. Half-precision training can underflow small gradients to zero; the 16-bit floating point (FP16) vs brain floating point 16 (BF16) page explains why. Normalization layers behave differently in training and evaluation modes; the batch norm vs layer norm page covers it. Very deep plain networks are harder to train, which residual connections were built to address.

Text description of the diagram

• A wrong step-zero loss points to a loss, initialization or class-count bug.
• A model that cannot overfit a tiny batch has a code or data bug.
• A loss that blows up points to a step size that is too large.
• Falling training loss with flat validation loss is a generalization gap.

How to answer it in an interview

The spoken structure: name the order, say what each check rules out, and end by separating a bug from a generalization gap.

Common mistakes

• "I would make the model bigger." (wrong) A bigger model cannot fix wrong labels or a broken loop. (right) "I would first prove the code can fit a tiny batch, then decide whether capacity is the problem."
• "No error message, so training is working." (wrong) Training fails silently. (right) "I check the step-zero loss and the curves, because errors do not announce themselves."
• "Accuracy is high, so the loss is fine." (wrong) In the toy run accuracy was 0.943 while the loss had barely moved. (right) "I watch the loss and a metric I can interpret together."
• "Training loss down and validation flat means lower the learning rate." (wrong) That pattern is a generalization gap. (right) "I add data or regularization and check that validation matches the training distribution."

Follow-up questions

Your validation loss has stopped decreasing: what do you do?

First check whether the training loss is still falling. If it is, the model is overfitting: add regularization, more data, or stop earlier. If both are flat, go back to the data, the learning rate and the tiny-batch test.

What is your approach to debugging a model that is not converging?

Follow the order of this page: data, step-zero loss, tiny-batch overfit, learning rate, then the validation gap. Change one thing at a time and keep a baseline so each change can be judged.

Training loss keeps reducing but validation loss is stagnant on a large dataset: how will you address it?

Falling training loss with flat validation loss is high variance (overfitting): the model fits the training data but not new data. The bias-variance trade-off is that a bigger model lowers bias but raises variance. Add regularization or data augmentation, shrink the model, stop earlier, and check that the validation split matches the training distribution.

What if the loss does not decrease when fine-tuning a decoder-only language model?

Measure the pretrained model's starting loss on your data, confirm that padding and prompt tokens are masked, and try a smaller learning rate than in pretraining. Then run the tiny-batch overfit test on a few examples.

How do you tell a bug from a model that is too weak?

Run the tiny-batch overfit test. A model that cannot reach about zero loss on a few examples has a bug or a setting problem, because capacity is not the limit there.

Sources

1. A Recipe for Training Neural Networks — Andrej Karpathy (blog), 25 Apr 2019
2. CS231n: Neural Networks Part 3, Learning and Evaluation (read 2026-10-26; page undated) — Stanford University, CS231n course notes, 26 Oct 2026
3. On the difficulty of training recurrent neural networks (arXiv 1211.5063) — arXiv (Pascanu, Mikolov, Bengio), 21 Nov 2012