How to debug neural network training when it is not learning

In this article

1. Before you start
2. Why interviewers ask this
3. Debugging neural network training from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the deep learning learning path. The loss arithmetic uses logarithms, which the probability and statistics guide refreshes.

Why interviewers ask this

A candidate who cannot say what to check when a network will not learn has probably never shipped one, because training runs often fail silently and the person who ran them has to find out why. A related question, whether to overfit or underfit a network first, is listed in the Machine Learning Interviews Book (read October 2026). Candidates also report debugging questions in design and coding rounds, though no public source ties one to a named company.

The question separates people by order and by evidence. A weak answer is a list of fixes: lower the learning rate, add dropout, try another optimizer. A strong answer is an ordered set of checks, each with an expected value. A Staff-level answer adds a stop rule: if a check fails, nothing after it is worth running, and the answer says why.

Debugging neural network training from first principles

Silent failure

A silent failure is a bug in a training pipeline that raises no error and lets the run finish, yet leaves the model worse than it should be. Code that is wrong in shape-compatible ways still runs and still lowers the loss a little, which is why the bug stays hidden.

Chance-level loss

Chance-level loss is the cross-entropy a classifier scores when it gives every class the same probability. It equals the natural logarithm of the number of classes, and it is the value a correctly built model should show before any learning has happened.

Single-batch overfit test

The single-batch overfit test is a check that trains a model on a handful of examples, repeatedly, until its training loss falls close to zero. A model that cannot memorize that few examples has a bug in its code, data or setup, not a tuning problem.

In plain words

Debugging training is like checking a car before a long trip: tires, fuel and brakes, one at a time, before you blame the driver. The analogy breaks in one place. A car with a flat tire shows a warning light, while a network with a flat tire drives on and arrives late.

That is the core fact. Karpathy (2019) writes that "Neural net training fails silently" and that a misconfigured network will throw exceptions "only if you're lucky". In one precise sentence, debugging training means replacing one large unknown, "why is my model bad", with a chain of small checks whose expected answers you already know.

Step 1: Look at the data and make the run repeatable

The first debugging step is to inspect the data and fix the random seed, before any model code is touched. Karpathy's recipe starts with the data: scan many examples, count the labels, look for duplicates and corrupted items, and view the exact tensor that enters the model. His list of silent bugs includes labels that drift away from their inputs after shuffling, and a view where a transpose was needed, which mixes information across the batch dimension.

Fix the seed so that two runs give the same outcome, and switch data augmentation off for now. Randomness hides small effects, so you cannot tell a fix from luck without a repeatable run.

Input: the raw dataset and the training script. Output: a tensor and a label you have looked at with your own eyes, and a run you can repeat. You now know the inputs are sane; next comes what the loss should read before any learning.

Step 2: Check the starting loss equals the chance value

A softmax classifier with n classes should start with a loss of about the natural logarithm of n. Karpathy's recipe says to verify that "your loss starts at the correct loss value" and gives the formula -log(1/n_classes). The Stanford course notes make the same check with small initial weights (CS231n notes).

Cross-entropy loss is minus the natural log of the probability the model gives the true class, so a probability of 0.1 gives −ln 0.1 = 2.3026.¹ The chance-level loss for 10 classes is therefore 2.3026, and for 1,000 classes it is 6.9078.¹ The table shows more cases. If a 10-class model starts at a loss of 10, it gives the true class a probability near 0.0000454 (e to the power minus 10), which means it is confidently wrong from the first step. Likely causes are an over-scaled final layer, a poor initialization, or a loss that is summed over the batch instead of averaged.¹

Text description of the diagram

• The chance-level loss for 10 classes is 2.3026.
• The chance-level loss for 1,000 classes is 6.9078.

Set the final bias for imbalanced data

For imbalanced data, set the final bias so the starting output matches the class frequency. For a sigmoid output, the bias is the log odds of the positive fraction; with a softmax, it is the log of each class frequency. If positives are 1 in 11 examples, the sigmoid bias is -2.3026; for a starting probability of exactly 0.1 it is -2.1972.¹ Karpathy writes a ratio of 1:10 and a probability of 0.1, two numbers that differ slightly, so compute from the fraction you actually have.

Step 3: Overfit one tiny batch before anything else

Overfit one tiny batch before tuning, because a healthy model can memorize a few examples and a broken one cannot. Karpathy's test is to "Overfit a single batch of only a few examples (e.g. as little as two)" and to check that the labels and predictions line up at the minimum loss; if they do not, "there is a bug somewhere and we cannot continue to the next stage". The Stanford notes use about 20 examples and ask for a cost of zero (CS231n notes).

Run it with regularization and augmentation off and a constant learning rate. Karpathy's safe baseline is Adam at 3e-4 (Karpathy, 2019).

Read the result by shape. A loss that never moves points to gradients that do not reach the weights, a learning rate far off, or a loss computed on the wrong tensor. A loss that falls and stalls above zero points to too little capacity, contradictory labels in the batch, or leftover regularization. Input: a few examples. Output: a yes or no on whether the pipeline can learn at all.

Step 4: Check that gradients flow and examples stay separate

Examples in a batch must not influence each other's outputs, and a short test can show whether they do. Karpathy's dependency test sets the loss to the sum of the outputs for one example i, runs backpropagation, and checks that only input i receives a nonzero gradient. A leak shows up as gradient on other inputs. For sequence models, run the same test over time positions as well as over examples, since a per-example test does not catch a leak between positions.

Run the test in evaluation mode. Batch normalization mixes statistics across examples in training mode, so a nonzero gradient there is expected; the batch norm versus layer norm page explains why. Real leaks come from the view versus transpose bug, from reductions over the wrong axis and from sequence code that lets position t see position t plus one.

Input: one batch and one chosen example. Output: a gradient map that is nonzero only where it should be. A model that passes Steps 1 to 4 has passed the checks that catch the common silent pipeline bugs.

Step 5: Watch gradients, updates and non-finite values during training

During a real run, log the gradient norm, the update-to-weight ratio and any non-finite value. Gradient clipping is a rescaling of all gradients together when their combined norm passes a limit, which caps the size of one update but does not repair the cause of a bad gradient. The PyTorch documentation (version 2.14) says clip_grad_norm_ modifies gradients in place and returns the total norm of the gradients, which is the value before clipping, so the call doubles as a monitor. Its error_if_nonfinite option defaults to False, so not-a-number (NaN) or infinite gradients pass silently unless you set it to True.

Read the update-to-weight ratio

The update-to-weight ratio is the size of one weight update divided by the size of the weights. The Stanford notes give a rough heuristic that it should be around 1e-3 (CS231n notes); an update of 0.00003 on weights of size 0.03 gives exactly that (illustrative).¹ A much lower ratio suggests the learning rate is too low. For spikes and divergence read exploding gradients; for NaN values under half precision see 16-bit floating point (FP16) versus brain floating point 16 (BF16); for what the optimizer keeps between steps see optimizer states.

Step 6: Tune only after every earlier check passes

Tune only after every earlier check passes, and start from something that already works. The Google Research tuning playbook advises: "When starting a new project, try to reuse a model that already works." Karpathy calls the same idea "Don't be a hero": copy the simplest established architecture.

Order matters here too. Overfit first, then regularize. If no model reaches a low training error, that points to a bug or misconfiguration, not to a need for more dropout. Karpathy keeps a constant learning rate until the very end and prefers random search over grid search for tuning, because networks are more sensitive to some hyperparameters than others.

Add one change at a time, so a regression has one suspect. Karpathy's warning is against "unverified" complexity added at once. You now have the full chain, shown in the figure.

1 · Data and seed2 · Starting loss3 · Overfit a batch4 · Gradients5 · Updates6 · TuneReplay

Step-by-step description

1. Inspect the data and fix the seed.
2. Compare it with the chance value.
3. A healthy model memorizes a few examples.
4. Gradients flow and examples stay separate.
5. Watch norms, ratios and non-finite values.
6. Tune last, one change at a time.

• Inspect the data and fix the seed first.
• Overfit one tiny batch before tuning.
• Tune only after every earlier check passes.

The Staff-level answer

What makes this Staff-level is the fixed order with a stop rule at each gate and an expected value to compare against, so every step either passes or names the bug.

The rubric

Three variants of this question

What if the loss is NaN from the first step?

Check the inputs and labels for NaN values first. The most common first-step cause is a log of zero or a hand-written softmax or log that is numerically unstable; a learning rate cannot cause NaN in the very first forward pass, though it can later. In half precision, look for overflow. Clipping does not fix a non-finite gradient, so set error_if_nonfinite to True to find the first bad step.

Should you overfit or underfit a network first?

Overfit first. A model that cannot reach a low training error signals a bug or misconfiguration, and only a model that can memorize its data is worth regularizing; the question is listed in the Machine Learning Interviews Book.

What if training looks fine but validation results are worse than expected?

Treat it as a possible silent failure first: confirm that training and validation data go through the same preprocessing and that the model is in evaluation mode. Then compare the training and validation losses to decide between a data problem and an overfitting problem.

Common mistakes

• "The loss is going down, so the code is correct." (wrong) Label-shuffle and batch-mixing bugs still train. (right) "A falling loss is necessary, not sufficient; I verify the starting loss and the one-batch overfit."
• "The model is not learning, so I will tune the learning rate." (wrong) A pipeline bug survives every learning rate. (right) "I prove the pipeline can learn one batch before tuning."
• "Gradient clipping fixes NaNs." (wrong) Clipping rescales gradients and error_if_nonfinite is off by default. (right) "I find the first non-finite step and fix its cause; clipping is a guard."
• "I run a grid search over the learning rate and weight decay." (wrong) Some hyperparameters matter far more than others. (right) "I use random search, one change at a time."
• "Validation loss leveled off, so I stop." (wrong) Karpathy notes networks often keep improving for an unintuitively long time, though this is his opinion. (right) "I look at the whole curve before stopping."

Follow-up questions

What does a flat loss curve tell you?

A loss that never moves from the first step points to gradients that do not reach the weights, a learning rate far off, or a loss computed on the wrong tensor. The one-batch overfit test separates these pipeline faults from a tuning problem.

How do you know the learning rate is too low?

The update-to-weight ratio is a rough guide: around 1e-3 is the Stanford heuristic, and a much lower ratio suggests the learning rate is too low. A loss that falls but very slowly in the one-batch test says the same.

Why use random search instead of a grid?

Networks are more sensitive to some hyperparameters than others, and a grid spends most of its points repeating values of the sensitive one. Random search tries a new value of every hyperparameter each time (Karpathy, 2019).

Which optimizer and learning rate do you start with?

Start from the most popular optimizer for the problem, as the tuning playbook advises. Karpathy's baseline is Adam at 3e-4 with a constant learning rate; more practice questions are in the deep learning interview questions.

Why do exploding gradients matter when debugging?

Huge gradients cause loss spikes and non-finite values that look like a data bug. Logging the gradient norm tells you which one you have; see exploding gradients.

Sources

1. A Recipe for Training Neural Networks (Andrej Karpathy) — karpathy.github.io, 25 Apr 2019
2. CS231n notes: Neural Networks Part 3, learning and evaluation (official repository, source markdown) — Stanford course notes (Karpathy), 26 Oct 2026
3. Deep Learning Tuning Playbook, v1.0 (2023) — Google Research (Godbole, Dahl, Gilmer, Shallue, Nado), 19 Jan 2023
4. torch.nn.utils.clip_grad_norm_, PyTorch 2.14 documentation — PyTorch, 26 Oct 2026
5. Machine Learning Interviews Book, 8.3 Training neural networks — Chip Huyen, 26 Oct 2026