Deep Learning Fundamentals

How would you optimize a neural network model?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

How would you optimize a neural network model? Lower its loss on data it has not trained on, as a sequence: baseline, overfit, regularize, tune, squeeze out gains, one change at a time. The step most often skipped is overfitting one small batch on purpose: if the loss cannot reach near zero there, the problem is a bug, and no amount of tuning fixes it.

In this article

Before you start
Why interviewers ask this
Optimizing a neural network from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

The basics are in how neural networks work, and the statistics underneath, such as variance and sampling, are in probability and statistics.

Why interviewers ask this

A list of tricks answers a different question from the one asked, so the follow-ups that come next expose it. "Use Adam, add dropout, make the model bigger" names tools but gives no method, and the interviewer cannot tell whether you have ever debugged a run that would not learn.

The question is open on purpose, and the order of your answer shows how you work. A weak answer lists every technique at once. A strong answer gives an order, says what each step proves, and says how you would know a change helped. A Staff-level answer also separates three jobs: making the training loss fall, making the validation loss fall, and making the trained model cheaper to run. Each job has its own tools, and mixing them is how teams waste weeks.

Optimizing a neural network from first principles

Weight decay: Weight decay is a regularization technique that pulls weights toward zero at each training step, through a penalty on large weights added to the loss or directly in the weight update, and reduces overfitting by discouraging the model from fitting noise in the training data.

In plain words

Think of a student preparing for an exam. Past papers are the training set. Memorizing their answers word for word is overfitting: excellent scores on old papers, poor scores on a new one. A fresh practice paper kept in a drawer is the validation set. The analogy breaks in one place. A student knows when they are memorizing, but a network gives no warning, so only the held-out score tells you.

In one precise sentence, optimizing a neural network is the search for weights and training settings that minimize the validation loss, not the training loss.

The order of work is data and baseline, overfit, regularize, tune, then squeeze out gains. Change one thing at a time so the effect of each change is clear. This is the shape of Karpathy's well-known recipe (Karpathy, 2019), which also stresses patience: "patience and attention to detail" correlate most with success in his experience (Cornell course print, 2019).

Step 1: Look at the data and build a dumb baseline

The first step is to read the data by hand and build the simplest model that runs end to end. Look at real examples, label counts and odd values, and check that no example sits in both the training set and the validation set. Then write the whole loop: load data, train, measure validation loss, and fix a random seed so two runs of the same code agree.

Karpathy's recipe says to pick "some simple model that you couldn't possibly have screwed up somehow" and to switch off unnecessary extras (Karpathy, 2019). Record the baseline's validation loss. Every later change is judged against that one number. You now have a trusted yardstick and a loop that works, so a worse result later points at your change and not at a hidden bug.

Step 2: Overfit one batch, then fit the full training set

Overfitting one batch on purpose is a test that the model, the loss and the training loop can learn at all. Take one small batch, turn off every regularizer, and train until the loss on that batch is close to zero. Karpathy's recipe lists "overfit one batch" (Karpathy, 2019).

If the loss refuses to fall, the cause is a bug or a setting: misaligned labels, a learning rate far too small or large, a loss on the wrong output, or a model too small. If it falls, the machinery works.

Then fit the full training set. Pick a known architecture rather than inventing one, start from a safe optimizer default (Karpathy suggests Adam at a learning rate of 3e-4, stage 3 of the recipe), and add complexity one piece at a time until training loss is low on the full set. High training loss there is underfitting, meaning the model is too weak or too briefly trained to fit even its own data. The fix is a better or larger model, a sane optimizer and longer training, not regularization.

Step 3: Regularize and stop at the right moment

Regularizing means making the model fit the training set a little less closely so it does better on the validation set. The options Karpathy lists include more data, data augmentation, pretraining, a smaller model, dropout, weight decay and early stopping (Karpathy, 2019). Add them one at a time. He calls augmentation "just another opportunity to introduce some dumb bug" if added too early, so it comes after the baseline works.

Weight decay is a penalty that grows with the size of the weights, pulling them toward zero; it is added to the loss for plain stochastic gradient descent (SGD), while AdamW applies it directly in the weight update (Loshchilov and Hutter, 2017). Dropout randomly switches off a share of units during training so no unit can be relied on alone. Early stopping is the rule of halting training when the validation loss stops improving and keeping the weights from the best epoch.

What early stopping looks like in numbers

Here are illustrative numbers. Validation loss is lowest at epoch 5, at 0.5, while training loss keeps falling to 0.06 at epoch 8.¹ The gap between them grows from 0.29 at the best epoch to 0.61 at the end, and validation loss at epoch 8 is 0.17 above its best.¹ With a patience of two epochs, training stops at epoch 7 and the weights from epoch 5 are restored.¹

When validation loss turns up while training loss keeps falling, the model has started to memorize.CC BY 4.0 — source: How would you optimize a neural network model?

Text description of the diagram

Validation loss is lowest at epoch 5 and then rises.
Training loss keeps falling to the last epoch.

Step 4: Tune the learning rate first, then the rest

Tuning starts with the learning rate, because it usually matters most: Goodfellow, Bengio and Courville (2016) write that it is "perhaps the most important hyperparameter". Steps 1 to 3 run at a sane default rate, and "first" means first among the settings searched in this step. The learning rate is the size of the step that gradient descent takes against the gradient.

Too small and training crawls; too large and the loss bounces or blows up. Stochastic gradient descent (SGD) takes that step on small batches of data. Variants such as momentum and Adam change how the step is built from past gradients, and optimizer states are the extra numbers they store. Most runs also decay the rate over training, often after a short warmup, and Karpathy's list warns against trusting learning-rate decay defaults.

A one-weight toy shows the effect. Let the loss be the square of one weight, starting at 1, so each step multiplies the weight by one minus twice the learning rate. After ten steps the loss depends on the rate as shown (illustrative toy, computed in the script).¹

A learning rate that is too small barely moves the loss, one near the sweet spot nearly removes it, and one above the stable range makes it grow.
Learning rate	Loss after ten steps	What happens
0.01	0.6676	slow progress
0.1	0.0115	steady progress
0.5	0.0	lands on the minimum
1.0	1.0	bounces, never improves
1.1	38.3376	diverges

Why random search beats grid search

For the other settings, such as width, weight decay and batch size, Karpathy's list says "random over grid search" (Karpathy, 2019). Bergstra and Bengio (2012) argue that only a few settings usually matter for a given problem. Random search tries a new value of every setting on every run, while a grid repeats the same few values of each, so random search covers the important ones more densely for the same budget. Layers such as batch normalization and residual connections make training easier to tune, and exploding gradients are the failure to check when the loss turns into a huge number.

Step 5: Squeeze out the last gains and make it cheap to run

The last step takes small, safe gains once the model is tuned. Karpathy's list ends with ensembles and longer training (Karpathy, 2019). An ensemble is several models whose outputs are averaged; it usually helps a little and multiplies the cost of serving.

A separate job is making the finished model cheaper to run. Mixed precision is training and running with half-width numbers where it is safe, which cuts memory, keeps full-precision master copies of the weights, and speeds up training on hardware with half-precision matrix units; 16-bit floating point (FP16) versus brain floating point 16 (BF16) covers the formats. Quantization is storing weights with fewer bits. Pruning is removing weights that matter little. Distillation is training a small model to copy the outputs of a large one. Each trades a little quality for speed or size, so measure validation loss again after each.

Optimization runs in a fixed order, and each step proves something the next step relies on.CC BY 4.0 — source: How would you optimize a neural network model?

Step-by-step description

Read the data and build a simple baseline.
Overfit a tiny set to prove it can learn.
Fit the training set a little less closely.
Tune the learning rate first.
Take the last small gains.

The order of work is data and baseline, overfit, regularize, tune, then squeeze out gains.
Change one thing at a time so the effect of each change is clear.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.