How do neural networks work?

In this article

1. Start here: the learning path
2. How does a neural network work, explained from zero?
3. Why do we need many layers if one non-linear unit can learn?
4. Why can't all the weights start at the same value?
5. Why do networks with more parameters than data still generalise?
6. What are the different types of neural networks?
7. Activation functions and numerical precision
8. Questions interviewers ask on deep learning
9. All 36 guides in Deep learning fundamentals
10. Follow-up questions
11. Sources

Start here: the learning path

Neural network

A neural network is a function made of stacked layers of units, each unit computing a weighted sum of its inputs plus an offset and then applying an activation function. Its behaviour is set entirely by its weights and offsets.

Weight

A weight is a number a unit multiplies one input by, and weights are the quantities training adjusts. A network's size is counted in its weights and offsets, together called parameters.

Activation function

An activation function is a non-linear function, such as the sigmoid or the rectified linear unit (ReLU), applied to a unit's weighted sum. Without it, any number of layers would collapse into one linear map.

Read the worked example under "How does a neural network work, explained from zero?" first: activation functions, numerical formats and backpropagation are parts of its training loop, and the network families are built from those parts. The gradient descent article covers the update rule and the cross-entropy loss article covers the loss most classifiers use.

Text description of the diagram

• Every other topic is one part of the training loop: activation functions, numerical formats and backpropagation.
• Convolutional networks, diffusion models and sequence models are network families built from these parts.

1. What are diffusion models? How they work, step by step
2. What are convolutional networks, and how do they work?
3. What are multimodal models, and how do CLIP and BLIP-2 work?
4. What are activation functions in a neural network?
5. FP16 vs BF16
6. What is backpropagation, and how does it work?
7. What are exploding gradients, and how do you fix them?
8. LSTM explained: how a long short-term memory cell works
9. What is batch normalization, and how is it different from layer normalization?
10. Debugging a model that will not learn: what to check first
11. Exploration vs exploitation: what is the difference?
12. What is your opinion about text diffusion?
13. What are the different types of neural networks?
14. Why is a deep neural network better than a shallow neural network?
15. What is layer normalization, and how does it work?
16. Why does softmax need numerical stabilization?
17. How would you optimize a neural network model?
18. What is RLHF, and how does it train a language model?

How does a neural network work, explained from zero?

A neural network works by computing a guess from its input, measuring how wrong the guess is, and nudging every weight in the direction that makes the guess less wrong, repeated many times. That is the whole trick. Four steps make it concrete, using a network small enough to check by hand.

The network. Two inputs feed two hidden units, which feed one output unit. Every unit applies a sigmoid, a smooth curve between 0 and 1. All numbers below are illustrative: input (1.0, 0.5), target 1, and starting weights chosen by hand.

Forward pass. The hidden units compute weighted sums of 0.30 and 0.15 and apply the sigmoid, giving 0.574 and 0.537. The output unit combines them into 0.115, and the sigmoid turns that into 0.529. The network guessed 0.529 where the target was 1¹.

Loss. The loss measures the gap. With half the squared difference, the loss is 0.111¹. For classification the usual choice is cross-entropy, but squared error keeps the arithmetic short.

Backward pass. Backpropagation applies the chain rule from the output back toward the input. The error signal at the output is −0.117. Multiplying by the output weights and the sigmoid's slope gives signals of −0.0086 and 0.0058 at the two hidden units, and each weight's gradient is its unit's signal times the input it carried¹. To check the algebra, a finite-difference test, which nudges each of the four first-layer weights by a tiny amount and measures the loss change, agreed with the backpropagated gradients to within $\approx 2.3\times {10}^{-11}$¹.

Update. Each weight and offset moves a small step against its gradient. With a step size of 1.0, one update lifts the output to 0.576 and cuts the loss to 0.090. After 100 updates the output is 0.943 and the loss is 0.0016¹. See gradient descent for how the step size is chosen and what can go wrong.

1 · Forward pass2 · Loss3 · Backward pass4 · UpdateReplay

Step-by-step description

1. The network turns the input into a guess, 0.529.
2. The loss measures the gap to the target, 0.111.
3. The chain rule gives every weight's gradient.
4. Each weight moves against its gradient; the loss drops to 0.090.

• The forward pass gives an output of 0.529 where the target was 1.
• The loss of the tiny network is 0.111 at the start.
• The error signal at the output is -0.117 and the backward pass applies the chain rule.
• One update with a step size of 1.0 cuts the loss to 0.090.

Text description of the diagram

• After 100 updates the output is 0.943 and the loss is 0.0016.
• After one update the output is 0.576 and the loss is 0.090.

Real networks do the same with many more weights. A network with 784 inputs, a hidden layer of 128 units and 10 outputs has 784 times 128 plus 128, which is 100,480, weights and offsets in the first layer, and 1,290 in the second, so 101,770 in total¹.

Why do we need many layers if one non-linear unit can learn?

Depth lets a network build complicated functions from simple ones, and for networks of rectified linear units (ReLU) there is evidence that depth adds more expressive power than width. Lu et al. (2017) show that ReLU networks of width $n+4$, with $n$ the input size, are universal approximators while width-$n$ networks cannot approximate most functions, and they conclude that their results give evidence that depth is more effective than width for the expressiveness of ReLU networks.

Depth also has a training cost. He et al. (2015, residual learning) note that deeper networks are more difficult to train and propose residual learning, which let them train 152-layer networks, about eight times deeper than Visual Geometry Group (VGG) nets, and reach 3.57% error on the ImageNet test set with an ensemble. Residual connections are also used inside the blocks of the models in the transformers guide.

Why can't all the weights start at the same value?

Identical starting weights make identical units, and identical units receive identical gradients, so they never differ. Take the tiny network with every input-to-hidden weight set to 0.2, both hidden-to-output weights set to 0.3 and every offset 0: each hidden unit's gradient row comes out as −0.00738 and −0.00369, and every update keeps the two units equal¹. Both the incoming and the outgoing weights must match; with output weights of 0.3 and −0.2, as in the example above, the two units would get different gradients and drift apart. Two units then do the work of one.

Random initialisation breaks the tie, and its scale matters. Glorot and Bengio (2010) found that sigmoid activations with random initialisation can push the top hidden layer into saturation, and proposed a scheme that sped up convergence. He et al. (2015, rectifier initialisation) designed an initialisation for rectifier units that made very deep rectified networks trainable from scratch. The activation functions article explains why saturation kills gradients.

Why do networks with more parameters than data still generalise?

Over-parameterised networks generalise well in practice, and classical theory did not predict it. Zhang et al. (2016) showed that standard image networks trained with stochastic gradient methods can fit a completely random labelling of the training data, and that explicit regularisation does not change this, so capacity alone cannot explain good test performance. Belkin et al. (2018) proposed a double-descent curve: test error rises as the model nears the point where it fits the training data exactly, then falls again as capacity grows further. Neither paper settles why large networks generalise, so treat it as an open research area.

What are the different types of neural networks?

The main families differ in how units are connected. A multilayer perceptron connects every unit in one layer to every unit in the next and is the network of this page. A convolutional network shares small filters across positions and suits images. Recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) process a sequence one step at a time, and the transformer replaces that with attention, covered in the transformers guide. Diffusion models generate images by learning to remove noise step by step, and multimodal models connect text and image encoders. Chapters 6 to 10 of the Goodfellow, Bengio and Courville book cover feedforward, regularisation, optimisation, convolutional and sequence networks in turn.

Activation functions and numerical precision

Activation functions give a network its bends, and numerical precision decides whether training survives. Sigmoid saturates at both ends, which is why ReLU and its relatives replaced it in hidden layers. The activation functions article covers ReLU, the Gaussian error linear unit (GELU), SwiGLU (a gated linear unit built on the Swish function) and a numerically stable softmax. A loss that turns to not-a-number is usually a precision or stability problem, which the fp16 versus bf16 article treats, including mixed-precision training. For how attention kernels are made fast, see FlashAttention.

Questions interviewers ask on deep learning

Deep learning interview questions come in a few shapes: explain a neural network as if I know nothing, say why depth helps, say what happens with identical initial weights, say what a non-linearity and a residual connection are for, and say why ReLU instead of sigmoid. The follow-ups below answer the common ones. For classical models see the classical machine learning guide.

Follow-up questions

Which type of neural network suits images, text or sequences?

Convolutional networks suit images because they share small filters across positions. Transformers suit text and other sequences because attention relates every position to every other, and recurrent networks and LSTMs process a sequence one step at a time. The transformers guide covers the one behind current language models.

Why is a deep neural network better than a shallow neural network?

Depth builds complex functions from simple parts, and Lu et al. give evidence that, for ReLU networks, depth is more effective than width for expressiveness. The cost is harder optimisation, which residual connections address.

What happens if all weights start at the same value?

When both their incoming and outgoing weights match, identical units get identical gradients and stay identical forever, so the layer behaves as a single unit. Random initialisation, scaled with a Glorot or He scheme, breaks the symmetry.

What sorts of layers would you use, and why ReLU instead of sigmoid?

Use linear or convolutional layers for the learned maps, a non-linearity after each so layers do not collapse into one, and residual connections for depth. ReLU is preferred in hidden layers because the sigmoid saturates, as Glorot and Bengio observed. See activation functions.

If the loss is NaN, what are the possible reasons?

Common causes are a learning rate that is too high, an overflow in a half-precision format, a log of zero in the loss, or bad input values. The fp16 versus bf16 article walks through the precision causes.

Sources

1. Understanding the difficulty of training deep feedforward neural networks — PMLR vol. 9, AISTATS 2010 (Glorot and Bengio), 13 May 2010
2. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (arXiv 1502.01852) — arXiv (He, Zhang, Ren, Sun), 6 Feb 2015
3. Deep Residual Learning for Image Recognition (arXiv 1512.03385) — arXiv (He, Zhang, Ren, Sun), 10 Dec 2015
4. Understanding deep learning requires rethinking generalization (arXiv 1611.03530) — arXiv (Zhang, Bengio, Hardt, Recht, Vinyals), 10 Nov 2016
5. Reconciling modern machine learning practice and the bias-variance trade-off (arXiv 1812.11118) — arXiv (Belkin, Hsu, Ma, Mandal), 28 Dec 2018
6. The Expressive Power of Neural Networks: A View from the Width (arXiv 1709.02540) — arXiv (Lu, Pu, Wang, Hu, Wang), 8 Sep 2017
7. Deep Learning (contents: Part II, Modern Practical Deep Networks) — MIT Press (Goodfellow, Bengio, Courville), 18 Nov 2016