Why is a deep neural network better than a shallow neural network?

In this article

1. Before you start
2. Why interviewers ask this
3. Deep versus shallow networks from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Start from the deep learning learning path if the basics are new.

Why interviewers ask this

Candidates who say "deeper networks are more powerful" lose the point on the next question. The question separates people who memorized a slogan from people who know what depth buys. It is reported by candidates as a deep learning basics question, and it leads straight to the universal approximation result and to the cost of training deep networks.

A weak answer says that deep networks learn more complex features. A strong answer admits that a single hidden layer can already approximate any continuous function on a bounded domain, then says the difference is size: a shallow network may need exponentially many units. A Staff-level answer shows the count, explains why layers multiply while units add, and says what the theory does not prove about training real data. For more practice questions in this area, see deep learning interview questions and answers.

Deep versus shallow networks from first principles

Shallow neural network

A shallow neural network is a network with exactly one hidden layer between its input and its output, so it computes a weighted sum of simple bent functions, each of one weighted combination of the inputs; it is the standard comparison case for deep networks.

Depth separation

Depth separation is a proven gap in which a function that a small network with more layers computes cannot be approximated by any network with fewer layers unless that shallower network is exponentially wider, which makes depth more valuable than width for that function.

Linear piece

A linear piece is a stretch of the input over which the output of a rectified linear unit (ReLU) network is a straight line, so the number of pieces counts how many times the learned curve can bend; more pieces mean a more flexible function.

In plain words

Think of describing pictures with words. A shallow network is like a dictionary with one separate entry for every possible picture. A deep network builds letters into words, words into sentences, and reuses each part many times. The analogy breaks in one place: a real network learns its own parts, and nothing forces them to be as tidy as letters and words.

Put precisely, depth lets a network compose simple functions. The number of distinct shapes it can make then grows multiplicatively with layers, while adding units to one layer grows it additively.

Step 1: One hidden layer can already approximate any continuous function

Universal approximation is the proven fact that a network with one hidden layer and a reasonable activation function can approximate any continuous function on a bounded domain, to any accuracy. Eldan and Shamir (2016), introduction cite Cybenko (1989) and Hornik and colleagues (1989) for it.

So depth is not needed for what a network can approximate in principle. The catch, in the same introduction, is that the required size of such a network can be exponential in the input dimension. The question is therefore about efficiency: how many units and parameters you need. You now know the real claim: depth is about cost, not possibility.

Step 2: Width adds bends one at a time

Take the simplest case, one number as input. The unit used here is called ReLU (rectified linear unit): it outputs its input when the input is positive and zero otherwise, so each unit contributes one bend. A one-hidden-layer ReLU network with n units is a sum of n bends, so its output has at most n + 1 linear pieces.

Input: n units. Output: at most n + 1 straight segments. Seven units give at most 8 pieces.¹ Doubling the width roughly doubles the bends. That is the best a single layer can do, and it matters in the next step, where depth does something much faster.

Step 3: Each extra layer doubles the bends

The tent map is the function that rises from 0 to 1 over the first half of the interval and falls back to 0 over the second half. One tent needs two ReLU units: 2·ReLU(x) − 4·ReLU(x − 0.5) on the interval from 0 to 1.¹ Feed the output of one tent into another and each of the two pieces of the first tent is itself squeezed into a tent, so the pieces double.

Input: the previous layer's curve. Output: twice as many pieces, for two more units. Three layers cost 6 units and give 8 pieces, and the count was checked by composing the function exactly for up to 10 layers.¹ This is the sawtooth style of construction used in depth-separation arguments; Telgarsky (2016) proves related results for networks with many layers and only a constant number of nodes per layer. Layers multiply; units add.

Step 4: The same sawtooth costs exponentially more when shallow

A sawtooth with 2^L pieces comes from L layers and 2L units. A one-hidden-layer network with n units has at most n + 1 pieces, so reproducing the sawtooth exactly needs at least 2^L − 1 units. At 20 layers the deep network has 40 units and the sawtooth has 1,048,576 pieces; the shallow network needs at least 1,048,575 units.¹ At 2 layers the shallow network is cheaper, 3 units against 4, so the crossover comes early and the gap then widens fast.¹

In higher dimensions, Eldan and Shamir (2016) show that a simple function from a small 3-layer network (two hidden layers) cannot be approximated by any 2-layer network unless its width is exponential in the dimension. Depth, "even if increased by 1 – can be exponentially more valuable than width". These are theorems about constructed functions, not about training on real data.

1 · One layer2 · Width3 · Depth4 · Same countReplay

Step-by-step description

1. Each unit adds one bend.
2. Seven units give at most eight pieces.
3. Each tent layer doubles the pieces.
4. Twenty layers need 40 units, shallow needs about a million.

• A one-hidden-layer ReLU network with n units has at most n + 1 linear pieces.
• Seven units give at most 8 pieces.
• Three tent layers use 6 units and give 8 pieces.
• Twenty layers use 40 units and give 1,048,576 pieces, while one hidden layer needs at least 1,048,575 units.

Text description of the diagram

• From 2 to 10 layers the deep network needs 4 to 20 units.
• One hidden layer needs at least 3 units at 2 layers and at least 1,023 units at 10 layers.

The chart shows the same arithmetic: from 2 to 10 layers the deep network needs 4 to 20 units, and one hidden layer needs at least 3 to 1,023.¹ The sawtooth is a toy chosen to make the effect visible. It is illustrative of the mechanism, not a measurement of any trained model.

The Staff-level answer

What makes it Staff-level: it concedes universal approximation first, gives a count that can be checked, and names the limit of the theory before the interviewer asks.

The rubric

Three variants of this question

Why not just make a shallow network wider?

Width works in principle, but the units needed can be exponential for some functions. To reproduce the sawtooth exactly, the one above needs at least 1,048,575 units shallow against 40 deep.¹ A wide network is also a poor reuser of features: every piece is built separately.

Is a deeper network always better?

No. The theorems say depth can be exponentially cheaper for some functions, not that every added layer helps. Deeper networks are harder to optimize, and residual connections exist largely to make depth trainable.

Does depth matter for real data such as images?

A common intuition is that deep image models build features in stages, from simple local patterns to larger shapes; this is an intuition for feature reuse, not something the theorems prove. See convolutional networks. How a trained model generalizes is a statistical question, and the statistics for machine learning (ML) and data roles guide gives the background.

Common mistakes

• "A shallow network cannot represent complex functions." (wrong) Universal approximation says one hidden layer can approximate any continuous function on a bounded domain. Say instead: "A shallow network can approximate it, but may need exponentially many units." (right)
• "More layers is always better." (wrong) Training gets harder with depth, so the benefit is not automatic. Say instead: "Depth is cheaper for some functions, and I pay for it in optimization difficulty." (right)
• "The theorem proves deep networks generalize better." (wrong) The results concern approximating specific functions, not training on real data. Say instead: "They show a representation gap; generalization is a separate, empirical question." (right)
• "Depth and width are interchangeable." (wrong) Eldan and Shamir (2016) show depth can be exponentially more valuable. Say instead: "Adding a layer can beat adding a huge number of units." (right)

Follow-up questions

What happens if a deep network has no activation functions?

It collapses into a single linear map, because a product of linear layers is one linear layer, so depth adds nothing. The bends from activation functions are what let layers compose into new shapes.

Why do very deep networks need residual connections?

Plain deep stacks are harder to optimize, and a residual connection adds a shortcut around a block so the signal and gradient can pass through. The page on residual connections covers why.

Is it better to add width or depth in practice?

The theory favours depth for some functions, but practice balances the two: depth costs optimization difficulty, such as the growing gradients covered on the exploding gradients page, while width costs parameters and memory. Teams compare sizes on held-out data rather than trusting a rule.

What changes when the input has many dimensions?

The width a shallow network needs can grow exponentially with the input dimension, which is the case Eldan and Shamir (2016) analyze. See high-dimensional data for why dimension makes problems hard.

Sources

1. The Power of Depth for Feedforward Neural Networks (arXiv 1512.03965) — arXiv (Eldan and Shamir), 23 Jun 2016
2. The Power of Depth for Feedforward Neural Networks — PMLR, COLT 2016 (Eldan and Shamir), 23 Jun 2016
3. Benefits of depth in neural networks — PMLR, COLT 2016 (Telgarsky), 23 Jun 2016