What is a feed forward network, and how does it work?

In this article

1. Before you start
2. Why interviewers ask this
3. Feed forward network from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the deep learning learning path. A little probability helps, because backpropagation minimises an error that is often measured as a probability; the probability and statistics guide covers it.

Why interviewers ask this

The feed forward network is the building block under most of deep learning, so a gap here shows up in every later answer. A candidate who cannot say why a network needs an activation function will struggle with attention, normalisation and training questions that follow.

The question also appears in a second form: "what is the feed forward layer in a Transformer?" A weak answer says "it is a dense layer after attention." A strong answer says what it computes, that it works on each token separately, and what would break without the activation. A Staff-level answer adds the parameter count and where the layer sits next to attention and the residual path. The deep learning interview questions page lists neighbouring questions.

Feed forward network from first principles

Feed forward network

A feed forward network is a neural network in which information flows from the input, through intermediate computations, to the output, with no feedback connections that return outputs to the model. It approximates a function from inputs to outputs, as a chain of functions whose parameters are learned from examples.

Hidden layer

A hidden layer is a layer between the input and the output of a network. Its values are neither given as data nor read as the final answer; they are intermediate features that the network learns, and a network can have one hidden layer or many.

Position-wise feed forward layer

A position-wise feed forward layer is the sub-layer of a Transformer that applies the same small two-layer network to every token position separately. It does not move information between tokens, and one layer uses the same weights at every position.

In plain words

Picture an assembly line. Each station takes the parts it receives, combines them by a fixed recipe and passes the result on. Nothing ever travels backwards along the belt.

The analogy breaks in one place. In a network the recipes are numbers that the training process adjusts. A station does not follow a rule written by hand.

Precisely, a feed forward network is a chain of functions. Each takes the previous output as input. The same shape is also called a multilayer perceptron (MLP) in Goodfellow et al.. A recurrent neural network is a network with feedback connections that return outputs to the model, which gives it a memory of earlier inputs.

Step 1: A layer is a weighted sum plus a bias

A weight matrix is a grid of numbers that turns an input vector into a new vector, one weighted sum per output slot. A layer computes $z=xW+b$, where $x$ is the input, $W$ the weight matrix and $b$ a bias vector added at the end.

Input: a vector. Output: a new vector, possibly of a different length. Take the first layer of the small network used later, with weights of one and a bias of zero and minus one. For the input (1, 1) it gives (2, 1).¹

You now know what one layer does: it mixes the inputs by weighted addition. The next step shows why that alone is not enough.

Step 2: An activation function stops layers collapsing into one

An activation function is a simple nonlinear function applied to each number a layer produces. The rectified linear unit (ReLU) is $\max (0,z)$: it keeps positive numbers and turns negative ones into zero.

Without it, stacking does nothing new. Take the input (1, 2) and two weight matrices, [[1, 2], [3, 4]] and then [[1, 0], [1, 1]] (illustrative). Passing through both gives (17, 10). The single matrix [[3, 2], [7, 4]], their product, gives the same (17, 10).¹ Two linear layers are one linear layer in disguise; with biases counted, the pair is one affine layer.

A nonlinearity between the layers breaks that equivalence. The activation functions page compares the common choices.

Step 3: Stacked layers with an activation learn curved patterns

A network feeds each layer's output into the next, so hidden layers build on each other. The network used below has two inputs, two hidden units with ReLU and one output. That is nine learnable numbers: four weights and two biases in the first layer, then two weights and one bias in the second.¹

Input: two numbers. Output: one number. The task is XOR: the output is one when exactly one input is one. Goodfellow et al. open chapter 6 with this task to make the idea of a feed forward network concrete.

"Feed forward" describes the direction of data flow, not the way of training. Weights are learned with backpropagation, which sends error signals backwards during training only.

Step 4: In a Transformer, the same idea runs on each token

In a Transformer, Vaswani et al. (2017) put a fully connected feed-forward network in each layer, "applied to each position separately and identically". It is two linear maps with a ReLU between:

The input and output size is $d_{model}$ = 512 and the inner layer is $d_{ff}$ = 2048, four times wider. The weights are shared across positions in one layer, but each layer has its own. The paper also describes the block as two convolutions with kernel size one. Its output joins the input through a residual connection and layer normalisation, written $\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x))$; see batch norm vs layer norm.

1 · Weighted sum2 · Activation3 · Second layer4 · OutputReplay

Step-by-step description

1. The first layer computes a weighted sum.
2. ReLU keeps positives and zeroes negatives.
3. A second weighted sum follows.
4. One output vector per input token.

• A feed forward block has a first linear layer, a ReLU activation and a second linear layer.
• The activation sits between the two layers.
• The output has the same size as the input.

Worked example

Take the nine-number network of Step 3 with the solution given in Goodfellow et al., section 6.1: first-layer weights all one, first-layer biases (0, −1), output weights (1, −2) and output bias zero. The forward pass is computed for all four inputs.¹

For (1, 1) the hidden values are 2 and 1, and the output is 2 × 1 + 1 × (−2) = 0.¹ For (0, 0) the second hidden value is max(0, −1) = 0, so ReLU cleans it; without ReLU the output is 2.¹

Now the real block. With the sizes of Vaswani et al., $W_1$ has 1,048,576 numbers, $W_2$ has 1,048,576, and the biases add 2,048 and 512.¹

The stack of N = 6 layers is from the same paper, so six layers hold 12,598,272 parameters, growing by 2,099,712 per layer.¹

Text description of the diagram

• One feed forward block holds 2,099,712 parameters.
• Six layers hold 12,598,272 parameters.

Trade-offs and edge cases

A feed forward network has no memory. It reads a fixed-size input and forgets it, so on its own it cannot follow a sequence. Adding feedback connections gives a recurrent network; the long short-term memory (LSTM) network is one such design. Transformers avoid recurrence by letting attention move information between tokens.

Hornik, Stinchcombe and White (1989) show that multilayer feedforward networks can approximate a wide class of functions to any accuracy given enough hidden units, though that says nothing about how easily training finds the weights. A fully connected layer treats every input number alike. For images, convolutional networks reuse small filters instead, which suits grid data better. A convolutional network is itself a feed forward network, because its data also flows one way with no feedback; it only restricts which weights exist.

Self-attention is the first sub-layer of an encoder layer and the feed forward layer is the second. Attention mixes information across tokens, while the feed forward layer is applied to each position separately. Both sub-layers sit inside a residual connection and LayerNorm.

Text description of the diagram

• Self-attention is the first sub-layer of an encoder layer and the feed forward layer is the second.
• The feed forward layer is applied to each position separately.
• Both sub-layers sit inside a residual connection and LayerNorm.

Cost scales with width. With $d_{ff}$ kept at four times $d_{model}$, doubling $d_{model}$ from 512 to 1024 gives 8,393,728 parameters, about four times the 2,099,712 of the smaller block.¹ Depth also matters: very deep stacks can suffer exploding gradients. The block above is the 2017 form, and it has moved in two ways. First, gated layers: Shazeer (2020) replaces ReLU with gated variants such as the gated linear unit with a Swish activation (SwiGLU). These use three weight matrices instead of two, and the hidden width is cut to about two-thirds to keep the parameter count equal.

The Llama language models of Touvron et al. (2023) use SwiGLU with a hidden size of 2/3 of 4d instead of 4d. Second, mixture-of-experts: Fedus et al. (2021) replace the single feed forward layer with many expert feed forward layers, and a router sends each token to one of them. For scale, in the 2017 layer the feed forward block holds 2,099,712 parameters and the four attention projections hold 1,050,624, so the feed forward block has about two-thirds of the sub-layer parameters (layer normalisation excluded).¹

How to answer it in an interview

The spoken structure: define it, say what one layer does, say why the activation is needed, then name the Transformer use.

Common mistakes

• "Feed forward means there is no backpropagation." (wrong) The name describes the direction of data flow, not training. "Data flows forward; backpropagation trains the weights." (right)
• "Deeper linear layers can learn more." (wrong) They collapse into one matrix. "I need a nonlinearity between layers." (right)
• "The Transformer feed forward layer mixes tokens." (wrong) Each position is processed separately. "Attention mixes tokens; the feed forward layer does not." (right)
• "All layers share the same feed forward weights." (wrong) Weights are shared across positions within a layer, not across layers. "Each layer has its own parameters." (right)

Follow-up questions

What is the difference between a feed forward network and a recurrent network?

A feed forward network has no feedback connections. A recurrent neural network adds connections that return outputs to the model, which gives it a memory of earlier inputs.

What happens if you remove the activation function?

The layers multiply into one matrix, so the whole network is one linear map. In the worked example the XOR network then outputs 2 for the input (0, 0) instead of 0.

Does the feed forward layer in a Transformer mix tokens?

No. Vaswani et al. apply it to each position separately and identically. Self-attention is the sub-layer that moves information between positions.

Is a feed forward network the same as a multilayer perceptron?

In Goodfellow et al. the two names refer to the same family, along with feedforward neural networks. A multilayer perceptron is usually fully connected.

What do modern LLMs use instead of ReLU in the feed forward layer?

Many use a gated variant such as SwiGLU. Shazeer (2020) describes gated layers with three weight matrices instead of two, and the hidden width is cut to about two-thirds so the parameter count stays the same.

What if the model dimension doubles?

With the hidden width kept at four times the model dimension, the block's parameters grow about four times, from 2,099,712 to 8,393,728.¹

Sources

1. Attention Is All You Need (arXiv 1706.03762), section 3.3 Position-wise Feed-Forward Networks — arXiv / NeurIPS 2017 (Vaswani et al.), 12 Jun 2017
2. Deep Learning, chapter 6: Deep Feedforward Networks — MIT Press (Goodfellow, Bengio and Courville), 18 Nov 2016
3. Multilayer feedforward networks are universal approximators — Neural Networks 2(5), 1989 (Hornik, Stinchcombe and White), 1 Jan 1989
4. GLU Variants Improve Transformer (arXiv 2002.05202) — arXiv (Shazeer), 12 Feb 2020
5. LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971) — arXiv (Touvron et al.), 27 Feb 2023
6. Switch Transformers: Scaling to Trillion Parameter Models (arXiv 2101.03961) — arXiv (Fedus, Zoph, Shazeer), 11 Jan 2021