What is layer normalization, and how does it work?

In this article

1. Before you start
2. Why interviewers ask this
3. Layer normalization from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits inside the transformer architecture overview and assumes only a little of the neural network basics.

Why interviewers ask this

Layer normalization appears in every block of a transformer, so a question about it is a quick test of whether a candidate understands the machinery around attention and not only attention itself. The question is often phrased as a contrast with batch normalization, or as "why do transformers not use batch norm?".

A weak answer says "it normalizes the activations to make training stable" and stops. A strong answer says which numbers are averaged (the features of one example, not the batch), writes the formula including the learned gain and bias, and explains the consequence: the layer behaves the same at training and test time and does not care about batch size. A Staff-level answer adds where the layer sits in the block and what that placement changes, and names the cheaper variant that drops the mean. The interview questions on this topic are reported by candidates in loops for machine-learning roles; the large language model (LLM) interview questions page collects neighbouring ones.

Layer normalization from first principles

Layer normalization

Layer normalization is a layer that computes the mean and variance of all the features of one example, subtracts the mean, divides by the standard deviation, and then applies a learned gain and bias, so its output does not depend on any other example in the batch.

Epsilon

Epsilon is a tiny constant added to the variance before the square root, so that the division stays defined when every feature of an example is equal and the variance is zero.

In plain words

Think of auto-levelling one photo using only that photo's own pixels. You shift its brightness so the average pixel sits in the middle. You stretch its contrast so the spread is standard. No other photo is consulted. That is layer normalization, with the features of one example playing the pixels. The analogy breaks in one place. Afterwards the network applies a learned gain and bias, a per-feature multiplier and offset, so it can undo or reshape the conversion.

In one precise sentence, layer normalization rescales a vector using the mean and variance of its own entries.

The formula

Layer normalization maps a vector $x$ to $\gamma ·(x-\mu )/\sqrt{{\sigma }^2+\epsilon }+\beta$, where $\mu$ and ${\sigma }^2$ are the mean and variance of the entries of $x$. Ba, Kiros and Hinton (2016) introduced it by transposing batch normalization: they compute the mean and variance "from all of the summed inputs to the neurons in a layer on a single training case".

Read it left to right: centre the vector, divide by its spread, then scale and shift each feature with learned numbers. The same formula is documented for undefined (version 2.14 of the PyTorch documentation), where the statistics run over the last dimensions of the input and the default epsilon is 1e-5.

Step 1: Compute the mean of the features

The mean is the sum of the features divided by their count. The input is one vector, such as the features of a single token. The output is one number. In the worked example below the vector is 1, 2, 3, 4 (illustrative), and its mean is 2.5.¹

The key point is which numbers are added. Layer normalization adds the features inside one example. It never adds numbers from a different example, and that is why the next example in the batch cannot change the answer. You now have the centre of this one example; the next step measures its spread around that centre.

Step 2: Compute the variance of the features

The variance is the average of the squared distances from the mean. For the vector 1, 2, 3, 4 the distances from 2.5 are -1.5, -0.5, 0.5 and 1.5, the squares are 2.25, 0.25, 0.25 and 2.25, and the variance is 5 divided by 4, which is 1.25.¹

Note the divisor. Layer normalization divides by the number of features, not by that number minus one, so it is the population variance, the form shown in the PyTorch documentation. The input to this step is the vector and its mean. The output is one number that says how spread out the features are. A large variance means the example's numbers were far apart. You now have both statistics, and only the division is left.

Step 3: Divide by the standard deviation, with a small epsilon

Each centred feature is divided by the square root of the variance plus epsilon, so the result has a standard deviation of about 1. For the example, the values become about -1.34, -0.45, 0.45 and 1.34 (illustrative input; epsilon of 1e-5, the PyTorch default).¹

The epsilon matters in one case. If every feature of an example is the same, such as 3, 3, 3, 3, the variance is 0 and the division would be by zero. With epsilon the denominator stays positive and the output is exactly 0 for every feature.¹ Epsilon also means the output variance is very slightly below 1, which is why the output is "about" unit variance and not exactly. The input to this step is the centred vector and the variance; the output is the normalized vector.

Step 4: Apply the learned gain and bias

After normalizing, each feature is multiplied by its own learned gain and shifted by its own learned bias. As Ba et al. (2016) put it, "we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity." With an illustrative gain of 0.5 and bias of 1 for every feature, the normalized values become 0.33, 0.78, 1.22 and 1.67.¹

These two vectors, the gain and the bias, are the only learned parameters of the layer. Without them the layer would force every output to the same shape and the network could not choose a different scale for a feature. They start at a gain of 1 and a bias of 0, the PyTorch defaults, so training begins from plain normalization.

1 · Mean2 · Variance3 · Divide4 · Gain and biasReplay

Step-by-step description

1. Average the features of one example.
2. Average the squared distances from the mean.
3. Divide by the standard deviation plus epsilon.
4. Scale and shift each feature.

• The mean of 1, 2, 3 and 4 is 2.5.
• The variance of 1, 2, 3 and 4 is 1.25.
• The normalized values are about -1.34, -0.45, 0.45 and 1.34.
• A gain of 0.5 and a bias of 1 give 0.33, 0.78, 1.22 and 1.67.

Worked example

The script behind this page normalizes three illustrative examples of four features each: 1, 2, 3, 4; then 2, 4, 6, 8; then 0, 1, 0, 1. It uses an epsilon of 1e-5, the PyTorch default, and a gain of 1 and bias of 0 unless stated.¹

Example 2 is example 1 multiplied by 2, and the layer-normalized outputs match to four decimals: dividing by the spread removes the scale.¹ The standard deviations before normalization are 1.118, 2.236 and 0.5, and after normalization all three are about 1.¹ The last row is arithmetic: a gain and a bias for each of 512 features (an illustrative size) is 2 times 512, which is 1,024 learnable numbers.¹

The batch-normalized row normalizes the same three examples down each feature column instead of across each row. Example 1 then gets 0, -0.2673, 0 and -0.1162, because its first and third features happen to equal their batch mean and the others sit slightly below it.¹ Change the other examples in the batch and that row changes; the layer-normalized row does not.

Text description of the diagram

• Before normalization the standard deviations are 1.118, 2.236 and 0.5.
• After normalization every example has a standard deviation of about 1.

Trade-offs and edge cases

Layer normalization trades the information carried by an example's overall scale for independence from the batch. The comparison with batch normalization is the one interviewers ask for.

Text description of the diagram

• Layer norm computes statistics over one example's features.
• Batch norm computes statistics across the batch for each feature.
• Layer norm performs the same computation at training and test time.

Layer norm computes statistics over one example's features, and batch norm computes statistics across the batch for each feature. Ba et al. (2016) note that batch normalization's effect "is dependent on the mini-batch size" and that it is not obvious how to apply it to a recurrent neural network (RNN), a network that reads a sequence one step at a time. They add that, unlike batch normalization, layer normalization "performs exactly the same computation at training and test times", and that for an RNN it can compute statistics separately at each time step.

In a transformer the layer is applied to each token's feature vector separately, so one token's result does not depend on how many other tokens or examples are in the batch. That is general reasoning from the definition, not a measured claim. It lets the same code run with a batch of one at serving time.

Placement is the second edge case. The original Transformer (Vaswani et al., 2017) adds a residual connection, and a residual connection is a shortcut that adds a sublayer's input to its output; it then applies layer normalization to the sum. That arrangement is called post-norm. Pre-norm instead normalizes the input of each sublayer and leaves the shortcut untouched. Xiong et al. (2020) show that post-norm has large gradients near the output at initialization, so it needs a learning-rate warm-up (a schedule that starts with a small learning rate), while pre-norm can train without one.

A lighter variant, root mean square normalization (RMSNorm), drops the mean subtraction and divides only by the root mean square of the features; Zhang and Sennrich (2019) propose it as a cheaper alternative that does not need re-centering. The sibling page on how LLaMA's normalization differs from the original Transformer's covers that contrast, and the self-attention and multi-head attention pages cover what sits inside each normalized block.

How to answer it in an interview

The spoken structure: say what is averaged, give the formula in words, name the consequence for batch size and test time, then the trade-off.

Common mistakes

• "Layer norm normalizes across the batch." (wrong) It normalizes across the features of each single example. (right) "The mean and variance come from one example's features; no other example is involved."
• "It behaves differently at inference." (wrong) It keeps no running statistics. (right) "It performs the same computation at training and test time."
• "After layer norm the output is exactly zero mean and unit variance." (wrong) The learned gain and bias rescale and shift it, and epsilon shrinks the variance slightly. (right) "Before the gain and bias it is about zero mean and unit variance."
• "Transformers use batch norm." (wrong) The original Transformer applies layer normalization after each residual addition. (right) "Transformers normalize each token's vector with layer norm."
• "Pre-norm and post-norm are the same thing in a different order." (wrong) Where the layer sits changes what the shortcut carries. (right) "Placement is a design choice that affects training behaviour, so I name which one a model uses."

Follow-up questions

What is the difference between batch norm and layer norm?

Batch norm averages each feature over the examples in the batch; layer norm averages the features of each example. Layer norm therefore ignores batch size and needs no running averages at test time.

Why do transformers use layer norm instead of batch norm?

Transformers process sequences of varying length and are often run with small batches or a batch of one, where batch statistics are awkward. Layer norm computes everything from one token's vector, so it behaves the same in every setting.

What is the difference between pre-norm and post-norm?

Post-norm applies layer norm after the residual addition, as in the original Transformer. Pre-norm applies it before each sublayer, inside the branch. They differ in training behaviour: post-norm needs a learning-rate warm-up, while pre-norm can train without one (Xiong et al., 2020). Say which one you mean.

What is root mean square normalization?

Root mean square normalization skips the mean subtraction and divides each feature by the root mean square of the vector, then applies a learned gain. Zhang and Sennrich (2019) propose it as a cheaper variant of layer norm, and LLaMA applies it to the input of each transformer sub-layer (Touvron et al., 2023); see the LLaMA comparison page.

Why is there an epsilon in layer norm?

Epsilon keeps the denominator above zero. If all features of an example are equal, the variance is 0, and without epsilon the division fails; with it the output is 0.¹

How many learnable parameters does layer norm have?

Two per normalized feature: one gain and one bias. For 512 features (an illustrative size) that is 1,024 numbers.¹ The transformer overview shows where these layers sit in a block.

Sources

1. Layer Normalization (arXiv 1607.06450) — arXiv (Ba, Kiros, Hinton), 21 Jul 2016
2. torch.nn.LayerNorm (PyTorch 2.14) — PyTorch documentation, 26 Oct 2026
3. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
4. On Layer Normalization in the Transformer Architecture (arXiv 2002.04745) — arXiv (Xiong et al.), 12 Feb 2020
5. Root Mean Square Layer Normalization (arXiv 1910.07467) — arXiv (Zhang and Sennrich), 16 Oct 2019
6. LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971) — arXiv (Touvron et al.), 27 Feb 2023