In this article
Before you start
This page is one question inside the transformer architecture guide, and it builds on the basics in how neural networks work. For the full background on the operation being changed, see what layer normalization is and how it works.
Why interviewers ask this
The question is a quick test of whether a candidate has read a modern model's architecture rather than only the original paper. The original Transformer and LLaMA share most of their structure, so the normalization layer is one of the few places where a careful answer shows real knowledge, and it often leads into questions about training stability.
A weak answer says "LLaMA uses RMSNorm, which is faster." A strong answer writes both formulas and says what RMSNorm leaves out. A Staff-level answer also names the placement change, explains why it matters for gradients, and says what is and is not proven about the speed gain. The same question appears in lists of large language model (LLM) interview questions, usually next to rotary embeddings and attention variants.
Layer normalization from first principles
- RMSNorm
- RMSNorm is a normalization layer that divides each token's vector by the root mean square of its numbers and multiplies by a learned gain, with no mean subtraction and no bias, which makes it cheaper than layer normalization.
- Pre-norm
- Pre-norm is a block layout in which the normalization is applied to the input of each sub-layer, so the residual path carries the unnormalized sum straight through the block.
In plain words
Picture a class where every exam is marked on a different scale. One teacher grades out of ten. Another grades out of a thousand. Before comparing the papers you rescale each one to a common scale. Normalization does that to the numbers inside a network. No token's vector then arrives far larger or smaller than the next layer expects. The analogy breaks in one place: the network also learns a gain, so it can undo the rescaling where that helps.
In one precise sentence: both layers rescale every token's vector on its own. They differ in whether they first subtract the mean, and in where in the block they sit.
Step 1: Layer normalization centers and then rescales each token
Ba, Kiros and Hinton (2016) compute the statistics from the numbers of one example, so the result does not depend on the batch size. For a token vector, layer normalization subtracts the mean μ, divides by the standard deviation, then applies a learned gain g and bias b:
Here ε (epsilon) is a tiny constant that stops the square root from dividing by zero. The original Transformer uses this layer.
Take the vector [1, 2, 3, 4] with g = 1, b = 0 and ε ignored. Its mean is 2.5, its variance is 1.25 and its standard deviation is 1.118, so the output is [-1.3416, -0.4472, 0.4472, 1.3416].1 Input: one vector. Output: a vector with mean zero and variance one. You now know what the old layer does; the next step is where it sits.
Step 2: Pre-norm and warm-up, with the norm before each sub-layer
The original Transformer applies the norm after the residual add: each sub-layer's output is added to its input and the sum is normalized, written LayerNorm(x + Sublayer(x)) in Vaswani et al. (2017). LLaMA instead normalizes the input of each transformer sub-layer (the attention sub-layer, see multi-head attention, and the feed-forward sub-layer), with RMSNorm, as Touvron et al. (2023) describe. That layout is called pre-norm; the original is called post-norm. Touvron et al. credit the pre-normalization idea to generative pre-trained transformer (GPT)-3, so LLaMA adopted it rather than invented it.
Placement matters because of gradients. Xiong et al. (2020) show that with the norm before the sub-layer, gradients are well behaved at the start of training, so the learning-rate warm-up can be shortened or removed; with the norm after the add, gradients near the output are large. Xiong et al. study training stability, not final quality, so a claim about quality needs its own evidence.
Input: a sub-layer's input. Output: a sub-layer result added to the unnormalized path. You now know the second difference.
Step 3: RMSNorm drops the mean and the bias
Zhang and Sennrich (2019) hypothesize that the re-centering in layer normalization is dispensable. RMSNorm keeps only re-scaling, dividing by the root mean square (RMS), the square root of the average of the squared numbers:
The authors describe it as computationally simpler and more efficient than layer normalization.
For [1, 2, 3, 4] the mean of the squares is 7.5, the RMS is 2.7386 and the output is [0.3651, 0.7303, 1.0954, 1.4606].1 Unlike layer normalization, the output is not centered.
| Check on the toy vector | Result |
|---|---|
| Vector [-3, -1, 1, 3] (mean zero): the two outputs | identical, [-1.3416, -0.4472, 0.4472, 1.3416] |
| Multiply [1, 2, 3, 4] by 10 | both outputs unchanged |
| Add 100 to [1, 2, 3, 4] | layer normalization unchanged; RMSNorm gives [0.9853, 0.9951, 1.0048, 1.0146] |
| Norm layers in LLaMA-7B (two per block, 32 blocks, one final) | 65 |
| Learned values with a hidden size of 4,096 | RMSNorm 266,240; layer normalization with bias 532,480 |
The hidden size of 4,096 and the 32 layers of LLaMA-7B come from the model table of Touvron et al. (2023); the count of 65 norm layers assumes the usual layout of two per block plus a final norm.1 The saving is real but tiny against the billions of weights in the model.
Step-by-step description
- Layer normalization subtracts the mean first.
- The norm moves before the sub-layer.
- RMSNorm drops the mean and the bias.
- Layer normalization subtracts the mean, divides by the standard deviation, then applies a gain and a bias.
- The original Transformer normalizes after the residual add; LLaMA normalizes the input of each sub-layer.
- RMSNorm divides by the root mean square and applies only a gain.
The table below puts both differences side by side, which is also the shape of a good spoken answer.
Text description of the diagram
- RMSNorm has no mean subtraction and no bias.
- The original Transformer normalizes after the residual add; LLaMA normalizes before each sub-layer.
LLaMA also changes other parts of the block, such as rotary positional embeddings, but those are separate answers from the normalization question.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What are the differences between the Transformer's and LLaMA's layer normalization?
There are two. First, the formula: the original uses layer normalization, which subtracts the token's mean, divides by the standard deviation, then applies a gain and a bias. LLaMA uses RMSNorm, which only divides by the root mean square and applies a gain, so there is no mean and no bias, and it is cheaper. Second, the placement: the original normalizes after the residual add, LLaMA normalizes the input of each sub-layer, which keeps the residual path clean and gives better-behaved gradients early in training. The catch is that RMSNorm bets that re-centering is not needed.
What makes this Staff-level is that it names both differences, ties placement to training stability, and states the bet RMSNorm makes instead of calling it simply "faster".
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "LLaMA removed normalization." | LLaMA uses RMSNorm in place of layer normalization. | Also says RMSNorm is not shift-invariant. |
| Depth | Gives one formula. | Writes both formulas and names what is dropped. | Explains re-scaling versus re-centering invariance. |
| Trade-offs | "RMSNorm is faster." | Fewer operations and fewer learned values. | Says the gain depends on the setup: Zhang and Sennrich report 7%–64% less running time in their tests. |
| Communication | Lists facts in no order. | Formula first, then placement. | Ends with the bet the design makes. |
| Follow-up readiness | Stuck on "why pre-norm?" | Knows pre-norm makes warm-up less critical. | Cites the gradient analysis and its limits; warm-up is less critical, not gone. |
Three variants of this question
Why does LLaMA use RMSNorm instead of LayerNorm?
RMSNorm is computationally simpler, and Zhang and Sennrich (2019) report that the re-centering step can be removed without losing the benefits they test. Say it as a bet backed by experiments, not as a proof.
What is pre-norm, and why do LLaMA-style models use it?
Pre-norm normalizes the input of each sub-layer and leaves the residual path alone. Xiong et al. (2020) show it gives well-behaved gradients at initialization, which is why warm-up becomes less critical. It does not disappear: LLaMA still uses 2,000 warm-up steps (Touvron et al., 2023, section 2.3).
What if the input already has a mean of zero?
Then RMSNorm and layer normalization give the same output, as the toy vector [-3, -1, 1, 3] shows.1 The two differ only when the mean is not near zero.
Common mistakes
- "LLaMA removed layer normalization." (wrong) It still normalizes at every sub-layer. (right) "LLaMA replaces layer normalization with RMSNorm."
- "RMSNorm is just layer normalization with a different name." (wrong) It has no mean subtraction and no bias. (right) "RMSNorm keeps re-scaling and drops re-centering."
- "The only difference is the formula." (wrong) The placement also changed. (right) "The formula changed and the norm moved before each sub-layer."
- "RMSNorm makes LLaMA much faster overall." (wrong) The paper reports gains in specific setups. (right) "It is cheaper per norm layer; I would measure the end-to-end effect rather than quote a number."
Follow-up questions
Does RMSNorm have a bias term?
No. RMSNorm applies only a learned gain, which is why it has one learned value per feature per norm instead of two.
Is pre-norm always better than post-norm?
Not proven. Pre-norm trains stably with less warm-up, according to Xiong et al. (2020), but that is a statement about training stability, not about final quality.
How many parameters does LLaMA-7B spend on normalization?
With a hidden size of 4,096 and 65 norm layers, RMSNorm holds 266,240 learned values, against 532,480 if each norm also had a bias.1
Why is there a final norm before the output layer?
In a pre-norm stack the residual path is never normalized inside the blocks, so a last norm rescales it before the output layer reads it. See the transformer architecture guide for the full stack, and why attention scores are divided by sqrt(d_k) for another scale-control detail.
- The Transformer uses layer normalization after the residual add; LLaMA uses RMSNorm before each sub-layer.
- RMSNorm drops mean subtraction and the bias, keeping only re-scaling and a gain.
- With a zero mean the two layers give the same output; with a shifted input they do not.
- Pre-norm gives better-behaved gradients at the start, so warm-up matters less.
- Say the formula, then the placement, then the bet RMSNorm makes.
Sources
- Root Mean Square Layer Normalization (arXiv 1910.07467) — arXiv (Zhang and Sennrich), 16 Oct 2019
- LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971) — arXiv (Touvron et al.), 27 Feb 2023
- Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
- Layer Normalization (arXiv 1607.06450) — arXiv (Ba, Kiros, Hinton), 21 Jul 2016
- On Layer Normalization in the Transformer Architecture (arXiv 2002.04745) — arXiv (Xiong et al.), 12 Feb 2020
