What is batch normalization, and how is it different from layer normalization?

In this article

1. Before you start
2. Why interviewers ask this
3. Batch and layer normalization from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the deep learning guide, which places normalization between a layer's weights and its activation function.

Why interviewers ask this

Normalization looks like a detail, but it is where training stability, batch size and train-versus-test behaviour meet. Interviewers ask the question to see whether you know which dimension each method averages over and what that implies when the batch is small, when the model is recurrent, or when the model is deployed. Reported in interviews at AI labs and large tech companies, it comes up in deep learning screens and in questions about why transformers use one method and image models often use the other.

A weak answer says both "normalize the activations". A strong answer names the axis, the inference behaviour, and the batch-size consequence.

Batch and layer normalization from first principles

In plain words

Batch normalization

Batch normalization is a layer that subtracts the mean and divides by the standard deviation of each feature, computed across the examples of the current mini-batch (in a convolutional layer the feature is a channel and the average also runs over every height and width position; for a sequence it also runs over every token), then applies a learned scale and shift. At inference it uses running statistics saved during training instead of the batch.

Running statistics

Running statistics are the moving averages of the batch means and variances that a batch normalization layer updates during training and then reuses to normalize at inference.

Imagine marking exam papers from several classes. Batch norm grades on a curve within each question. Every score on question 3 is compared with the other students' scores on question 3. Layer norm compares one student's scores across questions. It asks how far each question sits from that student's own average.

The analogy breaks at deployment. A curve needs a class, but a deployed model often scores one example at a time. So batch norm needs a stored class average.

In one precise sentence, both layers compute (x minus mean) over the square root of (variance plus a small epsilon), then scale and shift. They differ only in which numbers the mean and variance come from.

Step 1: Why activations are rescaled at all

Normalization keeps the numbers flowing through a deep network in a stable range, so training tolerates higher learning rates. Ioffe and Szegedy (2015) argued that the distribution of each layer's inputs changes during training as earlier layers change, which they called internal covariate shift, and that this slows training and forces careful initialization. They proposed normalizing layer inputs for each mini-batch and reported reaching the same accuracy on an image classifier with 14 times fewer training steps.

The input is a layer's activations. The output is the same activations rescaled to roughly zero mean and unit variance, so the next layer sees a steadier input. Internal covariate shift is the 2015 authors' explanation of why this helps, and a later paper disputes it (see the trade-offs and the follow-up questions). The two methods below differ in how they compute that rescaling.

Step 2: Batch norm averages across the examples in the batch

Batch norm computes one mean and one variance per feature, over every example in the mini-batch. In a convolutional layer the feature is a channel and the average also runs over every height and width position, and for a sequence it runs over every token, which is how one feature can have thousands of values behind its mean. The PyTorch 2.14 BatchNorm2d page gives the formula as (x minus mean) divided by the square root of (variance plus epsilon), times gamma, plus beta, with the mean and standard deviation calculated per dimension over the mini-batches and a default epsilon of 1e-5.

Gamma and beta are learnable vectors, one value per channel, that start at 1 and 0 so the network can undo the rescaling if that helps. Because the statistics come from the batch, each example's output depends on the other examples in it. The input is a batch of activations. The output is each feature rescaled across the batch.

Step 3: Layer norm averages across the features of one example

Layer norm computes one mean and one variance per example, over all of that example's features. Ba, Kiros and Hinton (2016) describe it as computing the statistics from all the summed inputs to the neurons in a layer on a single training case, so it does not depend on the mini-batch, and for recurrent networks it computes statistics separately at each time step.

The PyTorch 2.14 LayerNorm page says the statistics are computed over the last dimensions given by normalized_shape, with the same default epsilon of 1e-5. For a transformer's activations of shape batch, tokens, width, a normalized shape of the width gives one mean and variance per token.

The input is one example's activations. The output is that example rescaled across its own features, whatever else is in the batch.

Step 4: Training versus inference

Batch norm behaves differently in training and inference, and layer norm does not. During training, a batch norm layer keeps running estimates of the mean and variance, and in evaluation it normalizes with those estimates instead of the batch. The PyTorch page gives the update as the old estimate times (1 minus momentum) plus the new batch value times momentum, with a default momentum of 0.1 that is unrelated to the optimizer's momentum.

PyTorch also notes a detail that surprises people: the forward pass normalizes with the biased variance, while the value stored in the moving average uses the unbiased one. Layer norm needs no stored statistics: the LayerNorm page says it uses statistics computed from the input data in both training and evaluation modes. A forgotten switch to evaluation mode is a common batch norm bug; layer norm cannot have it.

The input is a new example at inference. The output is its normalized value, using stored statistics for batch norm and its own features for layer norm.

Step 5: Choosing between them

Choose by batch size and architecture. Batch norm works well when batches are large and every example in a batch is drawn from the same distribution, which is typical of image models with convolutional layers. Layer norm is preferred when batches are small, vary, or when the layer sees a sequence, because it needs no batch and, in the words of Ba and colleagues, does exactly the same computation at training and test times.

Two more families sit beside them. Group normalization is a layer that splits a layer's channels into groups and averages within each group of one example, and instance normalization averages over the height and width of one channel of one example. For small batches in convolutional models, Wu and He (2018) report group normalization reaching 10.6% lower error than batch norm at a batch size of 2 on ResNet-50 trained on ImageNet.

The input is your constraints, and the output is the method: batch statistics when batches are large and uniform, group normalization for small-batch convolutional models, and per-example statistics (layer norm) for sequences.

RMSNorm and pre-normalization

A third option for sequences is RMSNorm. Zhang and Sennrich (2019) proposed it as a variant that drops the mean subtraction, and reported comparable performance with running time reduced by between 7 percent and 64 percent across the models they tested. The root mean square (RMS) of a list is the square root of the average of its squares, and RMSNorm outputs x divided by the square root of (mean of x squared plus epsilon), times a learned gain.

For the example (2, 4, 6) below the RMS is 4.3205 and the output is 0.4629, 0.9258 and 1.3887¹, which is not centred on zero. Touvron et al. (2023) describe LLaMA as normalizing the input of each transformer sub-layer (pre-normalization) with RMSNorm.

1 · Rescale2 · Batch norm3 · Layer norm4 · Inference5 · ChooseReplay

Step-by-step description

1. Four examples, three features each.
2. Statistics per feature down a column.
3. Statistics per example along a row.
4. Batch norm swaps to running statistics.
5. Pick by batch size and architecture.

• Normalization rescales activations to roughly zero mean and unit variance.
• Batch norm computes one mean and variance per feature over the mini-batch.
• Layer norm computes one mean and variance per example over its features.
• Batch norm uses running statistics at inference and layer norm does not.
• Choose batch norm for large uniform batches and layer norm for sequences and small batches.

Worked example

Take a batch of four examples with three features each; these values are illustrative, computed in a script with the same formula and biased variance as PyTorch, with epsilon 1e-5¹. The rows are the examples: (2, 4, 6), (1, 3, 5), (3, 5, 7) and (2, 6, 10).

Batch norm works down the columns. The three feature means are 2.0, 4.5 and 7.0, and the variances are 0.5, 1.25 and 3.5. For the first example the normalized output is 0.0, minus 0.4472 and minus 0.5345¹. Layer norm works along the rows. The first example has a mean of 4.0 and a variance of 2.6667, so its normalized output is minus 1.2247, 0.0 and 1.2247, which has mean 0 and variance 1¹.

Now change the batch mates. In a second batch the first example is unchanged but the other three examples change in their first feature. Its first feature is 0.0 under batch norm in the first batch and minus 1.6713 in the second, so it moves by 1.6713, while under layer norm it stays at minus 1.2247 in both and moves by 0.0¹.

Text description of the diagram

• It moves by 1.6713 under batch norm when the batch mates change.
• Under layer norm it moves by 0.0.

The batch-of-one claim needs its qualifiers². In training mode, for a layer with no spatial or token positions, each feature's variance would be zero and the example would equal its own mean, so the normalized part is 0.0 and the output is just beta, which is 0 only at initialisation. PyTorch's BatchNorm1d does not even compute it: it raises ValueError ("Expected more than 1 value per channel when training").

A convolutional BatchNorm2d pools over height and width as well, so a single image still normalizes, using noisy statistics from that one image. At inference a batch of one is fine, because the stored running statistics are used. Layer norm, meanwhile, normalizes across the example's three features whatever the batch size. This is why batch norm degrades with tiny batches.

Running statistics converge slowly. Starting at 0 with a momentum of 0.1 and batches of 32 values drawn from a distribution with mean 5 (illustrative, seed 0), the running mean is 0.487 after one step, 2.047 after five, 3.237 after ten, 4.387 after twenty and 4.991 after fifty¹. A model evaluated too early, or trained for very few steps, normalizes with statistics that are still wrong.

Text description of the diagram

• The running mean is 0.487 after one step and 2.047 after five.
• It is 3.237 after ten steps, 4.387 after twenty and 4.991 after fifty.

Trade-offs and edge cases

Batch norm gives a strong training signal for large, uniform batches and acts as a regularizer, which Ioffe and Szegedy say can in some cases remove the need for dropout. Its costs are the batch dependence shown above, two code paths, and extra stored statistics. Layer norm removes all three costs and suits sequences, at the price of not exploiting cross-example statistics.

Three edge cases cause bugs. First, small or non-uniform batches: in training a batch of one is an error for a plain batch norm layer (a convolutional one gets noisy single-image statistics), and a batch of skewed examples gives noisy statistics. Second, evaluation mode: forgetting to switch a model to evaluation mode leaves batch statistics in use at test time, so predictions change with the batch. Third, distribution shift: running statistics reflect the training data, so a deployed model that sees different inputs is normalized with outdated numbers. For the sequence models behind these choices, see the transformers guide.

Researchers disagree on why batch norm works. The 2015 paper's explanation is internal covariate shift. Santurkar et al. (2018) argue that this is not the main reason and that batch norm instead makes the optimization landscape smoother, giving more predictable gradients. This page uses the 2015 account only as the authors' stated motivation.

How to answer it in an interview

The spoken structure is: say what both compute, name the axis that differs, then the inference behaviour, then your choice by batch size.

Common mistakes

• "Both layers normalize the same numbers" (wrong). Batch norm averages across examples per feature and layer norm across features per example (right: name the axis).
• "Batch norm works the same at inference" (wrong). It switches to running statistics, with a default momentum of 0.1 in PyTorch (right: say that it keeps running averages and needs evaluation mode).
• "Batch norm can be trained with a batch of one" (wrong). With no spatial positions the variance is zero and PyTorch raises an error; at inference a batch of one is fine because running statistics are used (right: a larger batch, group norm for convolutional models, layer norm for sequence models).
• "Layer norm has no learnable parameters" (wrong). It has a scale and, by default, a shift per element (right: 1,024 for width 512 with a bias).
• "Normalization removes the need for a good learning rate" (wrong). It tolerates higher rates, which is not the same as removing the choice (right: say it allows higher rates).

Follow-up questions

Does batch normalization improve accuracy?

It mainly speeds and stabilizes training, and the original paper reports reaching the same accuracy in 14 times fewer steps and a better final result for an ensemble on ImageNet. Whether the final accuracy improves depends on the model and data, so measure it on a validation set.

How are mean and variance obtained during training vs inference?

In training they are computed from the current mini-batch, and a moving average of them is saved. At inference the saved averages are used, so the output no longer depends on other examples. Layer norm computes them from the example itself in both modes.

When do you use batch norm vs layer norm and why?

Use batch norm for convolutional models with large, uniform batches, and layer norm for sequence models, recurrent layers and small batches, because it needs no batch. The reason is the axis: layer norm's statistics never leave the example.

Does batch norm really reduce internal covariate shift?

The 2015 authors say it does and that this explains the speedup. Santurkar and colleagues (2018) argue the gains come mainly from a smoother optimization landscape, so the question is still debated and a safe answer states both views.

What types of normalization are there and what are their functions?

Batch, layer, group, instance and root-mean-square normalization all rescale activations to stabilize training. They differ in which values the statistics come from: batch norm averages each feature over the batch, layer norm averages one example's features, group norm averages groups of channels within one example, instance norm averages one channel's height and width within one example, and RMSNorm scales by the root mean square without subtracting the mean. The classical machine learning guide covers feature scaling, which is a related idea applied to the inputs.

What happens to batch normalization with a batch of one?

It depends on the mode. In training, a layer with no spatial or token positions has zero variance, the normalized part is zero (the output is beta), and PyTorch raises an error. A convolutional layer still normalizes, with noisy single-image statistics, and at inference a batch of one works because running statistics are used. The fixes for training are a larger batch, group normalization for convolutional models, or layer norm for sequence models.

Sources

1. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (arXiv 1502.03167) — arXiv (Ioffe and Szegedy), 11 Feb 2015
2. Layer Normalization (arXiv 1607.06450) — arXiv (Ba, Kiros, Hinton), 21 Jul 2016
3. Root Mean Square Layer Normalization (arXiv 1910.07467) — arXiv (Zhang and Sennrich), 16 Oct 2019
4. torch.nn.BatchNorm2d, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
5. torch.nn.LayerNorm, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
6. How Does Batch Normalization Help Optimization? (arXiv 1805.11604) — arXiv (Santurkar, Tsipras, Ilyas, Madry), 29 May 2018
7. Group Normalization (arXiv 1803.08494) — arXiv (Wu and He), 22 Mar 2018
8. LLaMA: Open and Efficient Foundation Language Models (arXiv 2302.13971) — arXiv (Touvron et al.), 27 Feb 2023