What is dropout in a neural network? Dropout explained

In this article

1. Before you start
2. Why interviewers ask this
3. Dropout from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the deep learning learning path.

Why interviewers ask this

A candidate who says that dropout "turns off neurons" has given half an answer. Interviewers ask about dropout because it separates a memorized definition from an understanding of how training differs from prediction: what is random, what is scaled, and what is switched off afterwards.

A weak answer stops at "it reduces overfitting." A strong answer gives the mechanism, the scaling rule and the fact that dropout is off at test time. A Staff-level answer adds the costs and failure modes: slower training, lost capacity, underfitting when the rate is too high, and the keep-versus-drop confusion between the paper and the libraries. Neighboring questions on regularization and training are collected in the deep learning interview questions and answers.

Dropout from first principles

Dropout

Dropout is a regularization method that, on each training step, randomly sets each unit in a layer to zero with a fixed probability, so the network trains as a different, smaller network every time and units cannot lean on specific partners. It is switched off when the trained network makes predictions.

Thinned network

A thinned network is the smaller network that remains when dropout removes some units, along with their connections, for one training step. A network with n units can produce two to the power n different thinned networks, and all of them share the same weights.

Inverted dropout

Inverted dropout is the version of dropout that multiplies the kept activations by one over the keep probability during training, so no weight needs rescaling at test time and prediction is a plain pass through the layer.

In plain words

Picture a team where a random half of the members stay home each day. Nobody can count on one particular colleague, so every member has to become useful on their own. That is the idea of dropout.

The analogy breaks in two places. The absent members come back on the next step, and the whole team always attends on test day.

Regularization is any change to training that reduces overfitting. Co-adaptation is the habit of one unit being useful only when a few specific other units are present. Hinton et al. (2012) described dropout as "randomly omitting half of the feature detectors on each training case" and said it prevents complex co-adaptations. In one precise sentence: during training each unit is multiplied by a random zero-or-one mask and the survivors are scaled, and at test time the mask is removed.

Step 1: Flip a coin per unit, and know which p you mean

Each unit is kept with probability p, independently of other units. This p is the keep probability, the chance that a unit survives a given training step. A Bernoulli draw is a random choice that gives one with probability p and zero otherwise, and that coin flip decides each unit.

In the original paper (Srivastava et al., 2014, p. 1930), p is the retain probability. The authors report that 0.5 is close to optimal for hidden units across a wide range of networks, and that for real-valued inputs a typical value is 0.8 (App. A.4). The libraries flip the meaning. In PyTorch the argument p is the probability of zeroing an element, and in Keras the argument rate is the fraction of input units to drop.

The paper's p is the keep probability, but PyTorch's p and Keras's rate are the drop probability. Input dropout of 0.8 kept in the paper is 0.2 dropped in PyTorch, while 0.5 means the same in both conventions.¹

Text description of the diagram

• The paper's p is the keep probability, but PyTorch's p and Keras's rate are the drop probability.
• Input dropout of 0.8 kept in the paper is 0.2 dropped in PyTorch.

Step 2: Zero the dropped units and train the thinned network

A dropped unit outputs zero for that training step. The layer's output becomes the unit activations multiplied element by element by the mask, and the next layer receives that thinned vector, as in any feed-forward network.

Backpropagation then runs only through the sampled sub-network, so only the weights of kept units get an update on that step (Srivastava et al., 2014, Sec. 4 and 5.1). A new mask is drawn for each forward call, so the choice is not permanent (PyTorch documentation).

Because a network with n units can be seen as a collection of two to the power n thinned networks that share weights, training samples a different one at almost every step. With n equal to 10 that is 1,024 networks, and the count explodes for wide layers.¹ No unit can rely on a particular partner being present, which is the mechanism behind the drop in co-adaptation.

Step 3: Scale the kept outputs so training and testing agree

Kept outputs are scaled by one over the keep probability, so the average output of a layer stays the same with or without dropout. Without scaling, the next layer would see a smaller total input during training than at test time, when every unit is present.

The paper gives two forms, equivalent with an appropriately scaled learning rate and initialization. In the original form, a unit kept with probability p has its outgoing weights, the entries of the weight matrix that leave that unit, multiplied by p at test time (p. 1931). In the second form, the one now called inverted dropout, the retained activations are multiplied by 1/p during training and the weights are left alone at test time (Srivastava et al., 2014, Sec. 10).

Libraries use the second form. PyTorch scales the outputs by 1/(1-p) during training, where p is its drop probability, and Keras scales the inputs that are not set to 0 by 1/(1-rate) so that the sum over all inputs is unchanged.

Step 4: Switch dropout off at test time

At test time dropout does nothing and every unit is on. The paper's abstract says that the effect of averaging the predictions of all the thinned networks is approximated by "a single unthinned network that has smaller weights."

In PyTorch the module computes an identity function during evaluation, so you call model.eval() before predicting. In Keras the layer drops values only when training is true, and the documentation adds that trainable does not change the behavior of Dropout because the layer has no weights to freeze (Keras documentation).

You now have the whole mechanism: a mask per step, a thinned network trained, survivors scaled, and a full network at the end.

1 · Flip a coin2 · Train thinned net3 · Scale survivors4 · Test timeReplay

Step-by-step description

1. Each unit is kept with probability p.
2. Dropped units output zero this step.
3. Kept outputs are scaled up.
4. No dropout, every unit is on.

• Each unit is kept with probability p, independently of other units.
• A dropped unit outputs zero for that training step.
• Kept outputs are scaled by one over the keep probability.
• At test time dropout does nothing and every unit is on.

Worked example

Take one unit with activation 4 and a keep probability p of 0.8, both illustrative. The scale is 1/0.8 = 1.25, so a kept unit outputs 5 and a dropped unit outputs 0. The average output, weighted by probability, is 0.8 times 5 plus 0.2 times 0, which is 4, the original activation.¹ The variance of the output is 4, which matches the formula activation squared times (1 minus p) over p.¹ A seeded check over 10,000 masks, exactly 8,000 of them keeping the unit, reproduces a mean of 4 and a variance of 4.¹

Text description of the diagram

• A kept unit outputs 5 and a dropped unit outputs 0.
• The average output, weighted by probability, is 4, the original activation.

Now a layer of four units with illustrative activations 4, 2, 1 and 3, a keep probability of 0.5 and a scale of 2. There are 16 possible masks. Averaged over all 16, each unit's output equals its activation, so the expected sum is 10, and the sum varies from mask to mask with a variance of 30.¹

The original-paper form agrees. With keep probability 0.5, a weight of 2 and an input of 3 (illustrative), the expected training contribution is 0.5 times 2 times 3 = 3, and the test-time weight p times w = 1 gives 1 times 3 = 3.¹

The width rule comes from the paper: if an n-sized layer is optimal for a standard network, a good dropout network should have at least n divided by p units (Srivastava et al., 2014, App. A.1). With 512 and p = 0.5 that is 1,024 units.¹

Trade-offs and edge cases

Dropout trades training time and capacity for less overfitting. Training takes about 2 to 3 times longer than for a standard network of the same architecture, which the authors call a trade-off between overfitting and training time (Srivastava et al., 2014, Sec. 11 (Conclusion)). The paper reports that dropout also works best with max-norm, a high learning rate and high momentum: it suggests 10 to 100 times the learning rate of a standard network and a max-norm constant c between 3 and 4 (App. A.2 and A.3).

The paper's results show where the gain is large and small.

The street-number result also corrects a common belief: dropout in the fully connected layers only gave 3.02%, and adding it to the convolutional layers gave 2.55% (Srivastava et al., 2014, p. 1938). Dropout was also part of the model that won the 2012 ImageNet large-scale recognition challenge, with a top-5 test error of about 16% against about 26% for standard-feature methods (p. 1940).

Edge cases follow from the same paper. Too much dropout underfits: at small keep probabilities both training and test error are high; test error is flat for keep probabilities between 0.4 and 0.8 and rises again as p approaches 1, where there is too little dropout (Srivastava et al., 2014, Sec. 7.3). On extremely small data sets, 100 or 500 examples, dropout gave no improvement (Sec. 7.4). On the text set above, the authors say the gain was much smaller because the data set is big and overfitting is not a serious problem (Sec. 6.3).

Current practice in very large models differs from the 2014 recipe: PaLM was pretrained without dropout and used a dropout rate of 0.1 for most fine-tuning (Chowdhery et al., 2022, Sec. 4).

Dropout also interacts with layers that change activation statistics, such as batch normalization. Li et al. (2018) show that dropout placed before batch normalization shifts the variance of activations between training and test, and Ioffe and Szegedy (2015) report that batch normalization can reduce the need for dropout.

How to answer it in an interview

The spoken structure: say what is random, say what is scaled, say that it is off at test time, then name the cost and the keep-versus-drop trap.

Common mistakes

• "Dropout is also applied at inference." (wrong) Dropout is disabled in evaluation, and the layer becomes an identity. Keeping dropout on at inference is wrong for ordinary prediction; Monte Carlo dropout does it on purpose to estimate uncertainty (Gal and Ghahramani, 2016). (right) "I call model.eval() in PyTorch, or pass training=False in Keras, before predicting."
• "p means the same thing everywhere." (wrong) The paper's p is the keep probability, while PyTorch's p and Keras's rate are the drop probability. (right) "I read the documentation and convert: keep 0.8 is drop 0.2."
• "Dropout permanently removes neurons, so it prunes the model." (wrong) Units are dropped for one step and a fresh mask is drawn every forward pass. (right) "Every unit is back for the next step; none is removed from the trained model."
• "Freezing a layer turns dropout off." (wrong) In Keras, trainable does not affect the Dropout layer; the training flag does. (right) "I control dropout with the training flag."
• "More dropout always means more regularization that helps." (wrong) Too high a rate underfits, and tiny data sets gain nothing. (right) "I tune the rate on a validation set and widen the layer when I raise it."

Follow-up questions

What is the difference between dropout and L2 regularization?

L2 regularization adds a penalty on the size of the weights to the loss, while dropout injects random noise into the activations. For linear regression the two are linked: dropout is equivalent, in expectation, to ridge regression with a particular penalty (Srivastava et al., 2014, Sec. 9.1). On the handwritten-digit benchmark the paper reports 1.62% error for L2, 1.35% for max-norm, 1.25% for dropout with L2 and 1.05% for dropout with max-norm (Table 9), so the two can be combined.

Does dropout help convolutional layers or only fully connected layers?

It helps both. In the paper's street-number experiment, dropout in the fully connected layers lowered the error from 3.95% to 3.02%, and dropout in all layers lowered it to 2.55%.

Why does dropout slow training, and what do you change to compensate?

Each step trains a noisy, thinned network, so the gradient signal is noisier and convergence needs more steps; the paper reports 2 to 3 times longer training. It suggests a wider layer (at least n divided by p units), a much higher learning rate and a max-norm constraint.

When should you not use dropout?

Skip or reduce it when the model already underfits, when the data set is tiny (the paper saw no gain at 100 or 500 examples), or when overfitting is not the problem, as on a very large text set where the gain was small.

Is dropout the same as averaging an ensemble of networks?

No, it approximates one. The thinned networks share weights, and test-time weight scaling stands in for averaging their predictions. The paper reports that Monte-Carlo averaging, which samples thinned networks and averages them, becomes as good as weight scaling at around 50 samples (Sec. 7.5).

Sources

1. Dropout: A Simple Way to Prevent Neural Networks from Overfitting — Journal of Machine Learning Research 15 (Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov), 1 Jun 2014
2. Dropout: JMLR landing page (abstract and citation record) — Journal of Machine Learning Research, 1 Jun 2014
3. Improving neural networks by preventing co-adaptation of feature detectors (arXiv 1207.0580) — arXiv (Hinton et al.), 3 Jul 2012
4. torch.nn.Dropout — PyTorch documentation (read 2026-10-26), 26 Oct 2026
5. Keras Dropout layer — Keras documentation (read 2026-10-26), 26 Oct 2026
6. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (arXiv 1506.02142) — arXiv (Gal and Ghahramani), 6 Jun 2015
7. PaLM: Scaling Language Modeling with Pathways (arXiv 2204.02311) — arXiv (Chowdhery et al.), 5 Apr 2022
8. Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift (arXiv 1801.05134) — arXiv (Li et al.), 16 Jan 2018
9. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (arXiv 1502.03167) — arXiv (Ioffe and Szegedy), 11 Feb 2015