How does feature scaling help neural networks train?

In this article

1. Before you start
2. Why interviewers ask this
3. Feature scaling for neural networks from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the deep learning path.

Why interviewers ask this

A candidate who skips scaling, or scales before splitting the data, ships a model whose validation numbers cannot be trusted. That is why the question appears in data and machine learning rounds, as reported by candidates, often as a quick check inside a longer deep learning interview.

The question separates three levels. A weak answer says "scaling makes training faster" and stops. A strong answer names the formula, says which features it applies to, and says that the statistics come from the training split only. A Staff-level answer adds the cases where scaling is not needed (tree models, image pixels), what outliers do to min-max scaling, and how batch normalization relates to input scaling without replacing it.

Feature scaling for neural networks from first principles

Feature scaling

Feature scaling is a preprocessing step that rewrites each numeric input feature on a comparable scale, by subtracting a center and dividing by a spread, so that no feature dominates a model's training because of its units. Standardization and min-max scaling are the two common forms.

Standardization

Standardization, also called z-score normalization, is a feature scaling method that subtracts each feature's mean and divides by its standard deviation, leaving zero mean and unit standard deviation on the data used to compute them, while the shape of the distribution stays the same.

Min-max scaling

Min-max scaling is a feature scaling method that maps each feature linearly so its smallest training value becomes the bottom of a chosen range and its largest becomes the top, by default zero and one, which keeps relative spacing but removes the units.

In plain words

Imagine a mixing desk with two dials. One dial moves the sound in steps of one. The other moves it in steps of a thousand. You would struggle to set both well. A tiny turn of the second dial changes everything. The first dial barely seems to matter.

A network faces the same problem when one input is an age and another is an income. Scaling gives every dial the same step size. The analogy breaks in one place. A network can learn to compensate for bad units in the end. Scaling only makes the road there much shorter.

In one precise sentence, feature scaling replaces each feature value x with (x minus a center) divided by a spread, using numbers measured on the training data.

Why does feature scaling help a neural network train faster?

Feature scaling helps because it stops large-unit features from dominating the loss and the early weight updates. The scikit-learn documentation warns that a feature with a variance orders of magnitude larger than the others "might dominate the objective function".

Google's Machine Learning Crash Course gives a concrete case. If one feature spans ten times the range of another, the model at the start of training assumes the wider feature is ten times more important, and training takes longer than it should. With a far larger gap, such as one feature from -1 to +1 and another from 5000 to 1,000,000,000, training may take much longer to converge or fail to converge.

LeCun et al. (1998) write that scaling "helps to balance out the rate at which the weights connected to the input nodes learn", and that convergence is usually faster when each input averages close to zero (Efficient BackProp, 1998).

Step 1: Fit the scaler on training data only

A scaler is fitted when it computes its numbers: the mean and standard deviation of each feature, or the minimum and maximum. Fitting uses the training split only. Data leakage is the use of information from validation or test data while building a model or its preprocessing, which makes later scores look better than real performance.

The scikit-learn documentation describes its scaler as computing the mean and standard deviation on a training set. Stanford's neural network course notes say preprocessing statistics must only be computed on the training data and then applied to the validation and test data, and that computing the mean over the whole dataset before splitting "would be a mistake".

Input: the training columns. Output: one stored center and one stored spread per feature. The scaler learns its statistics from the training set only.

Step 2: Rescale each feature with its own numbers

Every feature gets the same formula with its own numbers. Standardization turns each value into a z-score. A z-score is the number of standard deviations a value sits above (positive) or below (negative) its feature's mean. With the training mean $\mu$ and training standard deviation $\sigma$ of that feature:

Min-max scaling uses the training minimum and maximum instead:

Standardization leaves the shape of the distribution as it was; scikit-learn notes that in practice we often ignore the shape and just center the data. Stanford's notes name two forms of the divide step: divide each zero-centered dimension by its standard deviation, or rescale each dimension so its minimum and maximum are -1 and 1. They call mean subtraction the most common form of preprocessing.

Step 3: Apply the same numbers to validation, test and live data

Validation, test and live inputs are scaled with the training numbers, never refitted. The same statistics are reused on validation, test and live data, so the model sees every input on the scale it was trained on. In scikit-learn this is a transform call on the fitted scaler, not a new fit.

The consequence is that unseen values can fall outside the training range. The scikit-learn page shows min-max scaling fitted on a three-row matrix and applied to the unseen row (-3, -1, 4); the result is (-1.5, 0, 1.667),¹ so values outside the range zero to one are normal and should not be clipped without a reason.

Input: a new row. Output: the same row rescaled with the stored numbers. Skipping this step in production, for example by rescaling each live request on its own, feeds the model inputs it never saw in training. A feed forward network multiplies exactly these scaled inputs by its first-layer weights.

Step 4: Batch normalization repeats the idea inside the network

Batch normalization applies the same zero-mean, unit-variance normalization to the inputs of hidden layers during training. Ioffe and Szegedy (2015) write that they "normalize each scalar feature independently, by making it have the mean of zero and the variance of 1", and that, as LeCun et al. showed, such normalization speeds up convergence.

Batch normalization repeats the normalization inside the network, using statistics of each mini-batch. Because each layer's outputs are the next layer's inputs, it is the same heuristic applied at every layer. It also adds a learned scale and shift, since normalizing the inputs of a sigmoid would constrain them to its linear regime; see activation functions.

Our reading, not a claim of the sources: it extends input scaling and does not replace it, because the first layer still receives your raw columns. The batch norm versus layer norm page covers the variants.

1 · Fit on training data2 · Rescale3 · Reuse4 · Batch normReplay

Step-by-step description

1. Statistics come from the training set only.
2. Each feature uses its own numbers.
3. Validation, test and live data use the stored numbers.
4. The idea repeats inside the network.

• The scaler learns its statistics from the training set only.
• The same statistics are reused on validation, test and live data.
• Batch normalization repeats the normalization inside the network.

Worked example

Take five people with ages 20, 30, 40, 50 and 60 and incomes 50,000, 20,000, 140,000, 80,000 and 110,000, in that order (illustrative).¹ Ages have mean 40 and standard deviation 14.1421. Incomes have mean 80,000 and standard deviation 42,426.4069, using the population formula that scikit-learn's scaler uses.¹

After standardization both columns have mean 0 and standard deviation 1. Age becomes -1.4142, -0.7071, 0, 0.7071 and 1.4142, and income becomes -0.7071, -1.4142, 1.4142, 0 and 0.7071, so the 3,000-fold gap between the income range and the age range disappears.¹

Why that matters for training: for a linear model with squared error and no bias term, the curvature of the loss along one weight (the diagonal entry of the Hessian, the matrix of second derivatives) is twice the average squared value of that feature (a standard calculus fact). The average squares are 1,800 for age and 8,200,000,000 for income, a ratio of about 4,555,556.¹ One learning rate cannot suit both: large enough to move the age weight, it overshoots on income; small enough for income, the age weight crawls. After standardization both average squares are 1.

The scikit-learn page uses a smaller toy matrix with rows (1, -1, 2), (2, 0, 0) and (0, 1, -1). Its column means are 1, 0 and 0.33, and its scales are about 0.8165, 0.8165 and 1.2472 (the page prints them truncated as 0.81, 0.81 and 1.24); the script reproduces them.¹

Outliers change the picture for min-max scaling. An outlier is a value far outside the range of the rest of the feature. Take incomes 30,000, 40,000, 50,000 and 60,000 plus a fifth person at 1,000,000 (illustrative). One income of 1,000,000 squeezes the other four min-max values below 0.031.¹ If the fifth income is 70,000 instead, the five values spread evenly from 0 to 1.

Text description of the diagram

• One income of 1,000,000 squeezes the other four min-max values below 0.031.
• Without that income the five min-max values spread evenly from 0 to 1.

Trade-offs and edge cases

When is feature scaling not needed?

Feature scaling is not needed for every model. The scikit-learn example says tree-based models are almost unaffected by scaling, while many other algorithms require normalized features. For images, Stanford's notes say the relative scales of pixels are already approximately equal, so the extra divide step is "not strictly necessary" and zero-centering is the part that matters.

The same notes apply the step only if you have reason to believe features have different scales or units. They add that whitening, which goes beyond per-feature scaling by decorrelating features through principal component analysis (PCA), is not used with convolutional networks.

How do outliers, sparse data and skewed shapes change the choice?

Outliers hurt min-max scaling most, because the extreme value fixes the range, as the chart shows. The scikit-learn documentation says scalers designed for data with outliers can be more appropriate. Standardization is also pulled by an outlier, since it inflates the spread.

For sparse data, centering destroys the sparseness structure, and the documentation says it is rarely sensible. Max-abs scaling suits sparse data because it does not center the data; its MaxAbsScaler keeps training values within -1 to 1. StandardScaler with with_mean=False also avoids centering. Max-abs scaling divides each value by the feature's largest absolute value, so an outlier squeezes the other values just as it does with min-max scaling. Min-max scaling shifts values by the training minimum, so zeros move unless the minimum is zero. Finally, standardization does not make a skewed feature normal; it only sets its center and spread.

Text description of the diagram

• Standardization subtracts the mean and divides by the standard deviation.
• Min-max scaling maps the smallest value to zero and the largest to one by default.
• Max-abs scaling suits sparse data because it does not center the data.

Standardization subtracts the mean and divides by the standard deviation. Min-max scaling maps the smallest value to zero and the largest to one by default. No source in our reading says one always beats the other, so test both on your data.

How to answer it in an interview

The spoken structure: say what scaling does, say why it helps training, then say the training-only rule, and finish with one case where you would not scale.

Common mistakes

• "I computed the mean and standard deviation on the full dataset, then split." (wrong) That is leakage. (right) "I split first, fit the scaler on the training split, then transform the others."
• "Standardizing makes the feature normally distributed." (wrong) It only sets the center and spread. (right) "Standardization keeps the shape and changes the units."
• "Batch normalization means I do not need to scale inputs." (wrong) Batch normalization acts on hidden layers, not on your raw columns, unless you place a normalization layer at the input, and then that layer is your scaler. (right) "I scale the inputs and also use batch normalization where it helps."
• "I refit the scaler on the test set so the numbers look right." (wrong) The model was trained on the training scale. (right) "I reuse the stored training statistics."

Follow-up questions

Does batch normalization make feature scaling unnecessary?

No: the first layer still receives the raw columns, so their units still matter, and Stanford's notes list input preprocessing as its own step. Batch normalization normalizes hidden-layer inputs per mini-batch, so I still scale the raw inputs. The batch norm page covers what it does.

Should you use standardization or min-max scaling for a neural network?

Standardization is the usual default for roughly centered features, and min-max scaling suits bounded inputs. With outliers, min-max suffers most, so check the range first. Compare both on a validation set.

Do tree-based models need feature scaling?

Not usually. The scikit-learn example says tree-based models are almost unaffected by scaling, while nearest-neighbor models get a completely different decision boundary on scaled data.

Why compute scaling statistics on the training set only?

Because statistics from validation or test data leak information about those splits into training and inflate the scores. The scaler is part of the model and is fitted only on training data.

Should you scale the target variable in regression?

Often yes when the target is in large units, because a large target makes the loss and its gradients large. Standardize it with training statistics only, train on the scaled target, and convert predictions back to the original units. This is common practice and is not covered by the sources on this page.

What is the difference between normalization and standardization?

The terms vary. scikit-learn calls z-scoring standardization, Stanford's notes use normalization for the divide-by-spread or range step, and Google uses normalization as an umbrella for linear, z-score, log and clipping methods. Define your term when you answer. If a loss still blows up after scaling, that is a separate failure; see exploding gradients.

Sources

1. 8.3 Preprocessing data (StandardScaler, MinMaxScaler, MaxAbsScaler), scikit-learn 1.9.1 — scikit-learn documentation, 26 Oct 2026
2. Importance of Feature Scaling, scikit-learn 1.9.1 — scikit-learn documentation, 26 Oct 2026
3. Neural Networks Part 2: Setting up the data and the model — Stanford course notes, 26 Oct 2026
4. Numerical data: Normalization, Machine Learning Crash Course — Google for Developers, 26 Oct 2026
5. Efficient BackProp — LeCun, Bottou, Orr, Müller, in Neural Networks: Tricks of the Trade (Springer), 1 Jan 1998
6. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (arXiv 1502.03167) — arXiv (Ioffe and Szegedy), 11 Feb 2015