In this article
Before you start
Why interviewers ask this
A candidate who answers "I would scale and encode the data" has named the tools and skipped the order. Machine learning rounds ask how to handle missing data, which transformations suit categorical data, and whether a neural network needs scaled inputs. Behind all three sits one test: whether you know that a preprocessing step is itself a model, fitted on data, and can leak.
A weak answer is a list of techniques. A strong answer gives a pipeline order, ties each technique to the model that needs it, and states where information can leak from the test rows into training. The same question appears inside a machine learning system design round, where the follow-up is how the steps are reproduced at serving time.
Feature engineering from first principles
In plain words
- Feature engineering
- Feature engineering is the practice of transforming raw columns into the numeric inputs a model learns from, by imputing, encoding, scaling and creating columns. A good transformation puts the signal in a form the chosen model can use.
- Imputation
- Imputation is the replacement of missing values with estimates inferred from the known part of the data, such as a column's mean, median or most frequent value. It keeps rows that discarding would lose.
Think of cooking from a recipe card. Raw ingredients arrive in different units, some are missing and some are written as words. Preparation puts everything in the same measures so the recipe works. The analogy breaks at one point: preparation here is fitted to the ingredients you were given, so the card for tomorrow's meal must reuse today's measures instead of recalculating them.
In one precise sentence, feature engineering builds model inputs so that every step that learns a statistic (imputing, encoding, scaling) is fitted on training rows and applied unchanged to every later row, at test time or in production. Stateless steps such as logs, ratios and date parts learn nothing and need no fitting.
The four steps below follow the order a pipeline runs them.
Step 1: Split first and learn every statistic from the training rows
The first step is to split the data into training and test rows before any transformation, because every fitted transformation must see the training rows only. The scikit-learn guide on common pitfalls states the general rule as "never call fit on the test data". It warns that the same risk applies to almost any transformation, including a scaler, an imputer or principal component analysis (PCA), and the demonstration below shows the leak with feature selection.
The size of the damage is easy to demonstrate, using the set-up of the scikit-learn common-pitfalls example re-run with logistic regression and 5-fold cross-validation. On 200 rows with 10,000 random features and random labels, the true accuracy is about 0.5. Choosing the best 25 features on all the data and then cross-validating (scoring by rotating which fifth of the rows is held out) scored 0.855; choosing them inside a pipeline, which refits the selection on each training fold, scored 0.485.1 A pipeline is an object that chains the transformations and the model so that each fit sees only the rows it should.
The input is the raw table. The output is a training set, a test set, and a rule: fit on one, transform both.
Step 2: Impute missing values, and say that a missing value can carry signal
Missing values are either dropped or imputed, and imputation usually keeps more of the data. The scikit-learn imputation guide says discarding rows and columns "comes at the price of losing data which may be valuable" and that a better strategy is to infer the missing values. Its simple imputer fills a column with a constant or with the mean, median or most frequent value; a k-nearest-neighbours imputer (k-nearest neighbours is a method that reads off the most similar rows) and an iterative imputer, which the guide labels experimental, use the other columns.
Missingness itself can be informative. In a synthetic set where a value is missing for 50.6 percent of positive rows and 9.7 percent of negative rows, dropping incomplete rows would discard 431 of 1,400 training rows and leave a model that cannot score an incomplete row later. Mean imputation scored an area under the curve (AUC) of 0.861 on the test set (0.5 is guessing, 1 is a perfect ranking), and adding an indicator column that marks which values were missing lifted it to 0.920.1
The input is a column with gaps. The output is a complete column plus, when missingness matters, a flag column.
Step 3: Encode categories without inventing an order
Categorical columns are encoded as numbers, and the method decides whether a false order or a wide table results. The scikit-learn preprocessing guide says an ordinal encoder gives each category an integer and "would interpret the categories as being ordered, which is often not desired". One-hot encoding is an encoding that turns a column with n possible values into n binary columns, exactly one of which is 1 in each row; it avoids that at the cost of one column per category.
For a column with hundreds of categories the guide describes target encoding, which replaces each category with the average target value of its rows, shrunk toward the overall average when a category has few rows, and is "useful with categorical features with high cardinality". Its fit-and-transform method cross-fits, encoding each fold with statistics from the other folds.
Here 300 categories gave 300 one-hot columns with a test AUC of 0.776, against one target-encoded column with 0.775; a plain fit then transform scored 0.828 on training and 0.775 on test rows, a leak that cross-fitting removed (0.767 and 0.775). Only the training score shows the leak here, since both encoders give the same test encodings, but an inflated training score misleads model choice.1
For target encoding, the input is a category column and the label. The output is numbers that carry the label's information and no false ordering.
Step 4: Scale numbers for the models that measure distance or learn weights by gradient descent
Scaling puts numeric columns on comparable ranges, and the models that need it are those that compare distances or are trained by gradient descent on feature weights, with tree ensembles excepted, even gradient-boosted ones. The scikit-learn guide says standardisation, which rescales each column to mean zero and unit variance, is a common requirement because objective functions such as the support-vector kernel or the l1 and l2 regularisers assume features centred near zero with similar variance, and that the mean and standard deviation come from the training set.
On the 178-row wine data, whose widest column range is 2,645 times its narrowest, a k-nearest-neighbours classifier scored 0.663 accuracy raw, 0.961 standardised and 0.972 min-max scaled; a decision tree scored 0.927 either way.1 A decision tree is a model that asks chained yes-or-no questions such as "is age above 30?", so any order-preserving rescaling does not change its splits.
For neural networks, Ioffe and Szegedy (2015) write that training "converges faster if its inputs are whitened", meaning zero means, unit variances and decorrelated, so scale the inputs; the gradient descent page covers the updates.
The input is numeric columns of different sizes. The output is columns a distance or gradient can treat fairly.
Step-by-step description
- Split first so no statistic sees the test rows.
- Fill gaps and flag which values were missing.
- Turn categories into numbers with no false order.
- Put numbers on comparable ranges for distance and gradient models.
- Splitting comes first so no step sees the test rows.
- The imputer, the encoder and the scaler are each fitted on the training rows only and reused on the test rows.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
A hand-checkable case
Take illustrative training ages of 20, 30 and 40 and one test age of 50 (made-up numbers, worked by hand). The training mean is (20+30+40)/3 = 30. The squared gaps sum to 200, so the variance is 200/3 = 66.7 and the standard deviation is 8.16. The test value scales to (50-30)/8.16 = 2.45, which says correctly that it lies far outside what training saw.
In this illustrative case, the leaked version recomputes the statistics with the test row included: ages 20, 30, 40, 50 give mean 35 and variance 500/4 = 125 (squared gaps of 225, 25, 25 and 225 from the mean), so standard deviation 11.18. The test value now scales to (50-35)/11.18 = 1.34 and looks milder, and every training row has moved too: 20 becomes -1.34 instead of -1.22. Redo it with your own numbers; the only change is whose rows feed the mean.
Measured results
Leakage, a missing-value flag and scaling each changed a measured score, and the tables below show how much.
| Technique | Setup | Result |
|---|---|---|
| Select features on all data | 200 rows, 10,000 random features, 5-fold accuracy | 0.855 |
| Select features inside a pipeline | same data | 0.485 |
| Mean imputation | 2,000 rows, informative missingness, test AUC | 0.861 |
| Mean imputation plus indicator | same data | 0.920 |
| Nearest neighbours, raw wine data | 178 rows, 5-fold accuracy | 0.663 |
| Nearest neighbours, standardised | same data | 0.961 |
| Decision tree, raw or standardised | same data | 0.927 |
The data are synthetic or built into scikit-learn, and the seeds are fixed so the run repeats.1
Text description of the diagram
- Selecting 25 features on all the data scored 0.855 accuracy on random labels.
- Selecting them inside a pipeline scored 0.485.
| Encoding | Columns | Train AUC | Test AUC |
|---|---|---|---|
| One-hot | 300 | 0.828 | 0.776 |
| Target encoding, cross-fitted | 1 | 0.767 | 0.775 |
| Target encoding, fit then transform | 1 | 0.828 | 0.775 |
A large gap between the train and test columns is the signature of leakage or overfitting; the cross-fitted row has almost none.1
Text description of the diagram
- Nearest neighbours scored 0.663 raw, 0.961 standardised and 0.972 min-max scaled.
- A decision tree scored 0.927 raw and 0.927 standardised.
Trade-offs and edge cases
Each technique fixes one problem and creates another, so the choice is a trade, and the table summarises it.
- Mean imputation shrinks variance. Every filled value sits at the centre of the column. Add an indicator when missingness can carry signal, or use a model-based imputer; the scikit-learn guide labels the iterative imputer experimental.
- Unseen categories at serving time. A one-hot encoder can be set to give all zeros for a category it never saw, and the target encoder uses the overall target mean, as the preprocessing guide describes.
- Scaling and outliers. The mean and variance are pulled by extreme values; the guide recommends an outlier-resistant scaler, which uses sturdier estimates of centre and range, when the data has many outliers.
- Sparse data. Standardising with a mean destroys sparsity, and the guide notes the scaler raises an error unless the mean is switched off; a max-abs scaler is its recommended choice for sparse data.
- Order of operations. Impute before encoding or scaling, and fit all of it inside one pipeline so cross-validation refits every step on each fold, the fix the pitfalls guide recommends.
- The classical-versus-deep split. Tree ensembles shrug off scaling, but distance-based, regularised linear and neural models do not; the classical machine learning overview shows which models fall on which side.
Text description of the diagram
- Imputation keeps rows that dropping would lose.
- Target encoding needs cross-fitting to avoid leakage.
How to answer it in an interview
How would you prepare raw tabular data with missing values and categories for a model?
I split first, then build one pipeline fitted on the training rows only. I impute numeric gaps with the median and add a missing-indicator column, because missingness can carry signal. I one-hot encode low-cardinality categories and use cross-fitted target encoding for high-cardinality ones, so the target does not leak. I scale only for distance-based, regularised linear or neural models, since trees do not need it. Then I cross-validate the whole pipeline, so every fit sees only its training fold. The failure I watch for is fitting any step on the test rows.
The spoken structure is: split first, then impute, encode and scale in order, name the model that needs scaling, and close on leakage.
Common mistakes
- "Scale the whole data set, then split" (wrong). The test rows shape the mean and variance (right: split, fit the scaler on the training rows, transform both).
- "Label-encode a city as 1, 2, 3" (wrong). A linear or distance-based model reads an order and a distance that do not exist, and a tree needs extra splits to undo the false order (right: one-hot, or target encoding with cross-fitting).
- "Drop rows with missing values" (wrong). It discarded 431 of 1,400 training rows in the example and the model still cannot score an incomplete row later (right: impute, and flag the gaps).
- "Always scale" (wrong). Tree models gain nothing, as the 0.927 against 0.927 result shows (right: scale for models that use distances or gradients).
- "Target-encode, then validate on the same rows" (wrong). The plain encoding scored 0.828 on training and 0.775 on test (right: cross-fit, then judge on held-out rows).
Follow-up questions
How do you handle missing data in a dataset?
Drop only when few rows are affected; otherwise impute with a constant, the mean, the median or the most frequent value, or with a model. Add a missing-indicator column when the gap may carry signal, as in the example where it moved AUC from 0.861 to 0.920.
What are some common transformations for categorical data?
One-hot encoding, ordinal encoding for genuinely ordered labels, and target encoding for columns with many categories. Rare categories can be grouped into one "other" level to keep the width manageable.
What if a categorical column has 10,000 levels?
One-hot would give 10,000 sparse columns, so use cross-fitted target encoding, or group rare levels into "other" first.
Do we need scaling when the input is a neural network?
Yes; standardised inputs (whitening also decorrelates them) make training converge faster according to Ioffe and Szegedy. Compute the scaling statistics on the training rows and reuse them in production.
Which models do not need feature scaling?
Decision trees and tree ensembles split on thresholds, so rescaling does not change them; the example scored 0.927 raw and standardised. Distance-based models, support-vector machines, regularised linear models and neural networks do need it.
What is target leakage in feature engineering?
Leakage is any feature or statistic built with information that is unavailable at prediction time. The classic case is a feature recorded after or because of the outcome, such as "account closed" when predicting churn; an encoder fitted with the test labels is the second. Every held-out metric, such as a confusion matrix, assumes the held-out rows stayed clean, and machine learning operations monitoring is how teams notice when they did not.
- Imputing, encoding and scaling each learn a statistic; apply the training-fitted version unchanged to every later row. Stateless steps such as logs and ratios need no fitting.
- Selecting features on all the data scored 0.855 on random labels; inside a pipeline it scored 0.485.
- Impute missing values and consider an indicator column; missingness can carry signal.
- Use one-hot for few categories and cross-fitted target encoding for many; avoid an ordinal code for unordered labels in linear and distance-based models.
- Scale for distance-based, regularised linear and neural models; trees do not need it.
Sources
- Preprocessing data, scikit-learn user guide (scalers, one-hot, ordinal and target encoders) — scikit-learn developers (page shows version only; read 2026-10-24), 24 Oct 2026
- Imputation of missing values, scikit-learn user guide — scikit-learn developers (page shows version only; read 2026-10-24), 24 Oct 2026
- Common pitfalls and recommended practices (data leakage), scikit-learn user guide — scikit-learn developers (page shows version only; read 2026-10-24), 24 Oct 2026
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (arXiv 1502.03167 v3) — arXiv (Ioffe and Szegedy), 2 Mar 2015
- Categorical feature support in gradient boosting, scikit-learn example — scikit-learn developers (page shows version only; read 2026-10-25), 25 Oct 2026
