In this article
Before you start
This page belongs to the machine learning algorithms overview, and the statistics behind sampling and averaging are in the statistics interview guide.
Why interviewers ask this
A tuning question shows whether a candidate separates what a model learns from what an engineer chooses. Candidates who say "I run a grid search" and stop leave the interesting parts unsaid: how the search is budgeted, how candidates are scored, and how the final number is kept honest.
A weak answer names a tool and a list of values. A strong answer explains why a random sample often beats a grid, scores every candidate with cross-validation and says that the winning score is optimistic. A Staff-level answer adds cost control: stop poor candidates early, use earlier trial results to pick the next one, and keep one final test set that tuning never touched.
Hyperparameter tuning from first principles
- Hyperparameter
- A hyperparameter is a setting of a model or its training procedure that is fixed outside the fitting step instead of being learned from the data, such as the regularization strength of a Lasso model or the kernel of a support vector classifier.
- Hyperparameter tuning
- Hyperparameter tuning is the search for the hyperparameter values that give the best score on data the model was not trained on, run as an outer loop that fits and scores the model once for each candidate setting.
- Nested cross-validation
- Nested cross-validation is a two-loop procedure in which an inner loop picks hyperparameters and an outer loop scores the whole picking procedure on folds the inner loop never saw, giving a less biased estimate of generalization.
In plain words
Think of baking. The crust and crumb form while the bread bakes; you never set them directly (those are like the weights the model learns). The oven temperature and the baking time are dials you set before it starts (those are hyperparameters). You try a few dial settings, taste each loaf and keep the best. The analogy breaks in one place: a model's dials interact, so the best temperature can change with the baking time.
In one precise sentence, tuning is an outer search over settings, and each candidate costs a full training run. The scikit-learn guide says: "Hyper-parameters are parameters that are not directly learnt within estimators." Its examples are C, kernel and gamma for a support vector classifier and alpha for Lasso.
Step 1: Define the parameter space
The parameter space lists which settings to try and their ranges. Each hyperparameter gets a list of values or a distribution. The same guide says that a search needs an estimator, a parameter space, a method for searching or sampling candidates, a cross-validation scheme and a score function.
Input: your model and your judgment about sensible ranges. Output: a space that every candidate is drawn from. Scale matters. Settings such as C, gamma and a learning rate act on a multiplicative scale, so a log-uniform distribution suits them; the guide calls it the continuous version of a log-spaced parameter. You now have the menu of settings.
Step 2: Pick a search method
The search method picks candidates from the parameter space. Grid search tries every listed combination. Random search samples a chosen number of candidates from distributions, and the guide's RandomizedSearchCV sets that number with n_iter.
Two other families cut the cost. Successive halving and Hyperband start many candidates on a small resource, such as a few epochs or a slice of the data, and stop the weak ones early. Hyperband allocates a resource such as iterations, data samples or features to randomly sampled configurations and stops poor ones early (Li et al., 2016/2018, Journal of Machine Learning Research (JMLR) 18). Sequential methods such as the Tree-structured Parzen Estimator (TPE) use earlier trial results to choose the next trial. Input: the space. Output: one candidate at a time. The exploration vs exploitation page covers the idea behind sequential choice.
Bayesian optimization in plain words
Bayesian optimization is a sequential search that builds a model of score versus settings from the trials run so far and uses that model to choose the next trial. The model is called a surrogate: a Gaussian process, or in TPE a pair of density models whose ratio ranks candidate values. An acquisition step, such as expected improvement, scores each possible next trial by how good it is likely to be and how uncertain the surrogate is about it.
That is the trade between exploring unknown regions and exploiting known good ones. The surrogate knows nothing at the start, so these methods begin with random trials. Each new trial depends on the previous results, so the search runs largely one trial after another and is harder to parallelise than random search, where every trial is independent.
Step 3: Score each candidate with cross-validation, then see why the best score is optimistic
Cross-validation is a method that splits the training data into folds, trains on all folds but one, scores on the held-out fold, and averages the scores across the folds. Each candidate gets one average score, and the guide calls searching for the best cross-validation score possible and recommended.
The trap is in the winner. Every candidate's score contains luck, and the maximum over many candidates collects the luckiest. The scikit-learn example puts it this way, where CV is short for cross-validation (CV): "Choosing the parameters that maximize non-nested CV biases the model to the dataset, yielding an overly-optimistic score." The same example says the magnitude of this effect depends primarily on the size of the dataset and the stability of the model. Input: a candidate. Output: one honest-looking but selected score. That selection is why the next step exists.
Step 4: Judge the winner on data tuning never touched
The best candidate is judged once on a test set that tuning never saw. Hold out a final test set before tuning begins, tune on the rest, refit the winner, and score it a single time. If you reuse the test set to choose again, it stops being untouched.
When data is too small for a held-out set, use nested cross-validation. In the scikit-learn example the inner loop, run by GridSearchCV, selects hyperparameters, and the outer loop, run by cross_val_score, estimates generalization error by averaging test-fold scores. Input: the winning procedure. Output: an estimate you can report. Preprocessing such as scaling belongs inside each fold; see feature engineering techniques.
Step-by-step description
- List the settings and their ranges.
- Pick candidates from the space.
- Score each candidate on held-out folds.
- Judge the winner once on untouched data.
- The parameter space lists which settings to try and their ranges.
- The search method picks candidates from the parameter space.
- Cross-validation scores each candidate on held-out folds.
- The best candidate is judged once on a test set that tuning never saw.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take the grid from the scikit-learn guide: C in 1, 10, 100 and 1000 with a linear kernel, plus those four C values crossed with gamma in 0.001 and 0.0001 for a radial basis function kernel. That is 4 plus 4 times 2, or 12 candidates, and with five folds it is 60 model fits.1
Grids grow fast. Four hyperparameters with five values each make 625 candidates, which is 3,125 fits at five folds. A random search with 50 candidates needs 250 fits, so the grid needs 12.5 times as many fits.1 Adding a fifth, useless hyperparameter multiplies the grid again but leaves the random budget unchanged; the guide lists this as a benefit of random search, together with a budget chosen independent of the number of parameters.
| Quantity | Value (computed) |
|---|---|
| Candidates in the guide's grid | 12 |
| Fits for that grid, five folds | 60 |
| Candidates, four hyperparameters with five values | 625 |
| Fits for that grid, five folds | 3,125 |
| Fits for random search, 50 candidates, five folds | 250 |
| Distinct values of each hyperparameter, nine-point grid | 3 |
| Distinct values of each hyperparameter, nine random points | 9 |
| Fits for nested cross-validation, five outer folds, two inner folds, 12 candidates | 125 |
| Share of uniform draws on 0.001 to 1000 that land above 100 | 0.9 |
| Share of log-uniform draws on the same range that land above 100 | 0.1667 |
The distinct-values rows are why random search works. If only one of two hyperparameters matters, a 3 × 3 grid tests just 3 values of it, because the 9 points collapse onto 3 when you look along that axis. Nine random points test 9 different values.1 Bergstra and Bengio (2012) show empirically and theoretically that randomly chosen trials are more efficient than trials on a grid, because few hyperparameters matter much for any one dataset and a grid wastes trials on the rest. Nested cross-validation costs outer folds times the sum of inner folds times candidates plus one refit: 5 times (2 times 12 plus 1) is 125 fits.1
Text description of the diagram
- Four hyperparameters with five values each make a grid of 625 candidates.
- With five folds that grid needs 3,125 fits, and random search with 50 candidates needs 250.
Early stopping changes the cost again. As an illustration, start 27 candidates on one epoch, keep a third each round and triple the epochs: the rounds run 27, 9, 3 and 1 candidates on 1, 3, 9 and 27 epochs. The total is 108 epochs against 729 for training all 27 to the end, a cut of 6.75 times.1
The log scale matters too. Sampled uniformly from 0.001 to 1000, only about 0.1% of C values fall below 1; sampled log-uniformly, half do.1
Trade-offs and edge cases
No search method wins everywhere. Grid search tries every listed combination, so it is simple and repeatable but misses any value you did not list. Random search samples candidates from distributions, so its cost is your budget, but it ignores what earlier trials showed. Successive halving stops weak candidates early, which saves compute when training is slow. Sequential methods use earlier results and suit costly single trials.
Text description of the diagram
- Grid search tries every listed combination.
- Random search samples candidates from distributions.
- Successive halving stops weak candidates early.
As of October 2026, Optuna's default sampler is TPE, which fits one Gaussian mixture to the parameter values with the best objective values and another to the rest, then picks the value that maximizes their ratio; random sampling is used first until a set number of startup trials has finished (TPESampler reference, Optuna 3.6.2). The documentation (version 5.0.0) recommends the Hyperband pruner with TPE (Optuna documentation).
The evidence is mixed, so claim no universal winner. In the Hyperband paper, experiments report over an order of magnitude speedups against Bayesian optimization competitors. Random search found statistically equal performance on four of seven data sets, and superior performance on one, against deep belief networks (DBNs) configured by manual and grid search (Bergstra and Bengio, 2012). Yet Bergstra et al. (2011) found random search unreliable for training DBNs and reported that sequential methods found significantly better results than the best previously reported.
Other edge cases: tune only the few settings that matter, since the scikit-learn guide notes a small subset can have a large impact while others stay at defaults; a regularization strength such as in L1 vs L2 regularization is a typical one; and a tuned model can still fail when data changes, as in distribution shift.
How to answer it in an interview
How do you tune the hyperparameters of a model?
Hyperparameters are settings the model does not learn, so I search for them. I define a space, using log scales for things like the learning rate, then run random search with a fixed budget, because it covers more distinct values per hyperparameter than a grid. I score every candidate with cross-validation. That best score is optimistic, so I keep a test set that tuning never touched, or use nested cross-validation. If runs are slow, I add early stopping or a sequential sampler.
The spoken structure: define the term, name the search method and why, say how candidates are scored, then name the optimism trap and the fix.
Common mistakes
- "Hyperparameters are learned from the data like weights." (wrong) They are set outside fitting and chosen by an outer search. (right) "The model learns weights; I search over hyperparameters."
- "Grid search is the most thorough, so it is best." (wrong) A grid only covers the values you listed and its cost multiplies. (right) "I default to random search at the same budget."
- "My best cross-validation score is my expected test score." (wrong) It is selected, so it is biased upward. (right) "I report the score of a held-out test set or nested cross-validation."
- "Random search is just luck." (wrong) It is a principled baseline with distributions and log scales. (right) "It is the baseline other methods must beat."
- "I tune every parameter." (wrong) Usually a small subset matters. (right) "I tune the few that move the score and leave defaults for the rest."
Follow-up questions
What is the difference between a parameter and a hyperparameter?
A parameter, such as a weight, is learned from the data during fitting. A hyperparameter is chosen before or outside fitting, and the training loop never updates it.
Grid search vs random search: which should you use?
Random search is usually the better first choice, since its budget does not depend on the number of parameters and extra useless parameters do not hurt it. Use a grid only for two or three hyperparameters with short lists.
What is nested cross-validation and why do you need it?
Nested cross-validation wraps the tuning loop inside an outer loop of folds. The inner loop picks hyperparameters and the outer loop scores that whole procedure, so the reported number is not inflated by selection.
What is Bayesian optimization for hyperparameters?
Bayesian optimization is a family of sequential methods that fit a surrogate model of score versus settings from past trials and use an acquisition step to choose the next candidate, balancing exploring and exploiting. TPE, the default in Optuna, is one example, and it starts with random trials.
Why search the learning rate or C on a log scale?
Settings such as the learning rate and C act multiplicatively, so a log-uniform distribution gives each order of magnitude equal attention. A uniform draw puts most samples in the largest decade, as the worked example shows. The gradient descent page explains the learning rate itself.
- A hyperparameter is a setting chosen outside fitting; tuning is an outer search over candidate settings.
- Random search usually beats a grid at equal cost and its budget does not grow with the parameter count.
- Score every candidate with cross-validation, and use log scales for multiplicative settings.
- The best cross-validation score is optimistic; judge the winner on untouched data or with nested cross-validation.
- Early stopping and sequential samplers cut cost, but no method wins everywhere.
Sources
- Tuning the hyper-parameters of an estimator (scikit-learn User Guide 1.9.1) — scikit-learn documentation, 26 Oct 2026
- Random Search for Hyper-Parameter Optimization (Bergstra and Bengio) — Journal of Machine Learning Research 13, 1 Jan 2012
- Nested versus non-nested cross-validation (scikit-learn example) — scikit-learn documentation, 26 Oct 2026
- Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (arXiv 1603.06560) — arXiv (Li et al.), 1 Mar 2016
- Efficient Optimization Algorithms (Optuna documentation) — Optuna documentation, 26 Oct 2026
- Algorithms for Hyper-Parameter Optimization (Bergstra, Bardenet, Bengio, Kegl) — Advances in Neural Information Processing Systems 24, 12 Dec 2011
- optuna.samplers.TPESampler (Optuna 3.6.2 reference) — Optuna documentation, 26 Oct 2026
