ML Coding From Scratch

How would you critique this regression model?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

How would you critique this regression model? Run four checks in order: the residuals, the assumptions and coefficients, the error metric, and the error on data the model has not seen. A critique is a structured review of whether a fitted model's numbers can be trusted. The line that separates a mid-level critique from a Staff one is the first move: plotting the residuals before reading any score.

In this article

Before you start
Why interviewers ask this
Critiquing a regression model from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page belongs to the machine learning (ML) coding round guide; the other pages there cover the code you write, and this one covers the model you are asked to judge.

Why interviewers ask this

A model that looks good on one number can still be wrong in ways that cost money, and an interviewer wants to know whether you would catch that before it ships. This prompt shows up in review-style rounds: you get a fitted model, a coefficient table or a scatter plot, and you are asked what is wrong with it. Candidates report it in ML coding and code-review rounds.

A weak answer reads the R² aloud and says "it looks good" or "add more features." A strong answer names what to inspect, in what order, and what each failure would look like. A Staff answer also says what it would do next with each finding, and separates "the model is wrong" from "the model is fine for prediction but its coefficients cannot be interpreted." The same habit is tested in code review interview questions: find the real defect, rank it, and propose a fix.

Critiquing a regression model from first principles

Variance inflation factor: A variance inflation factor is a diagnostic value that shows how much the variance of one coefficient is inflated because its input can be predicted from the other inputs, so it flags overlapping inputs that make coefficients unstable.

In plain words

Think of a weather forecaster. You do not judge the forecaster by one average score. You read the track record. Are the misses random? Or does the forecaster always miss on cold mornings? The analogy breaks in one place. A forecaster has no coefficients to interpret. A regression model also claims that each input has its own effect, and that claim needs its own check.

Here is the idea in precise terms. A regression model is acceptable when four things hold. Its residuals look like structureless noise. Its coefficients are stable and interpretable. Its error metric matches the cost of mistakes. Its error holds up on rows it never saw.

A critique runs four checks in order, and a later check can overturn a good score from an earlier one.CC BY 4.0 — source: How would you critique this regression model?

Step-by-step description

Plot residuals before reading any score.
Check spread, independence and coefficients.
Match MAE or RMSE to the cost of a miss.
Compare training error with error on new rows.

A residual plot shows each residual (actual minus predicted value) against the prediction or against an input.
Its residuals look like structureless noise.
A held-out set is used only to estimate how it behaves on new rows.

Step 1: Plot the residuals before reading any score

A residual plot shows each residual (actual minus predicted value) against the prediction or against an input, and a healthy model leaves a flat, even band around zero. The National Institute of Standards and Technology (NIST) handbook says residuals should be roughly normal, roughly independent, with mean zero and constant variance, and that departures usually mean the residuals contain structure the model does not account for. Find the structure, add a term for it, and the model improves.

Why a plot and not a score: Anscombe (1973) built four data sets of eleven points each where the ordinary least squares (OLS) fit is the same line while the scatter plots look strikingly different. All four fit y = 3 + 0.5x with R² near 0.67.¹

The four Anscombe sets share the same means, line and R², so only a plot separates the straight-line set from the curve, the outlier and the single high-influence point.
Set	Pattern in the plot	Mean y	Fitted line	R²
1	roughly linear	7.5	y = 3 + 0.5x	0.667
2	curved	7.5	y = 3 + 0.5x	0.666
3	straight with one outlier	7.5	y = 3 + 0.5x	0.666
4	one point sets the slope	7.5	y = 3 + 0.5x	0.667

In the chart below, set 2 curves away from the fitted line y = 3 + 0.5x, and set 3 lies on a straight line except one outlier at x = 13.

The curved set and the outlier set share one fitted line, so the line and its R² cannot show which pattern lies behind them.CC BY 4.0 — source: How would you critique this regression model?

Text description of the diagram

Set 2 curves away from the fitted line y = 3 + 0.5x.
Set 3 lies on a straight line except one outlier at x = 13.

You now know what to look at first. The next step asks what a bad residual plot would mean.

Step 2: Check the assumptions and the coefficients

Each shape in the residual plot names a specific fault. A curve means the model is missing a nonlinear term. A funnel, where the band widens as the prediction grows, is heteroscedasticity: in an NIST example the residuals show increasing variance as the predictor increases, which breaks the constant-variance assumption. A pattern over time means the errors are not independent. In an illustrative simulation of 400 rows, the residual spread was 0.84 in the lower half of the inputs and 1.63 in the upper half.¹

Then read the coefficients. Multicollinearity inflates their variance, and the variance inflation factor (VIF) is the factor by which the variance of a coefficient is inflated, equal to 1 / (1 − R²ₖ), where R²ₖ comes from regressing predictor k on the other predictors. Two predictors correlated at 0.95 give a VIF of 10.26.¹ VIF cut-offs are conventions, not a standard: Penn State's regression course, lesson 12 uses 4 and 10, and 5 and 10 are also common.

Anscombe sets 3 and 4 add two more checks. An outlier is a row whose target sits far from the pattern of the other rows. Leverage is how far a row's inputs sit from the inputs of the other rows. In set 4 one row at x = 19 has high leverage because every other row has x = 8, so that one row sets the slope.¹

Cook's distance is a measure of how much the fitted coefficients move when one row is dropped, so a high value marks an influential row. The NIST Dataplot manual describes Cook's distance as the combined impact of one observation on all of the regression coefficients, and notes that high-leverage rows tend to have small residuals even when they are outliers, so a residual plot alone can miss them; refit without the suspect row and see whether the story changes.

Normality is the softest of these checks: judge it last, because a mild departure matters less than visible structure or changing spread.

Step 3: Choose an error metric that matches the cost of mistakes

R² (the coefficient of determination) is the proportion of the variance of the target that the model explains. It is the score a scikit-learn regressor's score method returns (RegressorMixin.score: "Return coefficient of determination on test data"). R² computed on the training rows never falls when you add a predictor, so it cannot tell a useful input from a useless one. Adjusted R² is R² reduced for each extra predictor.

It penalises each extra predictor but does not remove chance gains. In an illustrative run with 30 rows, adding ten pure-noise predictors lifted R² from 0.764 to 0.877, and adjusted R² still rose, from 0.755 to 0.801.¹ Only held-out error settles whether the inputs help. Note also that the scikit-learn metrics page says r2_score calculates unadjusted R² without correcting for bias in sample variance of y.

Pick the error metric by what a miss costs. The mean absolute error (MAE) weighs every miss equally; the root mean squared error (RMSE) punishes large misses more. For ten illustrative errors, nine of size 1 and one of size 10, MAE is 1.9 and RMSE is 3.3.¹ If one big miss is far worse than several small ones, RMSE is the honest number; if outliers are noise, MAE is.

Step 4: Test on data the model has not seen

A held-out set is a portion of the data that the model never trained on, used only to estimate how it behaves on new rows. Overfitting is a fit so flexible that it learns the noise in the training rows, so training error looks good and held-out error does not. The check is a comparison: low training error with higher held-out error is overfitting; high error on both is underfitting.

In an illustrative simulation of 30 training rows drawn from a straight line plus noise, a straight-line fit scored RMSE 1.03 on training data and 0.99 on 200 fresh rows. A ninth-degree polynomial scored 0.88 on training data and 1.64 on the fresh rows.¹ The flexible model looked better on the data it had seen and was worse on new data. The straight line's test RMSE of 0.99 sits near the simulated noise level of 1, which is about the error no model can avoid.

The flexible model wins on training data and loses on new data, which is the gap a held-out set exposes.CC BY 4.0 — source: How would you critique this regression model?

Text description of the diagram

A straight line has training RMSE 1.03 and test RMSE 0.99.
A ninth-degree polynomial has training RMSE 0.88 and test RMSE 1.64.

The same discipline applies when you train a larger model; see gradient descent from scratch and learning-rate schedules and building generative pre-trained transformer (GPT) from scratch, where a rising held-out loss signals the same problem.

What if you are shown the code that trained it?

Read the code for pipeline defects. Look for a scaler or encoder fitted on all rows before the split. Look for target or future values leaking into the inputs, and for a random split on time-ordered data. Check whether the score is computed on the training rows, and whether hyperparameters were tuned on the test set. Each defect makes the reported held-out error look better than the error on new data will be.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.