In this article
Before you start
The take-home is one stage of the interview loop; the timed cousin of a take-home is covered in what an online assessment is.
Why interviewers ask this
A take-home is the round where your habits sit on the page instead of in your words. Candidates report briefs that ask them to review a junior's equipment-failure pipeline that claims 94% accuracy (an illustrative figure here), or to build a classifier from a supplied dataset, analyze its failure modes and propose improvements; these are reported by candidates, and no public source says which companies ask them.
The question separates people who can produce a number from people who can defend it. A weak submission reports one high score. A strong submission shows the split, a baseline, the metric choice and the rows the model gets wrong. A Staff-level submission also states what it did not test and what it would try next, so the reviewer can trust the number without rerunning anything.
Classical ML evaluation from first principles
- Failure analysis
- Failure analysis is a practice that inspects the rows a model gets wrong, grouped by shared traits, to learn why errors happen and what to fix, instead of trusting a single score that hides where the model fails.
In plain words
Think of a student who sees the exam questions the night before. A high mark then says nothing about what the student learned. Leakage works the same way: the test rows must stay unseen until the final score. The analogy breaks in one place. Leakage is quieter than reading the paper, because a scaler that learns the average of every row has already seen the test rows.
In one precise sentence, the classical-ML take-home grades whether your reported score estimates performance on rows the model has never touched, and whether you looked hard at the rows it gets wrong.
Step-by-step description
- Fit only on the training rows.
- Score always-ok first.
- Use precision and recall.
- Slice the errors, then improve.
- A pipeline keeps scaling and selection inside the training fold.
- Score a baseline first so a model's number has a reference.
- Choose precision and recall, not accuracy, when one class is rare.
Step 1: Split first, then fit everything inside a pipeline
Split the data before any fitting step. The scikit-learn guidance is blunt: the general rule is to never call fit on the test data. Its example is a normalization step: the average you divide by must be the average of the training subset, not of all the data.
A pipeline is a chain of preprocessing and model steps that is fitted as one object, so each step learns only from the rows it is given. The same page notes that pipelines help avoid leaking the test data into the training data, and a pipeline keeps scaling and selection inside the training fold.
To see the size of the error, the script draws 100 rows of pure noise with 2,000 features and keeps the 20 features that correlate best with the label. Selecting on all rows scores clearly above chance; selecting on the training rows only scores near chance, which is the honest answer for noise. The two mean accuracies are in the script output.1 You now have the rule every later step depends on.
Tuning and resampling without leakage
Tuning on the test set leaks it, because the test set then stops measuring generalization. The cross-validation guide says knowledge about the test set can leak into the model, so hold out a validation set or use cross-validation and score the test set once at the end. Resampling belongs inside the training part too, because imbalanced-learn warns against resampling the entire dataset before splitting. Related rows, such as one machine seen on many days, call for a time or group split instead of a random one.
Step 2: Score a baseline before any real model
Score a baseline first so a model's number has a reference. For a rare failure, the baseline is a model that always predicts the common class, "ok". Input: the training labels. Output: one score on the held-out rows that any real model must clearly beat.
Take an illustrative table of 1,000 rows with a 6% failure rate: 60 failures and 940 healthy rows. The always-ok model scores 0.94 accuracy, finds none of the failures and has an average precision equal to the failure rate, 0.06.1 A model that reports 0.94 therefore has shown nothing yet. Add a second reference, a simple linear model, so the reviewer sees what added complexity buys.
Do not assume complexity wins. Kapoor and Narayanan (Patterns, 2023)2 found leakage in 17 fields, affecting at least 294 papers; in their civil-war prediction case study, the claimed advantage of complex models over older regression models disappeared once the leakage was fixed. Next you need a metric that can tell the models apart.
Step 3: Choose a metric that fits the class balance
Accuracy is the proportion of all classifications that were correct, and it misleads on imbalanced data. Google's Machine Learning Crash Course shows that a model predicting negative every time scores 99% on a dataset with 1% positives, despite being useless. Its glossary works the same trap through: 36,499 correct negatives and 25 missed positives give 0.9993 accuracy and zero recall.
Precision is the share of predicted positives that are right. Recall is the share of real positives that are found. On the illustrative table, a model that finds 36 of the 60 failures with 28 false alarms scores 0.948 accuracy, 0.5625 precision, 0.6 recall and an F1 score of 0.5806.1 Accuracy moved by under one point; recall moved from zero to 0.6. Choose precision and recall, not accuracy, when one class is rare. If a missed failure costs ten times a false alarm (an assumption), the always-ok model costs 600 units and this model 268.1
Text description of the diagram
- The always-ok baseline scores 0.94 accuracy and zero recall.
- The illustrative model scores 0.948 accuracy, 0.5625 precision and 0.6 recall.
ROC versus PR curves
The area under the curve (AUC) summarizes a curve in one number. The receiver operating characteristic (ROC) AUC compares models well when classes are roughly balanced; on imbalanced data a precision-recall (PR) curve may offer a better view.
Step 4: Analyze failure modes, then propose improvements
Failure analysis is the part that most separates submissions. Input: the held-out predictions and the true labels. Output: a short list of where and why the model is wrong.
List the missed positives and the false alarms separately and look for a pattern: a sensor range, a machine type, a time window, a category the training rows barely contain. Slice the metric by those segments. Then pick the decision threshold from the cost of each error rather than leaving it at the default. Check the features for a proxy of the target and the rows for duplicates, because leakage arises from the data collection, sampling or preprocessing strategy, not only from scaler order.
End with improvements in order of expected value, each with how you would test it on the validation set, never the test set: more data for a weak segment, a better feature, a different threshold. Say plainly what you did not try. A reviewer reads that as judgment, not as a gap.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
You have a few days with this dataset. How do you approach the classical-ML take-home?
I restate the goal and the cost of each error, then split the data before touching anything, by time or group if rows are related. Every scaler, imputer and selector lives in a pipeline fitted on the training part only, and I tune with cross-validation so the test set is used once. I show a baseline first, report precision and recall because the failures are rare, and choose the threshold from the cost of a missed failure. Then I slice the errors by segment, list what I would try next and say which parts an assistant helped with and what I checked myself.
What makes it Staff-level: it states the cost of errors up front, keeps the test set untouched and ends with evidence about the model's weak spots instead of a single score.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Scales all rows, then splits | Splits first and fits inside a pipeline | Adds time or group splits and checks duplicates and proxy features |
| Metric | Reports accuracy only | Reports precision and recall | Ties the threshold to the cost of each error |
| Baseline | None | A majority-class baseline | A baseline and a simple model, with the gap explained |
| Failure analysis | "Some errors remain" | Lists missed positives and false alarms | Slices by segment and ranks fixes by expected value |
| Communication | Code with no write-up | Result first, then method | States what was not tested and what is next |
Three variants of this question
A junior's model reports 94% accuracy on equipment failure; do you approve it?
No, not yet. If failures are 6% of rows (illustrative), an always-ok model also scores 0.94 accuracy, so the number proves nothing.1 Ask for the baseline, the confusion counts, recall on failures and how the preprocessing was fitted.
Build a classifier from the provided dataset, analyze failure modes and propose improvements.
Follow the four steps above and make the third deliverable concrete: a ranked list of changes, each tested on validation data. The analysis is graded as much as the model, so give it its own section of the write-up.
Which parts did you use an AI assistant for, and what did you fix?
Answer honestly and specifically: name the parts, say what the assistant got wrong and how you caught it, for example a scaler fitted before the split. Check the brief's own rules first; the AI-assisted coding interview page covers how such rounds are run.
Common mistakes
- "I standardized the features, then split." (wrong) The scaler has seen the test rows. (right) "I split first and fit the scaler inside a pipeline on the training rows."
- "The model reaches 94% accuracy, so it works." (wrong) A majority-class model may match it. (right) "Against a baseline of 94%, recall on failures is the number that matters."
- "I tuned until the test score was best." (wrong) The test set stopped measuring generalization. (right) "I tuned on cross-validation and scored the test set once."
- "I oversampled the whole dataset, then split." (wrong) Copies of one row land on both sides. (right) "I resample only the training part."
Follow-up questions
How do you present an ML take-home?
Lead with the result and the metric that fits the problem, then the baseline, the split and the failure analysis, then next steps. Keep the code runnable and the write-up short enough that a reviewer can check every claim; no public source sets a length.
Should you resample an imbalanced dataset before splitting?
No. Resampling before the split is a documented leakage path, so resample the training part only, inside the pipeline.
Is ROC-AUC enough to report?
Not on imbalanced data. Add precision, recall and a PR curve, and state the threshold you chose.
Does a complex model beat a simple baseline?
Not by default. Claims of that kind failed to hold in one case study once leakage was fixed, so always show the baseline gap.
Where do the other take-homes in the AI loop fit?
The retrieval-augmented generation (RAG), machine learning operations (MLOps) and agent take-homes test different habits from the classical-ML take-home: see the RAG take-home, the MLOps take-home and the agent take-home; the AI engineer questions cover the live rounds.
- The classical-ML take-home grades a defended score, not a high one.
- Split first and fit every step inside a pipeline.
- Show a baseline before any real model.
- On rare classes, report precision and recall instead of accuracy.
- Finish with sliced failure analysis and a ranked list of tested next steps.
Sources
- Common pitfalls and recommended practices (scikit-learn documentation) — scikit-learn developers, 26 Oct 2026
- Cross-validation: evaluating estimator performance (scikit-learn documentation) — scikit-learn developers, 26 Oct 2026
- Classification: Accuracy, recall, precision, and related metrics — Google for Developers, 26 Oct 2026
- Machine Learning Glossary: Metrics — Google for Developers, 26 Oct 2026
- Classification: ROC and AUC — Google for Developers, 26 Oct 2026
- Leakage and the reproducibility crisis in machine-learning-based science — Patterns (Kapoor and Narayanan), 1 Sep 2023
- Common pitfalls (imbalanced-learn documentation) — imbalanced-learn developers, 26 Oct 2026
