How do you handle an imbalanced dataset?

In this article

1. Before you start
2. Why interviewers ask this
3. Imbalanced dataset from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the machine learning algorithms learning path and uses ideas from probability and statistics.

Why interviewers ask this

A model that scores high accuracy while finding none of the rare cases is the failure this question checks for. Interviewers use it to see whether a candidate reaches for the metric before the fix, knows more than one fix, and knows the order of operations.

Candidates report the question in fraud, payments and perception settings, with wordings such as a dataset where 1% of transactions are fraud, or one million rows with 50 positives (both illustrative figures from reported wordings).

A weak answer is one word: "synthetic minority over-sampling technique (SMOTE)". A strong answer names the metric, the split and two fixes with their costs. A Staff-level answer adds how the decision threshold is chosen from the cost of each error, and what the fix does to predicted probabilities.

Imbalanced dataset from first principles

Imbalanced dataset

An imbalanced dataset is a labeled dataset in which the classes have very different counts, so a majority class heavily outnumbers a minority class. Google's course calls a dataset balanced when the counts are about equal, and notes that imbalance is far more common in practice.

Precision-recall curve

A precision-recall (PR) curve is a plot of precision against recall as the score cut-off moves. Because precision is the fraction of true positives among the positive predictions, the curve reflects how many flagged cases are real.

In plain words

Picture a bank teller checking a cash drawer for counterfeit notes. Almost every note is genuine. A teller who calls every note genuine is right nearly every time and catches nothing.

The analogy breaks in one place. A teller knows what a missed fake costs. A model knows only the loss you give it.

Google's course says training must learn what each class looks like and how common each class is. It adds: "Standard training conflates these two goals."

In one precise sentence, class imbalance in machine learning means the model sees few minority examples, so it can lower its loss by favoring the majority unless you change the measurement, the data or the loss.

Step 1: Measure with precision and recall, not accuracy

Input: true labels and model outputs. Output: metrics that react to the rare class.

Take the 202-example set in Google's course: 200 majority and 2 minority examples. A model that always outputs the majority class gets 99.01% accuracy and 0% recall.¹ Accuracy rewards it for ignoring the class you care about.

Saito and Rehmsmeier (2015) show that ROC plots can mislead on imbalanced data, owing to a wrong reading of specificity. PR plots evaluate the fraction of true positives among positive predictions.

Illustrative case: 1,000,000 transactions, 1% fraud, so 10,000 frauds. A model with 90% recall and 95% specificity catches 9,000 frauds and raises 49,500 false alarms, so its precision is 15.38%.¹ At random, flagged cases would be right 1% of the time.¹ ROC is not wrong; it just hides this cost.

Text description of the diagram

• At 5% false alarms and 90% recall on 1% fraud, precision is 15.38%.
• Specificity of 95% still leaves 49,500 false alarms.

Step 2: Split first, then rebalance only the training part

Input: the full labeled data. Output: an untouched test set and a rebalanced training set.

A stratified split is a train-test split that keeps the same class ratio in every part. Do it first, so the test set keeps the real ratio and its metrics reflect the real class ratio.

Then rebalance the training part only. One option is downsampling the majority class and upweighting it. Google reports faster convergence and less disk use, but manual effort and a ratio to tune. In an illustrative set of 990 negatives and 10 positives, keeping 99 negatives with a weight of 10 each preserves the original ratio.¹

Another option is an over-sampling method, the synthetic minority over-sampling technique (SMOTE), which creates synthetic minority rows; the imbalanced-learn implementation has a k_neighbors setting that defaults to 5. A synthetic row is built from real training rows. If you resample before splitting, near-copies of a test row can sit in training and inflate the score.

Under cross-validation the same rule applies per fold: fit the resampler on each training fold only, for example inside an imbalanced-learn Pipeline, so the validation fold keeps only real rows.

Step 3: Reweight the loss with class weights

Input: a loss and class counts. Output: the same data with a loss that treats classes unequally.

The scikit-learn "balanced" mode uses n_samples / (n_classes * np.bincount(y)), as the documentation states. With 990 negatives and 10 positives, negatives get 0.505 and positives get 50, a ratio of 99.¹ Each class then carries equal total weight, and the data is unchanged. In scikit-learn's forest classifiers, these weights multiply any undefined you pass.

Weights multiply each example's term in the cross-entropy loss, so gradient descent takes bigger steps on rare-class mistakes.

Extreme case: with 50 positives in 1,000,000 rows, the balanced positive weight is 10,000, so each positive counts as much as about 19,999 negatives.¹ A single mislabeled positive then pulls hard on the model.

Step 4: Move the decision threshold

Input: model scores. Output: a yes or no label per example.

The decision threshold is the score above which the model labels an example positive. The usual default cut-off of 0.5, the one behind undefined, was not chosen for a rare class. Read the cut-off from the PR curve and the cost of each error: a missed fraud and a false alarm rarely cost the same.

Illustrative: if a better cut-off kept 90% recall with only 0.5% false alarms, precision on the 1% fraud set would rise from 15.38% to 64.52%.¹

Weights and resampling shift the scores upward, so the raw numbers stop matching real probabilities. If you need real probabilities, recalibrate on validation data with the true class mix. If the production mix drifts, see distribution shift.

In order: measure with precision and recall before changing anything; split first, then rebalance only the training part; class weights scale each example's loss; move the decision threshold to match the cost of each error.

1 · Measure2 · Split, then rebalance3 · Class weights4 · ThresholdReplay

Step-by-step description

1. Precision and recall replace accuracy.
2. Only the training part is rebalanced.
3. Rare-class mistakes cost more in the loss.
4. The cut-off follows the cost of each error.

• Measure with precision and recall before changing anything.
• Split first, then rebalance only the training part.
• Class weights scale each example's loss.
• Move the decision threshold to match the cost of each error.

The Staff-level answer

What makes it Staff-level: it fixes the order of operations, names the cost of each fix, and treats the threshold and the probabilities as part of the answer.

The rubric

Three variants of this question

What do you do first when 1% of transactions are fraud?

Check the baseline: always saying "legitimate" scores 99% accuracy, so accuracy tells you nothing.¹ Make a stratified split, then report precision, recall and PR-area under the curve (AUC), the area under the precision-recall curve, against a random baseline of 1% precision.¹

You have 1M samples but only 50 positives. How do you handle this?

Fifty positives is 0.005% of the rows, and five stratified folds hold only 10 positives each, so every metric is noisy.¹ Collect more labeled positives, report intervals, use class weights with care, keep any resampling inside each training fold, and consider framing it as anomaly detection. For a long-tail case such as a rarely seen vehicle type, also collect or mine more of the rare examples and sample them more often in training batches.

What happens to precision and recall if you don't do any sampling?

Without sampling or weights, a model tends to favor the majority class, so recall on the minority class is usually low while accuracy looks high. Lowering the decision threshold raises recall at the cost of precision.

Common mistakes

• "My model has 99% accuracy, so it works." (wrong) That is the score of a model that never flags the rare class. (right) "Accuracy is misleading here, so I report precision and recall."
• "I oversampled, then split into train and test." (wrong) Copies or synthetic rows can leak across the split. (right) "I split first and rebalanced only the training data."
• "ROC-AUC is wrong for imbalanced data." (wrong) The paper argues ROC plots can mislead, not that they are invalid. (right) "I report PR-AUC next to ROC-AUC, because PR shows the precision cost."
• "I always balance to 50/50." (wrong) Google says to experiment with rebalancing ratios. (right) "I tune the ratio on validation data, or use weights instead."
• "The model's scores are probabilities after reweighting." (wrong) Weighting shifts them. (right) "I recalibrate if I need real probabilities."

Follow-up questions

Which of the metrics is better for imbalanced data?

The precision-recall curve is usually more informative about the cost of false alarms, because it uses precision, as Saito and Rehmsmeier argue. Report it with ROC-AUC and with metrics at your chosen threshold, and see ROC curves in scikit-learn.

When is accuracy a good metric and when is it a bad one?

Accuracy suits a roughly balanced dataset where both errors cost about the same; Google's course calls 235 positives against 247 negatives balanced. It is a bad metric when the minority class is the point, as in the 200-to-2 example in Step 1.

Is SMOTE worth using?

It is one option, not a default. The imbalanced-learn guide lists SMOTE variants and a section on ill-posed examples. No primary source ranks it above weights or threshold tuning, so compare on validation data.

Are predicted probabilities still correct after reweighting?

Not usually. Weights and resampling change the class mix the model sees, so recalibrate on validation data with the real mix if the number is used as a probability.

What else should be in the cost function?

Beyond a per-class weight (Step 3), the loss can carry asymmetric costs for each kind of error, written as a cost matrix: a missed fraud and a false alarm get separate prices, and the same prices later set the threshold. Focal loss is a named alternative that shrinks the loss of easy examples so hard ones dominate. Heavy weights on few rows can overfit; the bias and variance trade-off explains why, and the base loss is in cross-entropy loss.

Sources

1. Datasets: Class-imbalanced datasets (Machine Learning Crash Course) — Google for Developers, 24 Oct 2026
2. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets — PLOS ONE (Saito and Rehmsmeier), 4 Mar 2015
3. sklearn.utils.class_weight.compute_class_weight — scikit-learn documentation (version 1.5.2), 26 Oct 2026
4. Over-sampling (imbalanced-learn user guide, version 0.14.2) — imbalanced-learn documentation, 26 Oct 2026
5. imblearn.over_sampling.SMOTE (imbalanced-learn API, version 0.14.2) — imbalanced-learn documentation, 26 Oct 2026
6. RandomForestClassifier (class_weight parameter) — scikit-learn documentation, 26 Oct 2026
7. Tuning the decision threshold for class prediction — scikit-learn documentation, 26 Oct 2026