Fraud detection class imbalance: what to do about it

In this article

1. Before you start
2. Why interviewers ask this
3. Fraud detection class imbalance from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page builds on probability and statistics, and the models it applies to are covered in machine learning algorithms.

Why interviewers ask this

A fraud model that looks excellent on accuracy can be useless, and interviewers ask about imbalance to see whether you notice before you build anything. Candidates report this as a standard setup: "you have a rare-event classifier, how do you train and evaluate it?"

A weak answer says "use SMOTE" and stops. A strong answer starts with the yardstick, because every later choice depends on it. It names a metric that focuses on the fraud class, ties the decision threshold to the cost of each error, and says what resampling does to the predicted probabilities. A Staff-level answer adds the process risks: resampling only inside training folds, labels that arrive late, and fraud patterns that drift. The rest of this page builds that answer from zero.

Fraud detection class imbalance from first principles

Class imbalance

Class imbalance is a property of a labeled dataset in which the classes are not approximately equally represented, so the class you care about, such as fraud, makes up a small share of the rows and a model can neglect it without paying much in overall error.

Class weight

A class weight is a multiplier on the training loss of every example of one class, so mistakes on a rare class cost more while the model is fitted, without adding or removing any rows.

SMOTE

The Synthetic Minority Over-sampling Technique (SMOTE) is a resampling method that creates new minority-class rows by interpolating between a real minority row and one of its nearest minority neighbors, instead of copying rows.

In plain words

Picture a night guard who tells every visitor "all clear". Fraud is rare, so the guard is right almost every night and catches nothing. The analogy breaks in one place: a classifier does not just say yes or no. It gives each transaction a score, and you choose where to draw the line.

Chawla et al. (2002) say a dataset is imbalanced if the classification categories are not approximately equally represented. They add that missing an abnormal example is often much costlier than the reverse error. A public card dataset shows the scale: it has 492 frauds out of 284,807 transactions, which is 0.172% (dataset card). That is about one fraud in 578 legitimate transactions.¹

In one precise sentence, imbalance makes the majority class dominate any average, so you must choose metrics, training weights and a threshold that look at the fraud class directly.

Step 1: Accuracy hides fraud, precision and recall do not

Accuracy is the share of all transactions classified correctly. Accuracy rewards a model that never flags fraud. On the public dataset such a model scores 99.83% and catches no fraud at all.¹

Two numbers look only at the fraud class. Precision is the share of flagged transactions that really are fraud, tp / (tp + fp) in scikit-learn's definition. Recall is the share of real frauds that get flagged, tp / (tp + fn). Here tp counts true positives, fp false positives and fn false negatives. The always-legit model has recall zero.

When you need one number, balanced_accuracy_score averages the recall of each class and, per the scikit-learn model evaluation guide, avoids inflated estimates on imbalanced data. You now have a yardstick that cannot be fooled by the majority class.

Step 2: ROC and precision-recall curves tell different stories

The ROC curve plots recall against the false-positive rate at every threshold. The false-positive rate is false alarms divided by all legitimate transactions, and that denominator is huge, so the rate stays tiny. The precision-recall curve plots precision against recall, and precision divides by the number of alerts instead.

Saito and Rehmsmeier (2015) show that ROC plots can deceive on imbalanced data because specificity, the share of legitimate transactions correctly cleared, is read the wrong way. Precision-recall plots evaluate the fraction of true positives among positive predictions. The precision-recall baseline is the prevalence, not one half. Prevalence is the share of rows that are fraud; scikit-learn's curve starts at precision equal to the class balance.

Average precision is a step-wise sum of precision weighted by each gain in recall. Per the scikit-learn average precision documentation, it is not interpolated, and the trapezoid rule "can be too optimistic". The ROC curve is not wrong, though: its points do not move when the class mix changes, so it measures ranking quality. The two answer different questions, and the scikit-learn ROC curve page shows how to compute the first.

Step 3: Set the decision threshold by cost

The decision threshold is the score above which a transaction is flagged. The default of one half assumes both errors cost the same, and for fraud they do not. The threshold should follow the cost of each error.

If a missed fraud costs c_fn and a false alarm costs c_fp, flagging a transaction with fraud probability p is worth it when p × c_fn exceeds (1 − p) × c_fp. That gives a threshold of c_fp / (c_fp + c_fn). With illustrative costs of 100 units for a missed fraud and 1 unit for a false alarm, the threshold is about 0.0099, far below one half.¹

The dataset's Amount column can supply an example-dependent cost (dataset card). In practice you tune the threshold on validation data, with the review capacity of your analysts as a second constraint. This step only works if the probabilities are trustworthy, which is the subject of the last step.

Step 4: Reweighting or resampling the training data shifts the probabilities

A class weight multiplies the loss of one class, as in the cross-entropy loss. In scikit-learn, class_weight="balanced" sets each weight to n_samples / (n_classes × bincount(y)). On the public dataset that is about 289.44 for fraud and 0.50 for legitimate rows.¹

Resampling changes the rows instead. Random oversampling duplicates frauds. SMOTE interpolates along the line segment between a fraud and one of its k nearest fraud neighbors; the default in imbalanced-learn is k_neighbors=5. A float sampling_strategy of 0.1 means one fraud per ten legitimate rows, which adds about 27,940 synthetic frauds here, while a full balance would add 283,823.¹

All of these change the training prior. Calibration is the match between a predicted probability and how often the event really happens. Dal Pozzolo et al. (2015) show that undersampling biases the posterior probabilities through a monotone formula, so for an ideal model the ranking is kept and only the threshold moves. A model retrained on resampled data, and SMOTE above all, can learn a different decision boundary and rank differently, so re-evaluate it on untouched validation data. Resampling distorts the probabilities; re-tune the threshold afterwards. Resample only the training folds, because a copy of a training fraud inside a validation fold inflates your metrics.

To recalibrate after undersampling, the same paper's correction is:

Here $p_s$ is the score from the model trained on undersampled data, and $\beta$ is the share of legitimate rows kept. In the example, $\beta$ = 492 / 284,315, so a score of one half maps to a true probability of about 0.00173.¹ For any model, the general tools are Platt scaling (a logistic curve fitted to the scores), isotonic regression (a monotone step-shaped fit) and scikit-learn's CalibratedClassifierCV, all fitted on untouched validation data (scikit-learn calibration guide).

1 · Metrics2 · Curves3 · Threshold4 · Training dataReplay

Step-by-step description

1. Use precision and recall, not accuracy.
2. The precision-recall baseline is the prevalence.
3. Set it from the cost of each error.
4. Resampling distorts the probabilities; re-tune the threshold.

• Accuracy rewards a model that never flags fraud.
• The precision-recall baseline is the prevalence, not one half.
• The threshold should follow the cost of each error.
• Resampling distorts the probabilities; re-tune the threshold.

Worked example

Take the public dataset: 284,807 transactions, 492 frauds and 284,315 legitimate ones.¹ Suppose a model, with hypothetical numbers chosen for illustration, catches 80% of the frauds, which is 394 caught and 98 missed, and raises 1,000 false alarms. Its recall is 80.08%, because 394 of 492 frauds are caught. Its precision is 394 / 1,394, which is 28.26%. Its false-positive rate is 1,000 / 284,315, which is 0.35%.¹

Accuracy is 99.61% and specificity is 99.65%, while precision is only 28.26%.¹ Notice that this useful model has lower accuracy than the always-legit model's 99.83%. The balanced accuracy of 89.86% is much more honest.

Now grow the legitimate traffic tenfold at the same false-positive rate. False alarms rise to 10,000 and precision falls to 3.79%, yet the ROC point has not moved.¹ That is the ROC curve's blind spot, and the reason to report precision as well.

Text description of the diagram

• Accuracy is 99.61% and specificity is 99.65%, while precision is only 28.26%.
• Recall is 80.08%.

One more computed link between steps 3 and 4: after undersampling to a perfect balance, a model score of one half corresponds to a true fraud probability of about 0.00173, which is the base rate. That is the correction of step 4 with $\beta$ = 492 / 284,315.¹

Trade-offs and edge cases

Each fix pays for itself differently. Class weights change the loss and leave the rows alone. Undersampling throws away legitimate rows and their variety. SMOTE adds synthetic rows and leaks into validation if you run it before the split. Only threshold moving leaves both the data and the probabilities untouched, but it needs a cost estimate you can defend.

Text description of the diagram

• Class weights change the loss and leave the rows alone.
• Only threshold moving leaves both the data and the probabilities untouched.

Edge cases matter more than the textbook list:

• Very few frauds. With only a few hundred positive rows, metrics are noisy and SMOTE interpolates between very few points. Prefer weights and a tuned threshold, and use repeated validation.
• Drift and late labels. Fraud tactics change and confirmed labels arrive weeks later, so a model validated today can degrade; see distribution shift.
• Alert budget. If analysts review a fixed number of alerts a day, report precision among the top alerts, not a global curve.
• Almost no labels. Unsupervised anomaly detection can complement a supervised model.
• Features. The public dataset hides its features as twenty-eight principal components plus raw Time and Amount (dataset card); real systems lean on feature engineering.

How to answer it in an interview

The spoken structure: say why accuracy fails, name the metric, set the threshold by cost, then mention weights or resampling and the recalibration they need.

Common mistakes

• "The model has 99.8% accuracy, so it works." (wrong) A model that never flags fraud gets that too. (right) "I check recall and precision on the fraud class."
• "The area under the ROC curve (ROC-area under the curve (AUC)) is the right metric for everything." (wrong) It does not move with prevalence and can hide poor precision. (right) "I report average precision and precision at a fixed recall as well."
• "I balanced the classes to even, so the probabilities are fine." (wrong) Resampling shifts the priors. (right) "For an ideal model only the threshold moves, but a retrained model can rank differently, so I recalibrate, re-tune the threshold and re-evaluate on untouched data."
• "SMOTE just copies minority rows." (wrong) Random oversampling copies; SMOTE interpolates between neighbors. (right) "I resample inside the training folds only."
• "A threshold of one half is the default, so keep it." (wrong) The two errors have different costs. (right) "I tune the threshold on validation data from the error costs."

Follow-up questions

Why is accuracy a bad metric for fraud detection?

Because the legitimate class dominates the count. On the public dataset a model that flags nothing scores 99.83% accuracy yet has recall zero, so use precision, recall and balanced accuracy instead.¹

Should you use ROC-AUC or average precision on imbalanced data?

Report both and say which question each answers. The area under the ROC curve (AUC) measures ranking quality independent of class mix, while average precision shows what analysts will experience, and its random baseline is the prevalence.

How do you choose the decision threshold for a fraud model?

Estimate the cost of a missed fraud and of a false alarm, flag when the fraud probability exceeds c_fp / (c_fp + c_fn), then tune on validation data within your team's review capacity.

What does class_weight balanced do in scikit-learn?

It weights each class inversely to its frequency using n_samples / (n_classes × bincount(y)), so fraud rows count far more in the loss without changing the data. See the machine learning algorithms overview for the models that accept it.

Does SMOTE work for fraud detection?

It can help some learners, but it does not replace a good metric or threshold, and it shifts the probabilities. Try weights and threshold tuning first, and treat SMOTE as an experiment judged on untouched validation data.

How do you validate a fraud model when you resample?

Split first, resample only the training fold inside each cross-validation round, and score on the untouched validation fold. Respect time order where labels arrive late.

Sources

1. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets — PLOS ONE (Saito and Rehmsmeier), 4 Mar 2015
2. SMOTE: Synthetic Minority Over-sampling Technique — Journal of Artificial Intelligence Research 16 (Chawla, Bowyer, Hall, Kegelmeyer), 1 Jun 2002
3. Credit Card Fraud Detection dataset card (ULB Machine Learning Group and Worldline) — Kaggle (original dataset listing), 1 Jan 2015
4. scikit-learn 1.9.1: precision_recall_curve, average_precision_score and model evaluation — scikit-learn documentation, 26 Oct 2026
5. scikit-learn 1.9.1: average_precision_score — scikit-learn documentation, 26 Oct 2026
6. scikit-learn 1.9.1: model evaluation (balanced accuracy) — scikit-learn documentation, 26 Oct 2026
7. scikit-learn 1.9.1: probability calibration — scikit-learn documentation, 26 Oct 2026
8. Calibrating Probability with Undersampling for Unbalanced Classification — IEEE Symposium Series on Computational Intelligence (Dal Pozzolo et al.), 1 Dec 2015
9. imbalanced-learn 0.14.2: SMOTE — imbalanced-learn documentation, 26 Oct 2026