ML System Design

How would you build a model to detect fraudulent host listings?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 11 min read

How would you build a model to detect fraudulent host listings? Train a classifier on risk signals such as host reputation and duplicate photos, score every new listing before it goes live, and act on the score in three tiers: block, human review or allow. A strong answer judges it by precision and recall, not accuracy, and the follow-up is why a model that is 99.5% accurate can catch zero frauds.¹

In this article

Before you start
Why interviewers ask this
Fraudulent host listings from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This question belongs to the machine learning (ML) system design interview path, and its closest neighbor is real-time fraud detection.

Why interviewers ask this

The question is a rare-event problem with a human cost on both sides, so one answer shows whether a candidate thinks about data, metrics and operations or only about algorithms. Candidates report it in machine learning (ML) system design rounds for marketplaces; no public source says which company asks it.

A weak answer names a model ("gradient boosting on listing features") and stops. A strong answer starts from the label and the base rate, says what happens to a listing at each score, and picks metrics that survive imbalance. A Staff-level answer adds the parts around the model: the review queue's capacity, the feedback loop that creates new labels, re-scoring over time, and the fact that fraudsters adapt. The interviewer is listening for those trade-offs, because a model is a small part of a trust system.

Fraudulent host listings from first principles

Listing-fraud model: A listing-fraud model is a classifier that reads the signals of a host and a listing, such as reputation, message templates and photos, and outputs a risk score that the platform turns into an action: block, review or allow.
Operating threshold: An operating threshold is the score cutoff chosen after training that decides which listings are blocked, reviewed or allowed, and moving it trades missed fraud against wrongly flagged honest hosts.

In plain words

Think of a hotel front desk that checks every new booking. It lets most guests in, turns away a few obvious impostors, and calls a manager for the unclear ones. The analogy breaks in one place. Here the desk is a model that learns its checks from past cases. Impostors also change their story once they learn the checks.

In one precise sentence: score each listing for fraud risk, then pick cutoffs that turn the score into actions at a cost you accept.

Step 1: Name the label and the class imbalance

Labels come first, and review outcomes and user flags become new labels that retrain the model.CC BY 4.0 — source: How would you build a model to detect fraudulent host listings?

Step-by-step description

Start from confirmed fraud labels.
Risk signals feed a risk model.
Cutoffs route to block, review or allow.
Outcomes become new labels.

Labels come first, then risk signals and a model.
Thresholds route each listing to block, review or allow.
Review outcomes and user flags become new labels for retraining.

The label is "this listing was later confirmed as fraud", and fraud is rare, so the data show class imbalance, which is a condition where one class is far rarer than the other. Airbnb said in 2017 that fake listings are rare but gave no rate, so this page assumes that 0.5% of new listings are fraud, an illustrative figure.¹ Out of 100,000 new listings that is 500 frauds. A model that always answers "legit" is right 99.5% of the time and finds none of them.¹

Accuracy therefore misleads. The scikit-learn course from Inria says accuracy should not be used on imbalanced classes, and the scikit-learn guide offers balanced accuracy instead. Precision is the share of flagged listings that are really fraud, tp / (tp + fp), and recall is the share of all frauds that get flagged, tp / (tp + fn), as the scikit-learn documentation defines them. Labels matter most: the Airbnb risk engineers' takeaways heading in their 2014 post reads "ground truth > features > algorithm choice".

Step 2: Build risk signals and a first model

Risk signals are the inputs, and the labels of Step 1 are what the model learns from. Airbnb described in 2017 a system that evaluates each listing against hundreds of risk signals, naming host reputation, template messaging and duplicate photos. A useful grouping is the host and account, the listing content, and behavior over time, such as bursts of new listings. Photos need image matching; the computer vision system design questions cover that part.

Start with rules and add a model once the rules get unwieldy. The 2014 Airbnb post says that for complex, evolving fraud, heuristics eventually become too complicated, and it reports that random forests had an appropriate precision-recall for its purposes. A tree ensemble on mixed features is a sound first model. Sudden spikes that no rule anticipated belong to anomaly detection, and non-ML controls still count: Airbnb screens out contact details until a booking and holds payment until after check-in, which makes fraud harder to cash out. In this step the signals and a model sit between the labels and the thresholds.

Step 3: Choose thresholds and three actions

The model outputs a score, and an operating threshold turns it into one of three actions. Airbnb described in 2017 that a high predicted likelihood leads to an automatic block, and other cases are delayed for extra review. That gives block, review or allow, with thresholds routing each listing to one of the three.

Moving the cutoff trades recall against precision. At the three assumed operating points below the model finds 300, 400 or 450 of the 500 frauds.¹ The cost uses an assumed ratio of 200 cost units for a missed fraud to 5 for a wrongly flagged honest host, and the review team can inspect 1,500 listings per 100,000, both illustrative.¹

To keep the arithmetic simple, each operating point here is a single review cutoff: every flagged listing goes to the review queue, and the 5-unit cost is the reviewer time plus the friction for an honest host (illustrative). In the full system a second, higher cutoff blocks automatically, and its false alarms cost far more.

A lower cutoff catches more fraud but drops precision from 60.1% to 13.1% and overflows a review queue that holds 1,500.
Quantity	High cutoff	Middle cutoff	Low cutoff
Recall	60%	80%	90%
False-positive rate	0.2%	1%	3%
Frauds caught	300	400	450
False alarms	199	995	2,985
Listings flagged	499	1,395	3,435
Precision	60.1%	28.7%	13.1%
Expected cost (cost units)	40,995	24,975	24,925

At the middle cutoff recall is 80% and precision is 28.7%.¹ At the low cutoff precision falls to 13.1% while recall rises to 90%. The low cutoff has the lowest cost by a hair, but it flags 3,435 listings, more than the queue can hold, so the middle cutoff wins. Capacity, not only cost, picks the threshold.

Lowering the cutoff buys recall at the price of precision, and the review queue grows with it.CC BY 4.0 — source: How would you build a model to detect fraudulent host listings?

Text description of the diagram

At the middle cutoff recall is 80% and precision is 28.7%.
At the low cutoff precision falls to 13.1% while recall rises to 90%.

Step 4: Score at the right time and close the loop

Scoring happens at listing creation and again as new events arrive, and the answers to those scores become new labels. The 2014 Airbnb post describes an asynchronous design, fine where a few seconds of delay has no negative effect, and notes that most of its scoring responses took under 10 milliseconds. Blocking a listing before it goes live needs the fast path with precomputed features. Re-scoring matters because a bad host may look fine at upload.

Models do not learn on their own. The Airbnb engineers wrote that "models cannot train themselves on-the-fly" and that a secondary mechanism must account for new ground truth. In the 2017 post, user flags feed the risk model, which decides between automatic removal and manual review. Review outcomes and user flags therefore become new labels for retraining. A related loop is described in harmful-content detection. A common pitfall is to treat labels from review only: reviewed cases are the ones the model already found suspicious, so keep a small random sample reviewed too.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.