LLM Evaluation

How would you approach building a churn prediction model from scratch?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

You would approach building a churn prediction model from scratch by treating it as a classifier of who leaves within a window: define churn, use only data known at prediction time, split by time and set the threshold by cost. One feature taken after the prediction date, such as days since last login, can make a weak model look near-perfect in testing and fail in production.

In this article

Before you start
Why interviewers ask this
Churn prediction from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

Churn prediction is a classical table-of-features problem, so no language-model knowledge is needed. The habit it shares with large language model (LLM) evaluation is deciding how to judge a model before building it.

Why interviewers ask this

A churn model that scores well in a notebook and fails in production costs a team months, so interviewers use this question to see whether you think about the whole system and not only the algorithm. The question has no single right model, which makes it a clean test of judgement.

A weak answer names an algorithm: "I would train gradient boosting on the customer table and report accuracy." A strong answer defines churn and the prediction window, protects against leakage, splits by time and picks a metric that suits rare events. A Staff-level answer goes one step further and ties the threshold to the cost of the retention offer, then asks how the team will test whether the offer changes behaviour. A strong answer shows this order of thinking; the model choice comes last and matters least.

Churn prediction from first principles

Churn prediction model: A churn prediction model is a classifier that takes what is known about a customer at a prediction date and outputs the probability that the customer leaves within a fixed window afterwards, so a team can decide whom to contact.
Right-censoring: Right-censoring is the situation where a customer has not yet left when the data ends, so you know they stayed at least this long but not when, or whether, they will leave.

In plain words

Think of a doctor's check-up. You record a patient's measurements today and ask who will be ill within a year. The analogy breaks in one place. Your business action changes the outcome you are predicting: a customer who gets an offer may stay. So the useful output is not only "who leaves" but "who is worth contacting".

In precise terms, you freeze each customer's history at a prediction date. In a monthly snapshot, that is the day you freeze the table, such as the first of the month, and everything after it counts as the future. You attach a label saying whether they left in the following window. Then you learn the mapping from history to label.

Step 1: Define churn and its prediction window

Churn must be defined before any code, and the definition is a design choice you agree with the business. For a subscription it can be "no renewal within a set grace period"; for a shop with no contract it can be "no purchase in a chosen number of days". Then fix three windows: the history used for features, the prediction date, and the window in which leaving counts, such as the next three months (illustrative).

You also choose the framing. A binary classifier answers whether a customer churns in a given window, at one point in time (University of British Columbia applied machine learning course, 2023). Survival analysis models time until leaving and handles customers who have not left yet. Dropping them biases the answer: in an illustrative cohort of eight customers where four are still active, the average tenure of those who left is 5.25 months, against 8.625 months observed across all eight.¹ The lifelines survival analysis introduction, 2026 says that leaving right-censored people out severely underestimates the true average lifespan.

Step 2: Build features only from data known at the prediction date

Every feature must be computable on the prediction date, using nothing from after it. That includes usage counts, plan type, support tickets and billing events, each cut off at the date. A feature such as "cancellation reason", or "days since last login" measured after the prediction date, is leakage: imbalanced-learn common pitfalls, 2026 defines leakage as using information that would not be available at prediction time.

Leakage gives over-optimistic estimates and poorer production performance (scikit-learn common pitfalls, 2026). Suspiciously near-perfect scores are a warning sign, not a success. Two habits prevent it. First, fit scalers, encoders and imputers on the training data only, inside a pipeline, and never call fit on test data. Second, resample inside the training fold only; oversampling the whole dataset before splitting also leaks (imbalanced-learn common pitfalls, 2026). Validation has the same trap: the label window of every training snapshot must end before the test period's prediction date, which the gap in the next step handles. The features are now safe, so the split can be honest.

Step 3: Split by time and choose a metric for rare events

Churn data is ordered in time, so train on earlier snapshots and validate on a later period. Random folds can train on the future, which TimeSeriesSplit avoids by making each training set earlier than its test fold, and its gap parameter can model the lag before a label is known (scikit-learn TimeSeriesSplit documentation, 2026). It splits by row, so with many customers per date, split on snapshot dates instead; gap counts rows, not days.

Accuracy misleads when few customers leave. In an illustrative base with a 5% churn rate, a model that always predicts "stays" scores 0.95 accuracy and catches no churner.¹ Precision-recall analysis is the useful lens for imbalanced classes (scikit-learn precision-recall example, 2026). Precision is the share of flagged customers who truly leave; recall is the share of all leavers that get flagged. Precision also depends on the base rate (Williams, 2021): at an illustrative true positive rate of 0.6 and false positive rate of 0.1, precision is 0.6 when 20% churn but 0.24 when 5% churn.¹ Report the base rate beside precision.

Step 4: Set the threshold from the cost of the offer

The threshold decides whom you contact, and it comes from money, not from 0.5. Take illustrative values: an offer costs 20 currency units, a customer is worth 400 units and an offer saves 30% of those who would have left.¹ Contacting a customer then pays only if precision exceeds 20 / (0.3 × 400) = 0.1667, the break-even precision. Lift is the ratio of the contacted group's churn rate to the overall rate.

In this illustrative base, contacting only the top 10% of customers is the one cutoff that beats break-even.
Contact the top	Customers	Churners caught	Precision	Lift	Net value per contact (units)
10%	10,000	2,000	0.20	4.0	4.00
20%	20,000	3,000	0.15	3.0	-2.00
30%	30,000	3,600	0.12	2.4	-5.60

All figures in the table are illustrative.¹ Illustrative precision falls to 0.12 at the top 30%, below break-even, while illustrative precision at the top 10% is 0.2, above the break-even of 0.1667.

Marginal precision and calibration

The table shows cumulative group precision, but the decision to make one more contact depends on marginal precision. In this illustrative base, the second decile adds 1,000 churners in 10,000 contacts, so marginal precision is 0.10 in the second decile and 0.06 in the third, already below 0.1667 even though group precision at the top 20% is still 0.15.¹ The per-customer rule is to contact when calibrated churn probability × save rate × customer value exceeds the offer cost, which here is a score threshold of 0.1667. Calibrated means a score of 0.2 matches roughly one leaver in five such customers; check this with a reliability curve and recalibrate if it does not hold.

A model that predicts who leaves does not show who responds to an offer, so a controlled test of the campaign is the final check. After launch, monitor the score distribution and the churn rate for drift and retrain on a schedule, because precision moves with the base rate.

A churn model is built in four ordered decisions, and the last one, the threshold, sets the business value.CC BY 4.0 — source: How would you approach building a churn prediction model from scratch?

Step-by-step description

Fix the label and the window.
Use only data known at the date.
Train on the past, test on later.
Choose the cutoff from cost.

Define churn and its prediction window first.
Features use only data known at the prediction date.
Split train and test by time.
Set the contact threshold from the offer cost.

Group precision falls as more customers are contacted; the marginal decile drops below break-even sooner.CC BY 4.0 — source: How would you approach building a churn prediction model from scratch?

Text description of the diagram

Illustrative precision at the top 10% of customers is 0.2, above the break-even of 0.1667.
Illustrative precision falls to 0.12 at the top 30%, below break-even.
Marginal precision is 0.10 in the second decile and 0.06 in the third.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.