In this article
Before you start
Related: machine learning operations (MLOps) interview questions. This page assumes the basics of machine learning algorithms.
Why interviewers ask this
A model that beats the current one on a held-out test set can still lose on live traffic, because production data shifts and test sets age. Champion challenger deployment is how a team gets evidence from live traffic before it changes what any user sees, so it comes up in the deployment part of a machine learning (ML) system design round.
A weak answer says "deploy the new model and watch the metrics." A strong answer separates who answers users from who is only scored, explains how the two models are compared fairly, and names the cost. A Staff-level answer adds the label delay, the sample size needed to call a winner, and a promotion rule agreed before the test starts. The same habit of checking a model against live data runs through model drift and large language model (LLM) observability.
Champion challenger model deployment from first principles
- Champion
- In a champion challenger setup, the champion is the model that currently serves production traffic, the one whose predictions reach users or downstream systems, and it is the baseline that every candidate must beat before it is allowed to replace it.
- Challenger
- A challenger is a candidate model, deployed beside the champion, that scores the same live requests but has its predictions logged instead of returned, so it can be compared with the champion on real traffic without affecting any user. Several challengers can run at once.
- Shadow deployment
- Shadow deployment is a release technique that sends a copy of all or a sampled share of live requests to a second model and logs its responses without returning them, so the second model is tested on real traffic while it serves nobody. It is the form of champion challenger described here, also called shadow testing or a dark launch.
In plain words
Picture a restaurant. A new chef cooks every order beside the head chef. Only the head chef's plate goes to the table. The new chef's plate goes on a shelf. Later, the team checks which plate matched what the diner wanted. The analogy breaks in one place. A diner's reaction is visible at once. A model's true outcome may arrive weeks later. And nobody reacts to the challenger's output at all.
In one precise sentence, champion challenger deployment scores a candidate model on the same live requests as the production model, withholds the candidate's output from users, and compares the two once true labels exist.
Step 1: Duplicate each request and answer only with the champion
The champion answers the caller; the challenger scores a copy of the request. Seldon's documentation puts it as traffic that "is duplicated so that it goes to both a main model and also a shadow model", and the responses that reach the end user come only from the main model (Seldon docs, as of October 2026).
Dataiku's documentation describes this mode as deploying one or more extra models whose predictions are "simply logged for further analysis" (Dataiku Knowledge Base, as of October 2026).
Input: one live request. Output: one answer to the caller and one hidden score. SageMaker forwards a user-set traffic sampling percentage to the shadow variant (Amazon Web Services (AWS) Machine Learning Blog, as of October 2026). A sensible ramp, with illustrative steps of 5%, 25% and 100%, starts low and rises as confidence grows.
As design advice, not a platform requirement, call the challenger asynchronously with its own timeout, so a slow challenger never delays the champion. You now know how the two models see the same input; next comes the record that makes comparison possible.
Step 2: Log both predictions under one request id
Both predictions are logged under one request id. Without a shared id, the champion's score and the challenger's score for the same request cannot be paired later, and a paired comparison is the fair one.
Input: the two predictions. Output: one log row per request. A useful row holds the request id, a timestamp, the model versions, the input features (or a pointer to them), and both scores. SageMaker can log the shadow variant's response in Amazon Simple Storage Service (Amazon S3) or discard it (AWS Machine Learning Blog, as of October 2026). Keep it. A discarded response tests only that the challenger runs; a logged one lets you test whether it is right. The feature store design page shows how to make sure both models read the same feature values.
Step 3: Wait out the label delay, then compare on the same requests
The comparison waits for the true labels to arrive. A label is the true outcome for a request, such as "this payment was fraud." Until labels exist, the log holds two columns of guesses and no verdict.
Input: the log plus the labels. Output: a metric for each model on the same requests, and a test of whether the gap is real. Label delay decides how soon this step can start. For fraud, a chargeback can take weeks; for a click, seconds.
This is why the answer to "how long do we run the challenger?" has two parts: the time to collect enough labelled requests, and the label delay on top. The worked example below puts numbers on both, and a larger sample is what turns a small gap into evidence.
Step 4: Promote the winner or retire the loser
A winning challenger is promoted; a losing one is retired. Promotion makes the challenger the new champion, and the old champion becomes a baseline or is removed. Both Seldon and SageMaker offer promotion as a platform step: SageMaker, once you confirm, updates the endpoint to make the shadow variant the new production variant (AWS Machine Learning Blog, as of October 2026).
Before promoting, raise sampling to 100 percent. AWS says sampling 100 percent of requests lets you validate that the shadow variant can handle production traffic once promoted, and recommends the same instance type, size and count as production (SageMaker developer guide, as of October 2026). Write the promotion rule down before the test starts. The continuous delivery design page covers how that rule fits a pipeline.
Step-by-step description
- The champion answers; the challenger scores a copy.
- Both predictions share one request id.
- Compare on the same requests once labels arrive.
- The winner is promoted, the loser retired.
- The champion answers the caller; the challenger scores a copy of the request.
- Both predictions are logged under one request id.
- The comparison waits for the true labels to arrive.
- A winning challenger is promoted; a losing one is retired.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a fraud model whose challenger has been shadowed on 10,000 requests that now have true labels (illustrative). The counts below are illustrative; every figure derived from them was computed.1
The champion was right on 9,500 requests and the challenger on 9,580 (illustrative), so accuracy is 95.0% against 95.8%. Accuracy is used only to keep the arithmetic readable; a real fraud team would compare precision, recall and the cost of each error.
Only the requests where the two models disagree can separate them. There are 680 such requests: the challenger was right on 380 and the champion on 300. The McNemar test, a standard test for paired yes/no outcomes, asks whether a split of 380 to 300 could be luck if both models were equally good. Its statistic, without continuity correction, is (380 − 300)² / 680 = 9.41, and the p-value is 0.0022, so luck is an unlikely explanation. With continuity correction the statistic is 9.18, which still gives a p-value below 0.01.1
| Quantity | Value (illustrative inputs, computed results) |
|---|---|
| Labelled requests compared | 10,000 |
| Both models right | 9,200 |
| Champion right, challenger wrong | 300 |
| Challenger right, champion wrong | 380 |
| Both wrong | 120 |
| Champion accuracy | 95.0% |
| Challenger accuracy | 95.8% |
| Difference | 0.8 percentage points |
| 95% confidence interval of the difference | 0.29 to 1.31 percentage points |
| p-value, McNemar test | 0.0022 |
| Same rates with 1,000 requests | p-value about 0.33, interval −0.82 to 2.42 points |
With a tenth of the requests and the same rates, the interval includes zero, so no winner can be called.1 Sample size, not the size of the gap, makes the verdict.
Time matters too. At 20,000 requests a day (illustrative), a 5% sample reaches 10,000 requests in 10 days and a 100% sample in 1 day. A 30-day label delay (illustrative) adds to every case, so the verdict comes on day 40, 32 or 31.1
Text description of the diagram
- A 5% sample reaches 10,000 requests in 10 days and a 100% sample in 1 day.
- A 30-day label delay adds to every case, so the verdict comes on day 40, 32 or 31.
Cost is the other number. If the champion runs on 4 instances (illustrative), a 100% shadow adds 4 more, so total inference compute is twice the normal bill during the test.1
Trade-offs and edge cases
Shadow deployment trades money and time for safety. It exposes nobody to the challenger, but it needs a second fleet and cannot show how users react. Three release patterns are often confused, so keep them apart. Shadow sends a copy of all live traffic to the challenger and logs its responses without returning them (Seldon docs, as of October 2026), or a sampled share of it (AWS Machine Learning Blog, as of October 2026).
A canary release is a rollout that sends a small percentage of real traffic to the new model while most traffic stays on the old one (Seldon canary docs, as of October 2026). In an A/B test the candidate scores a portion of live requests and serves them (Dataiku Knowledge Base, as of October 2026).
Text description of the diagram
- Shadow sends a copy of live traffic to the challenger and logs its responses without returning them.
- A canary sends a small percentage of real traffic to the new model, and an A/B test serves a portion of live requests.
Edge cases to name:
- No business impact signal. The challenger's output never reaches users, so shadow cannot measure clicks or conversions. This follows from how the pattern works; no source states it directly.
- Side effects. A challenger must not send messages, write shared state or charge cards. Only the champion's response may act.
- Platform limits. A SageMaker shadow test pairs one shadow variant with the production variant (CreateEndpointConfig application programming interface (API) reference, ShadowProductionVariants, as of October 2026); testing several challengers needs several tests or a custom fan-out.
- Different vocabulary. The Federal Reserve's 2011 supervisory guidance on model risk management names "effective challenge" of models, meaning critical analysis by objective, informed parties, as a guiding principle (Federal Reserve Board, 2011). In banking, challenge is broader than live deployment.
Sources use the term differently, so state which form you mean. Some teams also use champion challenger for a live split in which the challenger makes real decisions for a small share of cases; Dataiku would file that under A/B testing (Dataiku Knowledge Base, as of October 2026). This page teaches the shadow form. Logged challenger data can also feed the next training run, as in a data flywheel built from production feedback.
How to answer it in an interview
How would you safely test a new model against the one in production?
I would run champion challenger deployment. The current model, the champion, keeps answering users while the new one, the challenger, scores a copy of the same live requests. Its predictions are logged under one request id and never returned. When the true labels arrive, I compare both models on the same requests and test whether the gap is real, not noise. Label delay sets how long that takes. If the challenger wins, I raise sampling to full traffic to prove capacity, then promote it. The trade-off is a second fleet's cost, and it cannot show how users react.
The spoken structure: say who answers and who is only scored, say how you compare, name the label delay, then give the cost and the limit.
Common mistakes
- "Champion challenger always means the challenger serves nobody." (wrong) The term has two forms. (right) "Say which form you mean: shadow, where it serves nobody, or a live split, where it serves a share."
- "The challenger wins if its test-set metric is higher." (wrong) The point is comparison on the same live requests. (right) "I compare both on logged production requests, with labels."
- "Shadow is free." (wrong) It needs a second fleet, and 100% sampling roughly doubles inference compute. (right) "I budget for the second fleet and match the instance type."
- "Two weeks of shadowing is enough." (wrong) The label delay may be longer. (right) "I size the sample and add the label delay before I set the date."
- "Shadow shows the business impact." (wrong) Users never see the challenger. (right) "Shadow shows prediction quality; impact needs a canary or an A/B test."
Follow-up questions
What is the difference between champion challenger and A/B testing?
In shadow champion challenger the challenger scores the same requests as the champion and serves none. In an A/B test the candidate scores a portion of live requests and serves them, so user reaction can be measured.
What is the difference between shadow deployment and a canary release?
Shadow sends a copy of live traffic to the new model and logs its responses without returning them, so nobody sees the new model. A canary sends a small share of real traffic to the new model, so some users do get its answers.
How long should you run a challenger?
Long enough to collect the labelled requests the test needs, plus the label delay. In the worked example 10,000 requests were needed, and a 30-day delay (illustrative) made the earliest verdict day 31 even at full sampling.
Does shadow deployment cost twice as much?
At 100 percent sampling with the same instance type as production, inference compute roughly doubles during the test, which AWS recommends so the load test is realistic. A lower sampling percentage costs less only if the shadow fleet autoscales with the sampled traffic; with a fixed instance count the fleet bill is the same at any rate, and a low rate also proves less about capacity.
What does challenger mean in banking model risk?
It points to the "effective challenge" idea in model risk management, critical analysis of a model by objective, informed parties, which is broader than live deployment. See the MLOps interview questions hub for the neighbouring deployment topics.
- The champion answers users; the challenger scores a copy of the same requests and is only logged.
- Log both predictions under one request id so the comparison is paired.
- Label delay and sample size, not a calendar guess, decide when a verdict is possible.
- Raise sampling to 100 percent before promotion to prove capacity.
- Shadow exposes no user but cannot measure user reaction and needs a second fleet.
Sources
- Monitoring and feedback in the AI project lifecycle — Dataiku Knowledge Base, 26 Oct 2026
- Testing models with shadow variants — Amazon SageMaker AI Developer Guide, 26 Oct 2026
- Minimize the production impact of ML model updates with Amazon SageMaker shadow testing — AWS Machine Learning Blog, 26 Oct 2026
- Best practices (shadow tests) — Amazon SageMaker AI Developer Guide, 26 Oct 2026
- Shadow (rollout strategies) — Seldon Enterprise Platform docs, 26 Oct 2026
- Canary (rollout strategies) — Seldon Enterprise Platform docs, 26 Oct 2026
- CreateEndpointConfig (ShadowProductionVariants) — Amazon SageMaker API Reference, 26 Oct 2026
- Supervisory Letter SR 11-7: Guidance on Model Risk Management — Federal Reserve Board, 4 Apr 2011
