In this article
Core ideas
What is MLOps, and why does model deployment need its own practice?
MLOps is a machine learning (ML) engineering culture and practice that aims to unify ML system development and operation, with automation and monitoring at every step, including integration, testing, releasing, deployment and infrastructure management (Google Cloud Architecture Center, last updated August 2024). Deployment needs its own practice because the real challenge is not building a model but building an integrated system and operating it continuously in production.
Why it is asked: it shows whether you see a model as one part of a running system. Read the full explanation: Google Cloud, MLOps pipelines and the MLOps interview questions path.
What are CI, CD and CT in machine learning?
Google frames the automation as continuous integration (CI), continuous delivery (CD) and continuous training (CT) for ML systems (Google Cloud Architecture Center, last updated August 2024). CI tests and merges changes to code and pipelines, and in ML it also tests and validates data, data schemas and models, not only code. CD ships the result, and CT retrains the model as part of the pipeline. A model-acceptance gate that a new model must pass sits between training and release.
Why it is asked: it separates software delivery knowledge from ML-specific steps. Read the full explanation: designing CI/CD/CT for ML and large language model (LLM) apps.
Why is deploying the model not the hard part?
Developing and deploying ML systems is relatively fast and cheap, but maintaining them over time is difficult and expensive (Sculley et al., 2015). The same paper notes that a mature system might be at most 5% ML code and at least 95% glue code. So most work is data pipelines, serving and monitoring around the model, not the model.
Why it is asked: it tests whether you plan for operations, not only accuracy. Read the full explanation: Sculley et al., 2015; the models themselves are covered in machine learning algorithms.
Rollout strategies
What is a shadow deployment?
A shadow deployment is a release test in which a copy of the live inference requests goes to the new model while only the production model's responses are returned to the calling application (Amazon Web Services (AWS) News Blog). The shadow predictions are logged for analysis, and shadow variants run on separate instances, so production performance is not affected (AWS Machine Learning Blog).
Why it is asked: it tests whether you can compare a model on real traffic without risking users. Read the full explanation: canary deployment for LLM applications, the next stage after a shadow test.
What is a canary deployment?
A canary deployment is a release in which a small slice of traffic goes to the new model first while the old fleet serves the rest (Amazon SageMaker AI Developer Guide, read October 2026). The canary size should be at most 50% of the new fleet's capacity. A baking period then runs under alarms: if none trips, all traffic shifts and a final baking period follows. Linear traffic shifting is the sibling mode that moves traffic to the new fleet in several equal steps, with a baking period after each step, instead of one canary slice followed by everything.
Text description of the diagram
- A canary slice takes real traffic first, then a baking period runs under alarms.
- If no alarm trips, all traffic shifts and a final baking period follows.
- If any alarm trips, all traffic returns to the blue fleet.
Why it is asked: it checks that you limit the blast radius of a bad model. Read the full explanation: canary deployment for LLM applications.
What is a blue/green deployment?
A blue/green deployment is a release in which a new (green) fleet containing the update is brought up and traffic shifts from the existing (blue) fleet in one shot (AWS What's New, November 2021). In SageMaker, canary and linear traffic shifting are modes inside blue/green, so they are not alternatives to it. The endpoint keeps answering during the update.
Why it is asked: candidates often treat canary and blue/green as rivals. Read the full explanation: canary deployment for LLM applications, which shows the canary stage inside a blue/green release.
What is the difference between shadow and canary deployment?
Shadow returns only production responses to users, canary sends a real share of traffic to the new fleet, and a full blue/green deployment shifts traffic from the old fleet to a new fleet in one shot. Canary is itself a mode of blue/green, so the third option here is blue/green with a one-shot shift. In shadow, nobody sees the new model's output; in canary, some users do. Shadow also costs a second copy of inference, because the shadow variant needs its own instances (AWS News Blog).
Text description of the diagram
- Shadow returns only production responses to users.
- Canary sends a real share of traffic to the new fleet.
- Blue/green shifts traffic from the old fleet to a new fleet in one shot.
Why it is asked: it is the standard check that you know who is exposed to the new model. Read the full explanation: canary deployment for LLM applications.
What is the difference between a canary release and an A/B test?
A canary release is a safety check: a small share of traffic goes to the new model while alarms on latency and errors decide whether to continue or roll back (Amazon SageMaker AI Developer Guide, read October 2026). An A/B test is an experiment: users are split between variants with set traffic weights, and a business metric such as conversion is compared once enough data has been collected to tell a real difference from noise (Amazon SageMaker AI Developer Guide, A/B testing). In SageMaker the split uses production variants with traffic weights. A canary asks "is it safe?" and an A/B test asks "is it better?", so a team can run the canary first and the A/B test after.
Why it is asked: the two are routinely confused, and the answer shows you know a canary is not a powered experiment. Read the full explanation: canary deployment for LLM applications.
What triggers a rollback?
Pre-set alarms on metrics such as latency or error rates trigger an automatic rollback to the blue fleet (AWS What's New, November 2021). If any alarm trips during the baking period, SageMaker initiates a rollback and all traffic returns to the blue fleet, and you must create the Amazon CloudWatch alarms before setting up the deployment (Amazon SageMaker AI Developer Guide, read October 2026).
Why it is asked: rollback should not wait for a human, and interviewers want to hear the alarm design. Read the full explanation: canary deployment for LLM applications.
What is champion/challenger deployment?
Champion/challenger deployment is a release pattern in which the current production model, the champion, is compared with one or more challengers on the same live traffic before any of them replaces it. The challengers score the same requests through a shadow test or a traffic split, as in the entries above. The promotion metric and the minimum sample size are agreed before the test starts, promotion is a deliberate decision rather than an automatic one, and the champion stays in place as the rollback target. The pattern is also common in batch scoring, where challengers score the same batch and the results are compared after the run.
Why it is asked: teams use the words loosely, so interviewers listen for the mechanism. Read the full explanation: canary deployment for LLM applications and the shadow entry above.
Why does a new model version need staged validation?
Because of the principle Sculley et al. call Changing Anything Changes Everything (CACE) (Sculley et al., 2015). A new feature or model version can shift behaviour in ways a unit test will not show. Vendor guidance orders the rollout: complete a shadow test, then use deployment guardrails such as canary or linear shifting to switch the model (AWS News Blog).
Why it is asked: it tests whether you can justify the extra rollout steps. Read the full explanation: Sculley et al., 2015.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Monitoring after release
After deployment, how do you monitor the model?
Combine live monitoring with automated response: comprehensive real-time monitoring of system behaviour plus automated response is critical for long-term reliability (Sculley et al., 2015). Unit and end-to-end tests are valuable but not sufficient in a changing world. Watch latency and errors, the distribution of predictions, input distributions and upstream data producers, and alert on each.
Why it is asked: it separates "ship and forget" from running a system. Read the full explanation: LLM observability: tracing, cost and quality signals for the generative-model version.
How do you monitor a model when the true labels arrive late?
Split the checks by whether they need a label. Label-free signals work at once: the distribution of the model's scores, the distribution of its inputs, action limits so that alerts fire when a system hits a limit for a given action, and monitoring of the upstream producers of your input data (Sculley et al., 2015). Checks that need observed labels, such as comparing the distribution of predicted labels with observed labels (prediction bias) and measuring accuracy, run once the labels arrive.
Why it is asked: real labels often lag, and monitoring only accuracy fails. Read the full explanation: Sculley et al., 2015.
Does matching predicted and observed label distributions prove the model is good?
No. Prediction bias is a gap between the distribution of predicted labels and the distribution of observed labels, and in a system working as intended the two should usually be equal (Sculley et al., 2015). But a model that ignores its inputs can pass that check by predicting the base rate. Treat it as a diagnostic that raises an alarm, not a proof of quality.
Why it is asked: it tests whether you know what a monitoring signal can and cannot prove. Read the full explanation: Sculley et al., 2015.
What is model drift, and how do you catch it?
Model drift is a decline in model quality because live data or the relationship between inputs and outcomes moves away from the training data. Catch it by monitoring input distributions against a training baseline, watching prediction distributions, and measuring accuracy as labels arrive. Sculley et al. list data testing, meaning sanity checks plus monitoring of input-distribution changes, among the starting points (Sculley et al., 2015). Retraining is the usual fix.
Why it is asked: it links monitoring to the retraining trigger. Read the full explanation: trained in January, bad in July: model drift explained.
Why do fixed decision thresholds go stale when a model updates?
A classifier's score distribution changes when the model or its features change, so a cut-off tuned for the old model may no longer mean the same thing. Sculley et al. suggest learning thresholds from held-out validation data instead of setting them by hand once (Sculley et al., 2015). Re-derive the threshold with every model version, and include it in the release checklist.
Why it is asked: it catches candidates who treat the threshold as a constant. Read the full explanation: Sculley et al., 2015.
Serving and system design
How do you design model deployment, monitoring and low-latency inference for a fraud-detection model?
Start from requirements: how fast a decision must be returned and what a missed fraud costs. Serve the model behind a real-time endpoint with features computed the same way as in training, release through shadow then canary, and alarm on latency, error rate and prediction distribution. Keep a fallback rule set so a failed model never blocks payments. The rollout arithmetic is in the worked example below.
Why it is asked: it combines serving, release safety and monitoring in one answer. Read the full explanation: batch vs streaming vs real-time inference and design a feature store for machine learning; if tail latency jumps after a release, see a sudden jump in tail latency: what do you do?.
How do you design a large-scale AI model deployment system with GPU scaling, versioning and caching?
Name the parts: a model registry that versions each artefact, a serving layer that loads versions onto graphics processing units (GPUs) and scales replicas with load, a router that splits traffic between versions, and a cache for repeated requests where inputs repeat. Add request batching to raise GPU use, and monitoring that feeds autoscaling. State the trade-off: caching and batching save cost but add staleness and queueing delay.
Why it is asked: it checks whether you can organise a large design into components. Read the full explanation: Kubernetes for ML infrastructure.
How do you keep feature computation identical at training and serving time?
Compute features from one shared definition, not two copies of the logic. A feature store gives training and serving the same feature code, with historical values for training and the latest values for low-latency lookup. A model-acceptance gate in the pipeline should also check that serving outputs match training-time outputs on the same inputs before a release is allowed.
Why it is asked: skew between training and serving is a classic cause of silent failures. Read the full explanation: design a feature store for machine learning.
How would you optimize a trained model for an edge device with limited compute?
Shrink the model and match it to the hardware. Common techniques are quantization, which stores weights in fewer bits; pruning, which removes weights or channels that matter little; and distillation, which trains a small model to imitate a large one. Measure accuracy and latency on the target device after each change, and keep the original model as the fallback.
Why it is asked: it tests whether you treat accuracy, size and speed as a trade-off. Read the full explanation: deploying models to robots and embedded devices.
How do you version models and handle systems that depend on their output?
Version every model artefact so a release can go back to the previous one. Sculley et al. warn that without access controls, some consumers of a model's output may be undeclared, and that these couple the model to other systems. They suggest access controls or strict agreements to limit that. A versioned copy of an unstable input signal carries its own costs, such as staleness (Sculley et al., 2015).
Why it is asked: it tests whether you see hidden coupling before it breaks a release. Read the full explanation: designing CI/CD/CT for ML and LLM apps and, for the data dependencies, how to design the data platform under ML.
A worked rollout for a fraud model
Take a fraud model that serves 400 requests per second (illustrative). A 25% canary sends 100 requests per second to the new fleet, so a 600-second baking period sees 60,000 requests.1 In SageMaker the canary is a share of the green fleet's capacity, so traffic share and capacity are the same number: on a 20-instance green fleet (illustrative), 5 instances is 25% of capacity, under the 50% limit in the SageMaker documentation.1
Suppose the old model's server-side invocation error rate is 0.2% (illustrative). This is the rate an alarm can watch during a ten-minute bake; the model's misclassification rate is not observable yet, because fraud labels arrive late. About 120 errors are expected in the window. The random noise is one standard deviation, σ = √(n·p·(1−p)) with n = 60,000 requests and p = 0.002, which is about 10.94 errors.1
An alarm at twice the baseline, 240 errors, sits 10.97 standard deviations out. It is almost never noisy, but it catches only gross failures: a 50% rise, to about 180 errors, is 5.48 standard deviations above the mean and never trips it.1 An alarm at the baseline plus 4 standard deviations, about 164 errors, catches a rise of about 36% at a small false-alarm risk.1 A shadow test of 72 hours on 10 shadow instances adds 720 instance-hours.1
| Quantity | Value (illustrative, computed) |
|---|---|
| Canary share of traffic and capacity | 25% |
| Canary instances on a 20-instance fleet | 5 (25% of capacity) |
| Requests in a 600-second baking period | 60,000 |
| Expected errors at a 0.2% baseline | 120 |
| Noise, one standard deviation | 10.94 errors |
| Alarm at baseline plus 4 standard deviations | about 164 errors |
| Alarm at twice the baseline | 240 errors |
| Distance of the 240-error alarm from the mean | 10.97 standard deviations |
| Extra instance-hours for a 72-hour shadow test on 10 instances | 720 |
Follow-up questions
How should I structure my answer to a deployment question?
Say the goal, the release path, the guardrail and the monitor, in that order: for example "validate with a shadow test, release with a canary under alarms, roll back on error or latency, then monitor prediction and input distributions". Name one trade-off, such as the cost of a shadow fleet against the safety it buys.
Do I need to know one cloud provider's tools to answer these?
No. The ideas of shadow, canary, blue/green and alarm-driven rollback are general, and the vendor pages above are examples of them. Say the concept first and name a tool only as an illustration. For the surrounding design skills, see designing CI/CD/CT for ML and LLM apps.
What deployment questions are asked for senior roles?
Senior questions go deeper on trade-offs: what a bad rollout costs, how you detect a silent regression without labels, how you version and roll back, and how production feedback returns to training. Read how to build a data flywheel from production feedback for the loop that closes after release.
- MLOps unifies development and operation of ML systems, with automation and monitoring at every step.
- Shadow shows users nothing new, canary exposes a small share, and full blue/green moves traffic to a new fleet in one shot; a canary checks safety, an A/B test checks a business metric.
- Alarms on latency or error rates trigger an automatic rollback, and they must exist before the rollout.
- Label-free signals are score and input distributions, action limits and upstream producers; prediction bias and accuracy need labels, and no single check proves quality.
- Compute your rollout numbers: share, window, expected errors and alarm threshold.
Sources
- MLOps: Continuous delivery and automation pipelines in machine learning — Google Cloud Architecture Center, 28 Aug 2024
- Hidden Technical Debt in Machine Learning Systems (Sculley et al.) — NeurIPS 2015 (Google), 7 Dec 2015
- Use canary traffic shifting — Amazon SageMaker AI Developer Guide, 26 Oct 2026
- New for Amazon SageMaker: Perform Shadow Tests to Compare Inference Performance Between ML Model Variants — AWS News Blog, 30 Nov 2022
- Announcing new deployment guardrails for Amazon SageMaker Inference endpoints — AWS What's New, 10 Nov 2021
- MLOps deployment best practices for real-time inference model serving endpoints with Amazon SageMaker — AWS Machine Learning Blog, 26 Oct 2026
- A/B testing ML models in production using Amazon SageMaker — Amazon SageMaker AI Developer Guide, 26 Oct 2026
