MLOps interview questions: take a model to production

In this article

1. The vocabulary of MLOps
2. Start here: the learning path
3. What each step of the path teaches
4. Questions interviewers ask on MLOps
5. All 26 guides in MLOps
6. Follow-up questions
7. Sources

The vocabulary of MLOps

MLOps

MLOps is an engineering practice that applies DevOps principles to machine learning systems, automating and monitoring every step from integration and testing to release, deployment and infrastructure, so that a model keeps working after it ships.

Model registry

A model registry is a store of trained models that are ready for serving, each with a version and the metadata that records how it was built, so a release or a rollback means pointing traffic at a named version.

Data drift

Data drift is a change over time in the distribution of the inputs a deployed model receives, so that live data stops resembling the data the model was trained on.

Start here: the learning path

MLOps questions follow one loop: version what you train, validate it, release it gradually, monitor it, and decide whether to keep, roll back or retrain. The sections below teach each step with the numbers behind it, from the model registry and canary sizes to drift tests and the p99 latency fan-out problem. The practice Google's guide treats as unique to machine learning systems is continuous training (CT); together with continuous integration (CI) and continuous delivery (CD) it forms CI/CD/CT, and the answer to "how would you decide when to retrain" sits at the end of the loop.

1 · Version everything2 · Validate3 · Canary4 · Serve5 · Monitor and reactReplay

Step-by-step description

1. Code, data snapshot, feature definitions, config and weights each get a version; the model goes to a registry.
2. Tests and a comparison against the current production model decide whether the release may proceed.
3. A small share of live traffic goes to the new model while error and latency are compared.
4. If the canary holds, traffic shifts to the new model; the old version stays ready for rollback.
5. Service, data, model and business signals decide whether to keep, roll back or retrain.

• Artifacts to version include code, data snapshot, feature definitions, configuration and model weights.
• Validation compares a new model with the current production model before release.
• A canary release sends a small share of live traffic to a new model version first.
• Monitoring covers service health, data, model quality and business impact, and can trigger a rollback or retraining.

The story usually appears in the machine learning system design round, leans on large language model (LLM) inference for serving cost and memory, and depends on data engineering for the pipelines that feed the model.

1. Design a feature store for machine learning
2. Design distributed training for a 10B to 1T parameter model
3. Batch vs streaming vs real-time inference: how to choose
4. How is Kubernetes used for ML infrastructure?
5. How do you design the data platform under ML?
6. What happens when one GPU of 128 fails at hour 16?
7. Trained in January, bad in July: model drift explained
8. What is LLM observability? Tracing, cost and quality signals
9. Deploying Models to Robots and Embedded Devices
10. How do you design CI/CD/CT for ML and LLM apps?
11. The p95 went from 100 to 2,000 milliseconds: what do you do?
12. How do you handle training-serving skew?
13. How do you estimate model training duration?
14. How do you design a scalable ML training pipeline on Azure?
15. What is distribution shift, and how do you detect it?
16. How do you build a data flywheel from production feedback?
17. How will you process 100,000 files across multiple servers on Hadoop?
18. What are the challenges of deploying AI models in the Saudi Arabia market?
19. What is canary deployment for LLM applications?
20. What are reproducibility and governance in machine learning?

What each step of the path teaches

What is MLOps in simple words?

MLOps is the work that keeps a model useful after the notebook: packaging it, testing it, releasing it safely, watching it and updating it. Google's architecture guide describes it as applying DevOps principles to machine learning systems and says the hard part is not building a model but running an integrated machine learning system in production (Google Cloud, page updated 2024-08-28). The reason it exists is that machine learning code is a small part of the system: the earlier "Hidden Technical Debt" paper lists entanglement between parts, hidden feedback loops, undeclared consumers of a model's output, data dependencies and configuration problems among the sources of long-term cost (Sculley et al., 2015).

What artifacts do you version, and where do they live?

Version five things: the code, a snapshot or pointer of the training data, the feature definitions, the configuration and the model weights. If any one is missing you cannot reproduce or roll back a model. The usual split is that weights, which are large binary files, live in object storage, while a model registry holds the version, a pointer to the weights and the metadata. Google's guide describes a metadata store that records pipeline runs, including versions, parameters, artifacts and evaluation metrics, to support lineage, reproducibility, rollback and debugging (Google Cloud, 2024-08-28). The test of a good answer is the rollback sentence: "I can repoint serving to version N−1 in minutes without retraining."

How do you release a model safely and roll it back?

Release in stages: validate against the current production model, then canary, then full traffic, keeping the previous version warm for rollback. The same guide describes automated model validation that compares a new model with the current production model, tests for infrastructure compatibility and consistency with the prediction service's application programming interface (API), load tests, and online validation of a new model in a canary deployment or A/B test before it serves all traffic. A canary only helps if it runs long enough to see a difference, and rarer errors need far more traffic.

Under a two-proportion test at 5% significance and 80% power, spotting a rise in error rate from 1% to 2% needs 2,319 requests per arm, spotting 1% rising to 1.5% needs 7,750 per arm, and detecting 0.1% rising to 0.2% needs 23,511 per arm. The same one-point rise from 5% to 6% needs 8,158 per arm, because a higher baseline is noisier.¹ At an assumed 1,000 requests per second with 5% on the canary, the first case takes about 46 seconds; the last takes far longer, roughly 470 seconds at the same traffic.

Text description of the diagram

• Spotting a rise in error rate from 1% to 2% needs 2,319 requests per arm.
• Spotting 1% rising to 1.5% needs 7,750 per arm.
• Detecting 0.1% rising to 0.2% needs 23,511 per arm.
• The same one-point rise from 5% to 6% needs 8,158 per arm, because a higher baseline is noisier.

What do you monitor after launch?

Monitor four layers: service health, data quality and drift, model performance, and business impact. Service health is latency, errors and saturation. Data monitoring compares live input statistics with the training data. Model performance needs labels, which often arrive late, so proxies stand in until they do. Business impact is the metric the model exists to move. Google's guide describes monitoring as tracking data statistics and live model performance to trigger retraining or rollbacks. The machine learning (ML) Test Score paper turns this into a checklist of 28 concrete tests and monitoring needs for production readiness (Breck et al., 2017). A good answer names a trigger for each layer, not just a dashboard.

How do you tell real drift from noise?

Compare live inputs with a reference window using a two-sample statistical test, and decide in advance what size of change matters. The "Failing Loudly" study found that a two-sample test applied after dimensionality reduction with pre-trained classifiers performed best across the shifts it tested (Rabanser et al., 2019). Sample size changes what a test says. In our synthetic example, two samples of 10,000 values with no shift gave a Kolmogorov-Smirnov p-value of 0.33; a shift of just 0.05 standard deviations gave 0.02, and a shift of 0.2 gave a p-value near 5 × 10⁻³¹.¹ With enough data almost any change is "significant", so pair the test with an effect-size threshold tied to model quality.

Why does the p99 latency matter more than the average?

A p99 latency target is a promise about the slowest 1 in 100 requests, and in a system that makes many calls per request it governs the typical experience. For 1 call the chance of meeting a slow call, one in the slowest 1 percent, is 1 percent. For 10 calls it is 9.6 percent and for 30 calls 26.0 percent; for 100 calls it is 63.4 percent (computed as 1 − 0.99^n).¹

Dean and Barroso observe that temporary high-latency episodes that are unimportant in moderate-size systems may come to dominate overall service performance at large scale, and argue that large online services need to create a predictably responsive whole out of less predictable parts (The Tail at Scale, 2013). In an interview, a latency target is diagnosed by measuring the baseline per stage first.

Text description of the diagram

• For 1 call the chance of meeting a slow call, one in the slowest 1 percent, is 1 percent.
• For 10 calls it is 9.6 percent and for 30 calls 26.0 percent; for 100 calls it is 63.4 percent.

What is training-serving skew, and how does a feature store help?

Training-serving skew means the model sees different feature values in production than it saw in training, so accuracy drops with no code error. The classic cause is a handoff: the data scientist computes a feature in a notebook and an engineer reimplements it in the serving service. Google's guide ties skew to this manual handoff in the least mature setup. A feature store is a central repository that standardises feature definitions, storage and access for both training and serving, so one definition feeds both paths.

Batch, streaming or real-time inference?

Choose by how fresh the prediction must be and how many predictions you need. Batch inference scores a whole dataset on a schedule and stores the results, which suits nightly recommendations. Real-time inference scores one request at a time behind an API, which suits fraud checks at payment time. Streaming sits between: events are scored as they arrive. Batch is usually cheapest per prediction and real-time most expensive, so the interview answer is the cost-versus-freshness trade-off, not a preference.

What changes for LLM applications?

The loop stays the same but the artifacts and the signals change: prompts and retrieval configuration become versioned artifacts, and quality signals include answer groundedness and cost per request as well as latency. The walkthrough in LLM observability covers what to log and how to alert on these.

Questions interviewers ask on MLOps

Interviewers often ask this as a three-part question: how would you deploy a model, how would you monitor it, and how would you meet a strict latency target such as a p99 of 50 milliseconds. Answer each part in the order of the release story above. Deployment: name the five versioned artifacts, the registry, the validation gate and the canary with rollback.

Monitoring: name a metric and a trigger for each of the four layers. Latency: clarify the target and where it is measured, measure the baseline per stage, then reduce the biggest stage first. For the wider set of questions across an AI engineering loop, see the AI engineer question list, and for how the loop is structured, interview loops.

Follow-up questions

How can I prepare for an MLOps interview?

Rehearse the five-step story out loud with one concrete system, such as a fraud model from the ML system design round, and prepare a number for each step: canary size, a drift trigger, a latency budget. The three computed examples above give you a starting set.

Is MLOps harder than DevOps?

It adds three moving parts DevOps does not have: data, models that degrade without code changes, and feedback loops. Google's guide names continuous training as the practice unique to machine learning systems.

What is the difference between CI, CD and CT?

Continuous integration tests code, data and models; continuous delivery deploys pipelines and services; continuous training retrains models on fresh data automatically. Google's guide treats continuous training as unique to machine learning.

What artifacts do you version, and how do you enable rollback?

Code, data snapshot, feature definitions, configuration and weights, with the registry pointing serving at a named version. Rollback is repointing to the previous version, and for LLM features the prompts are versioned too, as the observability walkthrough shows.

How would you decide when to retrain?

Retrain on a schedule, on new data, on measured performance degradation or on measured drift, as the guide lists, and prefer performance-based triggers where labels exist because a drift alarm alone does not prove the model got worse; the operations section of the system design hub places this in the 2am question.

Sources

1. MLOps: Continuous delivery and automation pipelines in machine learning — Google Cloud Architecture Center, 28 Aug 2024
2. Hidden Technical Debt in Machine Learning Systems — NeurIPS 2015 (Sculley et al.), 7 Dec 2015
3. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — IEEE Big Data 2017 (Breck et al.), 11 Dec 2017
4. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift — arXiv (Rabanser, Gunnemann, Lipton), v4 revised 2019-10-28; NeurIPS 2019, 29 Oct 2018
5. The Tail at Scale — Communications of the ACM 56 (Dean and Barroso), 1 Feb 2013