How do you design CI/CD/CT for ML and LLM apps?

In this article

1. Before you start
2. Why interviewers ask this
3. CI/CD/CT for ML and LLM apps from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the machine learning operations learning path.

Why interviewers ask this

A design answer that stops at "train, then deploy" misses the place where production machine learning breaks: the data changes while the code does not. Candidates report being asked to design a pipeline that validates data before training, computes features the same way in training and serving, and lets a new model ship only if it beats the one in production. They also report questions on testing a model for regressions and on what a canary deployment means for a language-model app.

A weak answer lists tools. A strong answer separates the three loops, names what each one checks, and says what blocks a release. A Staff-level answer adds the decision rule: a fixed evaluation set, a metric, a margin, and a rollback path, plus the point that the loops differ for a language-model app.

CI/CD/CT for ML and LLM apps from first principles

Continuous training

Continuous training (CT) is an ML-specific property of a delivery pipeline that automatically retrains a model on fresh data when a schedule or an event fires, validates the result and serves it, so a model stays current without a person starting each run.

Model acceptance gate

A model acceptance gate is an automated check in the training pipeline that compares a candidate model with the current production model on a fixed evaluation set and stops the release unless the candidate clears a stated bar.

In plain words

Picture a bakery that sells bread every morning. Before baking, someone checks the flour and the recipe. After baking, someone tastes the loaf against yesterday's before it goes on the shelf. A few customers get it first. If they complain, the old loaf comes back.

CI is the checking, CT is the baking on a schedule, the gate is the taste test and the canary is the first few customers. The analogy breaks in one place: flour does not change by itself, but data does. That is why ML needs a loop that code alone never needed.

In one precise sentence, CI/CD/CT automates testing, shipping and retraining so that code, data and models each pass a check before they reach users.

Step 1: CI tests code, data and models

Continuous integration (CI) is the practice of automatically building and testing every change when it is merged. For ML, Google Cloud's machine learning operations (MLOps) document says CI is no longer only about testing and validating code and components, but also data, data schemas and models .

Input: a merged change to feature code, training code or configuration. Output: a pass or fail. Typical checks are unit tests on feature code, a schema check on a data sample, and a short training run that confirms the model builds and its loss falls.

For an LLM app, the same slot runs evaluation flows. In Microsoft's GenAIOps pattern, a merge to the release branch triggers a CI pipeline that runs the experimentation and evaluation flows and registers the flows (Microsoft Learn GenAIOps with prompt flow, as of October 2026). You now know what runs on every change; the next step covers what runs when data changes.

Step 2: CT retrains when a trigger fires

Continuous training starts the training pipeline without a person. Google Cloud's ML guidelines describe the trigger as a schedule, for example daily or weekly, or an event, for example when new labeled data is available.

Input: a trigger. Output: a candidate model plus its metadata. Automation needs more than a scheduler. The same Google Cloud pipeline guidance lists automated data validation, automated model validation, pipeline triggers and metadata management as the additions (Google Cloud's MLOps document, 2024).

Data validation is a check, required before model training, that decides whether to retrain the model or stop the execution of the pipeline (Google Cloud's MLOps document, 2024). Training-serving skew is a gap between the features a model saw in training and the features it gets in production; it appears when engineers re-implement features for low-latency serving, so share one feature definition, as in a feature store.

Step 3: A model acceptance gate decides whether a new model ships

The gate turns "looks better" into a decision rule with three parts: a fixed evaluation set, a metric and a margin. In Google Cloud's MLOps document, model validation is the step after the model is trained on new data, where you evaluate and validate it before it is promoted to production, so that the new model performs better than the current one.

Input: the candidate and the production model, both scored on the same examples. Output: ship or hold. The margin matters because a small evaluation set gives a noisy score, so a difference inside the noise is not evidence. The worked example below computes one.

For an LLM app the metrics change but the rule does not. Microsoft's guidance points to metrics such as groundedness, relevance and similarity (Microsoft Learn GenAIOps maturity page, as of October 2026). The evaluation set is then the regression test for prompts and retrieval settings. A model that fails the gate is never deployed.

Step 4: CD deploys the pipeline, then the model through a canary

Continuous delivery (CD) is the practice of automatically releasing a change that has passed its checks. For ML, Google Cloud's MLOps document says CD is no longer about a single package but about a system, a training pipeline, that automatically deploys another service, the model prediction service.

Input: a model that cleared the gate. Output: a live version. In the Microsoft pattern a CD trigger deploys the flow to an online endpoint and then runs integration and smoke tests (Microsoft Learn GenAIOps with prompt flow, as of October 2026). Prompt flow is scheduled for retirement on April 20, 2027 (Microsoft Learn GenAIOps maturity page), so read it as a pattern, not a tool recommendation.

A canary sends a small share of traffic to the new version first. For an LLM app the version is the model name, the prompt and the retrieval settings together. Keep the previous version registered so rollback is one step.

Step 5: Monitoring feeds the next trigger

Monitoring closes the loop: what you observe in production decides when the next training run starts and whether the live version stays. The signals are data quality, input drift, output quality and cost; for LLM apps these include traces and quality scores, covered in LLM observability.

Input: production traffic and its outcomes. Output: alerts, new labeled data and, when thresholds trip, a trigger for Step 2. A model trained in January and bad in July is the classic case; model drift explains why the world moves under a fixed model. Feedback collected in production can also become training data, as in a data flywheel.

CI tests code, data and models. A trigger starts training, and data validation runs after ingestion. The acceptance gate compares the new model with the production model. A canary sends a small share of traffic to the new version. Monitoring watches the live version.

1 · CI tests2 · Trigger3 · Gate4 · Canary5 · MonitorReplay

Step-by-step description

1. Code, data and models are tested.
2. A trigger starts validated training.
3. The candidate must beat production.
4. A small share of traffic goes first.
5. Signals show how the version behaves.

• CI tests code, data and models.
• The acceptance gate compares the new model with the production model.
• A canary sends a small share of traffic to the new version.
• Monitoring watches the live version.

Worked example

Take a candidate model and the production model scored on the same evaluation set of 2,000 examples. The set size, the production accuracy of 0.870 and the candidate accuracy of 0.884 are illustrative. The candidate is 0.014 higher.¹

With the standard error of the difference at 0.0104, a 95% interval for the difference is 0.014 plus or minus 0.0204, which runs from -0.0064 to 0.0344.¹ The interval includes zero, so a gate that requires the lower bound to be above zero holds the release. Scoring both models on the same examples and comparing them in pairs narrows the interval, so treat this as a conservative estimate. About 4,200 examples would be needed before a 0.014 gap just touches the same bar if the accuracies stayed the same, and about 8,600 for the gate to catch a true gap of that size 80% of the time.¹

For the canary, assume 10,000 requests per hour and a faulty version that fails 2% of requests (both illustrative). A 1% stage sees 100 requests per hour, a 5% stage 500, a 25% stage 2,500 and the full stage 10,000. The chance that the 1% stage sees at least one failure in an hour is 1 minus 0.98 to the power 100, which is 0.8674.¹

Seeing one failure counts as a signal only when the old version almost never fails in that way, as with a crash or a server error; if the production version also fails some share of requests, compare the failure rates of the two versions instead. A small first stage limits harm and still gives you about a hundred requests an hour to watch.

Text description of the diagram

• A 1% canary stage sees 100 of 10,000 requests per hour.
• A 5% stage sees 500 and a 25% stage sees 2,500 requests per hour.

Trade-offs and edge cases

Automation has a price. Google Cloud's MLOps document describes a manual process in which a new model version is deployed only a couple of times per year, and a pipeline in which retraining runs automatically, on a schedule such as daily or weekly (Google Cloud's ML guidelines, as of October 2026). The pipeline buys freshness and repeatability; it costs engineering time and compute, so estimate the training duration before choosing a cadence.

That document frames the journey as maturity levels: level 0 is a manual process, level 1 is machine learning pipeline automation, which adds continuous training, and level 2 is CI/CD pipeline automation. Moving from a manual process to an automated pipeline changes how often a model is trained, how retraining starts and how validation happens.

Text description of the diagram

• A manual process deploys a new model version only a couple of times per year.
• An automated pipeline retrains on a schedule, for example daily or weekly, or on a trigger.
• An automated pipeline adds data validation and model validation steps.

Edge cases to name aloud:

• LLM apps lean on newer guidance. The evaluation-in-CI pattern for LLM apps comes from Microsoft's GenAIOps guidance, which is still evolving, so treat it as a pattern rather than a fixed standard.
• A stale evaluation set. If the set stops matching live traffic, the gate approves models that fail in production. Refresh it from production feedback.
• Noisy judges. When a language model scores outputs, the score itself varies; use a larger set and a margin.
• Retrain on every alert. Automatic retraining on each drift alert from day one is risky. Start with a schedule plus review, and automate triggers once the gates pass reliably.

The data platform under ML is where the validated data and its versions live.

How to answer it in an interview

The spoken structure: name the three loops, say what blocks a release, then give the canary and rollback and the LLM difference.

Common mistakes

• "CI/CD for ML is the same as for code." (wrong) CI also validates data, schemas and models, and CT exists only for ML. (right) "I add data validation, a model gate and retraining triggers to the code pipeline."
• "If the candidate scores higher, ship it." (wrong) The gap may sit inside the noise of a small evaluation set. (right) "I ship when the difference clears a margin on a fixed set."
• "I compute features separately for serving." (wrong) That creates training-serving skew. (right) "Training and serving share one feature definition."
• "Retrain automatically on every drift alert." (wrong) Unvalidated automation can ship a worse model. (right) "I start with scheduled retraining and review, then automate triggers once gates hold."
• "A canary is just a smaller deployment." (wrong) It is a staged rollout tied to health signals. (right) "Traffic grows only while the signals stay healthy, and rollback is ready."
• "Shadow, canary and A/B testing are the same thing." (wrong) A shadow version serves no user, a canary serves a few, and an A/B test compares two versions on a business metric. (right) "I shadow first, then canary, then A/B test if the metric needs it."

Follow-up questions

How to test AI models for regressions before they go into production?

Score the candidate and the production model on the same fixed evaluation set, including slices for important user groups, and hold the release unless the candidate clears a margin. For an LLM app, run the same prompts and retrieval settings through evaluation metrics such as groundedness and relevance.

What testing methodologies are commonly used to ensure reliability and performance of AI models in production environments?

A strong answer layers them: unit tests on feature code, data-schema checks, evaluation on a held-out set, smoke tests on the deployed endpoint, a canary on live traffic and monitoring afterwards. Each layer catches a failure the previous one cannot see.

What is canary deployment for LLM applications?

It sends a small share of live requests to a new combination of model, prompt and retrieval settings while the rest stays on the old one. Widen the share only while quality, latency and cost signals stay healthy, and roll back otherwise.

Design an ML experiment tracking and analysis platform?

Record, for every run, the code version, the data version, the parameters and the metrics, so any result can be reproduced and compared. The same metadata feeds the gate and the registry; Google lists metadata management as a requirement of automated pipelines. Add a comparison view across runs, a link from each registered model to the run that produced it, and access control so that results are shared safely.

What is the difference between shadow deployment, canary and A/B testing?

A shadow deployment (dark launch) sends copies of live requests to the new version and discards its answers, so users see nothing while you compare behavior. A canary serves a small share of real users and widens only while health signals hold. An online A/B test splits users between versions for long enough to compare a business metric with statistical care.

Should drift trigger retraining automatically?

Not at first. Begin with a schedule and a human review, and let drift trigger a run only after validation gates have passed consistently. Retraining on stale or bad data only repeats the problem; model drift shows how to tell the cases apart.

Sources

1. MLOps: Continuous delivery and automation pipelines in machine learning — Google Cloud Architecture Center, 28 Aug 2024
2. Guidelines for developing high-quality, predictive ML solutions — Google Cloud Architecture Center, 26 Oct 2026
3. Machine learning operations (MLOps v2) — Microsoft Learn, Azure Architecture Center, 26 Oct 2026
4. GenAIOps with prompt flow and GitHub — Microsoft Learn, Azure Machine Learning, 26 Oct 2026
5. Advance your maturity level for GenAIOps — Microsoft Learn, Azure Machine Learning, 26 Oct 2026