MLOps

What is canary deployment for LLM applications?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

A canary deployment for large language model (LLM) applications is a partial, time-limited release of a changed prompt, model or retrieval setting to a small share of traffic, judged against the untouched control before it widens. Candidates usually stop at the traffic split, but a bad answer comes back as a normal success, so the gate that decides promotion has to read quality scores; the second step shows how.

In this article

Before you start
Why interviewers ask this
Canary deployment from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page is part of the machine learning operations (MLOps) interview questions path and builds on the basics in machine learning (ML) algorithms and how they work.

Why interviewers ask this

A canary question separates candidates who ship changes safely from those who only build them. An LLM application changes often, because a prompt edit, a new model version or a new retrieval index can alter every answer without a code bug. Reported by candidates, the question appears in machine learning system design rounds, usually as "how do you roll out a new prompt or model?"

A weak answer says "send 5% of traffic to the new version and watch the errors." A strong answer names the control, explains that a language model returns a well-formed but wrong answer with no error, and picks quality, latency and cost metrics. A Staff-level answer also sizes the canary so the quality test has enough samples, writes the promotion gate before the release, and makes rollback automatic.

Canary deployment from first principles

Canary deployment: A canary deployment is a release method that sends a changed version of a service to a small, time-limited share of live traffic and judges it against the rest of the traffic, so a bad change is found while few users are exposed and can be undone before it spreads.
Control: The control is the part of the service that keeps running the unchanged version during a canary, and it is the baseline the canary's metrics are compared with, because both groups see the same kind of traffic at the same time. Without a control, a metric drop could be a busy hour rather than the change.

In plain words

Think of a restaurant that wants to change a recipe. It serves the new dish at one table and the old dish everywhere else. Then it compares the reviews. If the new dish does worse, the old recipe returns before most guests taste it. The analogy breaks in one place. A restaurant waits for reviews, while a canary reads metrics automatically and can reverse itself in minutes.

In one precise sentence, a canary is a gradual release. The changed and unchanged versions run side by side on live traffic. Measured differences decide whether the change spreads.

The Google Site Reliability Workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation." It calls the part that gets the change the canary and the remainder the control, and says the process is effectively an A/B testing process. It lists three requirements: a way to deploy to a subset, a process that judges the change, and integration of that judgment into the release process.

Step 1: Split traffic into a canary and a control

The traffic split sends a small share to the canary and the rest to the control. In an LLM application the thing that changes is rarely only code. It can be the prompt text, the model version, the retrieval-augmented generation (RAG) index that feeds documents into the prompt, a sampling setting or a guardrail rule. The workbook's definition covers a change in a service, and each of these is such a change.

Input: incoming requests and a weight. Output: two groups. Pin an exact model version rather than a moving alias, so the canary and the control differ in exactly one thing, and keep one conversation on one version so a user does not see answers flip. Rollout controllers such as Argo Rollouts let you declare weights and pauses; the Kubernetes for ML infrastructure page shows where that sits.

Step 2: Choose metrics that catch a bad answer

A bad LLM answer returns a normal success status, so the metrics must include a quality score, not only errors. A model can write a fluent, wrong answer without raising any error. Four groups cover it. Reliability: error rate and timeouts. Latency: p95 latency is the time within which 95% of requests finish. Cost: tokens per request. Quality: the pass rate from a judge model, plus thumbs-down feedback, refusal rate and guardrail triggers. A judge model is a second language model that scores each answer against a written rubric.

Keep the set small. The workbook advises watching only the top few metrics, perhaps no more than a dozen, because every extra metric adds noise and a chance of a false alarm. Calibrate the judge model against human labels first, or its own noise becomes the signal. The LLM observability page covers how to collect these signals. You now know what to measure; the next step is how much data you need.

Step 3: Size the canary and the time it runs

Sample size sets how long the canary must run. The workbook says the higher the processing rate, the less time is needed to get a representative sample, and that simple metrics such as query success read quickly while noisier ones need more time or a larger canary population.

Take an illustrative case: a new prompt, a baseline judge pass rate of 92% and a canary at 85%. A two-proportion test at a 5% significance level and 80% power needs about 325 judged answers in each group to detect that 7-point drop.¹ At an illustrative 600 requests a minute, a 5% canary receives 30 a minute and reaches 325 in about 11 minutes, while a 1% canary receives 6 a minute and needs about 54 minutes.¹

The control receives the rest of the traffic and reaches its sample far sooner, so the canary's rate alone sets the wait. A smaller canary is safer per minute but slower to give an answer. Run long enough to cover varied traffic, not only a quiet hour.

Step 4: Ramp with gates and roll back automatically

The promotion gate decides whether to proceed, hold or roll back. A typical illustrative ramp is 1%, 5%, 25%, 50% and then 100% of traffic,¹ and each stage must pass its gate before the canary gets more traffic. The gate compares the canary with the control on the same minutes of traffic, never with last week's numbers.

The workbook describes the automated response: when the canary's metric is too far from the control's, the process pauses and rolls back the deployment or calls a human. Rollback is cheap because the control never stopped running the old version; sending the weight back to zero restores it. An inconclusive reading means hold and wait for more samples, not proceed. Checking the gate at every ramp stage is a repeated look at the data and inflates false alarms, so use a sequential test or a stricter threshold. For LLM changes a shadow deployment can come before the canary: it runs the new version on copied requests without serving the answers, so users never receive its answers while you score it.

A canary splits traffic, adds a quality metric, is sized for enough samples, and only then is gated and ramped.CC BY 4.0 — source: What is canary deployment for LLM applications?

Step-by-step description

A small share goes to the canary.
Quality joins errors, latency and cost.
Samples set how long it runs.
Proceed, hold or roll back.

The traffic split sends a small share to the canary and the rest to the control.
The metrics must include a quality score, not only errors.
Each stage must pass its gate before the canary gets more traffic.

What a defect costs at each stage

Blast radius is the share of users or requests that a defect reaches. The workbook's example makes it concrete: a defect that fails 20% of requests gives 20% errors for everyone in a full rollout, but with a canary population of 5% it gives a 1% overall error rate, and the impact on the error budget is proportional to the traffic exposed.¹

The same logic applies to quality. In the illustrative ramp with the 7-point quality drop, at 5% of traffic a defective version causes about 126 extra bad answers an hour, and at 100% the same illustrative defect causes about 2,520 extra bad answers an hour.¹ Cost works the same way: if a new prompt adds an illustrative 300 tokens to each request, a full rollout adds about 259,200,000 tokens a day, and the 5% canary adds about 12,960,000.¹

A defect harms users in proportion to the traffic share it reaches, so a staged ramp caps the damage.CC BY 4.0 — source: What is canary deployment for LLM applications?

Text description of the diagram

At 5% of traffic a defective version causes about 126 extra bad answers an hour.
At 100% the same defect causes about 2,520 extra bad answers an hour.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.