In this article
Before you start
The wider topic is covered in machine learning operations (MLOps) interview questions; this page starts from the basics below.
Why interviewers ask this
A design round on a ranker, a recommender or a model-serving service can end with one question: what happens when part of it breaks? The question separates candidates who only describe the happy path from candidates who have thought about operating the system.
A weak answer says "we return an error" or "we add a fallback model" and stops. A strong answer names the core function the product must keep, lists the lower-quality modes in order, says what triggers each one and says how the team learns that it is running in one. A Staff-level answer adds the failure of the degraded path itself: it is rarely used, so it is rarely tested, and it can make an outage worse. Candidates report this kind of follow-up in system design rounds; no public source shows a particular company asking it.
Graceful degradation from first principles
- Graceful degradation
- Graceful degradation is a design in which a service keeps performing its core function during failure or overload by doing less work per request, for example by returning stale, cached or lower-quality results, instead of failing outright.
- Load shedding
- Load shedding is the practice of dropping some proportion of incoming requests as a server approaches overload, so that the requests it still accepts can finish.
In plain words
Picture a restaurant kitchen whose grill breaks mid-evening. The kitchen does not close. It cuts the menu to dishes it can still cook well. Guests still get dinner, just a smaller one. The analogy breaks in one place. A guest sees the shorter menu, but a user of an ML system often cannot see that a weaker model answered, so the loss goes unnoticed unless you measure it.
In one precise sentence: in a gracefully degrading ML system, each request moves down a ladder of cheaper, simpler ways to answer, and moves back up when the cause clears.
Degraded mode is the state of running on any rung below the full path. Graceful degradation is not load shedding. The Google Site Reliability Engineering (SRE) book says load shedding "drops some proportion of load" and that graceful degradation "takes the concept of load shedding one step further by reducing the amount of work that needs to be performed" (Google SRE book, ch. 22).
Step 1: Name the core function and the ladder of modes
The core function is the one thing the product must still do when everything else fails, such as returning some ordered list of items. Amazon Web Services (AWS) says components should keep performing their core function even if dependencies become unavailable, and that they might serve slightly stale data, alternate data or no data (AWS Well-Architected, reliability pillar). The same page lists "not identifying the core business functionality needed" as an anti-pattern.
For an ML system, write the ladder from best to cheapest: the full model, a lighter model, cached or stale results, a non-personalised popularity list, and last a human or an honest error. As design advice, not a citation, match each failure to a rung: a failed retriever gives a limited answer, stale features give a non-personalised result, and a slow model service gives a cached answer. Each step down the ladder does less work per request. Input: a failure or overload. Output: a cheaper answer that still serves the core function.
Step 2: Choose triggers and alert on how often they fire
A trigger is the measured signal that moves traffic down the ladder. The SRE book asks which metrics should trigger degradation, naming central processing unit (CPU) usage, latency, queue length and threads used, and whether the switch is automatic or manual (Google SRE book, ch. 22). In an ML system, add model-specific signals such as missing features, stale data and low model confidence.
Degradation should be rare. The SRE book says it should not trigger very often, usually only after a capacity-planning failure or an unexpected load shift, and advises alerting when too many servers enter these modes. The ML reason is our own reasoning: a system that works only because it constantly degrades can look stable on its dashboards while the intended model is not running. So track the quality of the primary path next to the degraded-mode rate. This is the follow-up from the lead. Count the share of requests per rung and alert when it rises; see large language model (LLM) observability for the signals worth tracing.
Step 3: Keep every degraded path simple, tested and exercised
A degraded path must be simpler than the primary path, and it must run for real before the day it is needed. AWS says "the pathways taken in case of component failure need to be tested and should be significantly simpler than the primary pathway" (AWS Well-Architected, graceful degradation). The SRE book warns that a code path that is never used often does not work, and advises exercising the path regularly.
A fallback is a different mechanism that achieves the same result, and the Amazon Builders' Library warns that fallbacks are harder to test and often make an outage worse (Gabrielson, 2019). Its case study: when a shipping-speed cache failed, web servers queried the database directly and the entire website went down. For ML, make the degraded path small, such as a cached list or a lighter model on the same stack, and run it in production often. A continuous integration (CI)/continuous delivery (CD)/continuous training (CT) design can include a regular drill.
Step 4: Stop the pile-up and keep an off-switch
A circuit breaker stops calls to a failing dependency and probes occasionally. AWS describes it this way: "If a high number of calls are failing, it will stop sending more requests to the downstream system and only occasionally let calls through to test whether the downstream system is available again" (AWS Well-Architected, graceful degradation page). Without a breaker, retries multiply the load: the SRE book describes clients that miss deadlines and retry, which causes more overload.
Keep a quick way to turn degradation off. The SRE book advises that complex degradation can trip into degraded mode when it is not wanted or enter feedback cycles. Last, decide what users are told. Microsoft's Human-AI eXperience (HAX) Toolkit pattern for falling back to other strategies suggests considering informing the user when the fallback is noticeable, especially in early or beta versions, in high-stakes scenarios and for power users (HAX pattern 10C, paraphrased).
Which requests to protect, and what a deadline triggers
Request criticality is a label on each request that says how much the product needs it answered. Under pressure, a server sheds the lowest labels first, so the shed share is chosen instead of random. A ranking call on a checkout page is critical; a background re-score for an email digest is sheddable. A deadline is the time by which the caller needs an answer, and it is the usual trigger for the cached rung: if the model call has not returned by then, return the cached or lighter-model answer.
Pass the remaining time to each dependency so a late call is dropped rather than finished for nobody; the SRE book describes clients that miss deadlines and retry, which causes more overload (Google SRE book, ch. 22). A missing feature needs a rule too: use a default or imputed value, and count each use. This is design advice, not a quotation.
Step-by-step description
- Name the cheaper modes.
- Measured signals move traffic.
- Lower paths are simple and tested.
- Stop calls to a failing service.
- Write the ladder from best to cheapest: the full model, a lighter model, cached or stale results.
- A trigger is the measured signal that moves traffic down the ladder.
- Every degraded path is simpler than the primary and is exercised regularly.
- A circuit breaker stops calls to a failing dependency and probes occasionally.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a ranker on eight servers. All inputs are illustrative: a full ranking model serves 100 requests per second per server, a lighter model serves 600, and peak traffic is 1,200 queries per second (QPS).1 The full model alone has a capacity of 800 QPS, so 400 QPS are over capacity. Shedding the excess serves two thirds of requests and fails the other third.1
Degradation spreads the work instead. If a share f of requests uses the full model, servers are fully used when f is 0.6, so 60% of traffic gets the full model and 40% the lighter one.1 With illustrative relevance scores of 0.80 for the full model, 0.70 for the lighter model and 0.40 for a popularity list, the average relevance is 0.76 with degradation against 0.53 with shedding, counting failed requests as zero.1 The popularity list alone would serve everyone at 0.40.
| Quantity (illustrative unless sourced) | Value |
|---|---|
| Peak load | 1,200 QPS |
| Capacity, full model only | 800 QPS |
| Requests failed when excess is shed | 33.33% |
| Share on the full model with degradation | 60% |
| Share on the lighter model | 40% |
| Average relevance, degraded | 0.76 |
| Average relevance, shed | 0.53 |
| Attempts if every failed request retries 3 times | 2,400 QPS |
| Time to handle a request when the queue is full (queue of 10 times the threads, 100 milliseconds per request; sourced) | 1.1 seconds (Google SRE book, ch. 22) |
The last row shows why retries are not a safe fix: 400 failed requests per second retried three times add 1,200 attempts, so the service sees up to 2,400 attempts per second.1 The SRE book gives a sourced queue example: if the queue is 10 times the number of threads and a thread handles a request in 100 milliseconds, a full queue means a request takes 1.1 seconds (Google SRE book, ch. 22).
Text description of the diagram
- Illustrative average relevance is 0.76 with degradation and 0.53 with shedding.
- A popularity list alone would serve everyone at 0.40.
Trade-offs and edge cases
Degradation, load shedding and fallback solve different problems, and the choice depends on which resource is short. Load shedding drops some requests. Graceful degradation does less work per request. A fallback uses a different mechanism to achieve the same result.
Text description of the diagram
- Load shedding drops some requests.
- Graceful degradation does less work per request.
- A fallback uses a different mechanism to achieve the same result.
Practice differs on fallback. The AWS reliability page recommends degradation and also says "Generally, fallback strategies should be avoided." Microsoft's HAX Toolkit pattern 10C says that when the system hits a determined threshold it should fall back to another automated solution or to a human (HAX pattern 10C), and guideline 10 of Amershi et al. (2019) asks systems to degrade gracefully when uncertain about a user's goals. Both fit one rule: keep the lower path small and tested.
Edge cases. Shedding back to just under the old failure point is not enough. The SRE book's example is a service that was healthy at 10,000 QPS and began a cascading failure from crashes at 11,000: dropping to 9,000 QPS will almost certainly not stop the crashes, because servers that come back online are hit with a very high request rate and fail almost immediately (Google SRE book, ch. 22).
Shed far more, let the system recover, then restore traffic gradually. Stale features degrade quality slowly, so pair this page's triggers with drift monitoring. Do not use a fallback that makes users believe the system worked normally when it did not; disclose it when it is noticeable. Which feature to drop is a product decision as well as a technical one. Where serving mode is a choice, see batch, streaming and real-time inference.
LLM serving. As of October 2026, common rungs for a large language model service are general practice rather than one vendor's documented design: route the request to a smaller model, cap the number of output tokens, skip retrieval or reranking, and serve a cached response for a repeated prompt. Each rung saves work and changes answer quality, so evaluate it on a held-out set before the rung is allowed to trigger, and count the share of requests on each rung as in Step 2.
How to answer it in an interview
How would you make an ML serving system degrade gracefully?
I would do less work per request instead of failing. First I name the core function, say returning some ranked list. Then I define a ladder: the full model, a lighter model, cached or stale results, and a popularity list. Triggers like queue length and latency move traffic down the ladder, and I alert on how often that happens. Each lower path is simpler than the primary and exercised regularly, and a circuit breaker stops calls to a failing dependency. The cost is lower relevance on the degraded share of traffic, which I measure. I would also tell users when the change is visible.
The spoken structure: say the core function, then the ladder, then the triggers and the alert, then the testing of the lower path.
Common mistakes
- "Graceful degradation is load shedding." (wrong) Shedding drops requests. (right) "Degradation reduces the work per request; I may shed only what remains."
- "A fallback model always makes the system more reliable." (wrong) An untested one can turn a partial outage into a full one. (right) "The degraded path is simple, tested and run in production regularly."
- "If degraded mode works, we are fine." (wrong) A system that degrades constantly has lost its intended model. (right) "I alert on the degraded-mode rate and the primary-path quality."
- "Retry until it works." (wrong) Retries multiply load. (right) "A circuit breaker stops pointless retries and probes for recovery."
Follow-up questions
What is the difference between load shedding and graceful degradation?
Load shedding drops some requests when a server nears overload. Graceful degradation reduces the work each request needs, for example a less accurate but faster ranking. Systems often use both, with degradation first.
Is a fallback model the same as graceful degradation?
No. A fallback model is one way to degrade, but degradation can also be a cache, stale features, a simpler code path or a human. Whatever the form, it must be simpler than the primary and tested.
How do you test graceful degradation in an ML system?
Run the degraded path in production on a small share of traffic, or push a few servers near overload, so the path is known to work. Include the drill in the delivery pipeline, as in CI/CD/CT for ML apps.
How do you monitor a system that degrades?
Track the share of requests on each rung and the quality of the primary path together, and alert when many servers enter degraded mode, as the SRE book advises.
Should users be told when the system degrades?
Consider it when the change is noticeable or the stakes are high, as the HAX pattern suggests, especially in early or beta versions and for power users. Never present a degraded answer as the normal one.
What if peak is 2,000 QPS?
The lighter model alone has a capacity of 4,800 QPS, so nothing needs shedding. The same arithmetic puts 28% of requests on the full model and 72% on the lighter one, for an average relevance of 0.728 (illustrative inputs).1 Past 4,800 QPS the lighter model also saturates, and the rest must be shed or served by a cheaper rung such as the popularity list.
- Graceful degradation does less work per request; load shedding drops requests.
- Name the core function, then a ladder of cheaper modes.
- Triggers should be measured signals, and the degraded-mode rate needs its own alert.
- Every degraded path is simpler than the primary, tested and exercised.
- A circuit breaker and an off-switch keep degradation from making an outage worse.
Sources
- Addressing Cascading Failures (Load Shedding and Graceful Degradation) — Google, Site Reliability Engineering book, ch. 22 (book published 2016; page undated; accessed 2026-10-26), 1 Jan 2016
- REL05-BP01 Implement graceful degradation to transform applicable hard dependencies into soft dependencies — Amazon Web Services, Well-Architected Framework, Reliability Pillar (page undated; year taken from the page's own citation list; accessed 2026-10-26), 1 Jan 2022
- Avoiding fallback in distributed systems — Amazon Builders' Library (Gabrielson; accessed 2026-10-26), 5 Dec 2019
- Guidelines for Human-AI Interaction, guideline 10: Scope services when in doubt — Microsoft Research (Amershi et al., CHI 2019; the paper defines guideline 10; accessed 2026-10-26), 1 Jan 2019
- Pattern 10C: Fall back to other strategies — Microsoft HAX Toolkit (page undated; year of the guidelines it accompanies; accessed 2026-10-26), 1 Jan 2019
