Model routing and cascades: how to cut LLM cost per query

In this article

1. Before you start
2. Why interviewers ask this
3. Model routing and cascades from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page assumes you know how a transformer reads and writes tokens.

Why interviewers ask this

Candidates report this topic in generative-AI system design rounds. The questions include how you would route between small and large models, cut cost for an app with a million queries a day, or move a production feature to a cheaper model safely. These are reported by candidates, not tied here to any named company.

The question tests whether you treat cost as a design variable instead of a bill. A weak answer says "use the cheaper model when the query looks easy" and stops. A strong answer says who decides, with what signal, what a wrong decision costs, and how the choice is measured before it ships. A Staff-level answer adds the arithmetic: the extra work a router or a cascade adds, and the point where the saving disappears.

Model routing and cascades from first principles

Model router

A model router is a small, fast component that reads a query and picks which language model should answer it before any answering model is called, so each query pays for only one answering model.

Escalation rate

The escalation rate is the fraction of queries in a cascade that the scorer rejects at the cheap model and passes up to the more expensive one, and it is the number that decides whether the cascade saves money.

In plain words

Think of a hospital front desk. A nurse looks at each arriving patient. Minor cases go to a clinic and serious ones to a specialist. That nurse is the router. Now picture a clinic doctor who sees everyone first. The doctor refers a patient up only when unsure. That is the cascade. The analogy breaks in one place. A patient is not billed twice, but an escalated query pays for the cheap attempt and then for the expensive one.

In one precise sentence, routing and cascading are two policies that choose a model per query, so average cost falls while quality stays above a target.

Step 1: A router picks the model before any call

A model router reads the query, or features of it such as length and topic, and outputs which model to use. Input: one query. Output: one model name. The RouteLLM paper trains such routers on human preference data to choose between a stronger and a weaker model, and reports cost reductions of over 2 times in certain cases without a loss of response quality on the benchmarks it tested.

The router's strength is that a query reaches only one answering model, so there is no wasted attempt. Its weakness is that it must guess difficulty before seeing any answer. A wrong guess to the weak model returns a poor answer; a wrong guess to the strong model wastes money. You now know the first policy: decide first, call once.

Step 2: A cascade calls the cheap model first

An LLM cascade skips the guess. It sends the query to the cheapest model and looks at what comes back. Input: one query. Output: a draft answer from the cheap model. The FrugalGPT paper names three families of cost-reduction strategies: prompt adaptation, LLM approximation and LLM cascade. It learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy, and the baseline it argues against is sending every query to a single expensive model. Its abstract reports that it can match the performance of the best individual LLM (e.g. Generative pre-trained transformer (GPT)-4) with up to 98% cost reduction (Chen, Zaharia and Zou, 2023); that is a benchmark result, not a typical saving.

Most real traffic is easy, so the cheap model's answer is accepted most of the time and the expensive model is never touched. The draft is not yet the final answer, though. Something must decide whether it is good enough, which is the next step.

Step 3: A scorer decides whether to escalate

The scorer is a small judge, often a learned reliability model, that rates the cheap answer. The FrugalGPT paper describes the rule: query the cheapest model, score its answer, and escalate to the next model only if the score falls below a threshold. Input: the query and the draft. Output: accept or escalate.

The threshold is the dial. Raise it and more drafts are rejected, which improves quality and raises the escalation rate. Lower it and cost falls, but weak answers slip through. Every escalated query has now paid for the cheap model, the scorer and the large model. You now know the whole mechanism; what remains is whether it pays.

Step 4: The break-even escalation rate

A cascade pays for the cheap model and the scorer on every query, and for the large model only on escalated ones. With relative prices that are illustrative (cheap model 1 unit, scorer 0.5 units, large model 20 units), the cost per query is 1.5 plus 20 times the escalation rate.¹ Cost grows with the escalation rate, because each escalated query adds the large price. At an escalation rate of 20%, that is 5.5 units against 20 for the large model alone, a saving of 72.5%.¹

The cascade stops saving money where its cost equals the large model's. The break-even escalation rate is:

With the prices above that is 92.5%.¹ Above it, the cascade is worse than not having one: at 100% escalation it costs 21.5 units, 7.5% more than the baseline.¹ The price gap sets the break-even. With a cheap model at 5 units, a scorer at 0.5 and a large model at 10 (illustrative), it falls to 45%.¹ A router's worst case is only its own fee above the baseline, because it never pays for a rejected attempt: sent entirely to the large model it costs 20.05 units.¹

1 · Router2 · Cheap first3 · Score4 · CostReplay

Step-by-step description

1. A router picks one model before any call.
2. A cascade tries the cheap model first.
3. A high score returns; a low score escalates.
4. Cost grows with the escalation rate.

• A router picks a model before any call.
• A scorer checks the cheap model's answer and escalates when the score is low.
• Cost grows with the escalation rate.

Worked example

Take an app with one million queries a day. All relative prices, rates and latencies here are illustrative inputs; the totals follow from them by arithmetic.¹ The large model costs 20 units per query, so sending everything to it costs 20,000,000 units a day.¹

A cascade with an escalation rate of 20% costs 5.5 units per query, or 5,500,000 units a day.¹ A router that costs 0.05 units per query and sends 20% of queries to the large model costs 0.05 + 0.8 + 4.0 = 4.85 units per query, or 4,850,000 units a day.¹ The router is slightly cheaper here because it never pays for a rejected attempt, but its 20% share is an assumption: a real router only achieves it if its guesses are good. With a narrower price gap (cheap model 5 units, scorer 0.5, large model 10, illustrative) the break-even falls to 45%, and a 20% escalation rate costs 7.5 units against 10 for the large model alone.¹

Latency moves the other way. With illustrative times of 400 milliseconds for the cheap model, 100 milliseconds for the scorer and 1,200 milliseconds for the large model, an escalated query takes 1,700 milliseconds against 1,200 for the large model alone, while the average at a 20% escalation rate is 740 milliseconds.¹

Text description of the diagram

• At an escalation rate of 20% the illustrative cascade costs 5.5 units against 20 for the large model alone.
• The break-even escalation rate is 92.5%.
• At 100% escalation the cascade costs 21.5 units, 7.5% more than the baseline.

Trade-offs and edge cases

Neither policy is free. The comparison below shows where each pays.

Text description of the diagram

• A router decides before the call and a cascade decides after the cheap answer.
• A cascade pays for the scorer and for rejected attempts, and escalated queries are slower.

• The scorer is the weak link. A cascade is only as good as its judge. If the scorer accepts a wrong cheap answer, the user gets it and no later model sees it. If it rejects good answers, the escalation rate rises and the saving shrinks.
• Tail latency. Escalated queries wait for two models in sequence, so the slowest queries get slower even as the average falls. If a response must arrive within a fixed budget, a router is easier to reason about; serving a 70B model at a p95 of 500 milliseconds covers how such tail targets are set.
• Distribution shift. A router trained on one traffic mix can drift when users change what they ask. Re-measure the escalation rate on live traffic, not only on the training set.
• Other levers exist. FrugalGPT lists two other families next to cascades: prompt adaptation (fewer in-prompt examples, batching queries into one prompt) and LLM approximation (a completion cache, or fine-tuning a smaller model on the large model's outputs). Outside that paper, quantization and distillation also give a cheaper model. Distillation is a training method that teaches a small model to imitate a larger model's outputs, giving a cheaper model for the easy majority of queries. Serving-side work such as the key-value (KV) cache and continuous batching lowers cost per token without changing which model answers.
• Transfer. The RouteLLM authors report that their routers kept their performance when the strong and weak models were swapped at test time, which suggests a router need not be retrained for every model pair, though you should verify this on your own traffic.
• Routing and cascading combine. A router can pick the starting model and a cascade can escalate from there. Dekoninck et al. (2024) derive an optimal cascading strategy and propose cascade routing, one framework that unifies the two. In their experiments it consistently outperforms routing or cascading alone, and good quality estimators are the critical factor, which is why the scorer above is the weak link.

How to answer it in an interview

The spoken structure: name the two policies, give the cost formula, name the break-even risk, then say how you would validate it.

Common mistakes

• "Routing is just picking the cheapest model." (wrong) The router or scorer must be learned or tuned against a quality target. "I pick the cheapest model that meets a quality threshold, and I measure that threshold." (right)
• "A cascade always costs less." (wrong) Escalated queries pay for every stage. "A cascade saves money only below its break-even escalation rate, so I compute that rate first." (right)
• "Cost is about which model I call." (wrong) Cost has separate levers besides model choice; the FrugalGPT paper lists prompt adaptation and LLM approximation next to cascades. "I also shorten prompts, cache repeated calls, use a smaller distilled or quantized model, and tune serving." (right)
• "The average latency is fine, so users are fine." (wrong) Escalated queries run two or three models in sequence. "I check the slowest queries, not only the mean." (right)

Follow-up questions

How are you going to decide which query will be going to which AI model or which LLM?

Use signals available at the moment you decide. Before any call, a query classifier or an embedding of the query can predict topic or difficulty, and a learned router such as the preference-data routers of the RouteLLM paper is one way to build that signal. After a cheap call, the confidence or log-probability of its answer, or a learned scorer, says whether to escalate. Test each signal against labeled quality on real traffic, then set the threshold from the cost formula above.

What if the cheap model costs half as much as the large one?

The break-even escalation rate falls. With illustrative prices of 5 for the cheap model, 0.5 for the scorer and 10 for the large model, it is 45%, so the cascade stays cheaper only while fewer than 45% of queries escalate.¹ A narrow price gap favors routing before the call or a cheaper cheap model.

When do you route to a small distilled model vs a large LLM?

Model tiering is arranging models in price and quality tiers and sending each query to the lowest tier that meets its quality bar. Route to the small model for narrow, repetitive, well-covered queries where a labeled evaluation set shows it matches the large model, and to the large model for open-ended, rare or high-stakes queries. The decision signal is measured quality on that query type, not a feeling about difficulty.

You need to migrate a production feature from an expensive model to a cheaper one. How do you do it safely?

Migrate behind an evaluation set and a shadow run, then a gradual release. Build an evaluation set from real queries, run the cheaper model beside the current one on copied traffic without showing users its answers, compare quality, then release to a small share of users with a rollback switch and widen it only while the metrics hold.

Your app gets one million queries a day: how do you optimize cost?

Measure first: tokens per query, share of easy queries and quality per model. Then stack levers: shorten prompts, cache repeated calls, route or cascade, quantize or distill, and tune serving with continuous batching. Prompt compression is shortening the text sent to a model, for example by trimming instructions or summarizing context, so fewer tokens are billed; it belongs to the family FrugalGPT calls prompt adaptation and lowers cost on every query regardless of routing. In the illustrative worked example, a cascade at a 20% escalation rate cuts daily cost from 20,000,000 to 5,500,000 units.¹

How do you build a cost-efficient agent that handles both simple and complex tasks?

Route each step of the agent, not just the whole task, so simple steps go to the cheap model. Cascade on low-confidence steps, escalating only when the scorer rejects the draft, and watch tail latency, because a step that escalates adds a second model call to the agent's chain.

Sources

1. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (arXiv 2305.05176) — arXiv (Chen, Zaharia, Zou), 9 May 2023
2. RouteLLM: Learning to Route LLMs with Preference Data (arXiv 2406.18665) — arXiv (Ong et al.), 26 Jun 2024
3. A Unified Approach to Routing and Cascading for LLMs (arXiv 2410.10347) — arXiv (Dekoninck et al.), 14 Oct 2024