What is model tiering?

In this article

1. Before you start
2. Why interviewers ask this
3. Model tiering from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the learning path on large language models.

Why interviewers ask this

A team that sends every request to its strongest model pays the strongest model's price for questions a small model answers just as well. Interviewers ask about model tiering because it tests whether a candidate can turn that observation into a design with a measurable quality target, a cost budget and a decision component. Candidates report the question in system design rounds, sometimes worded as "design a model tiering or routing system."

A weak answer says "use a cheap model first." A strong answer separates the two mechanisms, a router and a cascade, and says how each one decides. A Staff-level answer prices the whole system: the failed attempts, the scorer and the share of traffic each tier really handles. It also says how the quality target is measured on held-out examples. Interviewers listen as well for the boundary with other cost levers, such as distilling a small model or shortening prompts.

Model tiering from first principles

LLM cascade

An LLM cascade is a list of language models ordered from cheapest to most costly that are called one after another, stopping at the first answer judged reliable enough. Because every model tried before the winner has already been paid for, a cascade saves money only when the early models are cheap and usually good enough.

Generation scoring function

A generation scoring function is a function that takes a query and an answer produced by a model and returns a reliability score between zero and one. A cascade compares that score with a threshold to decide whether to accept the answer or escalate to the next, costlier model.

In plain words

Picture a help desk with two tiers. Front-line staff handle the common questions quickly and cheaply. Specialists handle the hard ones at a higher hourly cost. A manager decides which questions go where. Model tiering is the same arrangement, with language models in the roles of staff and specialists.

The analogy breaks in one place. A human front-line worker usually knows when a question is beyond them, but a language model returns a fluent answer even when it is wrong. So the system needs an explicit check or an explicit prediction to decide who answers.

In one precise sentence, model tiering chooses, for each request, a model from an ordered pool so that expected cost is minimized while answer quality stays at a target.

Step 1: A pool of models at different prices

The pool holds a cheap small model and a costly large model, and the price gap is the economic reason tiering exists. In their survey of application programming interface (API) fees, Chen, Zaharia and Zou (2023) found that popular models have fees that "can differ by two orders of magnitude." Their example: the prompt cost for 10 million tokens was 30 dollars for one generative pre-trained transformer (GPT) model, GPT-4, but only 0.2 dollars for another, GPT-J hosted by TextSynth. That is about 150 times apart.¹ Both prices are from 2023 and are historical.

Input: nothing yet, only the list of available models with their prices and measured quality. Output: an ordered pool. The papers call the mechanisms an LLM cascade and a router; this page uses "model tiering" for both, a label neither paper defines. You now know why a pool is worth building: if the cheap tier can answer a large share of traffic, the average price falls.

Step 2: A router picks one model before any answer exists

A router picks one model before any answer exists, using only the request. Ong et al. (2024), in the RouteLLM paper, train routers that "dynamically select between a stronger and a weaker LLM during inference." The objective, as the paper's text puts it, is to minimize cost while reaching a performance target, for example 90% of the strong model's performance.

A trivial router that always picks the strong model has perfect quality and no savings, so quality is a trade-off curve, not one number. At one operating point, the one the paper calls a call-performance threshold (CPT) of 50%, defined in its metrics section, the same paper reports in its evaluation text a chat-benchmark score of 8.8 against 9.3 for the strong model alone, which is 95% of it. The abstract also reports costs cut by over 2 times in certain cases without compromising response quality.

Input: one request. Output: a model choice. A wrong choice costs quality, not extra spend, because only one model runs.

Step 3: A cascade pays for every failed cheap attempt

A cascade calls models in order from cheapest to costliest and stops at the first answer that passes a check. Chen et al. (2023) build it from two parts: a generation scoring function that rates how reliable an answer is, and a router that chooses the ordered list of models. That router sets the cascade order; it is not the per-request router of Step 2. A threshold is the minimum score an answer must reach to be accepted.

A cascade scores the small model's answer. When the score is high, the cascade returns the small model's answer; when it is low, the request escalates to the large model. Every model tried before the winner has already been paid for, so cost is the sum of every model called. The paper notes that query cost drops significantly only if the first models are inexpensive and produce reliable generations.

Input: a request and a threshold. Output: the first answer whose score clears the threshold. This is the trap behind the follow-up in the lead: a cascade whose cheap tier is often wrong can cost more than never tiering at all, as the arithmetic below shows.

1 · A pool of models2 · Router picks one3 · Cascade escalatesReplay

Step-by-step description

1. A cheap small model and a costly large model.
2. The router chooses a model before any answer exists.
3. A high score returns the answer; a low score escalates.

• The pool holds a cheap small model and a costly large model.
• A router picks one model before any answer exists.
• A cascade returns the small model's answer when the score is high and escalates to the large model when it is low.

What the published cascade results show

Chen et al. (2023, Table 3) report the cost of a cascade that matches the accuracy of the best single model. Their savings range from 59% to 98% across the three datasets in Table 3.

The same paper reports that the cascade can improve accuracy over GPT-4 by 4% at the same cost. Three datasets are a small base, so treat these as evidence that savings are possible, not as a promise.

Router cost: the blended price of a tier mix

Blended cost is the average price per request when traffic is split across tiers, and for a router it is a weighted average. Let p be the share of requests sent to the small model, c_s the small price and c_l the large price.

With illustrative prices of 0.50 and 10 dollars per million tokens, sending 70% of requests to the small model gives a blended 3.35 dollars, which is 66.5% below the 10 dollars of always using the large model.¹ At 50% the blended price is 5.25 dollars and at 90% it is 1.45 dollars.¹

Text description of the diagram

• Blended cost falls linearly with the share sent to the small model.
• At a 70% share the illustrative blended cost is 3.35 against 10 for always using the large model.

Blended cost falls linearly with the share sent to the small model. At a 70% share the illustrative blended cost is 3.35 against 10 for always using the large model. The arithmetic assumes every request uses the same number of tokens; real vendors price input and output tokens differently. The formula also leaves out the router's own inference cost, which is usually small but not zero.

Cascade cost: failed attempts and the scorer are paid for

A cascade pays the small price on every request, the scorer price on every answer it checks, and the large price on each escalation. Let a be the share of answers accepted at the small tier and f_s the scorer cost per check.

With the same illustrative prices, a scorer cost of 0.05 dollars and a of 0.7, the cascade costs 3.55 dollars, which is 64.5% below always-large.¹ If it never accepts the cheap answer, it costs 10.55 dollars, which is 5.5% more than never tiering. It beats always-large here only when a exceeds 0.055.¹ The three-tier row adds an illustrative mid tier priced at 2.00 dollars, plus a scorer check at the mid tier: the small tier accepts 60% of requests, the mid tier accepts half of the rest, and the large tier answers the remaining 20%.¹

Text description of the diagram

• A router decides before any model answers, and it needs a trained predictor.
• A cascade pays for every failed cheap attempt, and it needs a scoring function.

A router decides before any model answers, and it needs a trained predictor. A cascade pays for every failed cheap attempt, and it needs a scoring function. Neither is better in general: a router wastes no spend on a miss but cannot see the answer, while a cascade sees the answer but pays to look.

The Staff-level answer

What makes it Staff-level: it names the failure cost of each mechanism and prices the whole system, not just the cheap model.

The rubric

This rubric is a study aid showing how answers differ, not any company's scoring sheet.

Three variants of this question

Design a model tiering or routing system.

Fix three things first: the quality target, the cost budget and the decision component, a router or a scorer. Chen et al. frame a cascade as maximizing quality within a budget, and Ong et al. frame a router as minimizing cost for a performance target. A strong design then evaluates on held-out traffic, escalates on low confidence and monitors quality in production.

How is model tiering different from distillation or fine-tuning?

Tiering routes among existing models at inference time. Distillation trains a new small model from a larger one, and fine-tuning an LLM changes a model's weights (a parameter-efficient version is low-rank adaptation (LoRA) fine-tuning); both can supply a cheap tier, such as one of the small language models. Prompt adaptation, the other cost lever in Chen et al., is covered in fine-tuning versus prompt engineering.

What if the strong model's price falls by half?

Blended cost falls, but the saving against always-large shrinks. With the large price halved to 5 dollars and 70% of requests on the small tier, the router costs 1.85 dollars, a 63% saving instead of 66.5%.¹ Re-measure the threshold and the share the cheap tier handles, because the best split moves when prices move.

Common mistakes

• "Tiering means we always use the smallest model." (wrong) The strong model still handles the hard queries; the threshold sets how many it gets. (right) "The cheap tier takes what it can answer well, and the strong tier takes the rest."
• "Savings are certain." (wrong) The published range of 59% to 98% comes from three datasets, and router gains vary by benchmark. (right) "I measure the share the cheap tier handles on my own traffic before promising a number."
• "A cascade costs the price of the model that answers." (wrong) It also pays for every failed attempt and every score. (right) "Cascade cost counts the small model on every request, plus the scorer, plus each escalation."
• "A router and a cascade are the same thing." (wrong) One decides before generation, the other after each answer. (right) "A router risks quality on a wrong pick; a cascade risks wasted spend."

Follow-up questions

Does model tiering lower answer quality?

Not necessarily. Chen et al. report a cascade that improved accuracy over GPT-4 by 4% at the same cost, and Ong et al. report 95% of the strong model's chat-benchmark score in one setting. Results are task-dependent, so measure quality on your own held-out examples.

Can a router work with models it was not trained on?

In the RouteLLM paper the same routers were reused with other strong and weak model pairs, with no retraining, on a chat benchmark. That is a limited claim; check a router on your own traffic before trusting it with a new pair.

How do you know the cheap answer is good enough?

In a cascade, a generation scoring function rates the answer and a threshold decides. A strong design checks that score against labeled examples first, because a scorer that cannot tell good answers from bad ones makes the whole cascade unreliable.

Does model tiering add latency?

It can. A cascade that escalates has already run the cheap model, so that request takes at least two model calls in sequence. A router adds one prediction step before the call but never runs a second model. The two papers cited here report cost, not latency, so measure it.

How does model tiering differ from a mixture-of-experts model?

A mixture-of-experts model, such as the one explained in how Mixtral works, routes tokens among expert sub-networks inside one model. Tiering routes whole requests among separate models that may come from different vendors.

Sources

1. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (arXiv 2305.05176) — arXiv (Chen, Zaharia, Zou; Stanford), 9 May 2023
2. RouteLLM: Learning to Route LLMs with Preference Data (arXiv 2406.18665) — arXiv (Ong et al.; ICLR 2025), 26 Jun 2024