In this article
Before you start
This page belongs to the generative AI system design interview guide learning path.
Why interviewers ask this
A cost model that ignores volume sinks a design, because the cheaper option at a thousand requests a day is often the dearer one at a million. This question appears in model-selection steps of generative AI design rounds, and it separates candidates who quote one price per call from candidates who price a system as a function of traffic.
A weak answer says "LLMs are cheaper because there is no training" or "a classifier is always cheaper at scale". A strong answer states the cost shape of each option, names the break-even volume and says accuracy has to be tested on the task. A Staff-level answer adds the levers that move the bill (prompt caching, batch pricing, routing), the cost of labels, and the hybrid in which an LLM labels data for a small model. Related questions about gateways and routing are in the LLM gateway interview questions.
What each one is
- Trained classifier
- A trained classifier is a small neural network, often a Bidirectional Encoder Representations from Transformers (BERT) encoder, trained on labelled examples to output one label per input, so its cost is a one-off training bill plus hosting rather than a charge per call.
- Zero-shot LLM classifier
- A zero-shot LLM classifier is a general large language model given the label definitions in a prompt and asked to pick one, with no training on your labels, so every request pays for the prompt and the reply in tokens.
- Break-even volume
- Break-even volume is the number of requests per day at which the classifier's monthly hosting bill equals the LLM's monthly bill, so above it the classifier is cheaper to run and below it the LLM is.
In plain words
Picture two ways to sort a mailroom. You can hire a generalist temp who reads an instruction sheet for every letter. Or you can train one specialist clerk who then sorts without a sheet. The temp costs nothing to start. You pay for every letter, and a longer sheet costs more per letter. The clerk costs a training period, then a steady wage. The analogy breaks in one place. The clerk's wage does not fall when the mail is light, and the clerk's accuracy depends on how many correct examples you could afford to give.
In one precise sentence, the LLM bill grows in a straight line with requests, and the classifier bill is a one-off cost plus a hosting charge that grows in steps.
The break-even formula: when the classifier pays back
Break-even volume follows from two numbers. Let c be the LLM cost of one request and H the monthly hosting bill of the classifier. The LLM bill for a month is c × days × V for V requests a day, so the bills are equal when V = H / (days × c). Below that volume the LLM is cheaper to run; above it the classifier is.
Break-even volume says nothing about the one-off cost O of labels and training. The payback time is months = O / (L − H), where L is the monthly LLM bill. If L is below H, the one-off cost never pays back, however long you wait. This second formula shows that a lower running cost does not help when the running costs are already close. The worked example below puts numbers on both formulas.
The LLM side: you pay per request
An LLM charges for every token it reads and writes, so one request costs input tokens times the input price plus output tokens times the output price.
Prices change often and differ between vendors, so this page uses illustrative prices shaped like a small model tier: 0.10 dollars per million input tokens and 0.50 dollars per million output tokens, the pair that one third-party price table and a second one list for a small model as of October 2026. A cache read costs a fraction of the input price; on the larger-model rows of the first table it is 5% of input, so the example uses 0.005 dollars. These figures were not checked on the vendor's own price page, so read that page before you quote one.
Instructions and examples sit in the prompt, so they are billed on every call. A classification reply is short, often a single label, so the input side dominates the bill.
Caching, batch and minimum length
Prompt caching is a vendor feature that stores a repeated prompt prefix so that later requests read it at a reduced input price. Batch pricing halves input and output prices for asynchronous jobs, and the batch discount stacks with cache pricing. Vendors also set a minimum cacheable length. OpenAI, for example, caches prompts automatically from 1,024 tokens (nOps price table, October 2026); other vendors set their own minimum, some higher, so check it for the model you price.
The classifier side: you pay up front
A classifier trained through fine-tuning is paid for once in labels and training, then in hosting. Training an encoder is small compared with training an LLM: a 2024 study of finance-tuned encoders lists training time in hours, for example 12 hours for one supervised model. Labels cost more than compute for most teams, because a person has to write each correct answer.
More labels help, with diminishing returns. On a two-class task, a 2024 study measured a tuned BERT model at 71.1% test accuracy with 200 samples, 73.4% with 500 and 73.9% with 1,000. After training, the model runs without a prompt, so a request sends only the text. Hosting is a server that stays up, so it costs the same on a quiet day as on a busy one until traffic needs a second server.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a ticket-routing task (illustrative) with 100,000 requests a day for a 30-day month, which is 3,000,000 requests. Each request carries 1,200 tokens of fixed instructions and examples (illustrative, chosen to clear that minimum), 100 tokens of ticket text and a 5-token reply (illustrative). The LLM prices are the illustrative pair above: 0.10 and 0.50 dollars per million tokens, 0.005 dollars per million for cached input, and half price in batch.1
The classifier side uses assumptions, not sourced prices. A small processor instance at 0.10 dollars per hour (illustrative) costs 72.00 dollars over 720 hours, and one graphics processing unit (GPU) at 0.60 dollars per hour (illustrative) costs 432.00 dollars. The one-off cost is 1,000 labels at 0.30 dollars each (illustrative), which is 300.00 dollars, plus 12 hours of GPU training at the same hourly rate, which is 7.20 dollars, so 307.20 dollars in total, with engineering time left out.1
| Scenario (illustrative inputs) | Bill for 30 days (dollars) | Break-even volume vs the small instance | Payback of the one-off cost (307.20 dollars) |
|---|---|---|---|
| LLM, plain prompt | 397.50 | 18,113 requests a day | 0.9 months |
| LLM, batch | 198.75 | 36,226 requests a day | 2.4 months |
| LLM, cached prefix | 55.50 | 129,730 requests a day | never at this volume |
| LLM, cached prefix and batch | 27.75 | 259,459 requests a day | never at this volume |
| Classifier, small instance | 72.00 | not applicable | not applicable |
| Classifier, one GPU | 432.00 | 108,679 requests a day against the plain LLM | not applicable |
The cached rows assume every request reads the cached prefix and ignore the small cost of writing the cache. The plain LLM crosses the small instance at 18,113 requests a day, and against a GPU it crosses at 108,679 requests a day.1
Volume moves the answer because hosting is flat while tokens are not. Assume one instance serves 20 requests a second, which is 1,728,000 a day (illustrative), and add instances as traffic grows. At 10,000 requests a day the plain LLM bill is 0.55 times the classifier hosting cost, at 100,000 it is 5.52 times, at 1,000,000 it is 55.21 times and at 10,000,000 it is 92.01 times. With a cached prefix the same four ratios are 0.08, 0.77, 7.71 and 12.85.1
Text description of the diagram
- At 100,000 requests a day the plain LLM bill is 5.52 times the classifier hosting cost.
- At 10,000,000 requests a day the plain LLM bill is 92.01 times the classifier hosting cost.
Trade-offs and edge cases
Cost is half the decision, because the cheaper option has to be accurate enough. Evidence on accuracy depends on the task. Bucher and Martini (2024, v2) report that smaller tuned models consistently and significantly outperform larger zero-shot prompted models in text classification. Zero-shot performance is decent for common tasks such as sentiment analysis, but tuned models always surpass zero-shot performance as the training set grows.
A study of South Slavic text found tuned BERT-like models ahead on genre and topic classification, while sentiment was broad enough for stronger LLMs to read label definitions without tuning. In a second task of the political-science study, zero-shot prompting outperformed BERT tuned on 200 samples, and few-shot prompting was equal to or slightly better than BERT tuned on 500 or 1,000 samples.
Treat these studies with care. The models tested were earlier generations, so test current models on your own labelled sample.
Text description of the diagram
- An LLM bills per request, in tokens, including the prompt text on every call.
- A trained classifier bills once for labels and training, then for hosting.
- Changing a label means editing a prompt for an LLM but relabelling and retraining for a classifier.
An LLM bills per request, in tokens, including the prompt text on every call. A trained classifier bills once for labels and training, then for hosting. Changing a label means editing a prompt for an LLM but relabelling and retraining for a classifier.
When each wins
The LLM wins when volume is low, labels do not exist yet, the label set changes often or the task is broad and easy. The classifier wins when volume is high, the task is specialised with many classes, and you can afford labels. Where labels are missing but volume is high, an LLM can label a sample and a small classifier is trained on it. Count requests per day first, because volume sets both bills. Test both options for accuracy on the same labelled sample before choosing.
Step-by-step description
- Requests per day set both bills.
- Price the plain, cached and batch cases.
- Add hosting, labels and training.
- Test both on one sample, then pick.
- Count requests per day first, because volume sets both bills.
- Test both options for accuracy on the same labelled sample before choosing.
Edge cases that change the bill
Several vendor rules and one hybrid pattern change the bill, so check each against your own case.
- Fine-tuning the LLM itself. In October 2024, OpenAI listed 25 dollars per million training tokens for generative pre-trained transformer (GPT)-4o fine-tuning, with inference at 3.75 dollars per million input tokens and 15 dollars per million output tokens. A price comparison says OpenAI charges higher inference rates for tuned models, while other providers it lists charge the base rate. These prices move, so check the current vendor page.
- Shorter prompts save little on cheap models. The same comparison puts the saving from removing a 500-token prompt at about 0.15 dollars per 1,000 requests at 0.30 dollars per million input tokens.
- Caching assumptions can fail. A cached prefix needs enough traffic to stay warm, and vendors set a minimum cacheable length, so a short prompt may not qualify. Check the vendor's rules.
- Batch needs patience. Batch pricing suits jobs that can wait, not a user waiting on a reply.
- Hybrid. An LLM can label a sample that people then check, and a small classifier is trained on those labels. This page has no benchmark for that pattern, so test it before relying on it.
Neighbouring cost levers are covered in how to make retrieval-augmented generation fast and cheap, and the same cost-first habit applies in deciding whether you need retrieval at all.
How to answer it in an interview
Would you use an LLM or a trained classifier for high-volume text classification, and how do the costs compare?
I'd price both as a function of volume. The LLM costs input tokens times the input price plus output tokens times the output price on every request, so the instructions are billed each time unless I cache them, and batch halves the bill if the job can wait. The classifier costs labels and training once, then hosting that is nearly flat. I'd compute the break-even volume. Below it I'd start with the LLM; above it I'd train the classifier, but only after testing accuracy on a labelled sample, because specialised multi-class tasks often favour the trained model.
The spoken structure: say the cost shape of each, give the break-even idea, then say accuracy is tested before the choice.
Common mistakes
- "An LLM is cheaper because there is no training." (wrong) The bill is per call and grows with volume and prompt length. (right) "The LLM has no up-front cost, but I pay tokens on every request, so I compare it with classifier hosting at my volume."
- "Fine-tuning always lowers inference cost." (wrong) At one vendor a tuned model has a higher per-token rate than the base model, and others charge the base rate. (right) "I check the inference rate for the tuned model before assuming a saving."
- "The list price is what I pay." (wrong) Batch and caching change the effective price. (right) "I price the plain, batched and cached cases separately."
- "A paper says tuned models win, so I don't need to test." (wrong) The papers used earlier models and specific tasks. (right) "I run both options on a labelled sample from my own data."
- "I only count token cost." (wrong) Labels, hosting that never shrinks and engineering time are real costs. (right) "I add labelling, training and hosting to the classifier side."
Follow-up questions
Is it cheaper to fine-tune or to prompt an LLM?
Prompting has no up-front cost but bills the prompt on every request, so it is cheapest at low volume. Fine-tuning adds a training bill and, at some vendors, a higher inference rate, and it saves little when the prompt is short and the model is cheap.
At what volume does a classifier become cheaper than an LLM?
At the volume where the classifier's monthly hosting equals the LLM's monthly bill: V = H / (days × c). In the worked example that is 18,113 requests a day against the plain LLM and 129,730 against the cached one (illustrative inputs).1
How does prompt caching change the comparison?
Prompt caching lowers the price of the repeated instructions, so the break-even volume rises. In the worked example, with a 1,200-token prefix assumed to clear the cacheable minimum, caching moves it from 18,113 to 129,730 requests a day.1
Can an LLM label the data for a classifier?
Yes, as a way to cut the labelling bill: an LLM labels a sample, people check it, and a small model is trained on the result. Test the labels first, because the classifier inherits their mistakes. The document processing pipeline design shows where such a step fits.
What if accuracy matters more than cost?
Test both options on the same labelled sample and pick the one that passes, even if it is dearer. A staged approach to measuring quality is in evaluating a system stage by stage.
- An LLM bills per request in tokens, including the prompt; a trained classifier bills once for labels and training, then for hosting.
- Break-even volume is the monthly hosting bill divided by the days times the cost of one LLM request.
- Prompt caching and batch pricing raise the break-even volume, so price the plain, batched and cached cases.
- Tuned small models often win on specialised multi-class tasks, and LLMs are competitive on easy broad ones; test on your own sample.
- Count labels, hosting and engineering time on the classifier side, and check vendor price pages.
Sources
- Small LLMs (still) significantly outperform zero-shot generative AI models in text classification (arXiv 2406.08660) — arXiv (Bucher and Martini, v2), 16 Aug 2024
- Selecting Between BERT and GPT for Text Classification in Political Science Research (arXiv 2411.05050) — arXiv, 1 Nov 2024
- State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting? (arXiv 2511.07989) — arXiv, 1 Nov 2025
- Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning (arXiv 2409.11408) — arXiv, 1 Sep 2024
- Claude API pricing, October 2026 — BenchLM (third-party price table), 23 Oct 2026
- Anthropic API pricing: caching and batch — Finout (third-party price table), 12 Oct 2026
- OpenAI API pricing, October 2026 — nOps (third-party price table), 21 Oct 2026
- LLM fine-tuning pricing comparison — pricepertoken.com (third-party price table), 26 Oct 2026
- Introducing vision to the fine-tuning API — OpenAI, 1 Oct 2024
