In this article
Before you start
This page is one step of the LLM inference optimization learning path.
Why interviewers ask this
A wrong cost estimate is a launch decision made on a wrong number. A cost question tests whether you can turn a vague product into a figure and say which term moves it.
A weak answer quotes one price per token and stops. A strong answer separates input, output and cached tokens, states its assumptions, and says when renting GPUs beats paying per token. A Staff-level answer also names the two terms that surprise teams: the cache hit rate for hosted models and the idle capacity of self-hosted ones. Such a question can arrive as a system design prompt or as a direct "estimate the monthly cost of this feature" request, so the method matters more than any one price.
LLM cost estimation in production from first principles
- Cost per million tokens
- Cost per million tokens is a price quoted as dollars for one million tokens of one kind, such as input, cached input or output, that lets you multiply a token count by a rate without handling tiny per-token fractions. Hosted providers publish it for each token type. For your own GPUs you compute it from GPU price and throughput.
- Utilization
- Utilization is the share of a GPU's paid time that it spends producing tokens for real requests, a number between zero and one that multiplies peak throughput in the cost formula, so cost per token scales as one divided by utilization. A GPU that is paid for but idle, or running batches too small to use its arithmetic units, lowers utilization and raises the cost of every token.
- Cache break-even
- Cache break-even is the number of reads of a cached prompt prefix after which writing it to the cache and reading it back costs less than sending the same prefix uncached each time. It depends on the write multiplier, the read multiplier and whether the prefix is reused before the cache expires.
In plain words
Think of a taxi and a rented car. A taxi charges by the kilometre, so a longer trip costs more. A rented car charges by the hour, and an empty seat costs the same as a full one. A hosted model is the taxi. Your own GPUs are the rented car. The analogy breaks in one place: the taxi meter has several tariffs, because input, cached and output tokens each have their own rate. In one precise sentence, an estimate multiplies each token type by its rate for a hosted model, and divides the GPU price by utilization-adjusted throughput for your own servers.
Step 1: Count tokens and price each kind separately
A hosted model bills per token, and the rate card quotes each price per million tokens, with separate columns for input, output and cached input (OpenAI pricing page, accessed October 2026). One tracker states the same rule in plain words: OpenAI bills per token (Morph, as of October 2026).
Take one request and count three numbers: new input tokens, cached input tokens and output tokens. The cost is new input times the input rate, plus cached input times the cached rate, plus output times the output rate, all divided by one million. In a multi-turn chat or agent, the history and tool definitions are resent as input on every call, which one tracker names as a main reason bills exceed simple token counts (nOps, as of October 2026).
Anthropic's price table lists Claude Sonnet 4.6, an older model still listed, with an output rate five times its input rate (Anthropic prompt caching documentation, accessed October 2026). Multiply by requests per day and by thirty for a monthly figure. Prices change often, so date every rate you use.
Step 2: Output tokens cost more than input tokens, and reasoning counts as output
Output tokens are priced higher than input tokens. For Sonnet 4.6 the output rate is five times the input rate (Anthropic documentation; ratio computed in the script).1
The mechanism is consistent with how the hardware works, though providers do not publish it as the reason for the gap. Prefill is the pass that reads the whole prompt at once, and its matrix multiplications are compute-bound. Decode is the loop that generates one token at a time, and its attention is memory-bandwidth-bound (scaling book, February 2025). The transformer architecture explains why each new token needs a fresh pass.
A reasoning token is a token a model generates while working toward an answer, before or alongside the visible text. One tracker reports that OpenAI bills reasoning tokens at the full output rate (nOps, as of October 2026), so a short visible answer can hide a long paid thinking trace.
Step 3: Prompt caching rewrites the input price, if the prefix repeats
Prompt caching lets a provider reuse the processed form of a repeated prompt prefix. Cache reads cost a fraction of the base input price. Anthropic prices it with three multipliers on the base input rate: 1.25 for a five-minute cache write, 2 for a one-hour write and 0.1 for a read (0.05 on Opus 5.5 and Sonnet 5.5, per the same table) (Anthropic documentation, accessed October 2026).
That gives the break-even. A five-minute write plus one read costs 1.35 prefix prices against 2 for two uncached sends, so one read wins. A one-hour write needs two reads: 2.2 against 3.1 A prefix that is never read costs 1.25 times the normal price, so caching can raise the bill.
A hit needs an identical prefix (Anthropic documentation, accessed October 2026). A changing element such as a per-request timestamp inside the cached block gives no hit and a fresh write on every request. Prompts below a model's minimum cacheable length are not cached; check the documentation for your model. Put static text first and changing text last. Caching LLM calls covers the design.
Step 4: GPU memory holds the weights plus the KV cache
GPU memory holds the weights plus the KV cache, so memory sets how many GPUs you rent. The KV cache per token is 2 times layers times KV heads times head dimension times bytes per element, where the 2 counts keys and values (prompt20 explainer; the same form appears in the scaling book). Grouped-query attention (GQA) is an attention design that shares each key-value head among several query heads, which shrinks that figure.
Multiply the per-token size by sequence length and by concurrent sequences to get the cache total. Weights are loaded first and the cache gets what is left, so GPU memory and quantization change the GPU count.
Waste matters too. The vLLM team reports that existing serving systems wasted 60% to 80% of memory through fragmentation and over-reservation, and that paged attention cut the waste to under 4% (vLLM blog, June 2023).
Step 5: Utilization sets the cost per token on your own GPUs
Utilization sets the cost per token on your own GPUs. Cost per million tokens equals the hourly price of all GPUs, divided by the tokens produced per hour at real utilization, times one million.
Utilization matters because decode only uses the hardware well in large batches. The scaling book gives a critical batch size of about 280 tokens for the linear layers on an H100 and notes that generation is much harder to saturate than prefill (scaling book, February 2025). Continuous batching keeps batches full by admitting new requests as others finish.
Batching has a price. More concurrent requests mean higher latency per request, and the gains shrink as the KV cache grows toward the weights in size. An idle GPU still bills, so a node busy a third of the time costs about three times as much per token as a busy one.
Step-by-step description
- Price input, cached and output tokens separately.
- Output tokens cost more than input tokens.
- Reads are cheap, writes cost extra.
- Weights plus KV cache set the GPU count.
- Idle GPUs raise every token's cost.
- Output tokens are priced higher than input tokens.
- Cache reads cost a fraction of the base input price.
- GPU memory holds the weights plus the KV cache.
- Utilization sets the cost per token on your own GPUs.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
A support assistant (an illustrative workload) sends a 20,000-token stable prefix, 500 new input tokens and 400 output tokens per request, at 100 requests a day, which is 3,000 requests a month. Rates are the Sonnet 4.6 row quoted above (an older model still listed in the table), expressed against the input rate: output 5, a five-minute write 1.25 and a read 0.1.1
| Case | Relative cost per request (no caching = 100) |
|---|---|
| No caching | 100 |
| Cache hit | 20 |
| Cache write (a miss) | 122.2 |
If every miss pays a write, caching pays once the hit rate passes about 21.7% for this workload.1 Take cost with no caching as 100. A cache that is never read costs more than no cache: at a 0% hit rate the index is 122.2. At a 50% hit rate it is 71.1, at a 90% hit rate the cost index falls to 30.2, and at a 100% hit rate it is 20.1 One caveat: at 100 requests a day the average gap between requests is 14.4 minutes, longer than the five-minute window, so this workload needs the one-hour write (2 times the input price, break-even at two reads) or bursty traffic to see hits.1
Text description of the diagram
- A cache that is never read costs more than no cache.
- At a 90% hit rate the cost index falls to 30.2.
| Uses of one five-minute write | Average prefix price, uncached = 1 |
|---|---|
| 1 | 1.25 |
| 2 | 0.675 |
| 5 | 0.33 |
| 10 | 0.215 |
| 20 | 0.1575 |
Now the self-hosted side. A 70-billion-parameter model such as Llama 3 70B with 80 layers, 8 KV heads, a head dimension of 128 and 2-byte numbers needs 2 × 80 × 8 × 128 × 2 = 327,680 bytes of KV cache per token (prompt20). Its weights take about 140 GB in 16-bit form (IntuitionLabs).1
| Quantity | Value |
|---|---|
| Weights, 70B at 2 bytes per number | 140 GB |
| KV cache per token | 327,680 bytes |
| KV cache, one 4,096-token sequence | 1.34 GB |
| KV cache, one 128,000-token sequence | 41.94 GB (39.06 GiB) |
| Memory left on 8 × 80 GB after weights (before activations and runtime overhead) | 500 GB |
| Concurrent 4,096-token sequences, no waste | 372 |
| Same, with 70% waste (illustrative, inside the reported 60% to 80%) | 111 |
| Same, with 4% waste | 357 |
The 4% case holds 3.2 times as many sequences as the 70% case, which is why paged memory means fewer GPUs for the same load.1
For utilization, assume (illustrative) 8 GPUs at a fixed hourly rate each and a node throughput of 2,400 generated tokens per second at full utilization. Taking the cost per generated token at full utilization as 100, it is 125 at 80% utilization, 200 at 50% and 333.3 at 30%.1 The figure counts generated tokens only and ignores prefill, so it is a floor, not a quote. Cost per token scales as one divided by utilization. At 30% utilization the cost index is 333.3.1
Text description of the diagram
- Cost per token scales as one divided by utilization.
- At 30% utilization the cost index is 333.3.
Trade-offs and edge cases
No deployment option is cheapest on every workload, because each one moves a different cost term.
| Option | What you pay for | Use it when | Main risk |
|---|---|---|---|
| Hosted, no caching | tokens at list rates | traffic is small or prompts do not repeat | long repeated prompts are billed in full |
| Hosted with caching | cheap reads, dearer writes | a long prefix repeats inside the cache window | a changing prefix turns every request into a write |
| Hosted batch discount | tokens at a reduced rate | results can wait | slower results |
| Self-hosted GPUs | GPU hours | traffic keeps the GPUs busy | idle capacity and engineering time |
Long context has its own cliff. One tracker reports that above 272K input tokens OpenAI doubles the input and cache rates and raises the output rate by 1.5 times, and that the higher rates apply to the whole request, not only the tokens past the threshold (nOps, as of October 2026). Read the long-context rules of your provider before you size a long-document feature.
A batch interface suits work that can wait. One tracker reports that Batch and Flex bill at half the standard rates (Morph, as of October 2026), and Anthropic says its caching multipliers stack with the batch discount (documentation). Bulk batch work cares about total cost and is blind to the latency of single requests (scaling book); see billion-document batch inference.
A latency target pushes the other way. A tight target forces smaller batches and lower utilization, so serving a 70B model at a tight p95 costs more per token than a batch run. The token formula also leaves out retries, tool calls, embeddings and logging, which should be added as separate lines once you can measure them.
How to answer it in an interview
How would you estimate the production cost of an LLM feature?
I start with the workload: requests per day, input tokens, output tokens and how much of the prompt repeats. For a hosted model I multiply each token type by its rate, remembering that output costs more than input, and that a cache read is cheap but a cache write costs extra, so I need the hit rate. For self-hosting, memory sets the GPU count, weights plus KV cache, and cost per token is GPU price divided by throughput at real utilization. I state my assumptions and recompute at low utilization, because idle GPUs still bill.
The spoken structure: name the workload, give the hosted formula, name the cache rule, then the self-hosted formula with utilization.
Common mistakes
- "Cost is parameters times tokens." (wrong) That ignores the price gap between input and output and the cached rate. (right) "I price input, cached input and output separately."
- "Caching always cuts input cost by nine tenths." (wrong) A write costs more than normal input, and a changing prefix never hits. (right) "Caching pays above a break-even hit rate, so I measure the hit rate first."
- "Self-hosting is cheaper per token." (wrong) It depends on utilization, and idle GPUs still bill. (right) "I compute cost per token at several utilization levels before comparing."
- "Weights are the only memory." (wrong) The KV cache grows with context and concurrency. (right) "I size weights plus KV cache for the peak number of concurrent sequences."
Follow-up questions
How do you estimate the cost of an LLM feature before launch?
Estimate requests, tokens per request and the repeated share of the prompt, then apply dated rates for each token type. Add a range for the hit rate and, if self-hosting, for utilization, and report the low and high bill.
How do you answer an LLM serving cost estimation interview question?
State the workload, write the formula for the option you pick, name the two or three assumptions that move it most, and give a number with a range. Then say what you would measure after launch to replace the assumptions.
When is prompt caching not worth it?
It is not worth it when the prefix changes on every request, when the prompt is shorter than the model's minimum cacheable length, or when reads are too rare to pass the break-even. The worked example puts the break-even hit rate near 21.7% for one workload.1
Is self-hosting an LLM cheaper than a hosted model?
It can be, but only at high utilization and for a model you are allowed to host. In the worked example, a node used 30% of the time costs 3.33 times as much per token as a fully used one.1 Against the output rate of the Sonnet 4.6 row, the illustrative node matches the hosted price per generated token at about 12% utilization, a floor that ignores input tokens and engineering time.1
Why do output tokens cost more than input tokens?
Input is read in one parallel prefill pass, while output is produced one token at a time in decode, which is memory-bandwidth-bound (scaling book). The mechanism fits the price gap, though providers do not state it as their reason.
What costs does a token formula leave out?
It leaves out retries, tool calls, embeddings, logging, and the engineering time of running GPUs. Add each as its own line once you have measurements, and read the LLM inference guide for the serving side.
- Price input, cached input and output tokens separately, and date every rate.
- Output costs more than input, and reasoning tokens bill as output.
- Caching saves money only above a break-even hit rate and with an identical prefix.
- Memory, weights plus KV cache, sets the GPU count; paged memory cuts the waste.
- Self-hosted cost per token rises as utilization falls, so test the low case.
Sources
- Pricing - OpenAI API docs — OpenAI, 26 Oct 2026
- Prompt caching - Claude Platform Docs — Anthropic, 26 Oct 2026
- Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al.), 12 Sep 2023
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — vLLM project, 20 Jun 2023
- All About Transformer Inference - How To Scale Your Model — Google DeepMind authors (online book), 4 Feb 2025
- KV cache explainer with per-token size formula (secondary) — prompt20, 16 May 2026
- KV cache memory and long-context inference cost (secondary) — IntuitionLabs, 26 Oct 2026
- OpenAI API pricing, as of October 15, 2026 (tracker, secondary) — Morph, 15 Oct 2026
- OpenAI API pricing: long-context and reasoning-token billing (tracker, secondary) — nOps, 26 Oct 2026
