Caching LLM calls: how prompt and semantic caching cut cost

In this article

1. Before you start
2. Why interviewers ask this
3. Caching LLM calls from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the LLM inference optimization guide, and it builds on the transformer architecture.

Why interviewers ask this

Cost and delay are the two bills of an LLM product, so caching is one of the first levers a design round reaches. Candidates report this topic in generative AI system design and infrastructure rounds, usually as "how would you cache LLM responses to reduce cost?" or as a follow-up to a retrieval-augmented generation (RAG) design.

A weak answer says "put Redis in front of the model" and stops. It does not say what the key is, what happens when two questions look alike but differ in meaning, or when the saving turns negative. A strong answer separates the two layers, names the exact-prefix requirement of the first one and the wrong-hit risk of the second, and gives a break-even rule. A Staff-level answer adds measurement: hit rate, wrong-hit rate and the routing that keeps trivial queries away from the model altogether.

Caching LLM calls from first principles

Semantic caching

Semantic caching is an application-side store of past queries and their answers that returns a saved answer when a new query has nearly the same meaning, judged by the similarity of their embeddings, so the model is not called at all.

Time to live (TTL)

Time to live (TTL) is a cache setting that limits how long an entry stays usable after it is created or last used. When it runs out, the entry is dropped and the next request pays full price again.

In plain words

Picture a tutor who reads a long textbook before every question. Reading it once and keeping notes would save time. Prompt caching is the notes. The tutor still thinks up a fresh answer each time. Semantic caching is different. It is a notebook of past questions with the answers already written. If a new question matches an old one, the tutor reads the old answer out loud.

The analogy breaks in one place. A tutor knows when two questions mean different things. A cache only compares numbers, and it can be fooled.

A cache hit is a request answered from stored work. A miss is a request that is not. Every design choice below tries to raise hits without raising wrong answers.

In one precise sentence: caching LLM calls reuses either the processed prompt prefix or a whole earlier answer, and each reuse has its own price, time to live and failure mode.

Step 1: Prompt caching reuses a repeated prefix

Prompt caching saves the model from paying again for a long prompt start that many requests share, such as a system prompt, a policy document or a tool list. Anthropic's documentation describes the feature as letting an application resume from a stored prefix of its prompts (Anthropic, accessed 2026-10-26).

Input: a prompt whose first part repeats. Output: a cheaper, faster request, and a fresh answer. The stored thing is the processed prompt, not the reply, so output tokens are billed in full. The processed prefix is the attention keys and values of those tokens, the same KV cache a model builds inside one request, kept after the request ends so the next request with the same start skips that work. Hosted application programming interfaces (APIs) sell this as prompt caching; self-hosted engines such as vLLM do the same with automatic prefix caching. You now know what the first layer saves: re-reading the prompt, not re-writing the answer.

Step 2: The prefix must match, and the entry must stay warm

A prompt cache resumes from a prefix, so the reusable text has to come first and the changing text last. A timestamp, a user name or a retrieved passage placed before the long static part changes the start of the prompt and removes the benefit. Put the system prompt, the documents and the tool list first; put the user's question at the end.

Entries also expire. Anthropic's default time to live is five minutes, and each use refreshes it at no extra charge; a one-hour option exists at a higher write price (Anthropic, accessed 2026-10-26; a production write-up). Sparse traffic therefore loses the entry between requests and pays the write cost again. Check the vendor's live page for minimum prompt length and per-model rates before you commit to a design.

Step 3: Semantic caching stores answers and can return the wrong one

Semantic caching works in front of the model, as in open-source libraries such as GPTCache. The application embeds the incoming query, searches a vector store for the nearest saved query, and returns that query's saved answer if the similarity is above a threshold. A miss goes to the model, and the new pair is saved.

A hit skips the model, so it saves the full cost and most of the delay. In a RAG system it can also skip retrieval. The risk is the threshold. "How do I cancel my plan?" and "How do I cancel my order?" are close in embedding space but need different answers. A low threshold raises hits and wrong answers together; a high one is safe and saves little. Tune it on real traffic, measure the wrong-hit rate, and never share saved answers across users when the answer depends on who asked. The worked example below counts what a small wrong-hit rate means at volume.

Step 4: A router decides which layer a query should use

Not every query needs a model. A greeting such as "hi" can be answered by a fixed rule, and an exact repeat can be served from a plain key-value lookup. A router checks these cheap paths first, then the semantic cache, then the model call, where prompt caching applies to whatever long prefix remains.

The order matters because each layer costs something to check. A rule costs almost nothing. An embedding lookup costs a little on every query, hit or miss. A model call costs the most. Deciding query complexity can itself use a small classifier or simple features such as length and keywords, so it does not need a large model. You now have the whole path: rules, answer store, then a model call with a cached prefix.

1 · Model call and prefix2 · Match and stay warm3 · Store answers4 · RouteReplay

Step-by-step description

1. Static text goes first so the prefix can be cached.
2. Same prefix, within the time to live, hits.
3. Similar queries get a saved answer; a miss calls the model.
4. A router tries cheap paths before the model.

• Prompt caching reuses a repeated prompt prefix and the model still writes a new answer.
• Semantic caching returns a saved answer for a similar query and skips the model call.
• A router sends trivial queries to rules, then the answer store, then the model.

What goes into the cache key

An exact-match or semantic entry is keyed not only on the query text but on everything that changes the answer:

• the model and its version;
• the system-prompt or template version;
• generation settings such as temperature and tools;
• the tenant or user when the answer is personal.

Change any of them and the old entries must be cleared. With temperature above zero, a cached answer also freezes one sample, which is fine for factual queries and wrong for creative ones.

Worked example

The numbers below come from a short script.¹ Prices are in multiples of the normal input price. Anthropic's documentation gives a five-minute cache write at 1.25 times the base input price, a one-hour write at 2 times, and a cache read at 0.1 times for standard models (Anthropic, accessed 2026-10-26; OpenRouter, accessed 2026-10-26). Some newer models use other read multipliers, so read the live table.

Take the break-even first. One write plus one read costs 1.25 + 0.1 = 1.35, against 2.0 for two uncached passes, a saving of 32.5%.¹ A single write that is never read costs 1.25 against 1.0, a loss. The five-minute cache therefore pays for itself after one read, a point also made by a production write-up. The one-hour write at 2 times needs two reads: after one read it costs 2.1 against 2.0 uncached, after two it costs 2.2 against 3.0.¹

Now a 12,000-token prefix reused by 8 requests (illustrative). Uncached it is 8 × 12,000 = 96,000 token-equivalents. Cached it is one write of 15,000 plus seven reads of 1,200 each, 23,400 in total, a saving of 75.6% on the prefix only.¹ Output tokens and uncached text are billed in full, so the whole bill falls by less.

Text description of the diagram

• At one request the cached prefix costs 15,000 against 12,000 uncached.
• At eight requests the uncached prefix costs 96,000 and the cached one 23,400 token-equivalents.

For the semantic layer, assume an uncached call costs 1.0 unit and the embedding plus lookup costs 0.02 units on every query (both illustrative). Cost per query is 0.02 + (1 − h), where h is the hit rate. At h = 0.3 that is 0.72, so 1,000,000 queries a day cost 720,000 units instead of 1,000,000, a saving of 28%.¹ If 1% of those 300,000 hits are wrong (illustrative), that is 3,000 wrong answers a day.¹ A small wrong-hit rate is still a large absolute number at volume.

Trade-offs and edge cases

Each layer wins on a different axis, as the figure shows. Prompt caching is safe, because the model still answers fresh, but it only trims the cost of the prompt. Semantic caching removes the whole call, but it can return a stale or mismatched answer.

Text description of the diagram

• Prompt caching stores the processed prefix and the model still writes a new answer.
• Semantic caching stores whole answers and its main risk is a wrong answer to a look-alike query.

Edge cases decide real designs:

• Sparse traffic. If requests arrive further apart than the time to live, a prompt cache pays the write premium again and again. Keep it warm with steady traffic or skip it.
• Personal answers. A saved answer that mentions one user must never reach another. Key the semantic cache by tenant or user, or cache only generic questions.
• Stale facts. A saved answer outlives the facts behind it. Give entries an expiry and clear them when the source documents change.
• Moving prices. Multipliers differ by vendor and model and change over time, so recompute the break-even from the live table.

For other serving-side savings on a self-hosted model, see continuous batching and paged attention, whose fixed-size KV blocks are also what let a server share a cached prefix across requests.

How to answer it in an interview

The spoken structure: name the two layers, say what each stores, give the trade-off of each, then say how you would measure it.

Common mistakes

• "Prompt caching saves the model's answer." (wrong) It stores the processed prefix, and the model writes a new response each time. (right) "Prompt caching reuses the prompt; a response cache reuses the answer."
• "Caching always saves about 90%." (wrong) Only the cached prefix is discounted, at Anthropic's rates writes cost more than normal input, and output tokens are billed in full. (right) "The read is cheap, the write has a premium, and I work out the break-even."
• "Any similar prompt will hit the prompt cache." (wrong) The cache resumes from a prefix, so changing text early in the prompt breaks it. (right) "Static content first, variable content last."
• "A high hit rate means the semantic cache works." (wrong) Hits can be wrong. (right) "I track hit rate and wrong-hit rate together."

Follow-up questions

How would you cache LLM responses to reduce cost?

Route first: rules and exact-match lookups for trivial queries, then a semantic cache for repeated meaning, then the model with a cached prompt prefix. Measure hit rate and wrong-hit rate, and compute break-even before you build.

What is semantic caching and how can it reduce cost and latency in a RAG system?

Semantic caching returns a saved answer for a query whose embedding is close to an earlier one. In a retrieval-augmented generation system a hit can skip both the vector search and the model call, which cuts cost and delay together.

Your app receives 1M LLM queries per day. How do you optimize cost?

Start by finding the repeated parts: shared long prefixes suit prompt caching, repeated questions suit a semantic cache. In the illustrative model above, a 30% hit rate cuts cost by 28% before wrong hits are counted.¹ Pair it with quantization or smaller models for the misses.

How would you manage rate limits in OpenAI or Azure APIs?

Retry with exponential backoff and jitter, queue bursts, and spread load across keys or regions where the terms allow. Caching helps because every hit is a call you never make, which lowers the pressure on the limit.

How do you log and monitor LLM API failures in production?

Log request id, model, status code, latency, token counts and cache outcome for each call, and alert on error rate and timeouts rather than single failures. Recording the cache outcome shows whether a latency spike is a cold cache or a provider problem.

Should a greeting like hi ever reach the model?

Usually not. A rule or a small classifier can answer greetings and other trivial queries, so the model sees only queries that need it. Likewise, cache the high-frequency in-scope queries and route off-topic ones to a fixed refusal rule.

Sources

1. Prompt caching (Claude developer documentation) — Anthropic (accessed 2026-10-26), 26 Oct 2026
2. Prompt Caching (OpenRouter documentation) — OpenRouter (accessed 2026-10-26), 26 Oct 2026
3. Prompt Caching in Production Agents — dev.to, tridaai (accessed 2026-10-26), 26 Oct 2026
4. Automatic Prefix Caching (vLLM documentation) — vLLM project (accessed 2026-10-26), 26 Oct 2026
5. GPTCache: a library for creating semantic cache for LLM queries — Zilliz, official repository (accessed 2026-10-26), 26 Oct 2026