In this article
Before you start
This page belongs to the distributed systems interview guide. It assumes no caching knowledge.
Why interviewers ask this
A cache is the cheapest way to cut load on a database, and it is also a common source of wrong answers shown to users. That pairing is why caching shows up in design rounds: the question separates a candidate who says "add Redis" from one who says what is cached, how it stays correct and what happens when it fills.
Microsoft's Cache-Aside guidance says applications use a cache to improve performance for repeated access to information in a data store. The same page says it works best for relatively static data or data that applications read frequently.
A weak answer names a tool. A strong answer shows the read path, the write path, the expiry rule and the eviction rule, and says what the system does when the cache is down. A Staff-level answer adds the number that matters, the hit ratio, and ties it to database load. The related backend interview questions and database interview questions pages cover the neighbouring ground.
Caching from first principles
- Cache
- A cache is a faster store that holds copies of data from a slower source of truth, so that repeated reads can skip the slow path. It never replaces the source: if a copy is missing, expired or wrong, the source decides, and the cache is rebuilt from it on demand.
- Hit ratio
- The hit ratio is the share of reads that the cache answers on its own, a number between 0 and 1. Each read that the cache cannot answer is a miss and goes on to the slower source, so the hit ratio sets how much load the source still carries.
- Time to live
- A time to live (TTL) is a time limit stored with a cached copy; after it passes, the copy counts as expired and must be refreshed from the source before use. A short TTL limits how long stale data can be shown, and a long one saves more reads.
- Eviction
- Eviction is the removal of cached entries by the cache itself when it runs out of space, using a rule such as removing the least recently used (LRU) entry. Expiry is different: it is driven by time, not by space, and a cache can do both at once.
In plain words
Picture a filing room across the building and a notebook on your desk. You copy the pages you read most into the notebook. Looking up a page there is quick. Going to the filing room is slow. The analogy breaks in one place: nobody edits the filing room while you are away, but a database changes all the time, so a copy in your notebook can go out of date.
In one precise sentence, caching trades a small amount of fast storage and some risk of stale data for a large cut in average read time and in load on the source.
Step 1: A read asks the cache first
Every read goes to the cache before the database. The cache is a key-value lookup: the application builds a key such as user:42:profile, and the cache returns the stored value or nothing. If it returns the value, that is a cache hit and the database is never touched.
Input: a key. Output: the value, or a miss. The key must hold everything that changes the answer. If two users see different pages, the user must be part of the key, or one user receives the other's data. You now know the fast path; the next step covers what happens when the key is absent.
Step 2: A miss loads from the database and fills the cache
On a miss the application reads the database, returns the answer and stores a copy in the cache for the next reader. This pattern is called cache-aside, or lazy loading: data is loaded on demand into the cache from the data store, so only data that is actually read takes up space.
Input: a miss. Output: the value, plus a new cache entry. The first read of each key is always slow, and a restart that empties the cache makes every key slow once more. Many commercial caches also offer read-through, write-through and write-behind modes, where the cache itself talks to the database; cache-aside is the version the application controls, and it works with any cache.
Step 3: A write must invalidate the cached copy
When data changes, the cached copy is wrong. In cache-aside the application writes to the database first and then deletes the cached entry, so the next read misses and reloads fresh data. Microsoft describes this as invalidating the entry on write and repopulating it on the next read.
Input: a write. Output: the database changed, the cache entry gone. Deleting is safer than overwriting the entry with the new value, because two concurrent writers can store their copies in the wrong order and leave the old value in the cache. Microsoft's Cache-Aside page warns that between the write and the next read, a reader can see a cache miss or briefly see stale data, and that replicating data across stores makes consistency harder.
Even this order has a race: a reader misses and reads the old row, the writer updates the database and deletes the entry, and then the reader fills the cache with the old value. That stale copy stays until its TTL runs out, so the TTL bounds it; leases or versioned writes are the stronger fix.
Step 4: Expiry and eviction keep the cache bounded
A cache is small, so it needs two rules. Expiry is time-based: each entry carries a time to live, and an expired entry in a key-value cache is not served. Eviction is space-based: when the cache is full, it removes entries. Microsoft notes that most caches adopt a least-recently-used policy to choose what to evict. A cache that evicts too eagerly loses useful entries, and one that evicts too slowly runs out of memory, so the policy and the size limit are chosen together.
In Redis, an in-memory cache and data store, the maxmemory setting is the limit, and when the cache exceeds it the server enforces the chosen eviction policy. You now have the whole loop: read, miss and fill, write and invalidate, then bound the size.
Step-by-step description
- A read asks the cache first.
- A miss loads from the database.
- A write invalidates the copy.
- Expiry and eviction bound the cache.
- A read asks the cache first.
- A miss loads from the database and fills the cache.
- A write changes the database and deletes the cached copy.
- Time to live and eviction limit what the cache holds.
Where else does caching appear?
Caching is not only for databases. In Hypertext Transfer Protocol (HTTP), the web's request protocol, caches reuse a prior response to satisfy a current request (Request for Comments (RFC) 9111, 2022). A response is fresh while its age is below its freshness window and stale after; with a window of 60 seconds (illustrative), a response aged 45 seconds is fresh and one aged 61 seconds is stale.1 A stale response can often be revalidated with the origin rather than fetched in full. A shared cache stores responses for more than one user, so private data must not go there.
Large language model (LLM) servers cache too. A key-value (KV) cache is a store of the attention keys and values of tokens already processed, and prefix caching is reusing those stored blocks when a new request starts with the same prompt prefix. The vLLM documentation states that it does not change model outputs.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Suppose a service receives 1,000 reads per second, a cache read takes 1 millisecond and a database read takes 50 milliseconds. All three values are illustrative assumptions. The average read time is the hit ratio times the cache time plus one minus the hit ratio times the database time, and the database load is the request rate times one minus the hit ratio.1
| Hit ratio (illustrative) | Average read time | Database reads per second |
|---|---|---|
| 0.5 | 25.5 milliseconds | 500 |
| 0.9 | 5.9 milliseconds | 100 |
| 0.99 | 1.49 milliseconds | 10 |
The average read time is a straight line in the hit ratio: it falls by 0.49 milliseconds for each point, so the chart's x-axis, with uneven spacing, is not to scale.1 What grows near the top is the relative change: from 0.9 to 0.99 the average drops about fourfold (5.9 to 1.49) and database reads drop tenfold.1 That is why interviewers ask for the hit ratio, not only for "a cache".
Text description of the diagram
- At a hit ratio of 0.5 the average read takes 25.5 milliseconds in this illustrative case.
- At a hit ratio of 0.99 the average read takes 1.49 milliseconds.
What hit ratio can you expect? It depends on how skewed the reads are. In an illustrative simulation, 10,000 items are read 200,000 times with read odds proportional to one over the item's rank, and the cache holds only 1,000 items. An LRU cache reaches a hit ratio of 0.675, and a cache that evicts a random entry reaches 0.633.1 A cache that holds a tenth of the items answers about two thirds of reads because a few items dominate, which is the case where Redis says undefined fits, when a subset of elements is read far more often than the rest.
For sizing, take 1,000,000 items, a hot share of 20% and 2,000 bytes per item (all illustrative): the hot set is 0.4 GB.1 The back-of-the-envelope estimation page shows how to do that arithmetic aloud.
Trade-offs and edge cases
Caching trades freshness and simplicity for speed and load. The main costs are stale reads, an extra component that can fail, and an eviction rule you must choose.
Redis offers several eviction policies. The allkeys policies can evict any key, the volatile policies evict only keys that have a time to live, and noeviction removes nothing, so commands that add data return an error. The Redis eviction documentation lists the policies, among them allkeys-lru, allkeys-lfu, volatile-lru, volatile-ttl and allkeys-random. The table below is a selection: recent Redis versions also have volatile-lfu, volatile-random and the allkeys-lrm and volatile-lrm policies.
Text description of the diagram
- The allkeys policies can evict any key.
- The volatile policies evict only keys that have a time to live.
- The noeviction policy removes nothing and writes that add data return an error.
Caches sit at several levels: the browser, a content delivery network (CDN) at the edge, the application, and the database's own memory buffer. For the application level, the main strategies differ in who writes the database and what a crash costs.
| Strategy | Who writes the database | Freshness after a write | Write latency | Data loss if the cache crashes |
|---|---|---|---|---|
| Cache-aside | the application | entry deleted, next read reloads; a stale fill is bounded by the TTL | database write time | none, the database has every write |
| Read-through | the application; the cache loads on a miss | as cache-aside, unless the application also invalidates | database write time | none, the database has every write |
| Write-through | the cache, synchronously | cache and database updated together, so fresh | database plus cache write | none, the database has every write |
| Write-behind (write-back) | the cache, later, in the background | fresh in the cache, the database lags | lowest, only the cache write | queued writes not yet saved are lost |
| Write-around | the application, skipping the cache | cache stale until expiry or invalidation | database write time | none, the database has every write |
Edge cases to name:
- Cache down. The application must detect that the cache is unavailable and fall back to the original data store, per the caching guidance. Size the database for that day.
- Never keep data only in the cache. The caching guidance says not to store valuable data in the cache alone and to keep the information in the original data store as well.
- Popular entry expires (cache stampede, or thundering herd). Many requests can miss together and hit the database. Common remedies are a lock so one request reloads the entry, and randomising time-to-live values so entries do not all expire together.
- Spreading over many machines. One node eventually fills, so keys are spread across nodes; the distributed cache design page covers consistent hashing.
How to answer it in an interview
What is caching, and how would you use it in this design?
Caching means keeping copies of frequently read data in a faster store, such as memory, so most reads skip the database. I would use cache-aside: the application checks the cache, on a miss loads from the database and fills the cache with a time to live, and on a write updates the database and deletes the cached entry. The database stays the source of truth. The trade-off is staleness, so I set the time to live from how wrong the data may be, choose an eviction policy such as least recently used, and watch the hit ratio, because database load depends on the misses.
The spoken structure: define the cache, give the read path, give the write path, then name staleness, eviction and the hit ratio.
Common mistakes
- "The cache is where the data lives." (wrong) A cache holds copies and can lose them. (right) "The database is the source of truth; the cache is a disposable copy."
- "Expiry and eviction are the same thing." (wrong) One is driven by time, the other by memory pressure. (right) "A time to live expires an entry; eviction removes entries when the cache is full."
- "From the application I set the cache to the new value and write the database." (wrong) Concurrent writers can leave the old value in the cache. (right) "I write the database, then delete the cache entry, and the TTL bounds any stale copy."
- "A stale entry is unusable." (wrong) In HTTP a stale response can be revalidated with the origin. (right) "Stale means check again, not necessarily fetch again."
Follow-up questions
What is a cache key?
A cache key is the name under which a cached value is stored, and it must contain every input that changes the answer, such as the user, the language or the page number. A key that leaves one out serves one user another user's data.
What is KV caching in an LLM?
KV caching stores the attention keys and values of tokens a model has already processed, so it does not recompute them. With prefix caching, the vLLM documentation says blocks of processed requests are reused when a new request shares the same prefix, and only full blocks are cached. The KV cache grows with context length, and it is reused only within one request unless prefix caching is on.
What is LLM response caching, and how does it differ from prefix caching?
A response cache returns a stored answer when the same prompt arrives again, so the model does not run. Prefix caching still runs the model and only skips repeated prompt computation. A response cache can serve an outdated answer, so key it on the exact prompt and model and give entries a time to live. Batching is a separate optimization: it groups several requests so one model pass serves them, while caching avoids repeating work.
Can you design a clustered caching system like an online store?
Spread keys over many cache nodes with consistent hashing, keep replicas for failover, cache pages at the edge with HTTP caching and let cache-aside protect the database. The distributed cache design page walks through it.
What is the difference between expiry and eviction?
Expiry removes an entry because its time to live ran out; eviction removes an entry because the cache is full. The volatile Redis policies evict only keys that have a time to live.
Does caching help when you rate-limit an API?
It can: a limiter for an application programming interface (API) keeps small counters that are read and written very often, so they live in a fast store. The distributed rate limiter page shows the design, and for representational state transfer (REST) interfaces the REST API interview questions page covers the topic of HTTP caching.
- A cache holds copies of data; the database stays the source of truth.
- Cache-aside reads the cache first, loads on a miss and invalidates on a write.
- Expiry is driven by time, eviction by memory pressure.
- Average read time and database load both depend on the hit ratio.
- HTTP responses and LLM KV blocks are cached too, with their own rules.
Sources
- Cache-Aside pattern — Microsoft Azure Architecture Center, 11 Sep 2025
- Caching guidance — Microsoft Azure Architecture Center, 26 Oct 2026
- Key eviction — Redis documentation, 26 Oct 2026
- RFC 9111: HTTP Caching — IETF (RFC Editor), 1 Jun 2022
- Automatic Prefix Caching — vLLM documentation, 26 Oct 2026
