In this article
Before you start
For the wider topic, see the generative AI system design interview guide. This page treats ChatGPT and Claude.ai as an interview abstraction, because no public primary source describes how either product is built inside.
Why interviewers ask this
A chat product looks simple, yet it exposes every hard part of serving a language model: slow responses, huge memory use per user and a safety question on every message. Candidates report this prompt in loops at AI labs and large technology companies, phrased as designing a chat assistant like ChatGPT or Claude.ai, or a production chat system built on a language model. The prompt often adds conversation state, streaming, safety, error handling and mobile clients.
A weak answer draws a web server, a database and a box labelled "model". A strong answer explains why the reply streams, where the history lives and why one GPU can hold only so many conversations. A Staff-level answer sizes the fleet from a stated workload, uses a cache to cut repeated work and discusses how safety checks trade against usefulness.
Chat system design from first principles
- Server-sent events
- Server-sent events (SSE) are a way for a web server to push a stream of small text messages to a client over one long-lived request, in one direction only, so a chat reply can appear piece by piece.
- Time to first token
- Time to first token is the delay between a user sending a message and the first piece of the reply arriving, and it is what a person feels as the system being slow.
In plain words
Think of a restaurant. Waiters take orders and keep each table's history. The kitchen cooks many dishes together on one set of burners. A waiter does not cook, and the kitchen does not remember tables. The analogy breaks at counter space. A dish takes counter space only while it is being cooked, but a chat reply keeps a growing block of GPU memory for every token so far, and that memory sets how many chats cook together.
In one precise sentence, a chat system is a stateless web layer that rebuilds the full prompt from stored history each turn, plus a GPU layer that batches many such prompts and streams tokens back.
Step 1: The client opens a streamed request to a stateless chat service
Before drawing boxes, state the requirements: a target time to first token, a tokens-per-second rate for each stream, an application programming interface (API) to send a message, stream the reply and fetch a conversation, and a history data model of conversations and ordered messages. The client sends the new message to a chat service. The service authenticates the user, loads the conversation from a history store, and keeps no state itself. It saves the finished reply to the store when generation ends, so any copy of the service can handle the next turn.
The reply streams. The Claude streaming documentation says that setting stream to true streams the response incrementally with server-sent events. The event order is a message start, then content blocks made of a start, several deltas and a stop, then message deltas and a message stop. A text delta carries a few characters, and the client appends each one to its block by index.
Streaming changes perceived latency, meaning time to first token, not total generation time. You now know how text reaches the screen; the next step is what the service sends onward.
Step 2: The chat service rebuilds the prompt and runs safety checks
The model does not remember earlier turns. On every message the service assembles a prompt: the system instructions, the stored history, then the new message. When that prompt exceeds the model's context window, the service drops or summarizes the oldest turns. This is a design decision you state aloud, because it changes what the model can recall.
Safety checks sit on both sides of the model. A fast check on the input can refuse or reroute a message before any GPU time is spent. A check on the output can stop a stream that goes wrong. Per-user rate limits protect the pool, and an error that happens after streaming began must be sent to the client as an event, because the status line has already gone out.
Input: stored history plus a new message. Output: one prompt and a verdict. The prompt then goes to the GPU layer.
Step 3: Continuous batching is limited by KV cache memory
A GPU is cheap per token only when it works on many requests together. The vLLM paper (Kwon et al., 2023) opens by saying that high throughput needs batching enough requests at a time. It adds that the KV cache memory of each request is huge and grows and shrinks, and that fragmentation and duplication waste it and limit batch size.
Batching works at a finer grain than the request. The Orca paper, from the Operating Systems Design and Implementation (OSDI) 2022 conference, introduced iteration-level scheduling, which schedules at the granularity of one iteration instead of one request, and reports 36.9 times the throughput of NVIDIA FasterTransformer on a 175-billion-parameter model at the same latency (reported in the paper's abstract). vLLM stores the cache in blocks, a method called PagedAttention, inspired by operating system paging, and reports 2 to 4 times the throughput of FasterTransformer and Orca at the same latency. The baselines differ, so do not compare the two multipliers.
Decode, which writes one token per iteration, is limited by memory bandwidth and cache size, so memory, not compute alone, sets how many chats run together.
Step 4: Prefix caching and routing cut repeated work
Every turn resends the whole history, so the same long prefix is processed again and again. Prompt caching is a feature that reuses an already processed prefix of a prompt, such as the system instructions or conversation history, across calls. According to the Claude prompt caching documentation, a cache read costs 0.1 times the base input price, a write with a five-minute time to live (TTL) costs 1.25 times, and a one-hour write costs 2 times; the pricing page repeats this and lists per-model exceptions for reads.
Caching is not free: only reads are discounted. If you run your own servers, send each conversation to the replica that holds its prefix, so the cache is hit. The large language model (LLM) gateway page covers routing, limits and caching in front of model providers.
Step-by-step description
- History lives in a store, tokens stream back.
- The prompt is rebuilt and checked.
- A scheduler batches work each iteration.
- Processed history is reused.
- The chat service stores history and streams tokens to the client.
- Safety checks run before and after the model.
- Continuous batching schedules at the granularity of one iteration.
- A prefix cache reuses the processed history across turns.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Size a chat feature for a million daily users. These inputs are illustrative assumptions, not measurements: 1,000,000 daily users send 10 messages each, a peak is 3 times the average, each reply is 400 tokens, each prompt is 2,000 tokens, each GPU sustains 2,500 output tokens per second when batched (a per-GPU rate, not a per-replica one), and one stream delivers 40 tokens per second.1
That is 10,000,000 messages a day, 115.7 a second on average and 347.2 at peak. Peak output is 138,889 tokens a second, which needs 56 GPUs if compute were the only limit. A reply lasts 10 seconds, so about 3,472 streams are open at peak, and the front end must hold that many connections.1
Memory is the second constraint, and here it wins. For an illustrative model with 80 layers, 8 KV heads and a head size of 128 in 16-bit numbers, each token costs 327,680 bytes of cache. A chat of 2,000 prompt plus 400 reply tokens, 2,400 in all, holds 0.73 GiB, so 3,472 open chats hold about 2,543 GiB.1 The same illustrative model has 70 billion parameters at 2 bytes each, so its weights take 140 GB, more than one 80 GB GPU. A replica therefore spans two GPUs and has 20 GB left for cache. Holding 2,543 GiB of cache needs 137 replicas, or 274 GPUs, against 56 for compute alone, so memory sets the fleet size.1
| Quantity (illustrative inputs) | Value |
|---|---|
| Messages per day | 10,000,000 |
| Peak messages per second | 347.2 |
| Peak output tokens per second | 138,889 |
| GPUs, compute bound | 56 |
| Open streams at peak | 3,472 |
| KV cache per token | 327,680 bytes |
| KV cache per 2,400-token chat | 0.73 GiB |
| Fleet-wide KV cache at peak | 2,543 GiB |
| Weights per replica (70 billion parameters, 2 bytes each) | 140 GB |
| GPUs, memory bound (two per replica) | 274 |
| Fleet size, the larger bound | 274 |
Now the cache. Assume a ten-turn chat where the prompt starts at 500 tokens and grows by 300 each turn (illustrative; real APIs set a minimum cacheable prompt length, so a short first turn may not be cached), and every turn arrives within the five-minute cache window. Without caching the chat costs 18,500 input-token equivalents; with a prefix cache that pays 1.25 times for new text and 0.1 times for the reused prefix it costs 5,530.1 Turn one is slightly dearer cached (625 against 500) and the cache pays back from turn two.
Text description of the diagram
- Without caching a ten-turn chat costs 18,500 input token equivalents.
- With a prefix cache the same chat costs 5,530 input token equivalents.
Trade-offs and edge cases
Batch size against latency. A bigger batch raises throughput but each stream gets fewer tokens per second, and the batch cannot outgrow KV cache memory. Say which you protect, usually time to first token.
Cache time to live. From the multipliers in the Claude documentation, two requests with the same prefix cost 2.0 times uncached and 1.35 times with the five-minute cache, so it pays back on the second request. The one-hour cache costs 2.1 times for two requests and 2.2 against 3 for three, so it needs at least two reads.1 Idle chats outlive a short cache.
Streaming transport. Server-sent events are one-way, simple and pass through ordinary web infrastructure. A two-way channel is only needed if the client must interrupt the model mid-stream; a stop button can also be a separate request.
Safety against usefulness. Tight filters refuse benign requests; loose ones let harm through. Layer cheap checks first, review refusals on harmless prompts and keep the policy configurable. Reported practice differs between products, and no public source details either one.
Long context. Every extra token of history costs cache memory and prefill work, so summarize or truncate on purpose.
Text description of the diagram
- Request-level batching schedules whole requests.
- Continuous batching schedules each iteration.
- vLLM stores the cache in blocks, inspired by operating system paging.
How to answer it in an interview
Design a production LLM chat system.
I'd put a stateless chat service in front of a pool of GPU model servers. The client opens a streamed request, the service loads the conversation from a store, builds the prompt, runs safety checks and streams tokens back as server-sent events. Throughput comes from continuous batching, but the limit is KV cache memory, so I'd size the fleet by concurrent streams, not just requests. I'd cache the shared prefix, since every turn resends the history. For safety I'd layer input and output checks and measure refusals on harmless prompts.
The spoken structure: say the stateless layer, then streaming, then batching and its memory limit, then caching, then safety.
Common mistakes
- "The model remembers the conversation." (wrong) The model server holds no history between calls. (right) "The service stores history and resends it as part of each prompt."
- "Streaming makes the model faster." (wrong) It changes when tokens are shown, not how long generation takes. (right) "Streaming lowers perceived latency, measured as time to first token."
- "A bigger batch always helps." (wrong) KV cache memory limits batch size, and fragmentation wastes it. (right) "I batch continuously and manage cache memory in blocks."
- "Caching is free." (wrong) Writes cost 1.25 or 2 times the base input price. (right) "Reads are cheap, writes cost extra, so I cache the stable prefix only."
Follow-up questions
How would you minimize harmful outputs while keeping the model useful?
Layer the checks: a cheap input filter, the model's own training and an output check that can stop a stream. Then measure both sides, harmful outputs that get through and harmless requests that are refused, and tune the policy against both.
How do you scale an AI chat feature to a million daily users?
Convert users to peak messages per second, then to output tokens per second and open streams, as in the worked example: about 3,472 streams, which need 56 GPUs for compute but 274 for memory under the stated assumptions, so memory sets the fleet.1 Then add headroom, autoscaling and a cache.
How do mobile apps interact with the chat backend?
The app sends the message to the same chat service and reads a streamed response. Because networks drop, the app should be able to refetch the stored conversation and show the finished reply.
What happens when an error hits in the middle of a stream?
The Hypertext Transfer Protocol (HTTP) status has already been sent, so the service emits an error event on the stream and the client shows a retry. The partial reply is stored or discarded by a rule you choose, and retries must not double-charge or duplicate the message.
Does streaming make the model faster?
No. Streaming delivers tokens as they are generated, so the perceived wait equals time to first token instead of the full generation time; time to first token itself and total generation time are unchanged.
What changes if the assistant also searches documents?
A retrieval step adds context to the prompt before generation, which grows prefill work and cache use. See designing retrieval over real documents and agentic retrieval-augmented generation for the next layer.
- A chat system is a stateless service that rebuilds the prompt from stored history on every turn.
- Replies stream over server-sent events, which lowers perceived latency without speeding up generation.
- Continuous batching lifts throughput, and KV cache memory limits how many chats run together.
- Prefix caching cuts the cost of resending history, though writes cost extra.
- Size the fleet as the larger of the compute bound and the memory bound (weights plus cache for open streams), and state your assumptions.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al.; SOSP 2023), 12 Sep 2023
- Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI 2022, 11 Jul 2022
- Streaming messages — Anthropic, Claude Platform Docs (read 2026-10-26), 26 Oct 2026
- Prompt caching — Anthropic, Claude Platform Docs (read 2026-10-26), 26 Oct 2026
- Pricing: prompt caching — Anthropic, Claude Platform Docs (read 2026-10-26), 26 Oct 2026
