AI Agents

Are LLMs stateful or stateless?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

Large language models (LLMs) are stateless between calls and stateful only inside one: each request must carry the whole conversation, and a key-value (KV) cache holds working state only for that request. The follow-up an interviewer asks next is how fast the input grows with each turn of a chat, and the section on what resending the history costs computes it.

In this article

Before you start
Why interviewers ask this
Stateless and stateful LLMs from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page belongs to the AI agent architecture path, where memory is one of the core design choices.

Why interviewers ask this

A wrong answer here leads to a wrong design. An agent that assumes the model remembers will lose facts the moment the application forgets to resend them, and it will meet a token bill it did not expect. Interviewers ask the question because it separates three layers that are easy to blur: the model, the programming interface in front of it, and the application around both.

A weak answer says "yes, ChatGPT remembers the conversation." A strong answer says the model is stateless between calls, the application owns the state, and the cost of resending grows with the chat. A Staff-level answer separates three different things people call state (the conversation, the KV cache and the prompt cache) and says which of them can change the output. The question is reported by candidates in agent and system design rounds.

Stateless and stateful LLMs from first principles

Stateless: Stateless is a property of an interface that treats every request as independent, keeping nothing from earlier requests, so the caller must send all the context the answer depends on each time.
Conversation state: Conversation state is the record of earlier turns, the alternating user and assistant messages, that an application stores and sends back so the model can answer a follow-up in context.

In plain words

Picture a helper who forgets everything after each phone call. To get a useful answer, you read the whole transcript of your earlier calls aloud every time. The helper replies and forgets again. That is how a stateless model application programming interface (API) works.

The analogy breaks in two places. Reading the transcript costs money per word. And you may edit the transcript, even invent lines in it, and the helper cannot tell.

In one precise sentence: an LLM call is a function from the prompt tokens to a distribution over the next token, with weights that do not change during the call.

State lives in the application; the caches around the model save compute but add no memory.CC BY 4.0 — source: Are LLMs stateful or stateless?

Step-by-step description

The model reads the whole prompt each call.
The app stores history and resends it.
Working state exists inside one request.
A repeated prefix is reused across requests.

The model receives the whole prompt in every request.
The application stores the history and resends it.
The KV cache lives inside one request.
Prompt caching reuses a repeated prefix across requests.

Step 1: The model is a fixed function of its input

The weights of an LLM are fixed once training ends, so nothing a user says in a chat changes the model. Input: a list of tokens. Output: the next tokens. Sampling can make two replies to the same prompt differ, but that is randomness in the output step, not memory of the first reply.

The Hugging Face documentation treats the cache as an inference feature: it says caching should only be used for inference (Hugging Face Transformers, accessed 2026-10-26). Training, which does change weights, is a separate process that never runs inside a chat request.

So "does it remember me?" has a clear answer at the model level: it can only use what is in the prompt it is handed. You now know where memory cannot live. The next step shows where it does live.

Step 2: The application resends the whole history on every call

The application owns the conversation state and sends it again with each request. Anthropic's documentation puts it directly: the Messages API is stateless, so you always send the full conversational history (accessed 2026-10-26). The same page notes that earlier turns need not come from the model at all, so "memory" is simply text the application chooses to include. If the application leaves a turn out of the request, the model cannot see that turn at all, and it answers as though the turn never happened.

OpenAI describes the same pattern: each text generation request is independent and stateless, and in Chat Completions you store the transcript and send the accumulated messages array on each request (both accessed 2026-10-26).

What does resending the history cost?

Resending the history makes the input of each turn grow by one exchange. Take an illustrative chat with a 300-token system prompt, 100-token user messages and 150-token replies. Turn 1 sends 400 input tokens and turn 10 sends 2,650, which over ten turns totals 15,250 input tokens against 1,300 if only new text were sent.¹

In this illustrative chat the input per turn grows by 250 tokens, so the resend cost over ten turns is about 11.7 times the new text alone.
Quantity (illustrative)	Value
Input tokens at turn 1	400
Input tokens at turn 10	2,650
Total input over 10 turns, history resent	15,250
Total input over 10 turns, new text only	1,300
Ratio of the two	11.73

Resending history makes each turn's input larger, while the new text stays the same size.CC BY 4.0 — source: Are LLMs stateful or stateless?

Text description of the diagram

Input grows from 400 tokens at turn 1 to 2,650 at turn 10.
The new text stays at 100 tokens per later turn.

Input per turn climbs from 400 tokens at turn 1 to 2,650 at turn 10, while the new text stays at 100 tokens per later turn.¹ Long histories also hit the context window, a limit the application must manage with trimming, summaries or retrieval; the context rot page covers how quality drops as the prompt fills.

Step 3: A KV cache is state inside one request, not memory

A KV cache is a store of the attention key and value vectors of tokens the model has already processed, which lets it skip recomputing them at each generation step. It is state, but it lives inside one request and holds no new knowledge. Hugging Face explains that the stored pairs are reused for later tokens and that each step appends the new key and value vectors (accessed 2026-10-26).

The cache grows with sequence length. For an illustrative model with 32 layers, 8 key-value heads, a head size of 128 and 16-bit numbers, one token needs 131,072 bytes, so 8,192 tokens need 1 gibibyte and 32,768 tokens need 4.¹ The Hugging Face cache guide says this memory can become a bottleneck for long contexts, that quantising the cache cuts memory at a cost in speed, and that undefined disables it (accessed 2026-10-26).

Turning the cache off makes generation slower, not different in meaning. That is the test for "is it memory?": removing it must not change what the model knows.

Step 4: Server-side helpers and prompt caching move the work, not the model

Server-side conversation features store and replay history for you; they do not make the model remember. OpenAI's Responses API can chain responses with previous_response_id, and its migration guide presents this as a way to have OpenAI manage prior response context (accessed 2026-10-26). The same page warns that stable instructions must be resent, because the chain does not carry over the previous top-level instructions.

Prompt caching lowers the price of resending. Anthropic documents that 5-minute cache writes cost 25 percent more than base input, 1-hour writes cost twice base, and reads cost 0.1 times base on most models, with per-model exceptions (accessed 2026-10-26). The standard read multiplier is 0.1; the page's table footnote prices cache reads on Claude Opus 5.5 and Claude Sonnet 5.5 at 0.05 times base.

Real requests are cached only when the prefix reaches the minimum length the documentation sets for each model, and only from a cache breakpoint you mark, so short early turns pay full price. Ignoring that minimum, the standard multipliers give 4,572.5 units against 15,250 for the illustrative chat, about 30 percent of the uncached cost, which is a best case rather than a prediction.¹ The request is still stateless: the prefix must be the same, and the context window still applies.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.