ML System Design

How do token limits impact GenAI application design?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 10 min read

How do token limits impact GenAI application design? A token limit caps what one generative AI (GenAI) request can hold, prompt, history, documents and answer together, so design starts with a token budget per call. The follow-up that separates levels is cost: a 20-turn chat resends its history, so total input grows with the square of the turns.¹

In this article

Before you start
Why interviewers ask this
Token limits from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

The question usually comes up inside a larger machine learning (ML) system design round, as a follow-up on a chat or document-question-answering design.

Why interviewers ask this

A token limit is the first hard constraint a GenAI feature meets, and it quietly decides cost, speed and what the product can remember. Interviewers use the question to see whether a candidate designs around the limit or only knows its number. It is reported by candidates in design rounds on chat assistants and document question answering.

A weak answer says "the model can only read so much text, so we cut the input." A strong answer names everything that counts against the limit, including the reply, and describes a budget. A Staff-level answer adds the cost curve of resending history, the quality problem of very long inputs, and a rule that limits come from configuration rather than being hard-coded.

Token limits from first principles

Token limit: A token limit is a fixed cap that bounds how many tokens a model can handle in one request, counting the prompt, the earlier messages and the generated reply together, so an application must choose what to include before each call.

In plain words

Think of a suitcase with an airline weight limit. Everything you pack counts: clothes, shoes, and the gifts you buy for the trip home. The analogy breaks in one place: a model does not keep the suitcase between trips. On every turn you repack the whole case, and the model reads all of it again. The reply is written into the same case. So reserve room for it first; a full case leaves the answer nowhere to go.

In one precise sentence, the application must choose, before each call, which tokens go into a fixed-size window that also has to hold the answer.

Step 1: Everything in the request counts, including the answer

The context window covers more than the user's message. Anthropic's context-window documentation says everything in the request counts toward the window: the system prompt, every message (including tool results, images and documents) and the tool definitions. The window also holds the output the model generates, and thinking tokens count toward it too. A separate cap, max_tokens, limits one response. Window sizes differ by model and change often, and small models, such as those in on-device and privacy-constrained ML, have small windows. Anthropic's help page says you can query token limits programmatically with the Models API, whose response includes max_input_tokens and max_tokens, so read limits from configuration rather than hard-coding them.

Token counts are not word counts. OpenAI's help page gives a rule of thumb of about 4 characters, or three quarters of a word, per English token, and says the relationship between characters, words and tokens differs by language, so test the budget in each language you support. You now know what the limit covers; the next step turns it into a budget.

Step 2: A token budget turns the window into a plan

The token budget is the window minus every part that is always present. Take an illustrative window of 200,000 tokens, a system prompt of 2,000, tool definitions of 3,000, a reserved reply of 4,000 and a safety margin of 2,000. What remains for history and retrieved text is 189,000 tokens.¹

Input: the window and the fixed parts. Output: the room left. That room is a ceiling, not a target. At an illustrative 500 tokens per retrieved chunk, 189,000 tokens would hold 378 chunks, yet a top-8 retrieval costs only 4,000 tokens, or about 16,000 characters of English.¹ Sending less is cheaper, faster and, as the later step shows, often more accurate.

Before sending, count rather than guess. Use the vendor's own token counter on the full request, because tokenizers differ between model families.

Step 3: Resending history makes input cost grow with the square of the turns

A chat is stateless: each turn the application resends the earlier turns, and Anthropic's documentation describes the input as containing all previous conversation history plus the current user message. If each turn adds an illustrative 1,000 tokens, turn t sends t times 1,000 tokens, so n turns send n(n+1)/2 times 1,000 in total. Over 20 turns that is 210,000 input tokens for only 20,000 new ones, 10.5 times as many.¹

A 5-turn sliding window cuts the input over 20 turns from 210,000 tokens to 90,000, while the full history grows with the square of the turns.
Turns	Full history resent (tokens)	5-turn sliding window (tokens)
5	15,000	15,000
10	55,000	40,000
20	210,000	90,000

Each figure is computed from the illustrative 1,000 tokens per turn.¹ The window limit is rarely the first thing a long chat hits; the bill and the delay are. Delay grows because the model processes the whole input before it writes the first token and then generates output one token at a time. Short prompts matter most in latency-bound features such as typeahead autocomplete. Providers also cap throughput in input and output tokens per minute; read those limits from your plan's configuration, because they shape queueing and retries. A sliding window stops the climb, at the price of forgetting old turns.

Resending everything grows with the square of the turns; a sliding window grows only linearly after its first turns.CC BY 4.0 — source: How do token limits impact GenAI application design?

Text description of the diagram

Over 20 turns, resending the full history costs 210,000 input tokens.
A 5-turn sliding window costs 90,000 input tokens over 20 turns.
Over 5 turns both policies cost 15,000 input tokens.

Step 4: Trim, summarize or retrieve to fit

Three patterns fit content into the budget. Trimming drops the oldest turns (the sliding window above). Summarizing replaces old turns with a short summary, which can lose detail. Retrieval-augmented generation (RAG) is a pattern in which the application searches a document store and puts only the best-matching chunks into the prompt, so a corpus larger than the window stays usable. The retrieval, ranking and re-ranking funnel shows the same retrieve-then-rank idea at scale, and chunks are embedded for search, as in an embedding service.

A document larger than the window is summarized chunk by chunk, then the summaries are summarized (map-reduce), or only relevant chunks are retrieved. A bigger window does not remove the need: Liu et al. (2023) found performance is often highest when relevant information sits at the start or end of the input and degrades significantly in the middle of long contexts. Prompt caching is a provider feature that reuses a repeated prompt prefix instead of processing it again; it changes the price, not the window, so budget as if nothing were cached.

A token limit is handled in four moves: count everything, set a budget, watch history grow, and choose how to fit.CC BY 4.0 — source: How do token limits impact GenAI application design?

Step-by-step description

Prompt, history, tools and reply all count.
Window minus fixed parts and reply.
History is resent every turn.
Trim, summarize or retrieve.

Everything in the request counts, including the reply.
The budget is the window minus the fixed parts and the reserved reply.
History is resent each turn, so input grows with the square of the turns.
Trimming, summarizing or retrieval keep the request inside the limit.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.