In this article
Before you start
This page belongs to the generative AI system design interview guide, which covers the neighbouring designs.
Why interviewers ask this
A deployment question separates candidates who can call a model from candidates who can run one for a whole company. Many teams, many applications and many kinds of data share the same models, so the answer has to cover who may call what, what it costs, where the model runs, what it can touch and how you notice trouble.
A weak answer draws one box labelled "LLM" and one arrow from the app. A strong answer names the layers, says what each one prevents, and gives one failure mode per layer. A Staff-level answer adds the numbers that drive the design, such as memory per request on a GPU and requests per minute per team, and says which controls fail open. For a related design that fixes the tenancy question, see how to architect a multi-tenant LLM deployment.
Enterprise LLM deployment architecture from first principles
- Virtual key
- A virtual key is a credential issued by the gateway to one user, team or project, carrying its own allowed models, budget and rate limit, so applications never hold the real provider keys.
- Serving engine
- A serving engine is the program that loads a model onto GPUs and runs requests through it, batching many requests together and managing the memory they need.
In plain words
Think of a company's phone system. Employees do not dial the outside world directly. Calls go through a switchboard that knows who you are, what you may spend and which carrier to use, and it keeps the bill. Behind the switchboard are the carriers themselves. The analogy breaks in one place: a model call is not just connected, it can also read company data and trigger actions, so the layers around it also need security checks.
In one precise sentence, enterprise LLM deployment architecture puts one controlled path between every application and every model, and adds serving, data access, security and monitoring layers along that path.
Step 1: Every model call goes through one gateway
The gateway is the single choke point for model calls. The LiteLLM documentation, one example of the pattern, says to use the gateway if more than one application or person sends requests to LLMs. It issues virtual keys, each with its own models, budgets and rate limits, and it records the cost of each request by key, team, user and tag. It also does load balancing, fallbacks and retries when a provider has a problem. Provider credentials stay in the gateway, not in applications.
Input: a request carrying a virtual key. Output: a forwarded request, or a refusal. Now the failure mode. In that product, budget checks normally read a fast cache counter (Redis); the documentation warns that a stale or missing counter can under-report spend, and offers a setting, fail_closed_budget_enforcement, that validates every budgeted request against the database so the budget stays a hard ceiling. Name where your spend counter lives, and whether budgets fail open or closed when it is degraded.
Step 2: A serving layer runs the model
The serving layer is where the model actually runs, and there are two choices. Call a hosted provider through the gateway, or self-host with a serving engine such as vLLM, an open-source engine that lists continuous batching, chunked prefill, prefix caching and quantization among its features.
Self-hosting makes memory the central problem. The key-value (KV) cache is the per-request store of attention keys and values for every token already processed, which spares the model recomputing them. The PagedAttention paper (arXiv 2309.06180, Kwon et al., 2023) says this memory for each request is huge and grows and shrinks dynamically, and that waste from fragmentation and duplication limits the batch size.
PagedAttention is a way of storing the cache in fixed-size blocks that are looked up through a table, so memory is allocated as a request grows. The Inside vLLM blog describes a KV-cache manager that keeps a pool of free blocks, and the blocks map tokens to their computed cache. The paper reports 2-4× higher throughput than earlier systems at the same latency level. A serving engine handles batching and KV cache memory on GPUs.
Step 3: Data and tools are reached with the caller's permissions
Data access and tools are the layer where a model stops being a text generator and starts touching company systems. Retrieval, as in retrieval-augmented generation (RAG), fetches documents to put in the prompt, and tools let the model take actions such as sending a message or running a query.
The rule is that the model never gets more access than the person asking. Retrieval must filter documents by the caller's permissions before the prompt is built, because anything in the prompt can appear in the answer. The permission-aware RAG page shows how. Tools get least privilege: only the functions the task needs, only the rights the caller holds, and a human confirmation step for actions that cannot be undone. The agentic RAG page covers designs where the model chooses its own retrieval steps, which makes these limits matter more.
Step 4: Security checks cover inputs, outputs and actions
Security checks are controls placed on what goes into the model, what comes out and what the model may do. Prompt injection is a vulnerability in which user prompts alter the model's behaviour or output in unintended ways, according to the Open Worldwide Application Security Project (OWASP) Top 10 for LLM Applications 2025, and an OWASP list summary places it first.
The same list, among its ten entries, covers sensitive information disclosure, supply chain risk, improper output handling, excessive agency, vector and embedding weaknesses and unbounded consumption. Each maps to a control in this architecture: gateway authentication and budgets for unbounded consumption, output validation before anything downstream runs it, least-privilege tools for excessive agency, and permission-aware retrieval for disclosure. Self-hosting does not remove these risks: OWASP notes supply chains are susceptible to vulnerabilities affecting the integrity of training data, models and deployment platforms.
Step 5: Monitoring observes every layer
Monitoring is the layer that records what every request cost, how long it took, which model answered and whether it failed. The gateway already records cost per request, so spend by team is a query, not a project. A serving engine adds its own metrics; the vLLM documentation lists production metrics with Prometheus and Grafana among its deployment topics.
Quality needs separate measurement, because a request can succeed technically and still answer badly. Track a sample of real traffic against a fixed test set, and measure each stage on its own when retrieval is involved, as in evaluating a RAG system stage by stage. Alert on spend, error rate and latency per team, so one runaway job is visible before the invoice arrives.
Step-by-step description
- One choke point checks identity, budget and input.
- A provider or a serving engine runs the model.
- Access follows the caller's permissions.
- The answer is checked before it returns.
- Cost, latency and errors are recorded.
- The gateway is the single choke point for model calls.
- A serving engine handles batching and KV cache memory on GPUs.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Two calculations show the numbers that drive the design. All inputs are illustrative assumptions, not measurements of a real deployment.1
First, traffic. Assume an internal tool where each of 500 employees sends 30 requests a day, each with 2,000 input tokens and 500 output tokens. That is 15,000 requests and 37,500,000 tokens a day. If the busiest hour carries a tenth of the day, the peak is 25 requests per minute (RPM) and 62,500 tokens per minute (TPM), which is the figure to set rate limits against.1
Second, memory. Assume a 70-billion-parameter model with 80 layers, 8 KV heads, head dimension 128 and 16-bit numbers, on four 80 GB GPUs with a tenth held back. The cache costs 327,680 bytes per token, or 2.5 GiB for one 8,192-token sequence. The weights take 140 GB, leaving 148 GB for cache.1 Reserving full length for every request holds 55 sequences. Allocating blocks as sequences grow, with an average length of 1,500 tokens, holds 301. A four-GPU node holds 55 sequences with reserved memory and 301 with on-demand blocks.1 Allocating blocks as sequences grow holds about 5.5 times more sequences. That gap is why memory management, not raw compute, limits the batch size.
| Quantity | Value (illustrative inputs, computed) |
|---|---|
| Requests per day | 15,000 |
| Tokens per day | 37,500,000 |
| Peak requests per minute | 25 |
| Peak tokens per minute | 62,500 |
| KV cache per token | 327,680 bytes |
| KV cache, one 8,192-token sequence | 2.5 GiB |
| Cache budget on four 80 GB GPUs | 148 GB |
| Sequences held, full length reserved | 55 |
| Sequences held, blocks on demand | 301 |
Text description of the diagram
- A four-GPU node holds 55 sequences with reserved memory and 301 with on-demand blocks.
- Allocating blocks as sequences grow holds about 5.5 times more sequences.
Trade-offs and edge cases
The main trade is control against effort. A hosted provider API is the least work, but data leaves your network and you accept the provider's limits. Self-hosting gives control over data and cost shape, but you now run GPUs, upgrades and a serving engine, and you carry the supply chain risk of the models and software you pull in.
| Dimension | Hosted provider API | Self-hosted serving engine |
|---|---|---|
| Where data goes | to the provider | stays in your network |
| Who runs GPUs | the provider | you |
| Cost shape | per token | per GPU-hour |
| Scaling unit | provider rate limits | engine replicas |
| Main failure | provider outage or rate limit | GPU memory exhaustion or a bad upgrade |
Self-hosted serving scales by adding engine replicas behind a load balancer and autoscaling on queue depth or GPU memory use. As of October 2026, the LiteLLM gateway documentation routes Model Context Protocol (MCP) tools and agents through the same gateway with the same virtual keys, and the vLLM documentation lists disaggregated serving among its deployment topics.
A common confusion is treating the gateway and the serving engine as one thing. The gateway handles identity, budgets and routing across providers; the serving engine handles batching and memory on GPUs. They are separate layers, though the vLLM documentation lists LiteLLM among its integrations.
Text description of the diagram
- A gateway handles identity, budgets and routing.
- A serving engine handles batching and KV cache memory on GPUs.
Edge cases to name. Budget checks can fail open when the spend counter is stale or down, as in Step 1. Bigger batches do not always mean slower answers: the PagedAttention gain came from using memory better, so more requests fit at similar latency. A gateway adds a network hop and becomes a single point of failure, so run several copies and test the fallback path. For a bank-style regulated setting, see RAG in a regulated bank. For a consumer-scale design, see designing ChatGPT or Claude.ai.
How to answer it in an interview
How would you architect an enterprise LLM deployment?
I split it into layers. Applications call one gateway that holds the provider keys, gives each team a virtual key with budgets and rate limits, and records cost. Behind it sits the serving layer: a hosted provider API, or a serving engine such as vLLM on GPUs, where KV cache memory limits batch size. Retrieval and tools sit beside it with the caller's permissions enforced. Security checks cover prompt injection, output handling and agent permissions, and monitoring tracks cost, latency and errors. One failure I would name: budget checks that read a cache counter can under-count spend if the cache degrades, so I'd make enforcement fail closed.
The spoken structure: name the layers in request order, say what each one prevents, then give one failure mode and one number.
Common mistakes
- "The gateway is the inference server." (wrong) They are different layers. (right) "The gateway handles identity, budgets and routing; the serving engine handles batching and GPU memory."
- "Budgets protect us from overspend." (wrong) They depend on where spend is stored. (right) "Budget checks read a fast counter, so I check whether they fail closed when it is stale or down."
- "Prompt injection only matters for chatbots." (wrong) It chains with output handling and agent permissions. (right) "Any model that reads untrusted text and can act needs limited tools and validated outputs."
- "Self-hosting makes it secure." (wrong) Supply chain and platform risks remain. (right) "Self-hosting moves risk to us, so we vet models and software."
Follow-up questions
Is a gateway the same thing as an inference server?
No. A gateway authenticates callers, applies budgets and sends requests to a provider or deployment. An inference server runs the model on GPUs. Production systems usually have both. The LLM gateway interview questions go deeper.
Do gateway budgets always stop overspending?
Not necessarily. In the LiteLLM documentation, budget checks read a fast cache counter that can under-report spend when stale; a fail-closed setting checks the database on every budgeted request.
Is self-hosting a model more secure than using a provider API?
Not automatically. It keeps data inside your network, but OWASP lists supply chain vulnerabilities in models and deployment platforms, so you inherit the work of vetting what you run.
How do you size GPU memory for a self-hosted model?
Add the weights to the cache for the sequences you want to hold at once. Cache per token is two times layers times KV heads times head dimension times bytes per number, then multiply by sequence length and concurrency. Leave room for activations and engine overhead; the worked example holds back a tenth of GPU memory for that (illustrative).
Which security risks does the architecture have to cover?
Prompt injection, sensitive information disclosure, supply chain risk, improper output handling, excessive agency and unbounded consumption all have an owner in the layers: gateway, data access, tools and output checks.
- Enterprise LLM deployment architecture is a gateway path, a serving layer, permissioned data and tools, input and output checks, and monitoring across all of them.
- The gateway is the single choke point for identity, budgets, routing and cost records.
- A serving engine manages batching and KV cache memory on GPUs; it is not the gateway.
- Give the model no more data or tool access than the caller has.
- Name the controls that fail open, such as budgets read from a stale spend counter.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al., SOSP 2023), 12 Sep 2023
- vLLM documentation — vLLM project, 26 Oct 2026
- Inside vLLM: Anatomy of a High-Throughput LLM Inference System — vLLM blog, 5 Sep 2025
- LiteLLM AI Gateway documentation — LiteLLM (BerriAI), 26 Oct 2026
- LiteLLM documentation: users, keys and budgets — LiteLLM (BerriAI), 26 Oct 2026
- LiteLLM documentation home — LiteLLM (BerriAI), 26 Oct 2026
- OWASP Top 10 for LLM Applications 2025 (v2.0 PDF) — OWASP GenAI Security Project, 14 Nov 2024
- OWASP Top 10 for LLMs: summary of the 2025 list (secondary) — Promptfoo, 26 Oct 2026
