In this article
Before you start
This question sits inside the generative AI system design interview guide, which shows where a shared platform fits among the other designs.
Why interviewers ask this
The question separates candidates who treat the model as the whole system from candidates who see the platform around it. Interviewers use it to hear whether you put the security boundary in code you control, whether one customer's traffic can hurt another's, and whether you can show a customer that a change helped or hurt them.
A weak answer says "we tell the model in the system prompt to show only that customer's data." A strong answer filters documents by tenant and role before the model sees them, and gives each tenant its own limits. A Staff-level answer adds the quiet leaks (caches, logs, traces), prices the isolation options honestly, and ends with how to measure impact per tenant. Candidates report variants of this question in system design rounds for AI platform roles.
Multi-tenant LLM deployment from first principles
- Weighted fair share is a way of dividing limited capacity in which every tenant is served up to its demand or its weighted portion, and capacity a light tenant leaves unused is passed to the heavy ones.
In plain words
Think of an apartment building. Everyone shares the pipes and the roof. Each flat has its own lock, meter and mailbox. A multi-tenant LLM service works the same way. It has shared graphics processing unit (GPU) servers and models. Each tenant gets its own key, meter and mailbox of data. The analogy breaks in one place. A model has no walls, so anything in its prompt can come back out in its answer. The lock has to be fitted before the data enters the prompt.
In one precise sentence, multi-tenancy means every request carries a verified tenant identifier, and every shared resource either filters by it, meters by it or is keyed by it. The Azure Architecture Center guide on multitenant AI and machine learning discusses the same shared-versus-dedicated choices for model hosting.
Step 1: A gateway identifies the tenant and attaches its identity
The gateway is the single front door. It authenticates the caller, looks up the tenant and the user's role, and attaches a tenant identifier to every request. Input: a raw call with a credential. Output: the same call carrying a tenant identifier and role that the layers below trust and that the client cannot overwrite.
Per-tenant keys, budgets and rate limits are also set here, so a tenant is a first-class object rather than a string in a log. Nothing below the gateway should ever accept a tenant identifier from user-supplied text. You now know where identity is created; the next step is where it is enforced. The LLM gateway interview page covers the gateway itself in more depth.
Step 2: Enforce data access in retrieval, not in the prompt
Data access control belongs in the retrieval layer, because a model cannot be trusted to withhold text it has been shown. The retriever receives the tenant identifier and role from the gateway and filters documents before ranking, so a document the user may not read never reaches the prompt. Input: query plus tenant and role. Output: only permitted chunks.
A system prompt that says "only show tenant X's data" is a request, not a control; a crafted input can argue with it. The OWASP GenAI Security Project entry on sensitive information disclosure is the standard reference for this risk class. For role-based access, store the allowed tenants and roles with each chunk when it is indexed, and filter on them in the vector query. The permission-aware RAG page builds this step by step.
Step 3: Limit, cache and meter per tenant
Shared capacity needs per-tenant limits, or one noisy neighbor starves everyone. Providers do this too: the Anthropic rate-limits documentation describes limits applied at the organization level, and your gateway applies the same idea per tenant. Weighted fair share is the fair rule.
Take an illustrative platform with capacity of 100,000 tokens per minute and five tenants. Tenant D asks for 400,000 and the others together ask for 50,000. Without per-tenant limits, capacity is split in proportion to demand, so D takes 88,889 and every other tenant gets about 22% of what it asked for.1 With weighted fair share, tenants A, B, C and E are served in full and D gets the remaining 50,000, which is 12.5% of its demand.1
| Tenant | Demand (tokens/min) | No limits | Weighted fair share |
|---|---|---|---|
| A | 10,000 | 2,222 | 10,000 |
| B | 20,000 | 4,444 | 20,000 |
| C | 15,000 | 3,333 | 15,000 |
| D (noisy) | 400,000 | 88,889 | 50,000 |
| E | 5,000 | 1,111 | 5,000 |
Text description of the diagram
- Without per-tenant limits the noisy tenant takes 88,889 of 100,000 tokens per minute.
- With weighted fair share it gets 50,000 and every other tenant is fully served.
Without per-tenant limits the noisy tenant takes 88,889 of 100,000 tokens per minute. With weighted fair share it gets 50,000 and every other tenant is fully served. Caches need the same care: the vLLM prefix-caching documentation explains how requests with a shared prefix reuse stored attention state, so scope cache keys by tenant.
Step 4: Measure customer impact per tenant
Customer impact is measured by running evaluations and metrics sliced by tenant. Each tenant gets its own small evaluation set of real question types, with answers its own reviewers accept, and every release is scored on all of them before and after the change. Input: a candidate model, prompt or retriever. Output: a per-tenant pass or regression report.
Every log line and evaluation result carries the tenant identifier, so you can ask "did this release hurt tenant C?" rather than reading one global average that hides the harm. Roll out to a few tenants first and watch their metrics. The same stage-by-stage thinking applies as in evaluating a RAG system. Per-tenant cost metering uses the same identifier, which also makes billing possible.
Step-by-step description
- The gateway identifies the tenant.
- Documents are filtered by tenant and role.
- Limits and caches are scoped per tenant.
- Logs and evaluations carry the tenant.
- The gateway attaches a tenant identifier to every request.
- Retrieval filters documents by tenant and role before the model sees them.
- Every log line and evaluation result carries the tenant identifier.
The gateway attaches a tenant identifier to every request. Retrieval filters documents by tenant and role before the model sees them. Every log line and evaluation result carries the tenant identifier.
Which isolation tier does a tenant need?
Isolation is a spectrum you price, not a switch. The shared pool is the cheapest and suits most tenants. A per-tenant adapter gives custom behavior without a whole model copy. The method is low-rank adaptation (LoRA). A LoRA adapter is a small set of extra low-rank weights that adjusts a shared base model for one tenant, so many tenants can share one base model while each keeps its own style and terms.
A dedicated deployment isolates best and wastes the most idle capacity. Pick the tier from each tenant's contract and data class, and revisit it when either changes. A tenant that starts small can move up a tier later without redesigning the platform, as long as the tenant identifier is already present in every layer.
| Tier | Isolation | Cost | Use when |
|---|---|---|---|
| Shared pool | Logical, by filters and keys | Lowest | Most tenants |
| Per-tenant adapter | Own behavior on shared base | Medium | Tenant needs custom style or terms |
| Dedicated deployment | Own hardware and cache | Highest | Contract or regulation demands it |
For a regulated customer, see RAG in a regulated bank.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How do you architect a multi-tenant LLM deployment?
I would put one gateway in front of a shared model pool. It authenticates the tenant, attaches a tenant identifier to every request, and applies that tenant's rate limit and budget, using weighted fair share so one tenant cannot starve the rest. Data access is enforced in retrieval, filtering documents by tenant and role before the model sees them, never in the prompt. Caches, logs and traces are keyed by tenant. Tenants that need stronger isolation get a per-tenant adapter or a dedicated deployment, priced accordingly. To measure customer impact, I keep a per-tenant evaluation set and slice every release metric by tenant. The trade-off is cost: dedicated capacity isolates best and utilizes worst.
What makes it Staff-level is that it places each control in a layer you own, names the quiet leaks, and closes with measurement and cost instead of stopping at architecture boxes. For the surrounding product shape, see how to design ChatGPT or Claude.ai.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Prompt tells the model to hide other tenants' data | Retrieval filters by tenant and role | Also scopes caches, logs and traces by tenant |
| Depth | One shared model, no limits | Gateway with per-tenant keys and limits | Weighted fair share with burst and reserved capacity |
| Trade-offs | Never mentions cost | Compares shared and dedicated | Prices tiers and assigns them by contract and data class |
| Communication | Lists components | Follows one request through layers | States the boundary first, then what breaks if it moves |
| Follow-up readiness | Stalls on evaluation | Proposes a test set | Slices release metrics by tenant and rolls out in stages |
Three variants of this question
How do you architect a multi-tenant LLM deployment with role-based data access?
Store the allowed tenants and roles with every chunk at indexing time, take the user's role from the gateway, and filter on both in the retrieval query. The model only ever sees what that user may read.
Design a multi-tenant Claude deployment architecture and show how you measure customer impact via evaluation harnesses.
The architecture is the same as above with a hosted model behind the gateway. For impact, run a per-tenant evaluation set before and after each change, slice live metrics by tenant, and roll out to a few tenants first.
How do you stop one tenant's traffic from slowing every other tenant?
Apply per-tenant limits at the gateway with weighted fair share, so light tenants are served in full and the heavy tenant receives only the leftover capacity. Queue or reject the excess for that tenant alone.
Common mistakes
- "The system prompt says to show only this tenant's documents." (wrong) A prompt is not an access control. (right) "I filter by tenant and role in retrieval, before the prompt is built."
- "One global rate limit protects the platform." (wrong) A single heavy tenant can use all of it. (right) "I set limits per tenant with weighted fair share."
- "Shared inference means no cross-tenant risk." (wrong) Shared caches and shared logs can still mix tenants. (right) "I scope cache keys, logs and traces by tenant identifier."
- "Each tenant gets its own fine-tune." (wrong) That is usually the costliest option. (right) "Per-tenant retrieval first, adapters when behavior must differ, dedicated deployments only when required."
Follow-up questions
How do you stop prompt caching from leaking across tenants?
Include the tenant identifier in the cache key so a prefix from one tenant can never be reused for another. Shared prefix reuse is a performance feature, and its documentation should be read for how keys are formed.
Should each tenant get its own fine-tune?
Usually not. Per-tenant retrieval covers most knowledge needs, and an adapter covers behavior at far lower cost than a full copy. A dedicated model is for tenants whose contract or regulator requires it.
How do you attribute cost to each tenant?
Meter tokens in and out at the gateway under the tenant identifier, and add retrieval and storage costs from the same identifier. The RAG cost and speed page covers where those costs come from.
How do you handle a tenant asking you to delete its data?
Because every index entry, log line and evaluation set carries the tenant identifier, you can find and remove them all by that key. Rebuild or purge any cache that held the tenant's content.
- A multi-tenant LLM deployment shares one platform and keeps identity, data, quota and cost separate per tenant.
- The gateway creates the tenant identity; every lower layer filters, meters or keys by it.
- Enforce data access in retrieval, never in the prompt.
- Use weighted fair share so a noisy neighbor cannot starve other tenants.
- Measure impact with per-tenant evaluation sets and tenant-sliced metrics.
Sources
- Architectural approaches for AI and ML in multitenant solutions — Microsoft Azure Architecture Center, 26 Oct 2026
- Automatic Prefix Caching — vLLM documentation, 26 Oct 2026
- LLM02:2025 Sensitive Information Disclosure — OWASP GenAI Security Project, 26 Oct 2026
- Rate limits — Anthropic API documentation, 26 Oct 2026
