Multi-tenant model serving: how to design it

In this article

1. Before you start
2. Why interviewers ask this
3. Multi-tenant model serving from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the large language model (LLM) inference optimization guide and builds on the transformer architecture.

Why interviewers ask this

The question reaches past a single model on a single GPU. It asks whether you can share expensive hardware between customers who each expect their own latency, written into a service-level agreement (SLA), their own data privacy and their own bill. Design prompts of this shape, such as a shared inference service with latency commitments for one customer, are reported by candidates at companies that run inference platforms.

A weak answer says "put a load balancer in front of the GPUs." A strong answer names the scarce resource, which is GPU memory for the KV cache, turns it into a number of concurrent sequences, and says what happens to a tenant that exceeds its share. A Staff-level answer separates tenants by contract: which ones share a batch, which get a hardware slice, which get a dedicated replica, and what each choice costs per query.

Multi-tenant model serving from first principles

Multi-tenant model serving

Multi-tenant model serving is a serving design in which many customers share the same GPUs, and often the same model weights, while the system enforces a separate latency target, memory share, rate limit and data boundary for each customer.

Admission control

Admission control is a gate in front of the GPUs that decides, for each incoming request, whether to run it now, hold it in a queue or reject it, so that accepted work still meets its latency target.

LoRA adapter

A Low-Rank Adaptation (LoRA) adapter is a small set of extra weights, trained on top of a frozen base model, that changes the model's behavior for one task or customer without copying the base weights.

In plain words

Think of an office building with one kitchen. Every team shares the stove. The building decides how many cooks use it at once, gives the catering team priority at lunch, and locks the cabinets of the legal team. The analogy breaks in one place. Cooks on a stove do not slow each other's pans, whereas requests in one GPU batch do slow each other's tokens.

In precise terms, multi-tenant serving turns shared GPU memory and compute into per-tenant promises. It does so by budgeting memory, gating requests and choosing how much to isolate. The limit matters because the shared batch is where a noisy neighbor can slow everyone else.

Step 1: Tenants share one set of base weights

The base model's weights are the largest fixed cost, so serving a separate copy per tenant wastes memory. Tenants that differ only by fine-tuning can share the base weights and each own a LoRA adapter. S-LoRA (Sheng et al., 2023) stores all adapters in main memory and fetches the adapters used by the currently running queries into GPU memory. It uses a unified memory pool, called Unified Paging, for both adapter weights and KV-cache tensors, and custom kernels to batch requests that use different adapters.

Input: a request tagged with a tenant id. Output: the same base model run with that tenant's adapter. You now know why the weights are not the per-tenant cost; the next step shows what is.

Step 2: The KV cache sets a memory budget per tenant

The KV cache grows with every token of every active sequence, so GPU memory left after the weights decides how many sequences run at once. The size per token is two (keys and values) times layers times KV heads times head dimension times bytes per number. Paged attention stores the cache in fixed-size blocks to cut fragmentation, as introduced by Kwon et al. (2023).

Divide the free memory by the size of one sequence and you get a slot count. Then split the slots by tier: a premium tenant gets reserved slots, others share the remainder. A tenant that needs a slot when none is free waits or is rejected; it never takes memory from another tenant's reservation. The worked example below computes real slot counts.

Step 3: Admission control and priority protect the latency target

Sharing a batch shares compute. With continuous batching, every token step runs all active sequences together, so each added sequence makes every other sequence's next token slightly slower. A premium tenant with a target of 50 milliseconds per token (an illustrative figure) can miss it purely because other tenants filled the batch, even though its own traffic never changed. That is the failure from the lead.

Two latency targets are involved. Time to first token (TTFT) is the wait from sending a request until its first output token arrives, dominated by queueing and prefill, the pass that reads the whole prompt. Inter-token latency, also called time per output token (TPOT), is the gap between later tokens, set by batch size and decode. A new long prompt's prefill can stall the decode steps of every co-batched tenant.

Admission control prevents both. It caps the sequences in the batch and the prefill tokens per step at the level where the premium targets still hold, keeps a queue per tier, and when load exceeds the cap it rejects or delays the lowest tier first. A per-tenant token bucket, which refills at a fixed rate and lets short bursts through, stops one tenant from consuming the whole queue. Priority scheduling orders the queue so premium work enters the batch first. Chunked prefill splits a long prompt across several steps (Agrawal et al., 2024), and recent systems go further with separate prefill and decode pools (Zhong et al., 2024).

Step 4: Choose an isolation level per contract

Isolation means one tenant cannot hurt or read another. There is a ladder: a shared batch with priority, an adapter per tenant, a hardware slice, a dedicated replica and a dedicated GPU or node. NVIDIA's Multi-Instance GPU (MIG) feature splits a supported GPU into separate instances, each with its own memory and compute; the MIG user guide lists the supported GPUs and configurations, so check your hardware against it. A 70B model does not fit a sub-GPU MIG slice, so MIG suits small models.

Data isolation is a separate axis from speed. A shared prefix cache can reveal that another tenant sent the same prompt start, because a cache hit is faster, a timing difference that audits of prompt caching in model application programming interfaces (APIs) have measured (Gu et al., 2025), so sensitive tenants get a cache keyed by tenant or no sharing; see caching LLM calls. Isolation level is chosen per contract, not per company. Input: a tenant's contract. Output: the cheapest level that meets it.

Step 5: Route, place and scale on the right signals

A router maps each request to a replica pool by tenant tier and model. A GPU-aware scheduler places models on GPUs by memory: a model that needs 70 GB (illustrative) cannot land on a GPU with less free space. When a fleet serves many open-source models, rarely used ones are loaded on demand and busy ones keep warm replicas.

Scale on queue depth and KV-cache use, not on average GPU utilization, because a GPU can look busy while its premium queue is already late. Cost per query falls when batches are fuller, so route cheap requests to smaller models where quality allows; see model routing and cascades. You now have the whole system: shared weights, memory slots, a gate, an isolation level and a router.

1 · Shared weights2 · Memory slots3 · Admission4 · Isolation5 · Route and scaleReplay

Step-by-step description

1. Tenants share base weights.
2. The KV cache sets slots per tier.
3. A gate protects the latency target.
4. Pick a level per contract.
5. Scale on queue depth.

• Tenants share one set of base weights and differ by adapter.
• The KV cache sets a number of memory slots per tier.
• Admission control protects the latency target of the premium tier.
• Isolation level is chosen per contract.

Worked example

Take a service on two GPUs with 80 GB of HBM each, a 70-billion-parameter model stored at one byte per parameter (8-bit floating point, 8-bit floating point (FP8)), and a 10% memory reserve for activations. All of these inputs are illustrative. The weights take 70 GB, the reserve 16 GB, leaving 74 GB for the KV cache.¹

Use an illustrative configuration of 80 layers, 8 KV heads, a head dimension of 128 and two bytes per cache number. One token then needs 2 × 80 × 8 × 128 × 2 = 327,680 bytes. A 4,096-token sequence needs 1.342 GB, so the 74 GB budget holds 55 concurrent sequences.¹ Split 40%, 30% and 30% across tenants A, B and C and the reserved slots are 22, 16 and 16, with one slot spare.¹

Adapters change the picture. With a hidden size of 8,192, rank 16 and four adapted projections per layer (a simplified, illustrative setup), one adapter holds 83,886,080 parameters, or 167.8 MB at two bytes each; 1,000 adapters take 167.8 GB, against 140,000 GB for 1,000 full copies of the 70B model at two bytes per parameter (140 GB each, double the one-byte storage used for the KV budget above).¹

Cost follows utilization. If a shared, well-batched service produces 1,800 tokens per second and a dedicated, mostly idle deployment of the same two GPUs produces 450 (both illustrative), the shared one spends 1.111 GPU-seconds per 1,000 tokens and the dedicated one 4.444, four times more.¹

Latency follows headroom. Model one replica as an M/M/1 single-server queue with a service rate of 20 requests per second (illustrative), a simplification of a batched LLM replica; the p95 response time (queueing plus service) is then ln 20, about 3, divided by the spare rate μ − λ. It is 300 milliseconds at 50% utilization, then 0.749, 1.498 and 2.996 seconds at 80%, 90% and 95%.¹ Going from 80% to 95% utilization quadruples the p95, and going from 50% to 95% multiplies it by ten, which is why a premium tier is not run full.¹

Text description of the diagram

• The illustrative p95 response time is 300 milliseconds at 50% utilization.
• The illustrative p95 response time is 2.996 seconds at 95% utilization.

Trade-offs and edge cases

Isolation trades cost for safety. A shared batch with priority is the cheapest level and isolates the least. A MIG slice splits memory and compute in hardware, and a dedicated GPU or node isolates the most and costs the most. Pick per contract, not per company: one customer can hold a premium replica for production and a shared pool for experiments.

Text description of the diagram

• A shared batch with priority is the cheapest level and isolates the least.
• A MIG slice splits memory and compute in hardware.
• A 70B model does not fit a sub-GPU MIG slice, so MIG suits small models.
• A dedicated GPU or node isolates the most and costs the most.

Edge cases decide real designs. First, a mixture of experts (MoE) model, which routes each token to a few expert sub-networks, still needs every expert resident in memory, so memory follows total parameters, not active ones. In an illustrative 70B model with 20B active, the FP8 weights still occupy 70 GB, not 20 GB.¹

Second, when the KV cache does not fit, the system can offload parts to host memory at the cost of transfer time, or shrink it with quantization. Third, long contexts eat slots fast: doubling the context halves the sequences that fit. Fourth, a replica failure mid-generation either restarts the request on another replica or resumes from saved state; restarting is simpler, resuming is cheaper for long outputs.

Read the GPU fundamentals guide for why memory bandwidth, not arithmetic, usually bounds decoding.

How to answer it in an interview

The spoken structure: name the scarce resource, convert it to slots, gate admission, then choose isolation per contract and say what it costs.

Common mistakes

• "Each customer gets its own model copy." (wrong) That multiplies the biggest memory cost. (right) "Tenants share base weights and differ by adapter; only contracts that demand it get a dedicated replica."
• "Continuous batching isolates tenants." (wrong) A shared batch shares compute, so one tenant's load slows another's tokens. (right) "Batching raises efficiency; admission control and priority protect the latency target."
• "Scale when GPU utilization is high." (wrong) A premium queue can be late while average utilization looks fine. (right) "Scale on queue depth, KV-cache use and tail latency."
• "A MoE model needs memory for its active parameters only." (wrong) Every expert must be resident. (right) "Memory follows total parameters; compute follows active ones."

Follow-up questions

How do you design inference serving for a 70B MoE model with p95 latency targets?

Size memory by total parameters, split experts across GPUs, and budget KV-cache slots from what is left. Then hold utilization below the point where the tail rises, as in the worked example. For a target of 10,000 requests per second, divide by what one replica carries: at 70% of an illustrative 20 requests per second that is 14, so about 715 replicas.¹ Check the budget against the 70B at p95 page; it walks through a latency budget.

How do you cover GPU scheduling, cost per query and tenant isolation in one design?

Scheduling is the router plus admission control, cost per query is batch fullness and model choice, and isolation is the ladder from shared batch to dedicated GPU. State which tier uses which level and why.

How do you serve 100+ open-source models with a GPU-aware scheduler?

Place models by free GPU memory, keep warm replicas for busy models, load rare ones on demand and share a base model with adapters where models are variants of one family. Track cold-start time as a separate latency class.

What if the KV cache does not fit in GPU memory?

Offload cold parts to host memory and overlap the transfer with compute, quantize the cache, or cap context length per tier. Each option trades latency or quality for capacity, so say which tier pays.

What happens when a GPU server fails mid-generation?

Restart the request on another replica and accept the repeated work, or save intermediate state and resume. Restart is simpler; resume pays off for long outputs; redundant parallel runs cost double and suit only the tightest tier.

Sources

1. S-LoRA: Serving Thousands of Concurrent LoRA Adapters (arXiv 2311.03285v3) — arXiv (Sheng et al.), 6 Nov 2023
2. NVIDIA Multi-Instance GPU User Guide — NVIDIA, 13 Oct 2026
3. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al.), 12 Sep 2023
4. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv 2403.02310) — arXiv (Agrawal et al.), 4 Mar 2024
5. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (arXiv 2401.09670) — arXiv (Zhong et al.), 18 Jan 2024
6. Auditing Prompt Caching in Language Model APIs (arXiv 2502.07776) — arXiv (Gu et al.), 11 Feb 2025