How do you serve a 70B model at p95 under 500 milliseconds?

In this article

1. Before you start
2. Why interviewers ask this
3. Serving a 70B model from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the large language model (LLM) inference optimization guide, and it assumes the transformer architecture at the level of "a model reads tokens and writes one token at a time".

Why interviewers ask this

Candidates report this question in different wordings in large language model (LLM) serving and infrastructure rounds, for example "serve a 70B-parameter model with high throughput, do the memory math" or "keep time to first token low for a real-time phone call". It separates people who recite optimization names from people who can do arithmetic on the back of an envelope.

The consequence of skipping the arithmetic is a design that cannot exist: a plan on one GPU that does not fit, or a latency promise the hardware cannot keep. A weak answer lists "quantize, batch, parallelize". A strong answer first asks which latency the target measures, then computes memory, then computes the per-token delay, then shows what load keeps the tail low. A Staff-level answer also says where each number is an assumption and how it would be measured.

Serving a 70B model from first principles

p95 latency

The p95 latency is the response time that 95 of every 100 requests beat, so the slowest 5 percent are allowed to be worse; it describes the tail of the distribution, which an average hides.

Tensor parallelism

Tensor parallelism (TP) is a way of splitting a model that divides each layer's weight matrices across several GPUs, so that every GPU holds a slice of the model and computes its part of each layer, then the GPUs exchange partial results.

In plain words

Picture a restaurant kitchen. The recipe book (the weights) is huge and must be read for every single dish, and the cooks (the GPU's compute units) are fast but wait on the page-turning. That page-turning is memory bandwidth. Serving many diners at once lets one reading of the book serve many dishes. The analogy breaks at the pantry: each diner also needs a private tray of prepared ingredients, the KV cache, and the pantry is finite.

Precisely: prefill is the first pass in which the model reads the whole prompt in one go, and it is limited by arithmetic speed. Decode is the loop that writes one token per pass, and it is limited by how fast the weights stream out of memory.

Step 1: Weights plus KV cache decide the GPU count

A 70B model has about 70 billion weights, and at 2 bytes each in 16-bit precision they take 140 GB.² At 1 byte (8-bit) they take 70 GB, and at half a byte (4-bit) 35 GB.² An 80 GB GPU cannot hold the 16-bit weights. The 80 GB per GPU is the H100 SXM figure in the specification table on the NVIDIA H100 page (read 2026-10-26), and this page treats 90% of it as usable, an illustrative share.

The KV cache adds a per-request cost. For a Llama-3-70B-shaped model with 80 layers, 8 key-value heads and 128 numbers per head (Table 3 of the Llama 3 paper, 2024-07-31, which uses grouped-query attention), each token needs 2 × 80 × 8 × 128 × 2 bytes, which is 327,680 bytes, and a 4,096-token sequence needs 1.25 GiB.² The KV cache page derives the formula.

What you now know: memory, not compute, fixes the minimum GPU count.

Step 2: Memory bandwidth sets the decode floor

Each decode step must read every weight once, so the step cannot be faster than weight bytes divided by the memory bandwidth of all GPUs together. With the H100 SXM bandwidth of 3.35 terabytes per second per GPU (NVIDIA H100 page, read 2026-10-26), 16-bit weights on 4 GPUs give a floor of 10.45 milliseconds per token, and 8-bit weights on 4 GPUs give 5.22 milliseconds.²

This is a floor: it ignores the KV-cache reads, the communication between GPUs and software overhead. Real serving rarely reaches it, so this page assumes (illustrative) that decode achieves 60% of the bound, which makes the realistic inter-token delay 17.41 milliseconds on 4 GPUs at 16-bit.² (The worked example's 17.58 milliseconds at batch 1 is 0.17 milliseconds higher because it also counts one sequence's KV-cache reads.²) The high bandwidth memory page and the GPU fundamentals guide explain why bandwidth, not arithmetic, dominates here.

Spreading the model over more GPUs with tensor parallelism raises the combined bandwidth, but the GPUs must exchange partial results in every layer, so the gain is less than proportional.

Step 3: Spend the budget, first token plus tokens times delay

A latency target is a budget, and the budget is spent in a fixed order: queue wait, then the prefill pass, which emits the first token, then one inter-token delay per further token. Prefill costs about 2 × 70 billion floating-point operations per prompt token (the extra attention arithmetic is ignored). With an illustrative 1 petaflop per second per GPU at 40% utilization on 4 GPUs, a 1,000-token prompt takes 87.5 milliseconds.² The 1 petaflop figure is deliberately round and below the 1,979 teraflops 16-bit figure on the NVIDIA H100 page, which is quoted with sparsity.

Add an illustrative 50 milliseconds of queue wait. Time to first token (TTFT) is then 137.5 milliseconds, well inside half a second.² The remaining 362.5 milliseconds at 17.41 milliseconds per token buys 20 more tokens.² A 200-token reply on the same system takes about 3,600 milliseconds: the first token at 137.5 plus 199 further tokens.²

So "p95 under 500 milliseconds" is realistic as a TTFT target, or as a full-reply target for very short outputs. For streaming chat, state the target on TTFT and on inter-token latency separately.

Step 4: Batching raises throughput, and queueing shapes the p95

Serving one request at a time wastes the weight read. Batching shares it: with continuous batching, a scheduler adds and removes requests at every decode step, an idea introduced by Yu et al. (2022). Each added sequence adds its own KV-cache reads, so each step grows a little longer while throughput climbs.

The tail is a different problem. Take an illustrative single-server queue (M/M/1) with random arrivals and service times that vary randomly (exponentially distributed) around a 100-millisecond average. At 50% utilization the mean response is 200 milliseconds and the p95 is 599.1; at 80% the p95 is 1,497.9; at 90% it is 2,995.7.² In this model the p95 of the response time is about three times its mean (the mean times the natural log of 20). Load, not model speed, often decides the p95.

1 · Memory fit2 · Decode floor3 · Budget4 · LoadReplay

Step-by-step description

1. Weights plus cache set the GPU count.
2. Bandwidth sets the delay per token.
3. The budget counts the tokens that fit.
4. Batching and queueing set the p95.

• Weights plus KV cache set the GPU count.
• Memory bandwidth sets the decode floor per token.
• A latency budget counts the tokens that fit.
• Batching and queueing set the p95.

Worked example

Take the setup above: 16-bit weights, 4,096-token sequences, 80 GB GPUs with 90% usable memory, and the KV cache costing 327,680 bytes per token.² One 4,096-token sequence therefore needs 1.34 GB of cache (the 1.25 GiB of Step 1), and the headroom column below is in decimal GB, so dividing headroom by 1.34 gives the sequence count. The table shows which GPU counts fit.

So the answer to "how many GPUs to serve a 70B model" is two at the bare minimum for 16-bit weights, and four or more for real concurrency; 8-bit weights make two GPUs useful. Now add batching on 4 GPUs at 16-bit, with the same 60% efficiency as Step 2 and the KV-cache reads of every sequence included: one decode step takes 17.58 milliseconds at batch 1, 22.75 at batch 32 and 35.78 at the memory limit of 110 sequences, so throughput rises from 57 to 3,075 tokens per second.² At the bandwidth floor alone, the same steps would take 10.55 and 21.47 milliseconds.²

Text description of the diagram

• One decode step takes 17.58 milliseconds at batch 1.
• One decode step takes 35.78 milliseconds at batch 110.
• Throughput rises from 57 to 3,075 tokens per second.

The twofold slowdown against a fifty-fold throughput gain is why batching is the first lever, and also why a full batch leaves no latency slack: at 35.78 milliseconds per step the 362.5-millisecond budget buys about 10 more tokens, against 20 at batch 1.²

Trade-offs and edge cases

Every lever moves one number and costs another. Lower precision halves weight memory and the decode floor, but quality must be checked on your own evaluation set before shipping. More GPUs add bandwidth and KV room, but communication means less-than-proportional speed-up and a higher bill. Bigger batches raise throughput and lengthen each step. Shorter outputs shrink total time at no cost to quality, if the product allows them.

Two further levers target the two phases. Speculative decoding lets a small draft model propose several tokens that the large model checks in one pass, which cuts the inter-token delay when decode is bandwidth-bound. Prefix caching, described in caching LLM calls, reuses the stored cache of a fixed system prompt so prefill skips it, and chunked prefill splits a long prompt into pieces so it does not stall other requests' decode steps.

Text description of the diagram

• Lower precision halves weight memory but needs a quality check.
• More GPUs add bandwidth but communication limits the gain.
• A bigger batch raises throughput but lengthens each step.
• Shorter output cuts total time.

Edge cases that break the textbook answer: a long prompt makes prefill the dominant cost, since it grows with prompt length (350 milliseconds at 4,000 tokens on the illustrative setup),² so TTFT depends on input size. Long contexts shrink the number of sequences that fit, because the cache grows with every token. KV-cache fragmentation wastes the headroom you computed; paged attention, introduced by Kwon et al. (2023), stores the cache in blocks to reclaim it. Finally, a p95 target needs headroom in load: size for the percentile at your peak traffic, not the average.

How to answer it in an interview

The spoken structure: define the metric, do the memory math, do the per-token math, then name batching, queueing and quantization as the levers, each with its cost.

Common mistakes

• "A 70B model fits on one 80 GB GPU." (wrong) The 16-bit weights alone are 140 GB. (right) "In 16-bit I need at least two GPUs, and one only with 8-bit weights and almost no cache room."
• "Weights are the only memory cost." (wrong) The KV cache grows with batch and context. (right) "I budget weights plus cache per sequence, and that caps concurrency."
• "The target is half a second, so I will just batch more." (wrong) Batching lengthens each step and, near full load, queueing inflates the tail. (right) "I cap utilization and measure the p95 under load."
• "Average latency meets the service-level objective (SLO)." (wrong) A service-level objective (SLO) on p95 is about the slowest 5 percent. (right) "I report the percentile, and say which latency it covers."

Follow-up questions

How many GPUs to serve a 70B model?

At least two 80 GB GPUs for 16-bit weights, and four or more when you want a useful number of concurrent long sequences; with 8-bit weights, one GPU holds the weights but leaves almost no cache room.²

How do you serve a model at 10k QPS with sub-100-millisecond latency?

Divide the queries per second (QPS) by the throughput of one replica to get the replica count, keep each replica well below full utilization so the tail stays flat, and check that one decode step plus prefill fit the budget at all. For a 70B model they usually do not, which pushes you to a smaller or quantized model.

How do you optimize LLM inference for higher throughput and lower latency?

Throughput comes from continuous batching, paged cache management and lower-precision weights; latency comes from fewer bytes per token, more bandwidth and lower load. Speculative decoding also lowers the delay between tokens when decode is bandwidth-bound. The two goals pull against each other, so state which one the target protects.

How do you keep time to first token under 200 milliseconds for a phone-call app?

Budget the prompt length first: prefill grows with it, and on the illustrative setup a 1,000-token prompt already costs 87.5 milliseconds.² Keep the queue near empty and shorten the prompt. Reuse the stored cache of the fixed system prompt (prefix caching, see caching LLM calls) so only the new part is prefilled, split long prompts with chunked prefill, and consider 8-bit weights or more GPUs.

Does tensor parallelism scale linearly with GPU count?

No. The GPUs exchange partial results in every layer, so doubling the GPUs less than doubles the speed; measure it on your interconnect rather than assume the ideal curve.

Sources

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al., SOSP 2023), 12 Sep 2023
2. Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI 2022 (Yu et al.), 11 Jul 2022
3. The Llama 3 Herd of Models (arXiv 2407.21783) — arXiv (Llama Team, Meta), 31 Jul 2024
4. NVIDIA H100 Tensor Core GPU — NVIDIA (page read 2026-10-26), 26 Oct 2026