What is a GPU cluster model serving SLA, and how do you meet it?

In this article

1. Before you start
2. Why interviewers ask this
3. GPU cluster model serving SLA from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the large language model (LLM) inference learning path, and it builds on how a transformer generates text one token at a time.

Why interviewers ask this

A serving design that ignores the latency promise fails in production even when its benchmark chart looks excellent. Serving-design questions use this topic to see whether you separate what you promise a customer from what you measure inside the system, and whether you size a cluster by the requests that were fast enough rather than by raw volume.

A weak answer says "we keep latency low and add GPUs when it is slow." A strong answer names the measured indicators (time to first token and time per output token), states the target as a percentile, and explains why the contract is looser than the internal target. A Staff-level answer adds the levers: bounded batching delay, admission control at the queue, and splitting prefill from decode when the two interfere. It also says what each lever costs.

GPU cluster model serving SLA from first principles

Service level objective (SLO)

A service level objective (SLO) is an internal target for a measured quantity of a service, such as "99% of requests get their first token within a set time", that a team tries to meet and uses to decide when to add capacity or stop shipping changes.

Goodput

Goodput is the number of completed requests per second that meet the service level objectives, so a request that finishes too slowly counts for nothing, unlike in plain throughput. It is measured as the highest request rate at which a chosen share of requests, such as 90%, still meets them.

In plain words

Picture a courier company. It promises customers delivery within a set time and refunds part of the fee if it fails. Inside, its managers aim for a tighter time so that bad days do not breach the promise. They also track how many parcels arrived on time per van, not just how many parcels the vans carried.

A model-serving cluster works the same way. The customer contract is the SLA. The internal target is the SLO. The things measured are indicators.

The analogy breaks in one place. A parcel arrives once. A model reply streams, so two clocks run: how long until the first words appear, and how fast the rest follow.

In one precise sentence, a serving SLA is a contract on a measured indicator, and the cluster is sized to meet a tighter internal SLO on the indicators that users feel.

Step 1: An SLA, an SLO and an SLI are three different things

A service level indicator (SLI) is, in the Google site reliability engineering (SRE) book, chapter 4 (read October 2026), "a carefully defined quantitative measure of some aspect of the level of service that is provided." Request latency, error rate and throughput are the common ones. An SLO is a target for an SLI. An SLA is a contract that attaches consequences to a target.

Google Cloud's SRE fundamentals post (read October 2026) says an SLA normally promises an availability level over a period, with a penalty if it fails, and that the SLA target is normally looser than the internal one. Its example is 99.9% in the SLA against 99.95% internally. An SLA is looser than the internal SLO, which leaves headroom.

Vendor contracts show the SLA layer. Amazon SageMaker AI's SLA (read October 2026) measures availability per 5-minute interval as the share of requests that do not fail with errors, averages it over a billing month, and pays service credits. You now know the three layers; the next step names the indicators a language model adds.

Step 2: Time to first token and time per output token are the two latency indicators

Time to first token and time per output token are the two streaming latency measures a language model service reports. Time to first token (TTFT) is what the user waits through before anything appears. Time per output token (TPOT) is how fast the text then flows, and the vLLM metrics documentation (read October 2026) defines it as end-to-end latency minus TTFT, divided by the number of output tokens minus one.

Inter-token latency (ITL) is the time between consecutive streamed outputs. ITL describes each gap, and TPOT describes the request as a whole, so they differ even though they are close when each streamed piece holds one token.

Each indicator is aggregated over a measurement window into a rate, average or percentile, as the SRE book notes. The vLLM benchmark command-line page (read October 2026) shows a sample report with the mean, the median and P99 for TTFT, TPOT and ITL. An SLO therefore reads like "99% of requests have TTFT under a set bound." A mean alone cannot express that, because it hides the slow tail.

Step 3: Goodput counts only requests that meet the SLO

Throughput can rise while users get a worse service. The Hao AI Lab post on goodput (read October 2026) argues that throughput, the number of requests or tokens completed across all users, overlooks latency requirements. It proposes goodput, the completed requests per second that adhere to the SLOs, as a measure that accounts for both cost and user satisfaction.

The DistServe paper (Zhong et al., 2024) makes it per GPU: the maximum request rate that can be served while meeting the SLO attainment goal, for each GPU provisioned. In practice goodput is measured as the highest request rate at which a chosen share of requests, for example 90%, meets both the TTFT and TPOT targets. Divide a target load by goodput per GPU and round up to get a GPU count; the worked example does that.

The Hao AI Lab example (rates illustrative here) uses one prefill worker at about 5.6 requests per second and one decode worker at about 10. With two prefill workers and one decode worker, goodput is the smaller of 11.2 and 10, which is 10 requests per second over 3 GPUs, about 3.3 per GPU. One prefill and one decode worker give 2.8 per GPU, so the second prefill worker lifts the figure. The slower phase caps the system. The same post reports that this simple disaggregation yields 2x goodput, which implies a colocated baseline near 1.7 per GPU.¹ The worked example below adds the other layouts.

Step 4: Bounded queue delay trades latency for throughput

Bounded queue delay trades latency for throughput. A GPU does more useful work per second when it runs a larger batch, but each request waits for the batch to form. The NVIDIA Triton dynamic batcher guide (read October 2026) says by default it forms batches as large as possible up to the maximum batch size and does not delay.

If latency is within budget, the Triton guide advises raising the maximum batch size or setting a non-zero batch delay to trade increased latency for increased throughput. The delay lets requests wait briefly for others to join. Batching is therefore not free throughput: it spends the latency budget, and the SLO says how much you may spend.

The Triton model configuration guide's queue policy section (read October 2026) adds a second lever: a maximum queue size and a timeout action can reject or defer requests that wait too long. That is admission control. Rejecting overload visibly protects the rest of the traffic, while silent queueing quietly breaks the latency SLO for everyone. For language models, continuous batching applies the same trade per generation step.

Step 5: Prefill and decode on separate GPUs removes interference

Colocated phases slow each other down, and giving each phase its own GPUs stops that. Prefill is the first pass in which the model reads the whole prompt and produces the first token; decoding then produces the rest one token at a time. The two phases stress a GPU differently.

The DistServe paper observes that existing systems colocate the two phases and batch them across all users. Under stringent latency requirements they must then prioritize one latency over the other or over-provision compute to meet both. DistServe assigns prefill and decoding to different GPUs, which eliminates prefill-decoding interference, and tunes each phase separately to maximize per-GPU goodput.

The cost is that the KV cache of each request must move from the prefill GPU to the decode GPU. How much that transfer costs depends on the bandwidth between the GPUs, so placement matters. The KV cache page shows how large that cache gets.

1 · Contract2 · Indicators3 · Goodput4 · Batch and queue5 · Split phasesReplay

Step-by-step description

1. The SLA is looser than the internal SLO.
2. TTFT and TPOT measure streaming latency.
3. Only requests meeting the SLO count.
4. Bounded delay protects the budget.
5. Separate GPUs remove interference.

• An SLA is looser than the internal SLO.
• Time to first token and time per output token are the two streaming latency measures.
• Goodput counts only requests that meet the SLO.
• Bounded queue delay trades latency for throughput.
• Prefill and decode on separate GPUs removes interference.

Worked example

The numbers below are illustrative inputs run through a short script, except the two phase rates, which are the Hao AI Lab example, and the credit tiers, which are quoted from the SageMaker AI SLA as read in October 2026 (check the current table before relying on it).

A 99.9% target over a 30-day month leaves 0.1% of 43,200 minutes. A 99.9% target allows 43.2 minutes of downtime in a 30-day month. A 99.95% target allows 21.6 minutes, and a 99% target allows 432.¹

Text description of the diagram

• A 99.9% target allows 43.2 minutes of downtime in a 30-day month.
• A 99.95% target allows 21.6 minutes.

Now the tail. Take 1,000 illustrative requests: 980 with a TTFT of 200 milliseconds and 20 with 5,000 milliseconds. The mean is 296 milliseconds, which looks fine against a 500-millisecond target. Yet 2% of requests exceed that target, so a "99% under 500 milliseconds" SLO fails, and the P99 is 5,000 milliseconds.¹ A P99 target on 10,000 requests lets at most 100 exceed the bound.¹

A TPOT example: a request with a TTFT of 400 milliseconds, an end-to-end latency of 6,400 milliseconds and 101 output tokens has a TPOT of (6,400 minus 400) divided by 100, which is 60 milliseconds. Against an illustrative 50-millisecond TPOT target it is a miss, even though its TTFT passes.¹

For goodput, use 5.6 requests per second per prefill GPU and 10 per decode GPU. One prefill and one decode GPU give 5.6 over 2 GPUs, 2.8 per GPU. Two prefill and one decode give 3.33 per GPU. One prefill and two decode give 1.87, and three prefill and one decode give 2.5.¹ The best layout balances the phases.

Sizing follows by division. For an illustrative target load of 100 requests per second, the two-prefill-one-decode layout serves 10 per group of 3 GPUs, so 10 groups and 30 GPUs. A one-prefill-one-decode pair serves 5.6, so 18 pairs and 36 GPUs. With 20% headroom (illustrative) the target becomes 120 requests per second, and the two-prefill-one-decode layout needs 12 groups, 36 GPUs. GPUs are bought in whole groups, so state the layout granularity.¹

Credits follow the quoted tiers. Say 13 of the 8,640 five-minute intervals in the month were fully down. Monthly uptime is then about 99.85%, below 99.95% and at least 99%, which is the 10% tier: 100 credit units on an illustrative bill of 1,000.¹

Trade-offs and edge cases

Every lever buys one indicator with another. Bigger batches raise throughput and lengthen waits. Rejecting overload keeps admitted requests fast and fails others on purpose. Splitting phases raises goodput and adds transfer and placement work.

Text description of the diagram

• Colocated serving shares GPUs between prefill and decode.
• Disaggregation adds KV-cache transfer between GPUs.

Colocated serving shares GPUs between prefill and decode, so a long prompt can slow the decode steps of other users. Disaggregation adds KV-cache transfer between GPUs, and it does not always win. A small, steady, low-load service may lose more to extra GPUs and transfer than it gains.

Edge cases: contracts differ from engineering targets. The SageMaker terms quoted above define availability, not latency, so that SLA defines availability only; read the contract for any latency terms. A percentile needs enough samples; a P99 from a handful of requests is noise. Shared clusters add noisy neighbors, which multi-tenant model serving addresses. Work the cache can skip, as in caching LLM calls, frees GPU time. The 70B at p95 walkthrough applies these levers to one design, and GPU memory explains why batch size is bounded.

How to answer it in an interview

The spoken structure: contract first, then the internal targets and their indicators, then goodput for sizing, then the levers and what each costs.

Common mistakes

• "The SLA and the SLO are the same number." (wrong) The SLA carries consequences and is normally looser than the internal target. (right) "I keep an internal SLO tighter than the SLA so there is headroom."
• "Our average latency is under the target, so we are fine." (wrong) A mean hides the tail. (right) "I state the target as a percentile and check P99."
• "Higher throughput means a better serving system." (wrong) Throughput ignores latency requirements. (right) "I compare layouts by goodput per GPU."
• "Turn batching up to get more throughput." (wrong) Batching delay spends the latency budget. (right) "I raise batch size only while the latency SLO still holds."
• "ITL and TPOT are the same thing." (wrong) One describes each gap and the other the whole request. (right) "TPOT is per request; ITL is per streamed gap."

Follow-up questions

What is the difference between an SLA, an SLO and an SLI?

An SLI is a measured quantity such as latency or error rate. An SLO is an internal target for it. An SLA is a contract with consequences, normally set looser than the SLO.

What is goodput in LLM serving, and why is it better than throughput?

Goodput is completed requests per second that meet the SLO. Throughput counts requests that finished too slowly, so it can rise while users get a worse service.

Why use P99 latency instead of the average?

The average hides slow requests. In the worked example a mean of 296 milliseconds sat beside a P99 of 5,000 milliseconds, so the SLO would have failed unseen.

What if the decode worker doubles to 20 requests per second?

The best layout shifts toward prefill. With 20 requests per second per decode GPU (illustrative), three prefill plus one decode gives 16.8 over 4 GPUs, 4.2 per GPU, against 3.33 for two prefill. Four prefill plus one decode gives min(22.4, 20) = 20 over 5 GPUs, 4.0, because decode now caps it.¹

Does batching hurt latency?

It can. Batching raises throughput by letting requests wait for company, so a delay setting trades latency for throughput; see continuous batching for how language models soften this.

When should you split prefill and decode onto separate GPUs?

When colocated serving forces you to prioritize one latency or to over-provision. Weigh the gain against KV-cache transfer and placement, as the LLM inference guide discusses.

Sources

1. Service Level Objectives (Site Reliability Engineering, chapter 4) — Google SRE, 1 Jan 2016
2. SRE fundamentals: SLAs vs SLOs vs SLIs — Google Cloud blog, 1 Jul 2018
3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (arXiv 2401.09670) — arXiv (Zhong et al., OSDI 2024), 10 Jul 2024
4. Throughput is Not All You Need: Maximizing Goodput in LLM Serving — Hao AI Lab, 1 Mar 2024
5. Metrics design (vLLM documentation) — vLLM documentation, 26 Oct 2026
6. Benchmark CLI (vLLM documentation) — vLLM documentation, 26 Oct 2026
7. Model Configuration, Queue Policy (Triton Inference Server user guide) — NVIDIA, 26 Oct 2026
8. Batchers (dynamic batcher), Triton Inference Server user guide — NVIDIA, 26 Oct 2026
9. Amazon SageMaker AI Service Level Agreement — Amazon Web Services, 26 Oct 2026