The p95 went from 100 to 2,000 milliseconds: what do you do?

In this article

1. Before you start
2. Why interviewers ask this
3. The p95 spike from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is about debugging a latency spike in a deployed model; the wider subject is machine learning operations (MLOps).

Why interviewers ask this

A latency spike is a common production incident for a language-model service, and the person on call has to find the cause with only dashboards. Candidates report this prompt, with a jump from 100 to 2,000 milliseconds, in interviews at AI labs; one reported variant adds that the spike appears only when a patient's conversation history passes 4,000 tokens (as candidates report it). These are candidate reports, not a public statement by any company.

The question separates people who stare at one number from people who split the traffic. A weak answer says "scale up the servers" or "the model got slower." A strong answer first asks how many requests are slow and which ones, then checks whether the system is full, then ranks fixes by the share of slow traffic each would remove. A Staff-level answer also designs the monitoring so the next spike is found in minutes.

The p95 spike from first principles

Latency histogram

A latency histogram is a table of request counts per latency range, such as under 100 or 100 to 300 milliseconds, that lets a monitoring system compute any percentile and add the counts of many servers together.

Saturation

Saturation is a measure of how full a service is, given as the share of its capacity in use, such as memory, compute or queue length, and it matters because response times climb steeply as it approaches the limit.

In plain words

Picture a coffee shop. Most customers get a drink in two minutes. Some order a complicated drink and wait ten. The staff report that the average wait is three minutes. Nobody sees the people waiting ten. The p95 is the wait that only one customer in twenty exceeds. It shows the complicated-drink customers. The analogy breaks in one place. In a service the slow group is often not a different order type. It can be a server that has filled up and made everyone wait.

In one precise sentence, a p95 of 2,000 milliseconds says at least 5% of requests took 2,000 milliseconds or more (illustrative).

Google's monitoring chapter gives the reason to look beyond averages: a web service with an average latency of 100 milliseconds at 1,000 requests per second might have 1% of requests take 5 seconds. Tail latency, the latency of the slowest few percent of requests, is therefore a separate quantity to watch.

Step 1: Read the p95 as a count of slow requests

A p95 of 2,000 milliseconds means that at least 5% of requests are at or above 2,000 milliseconds. At 1,000 requests per second, that is at least 50 slow requests every second, against a healthy baseline where the slowest 5% were still near 100 milliseconds.¹

Input: one percentile from a dashboard. Output: a count of affected users. This changes the question from "is the system slow?" to "what is special about those 50 requests?" Two checks come first. Compare the p95 with the median and the p99: if the median is flat, a subgroup moved or saturation is just starting. And look at errors separately, because the site reliability engineering (SRE) book notes that the latency of successful requests and of failed ones must be told apart; a fast error makes latency look better than it is.

Step 2: Segment latency by a request property to find the slow mode

Segmenting means splitting the same latency data by one property at a time and seeing which split shows a step. Useful properties for a language-model service are input token count, output token count, route or feature, model name, region, and server instance.

Input: the latency histogram per segment. Output: the property whose slow bucket matches the spike. If the spike appears only when the conversation history passes 4,000 tokens (a threshold as candidates report it), plot p95 against input token count in buckets and look for the step. That threshold has no general cause; it is a hypothesis to test, and it may match a context window, a memory or batch limit, or a branch in the application that truncates or retrieves differently.

A longer prompt costs more compute before the first token and a larger key-value (KV) cache per request. Kwon et al. (2023) describe that cache as huge and dynamically growing, which can limit the batch size. You now know which requests are slow; the next step asks whether the whole system is also full.

Step 3: Check saturation and queueing

Saturation and queueing can also lift the p95. Saturation shows first in the tail: Google's chapter says latency increases are often a leading indicator of saturation. As the system fills, the median rises too; in the queue model below it goes from about 23 to about 462 milliseconds.¹ The same chapter also says that many systems degrade before reaching 100% utilization.

Input: queue depth, graphics processing unit (GPU) memory use, concurrent requests and the share of capacity in use over the same minutes as the spike. Output: a yes or no on whether requests wait. In a model server, a full KV-cache memory pool forces fewer requests per batch or pauses some, so requests wait in the queue even though each one, once started, runs at normal speed. A retry storm adds more requests exactly when the service is full, so check the request rate and the retry count too. The sibling page on large language model (LLM) observability covers which traces and signals to record for this check.

Step 4: Fix the biggest slow bucket and add monitoring

Prioritize by share of slow traffic times seconds saved: fix the bucket that removes the most slow requests per unit of effort. If long prompts are 10% of traffic and cause the whole spike, shorten or summarize history, cache the shared prompt prefix, or route those requests to a pool with more memory; if saturation is the cause, add capacity or cap concurrency.

Input: the ranked list of causes. Output: one change and a way to prove it worked. Then record latency as histogram buckets per segment, alert on the p95 and on saturation, and keep success and error latency apart. The SRE book recommends counting requests per latency bucket rather than storing raw values or averages, with bucket boundaries spaced exponentially. How you serve the model, in batch, streaming or real time, also shapes the target; see batch versus streaming versus real-time inference.

1 · Count2 · Segment3 · Saturation4 · FixReplay

Step-by-step description

1. The p95 means at least 5% of requests are slow.
2. Split latency by request property.
3. Check whether requests wait in a queue.
4. Fix the biggest bucket and monitor it.

• A p95 jump means more than 5% of requests are slow.
• Segmenting by request property finds the slow group.
• Saturation and queueing can also lift the p95.
• Fix the biggest slow bucket first and add monitoring.

Worked example

This synthetic simulation shows why the median hides the problem. It generates 100,000 requests per run: normal requests take about 80 milliseconds and slow requests, such as long prompts, about 2,500 milliseconds, both with random spread. Only the share of slow requests changes between runs.¹ The numbers are illustrative, not measurements of a real service.

The rule is arithmetic: the p95 is the value with 5% of requests above it, so it stays in the fast mode while the slow share is under 5% and jumps into the slow mode once the share is above 5%.¹ The p99 flips earlier, at about 1%, which is why a p99 alert would have fired first. A p95 move from 100 to 2,000 milliseconds therefore tells you the slow share crossed 5%, and the mean of 207 to 231 milliseconds at that point would have looked like a mild drift.¹

Text description of the diagram

• At a 4% slow share the p95 is 128 milliseconds.
• At a 6% slow share the p95 is 2,070 milliseconds.

Trade-offs and edge cases

Saturation can produce the same jump without any slow subgroup. In a simple single-server queue with random arrivals, the p95 of the time in the system is about ln(20) times the service time divided by one minus the utilization (an illustrative textbook model, not a model server). With a 20-millisecond service time, the p95 is about 100 milliseconds at 40% utilization and about 1,997 at 97%.¹ Going from 100 to 2,000 needs no slow subgroup at all: a rise in load from 40% to 97% of capacity is enough, so a traffic surge or a lost server can cause it alone.

Text description of the diagram

• The p95 is about 100 milliseconds at 40% utilization.
• The p95 is about 1,997 milliseconds at 97% utilization.

Other cases that break the textbook answer:

• Fan-out. If one page waits on many calls, the tail dominates. With 5% of calls slow and 10 parallel calls, 40.1% of pages see at least one slow call; with 20 calls it is 64.2%.¹ Google's chapter notes that one backend's 99th percentile can become the frontend's median.
• Variability cannot be removed. Dean and Barroso (2013) argue that eliminating every source of latency variability is impractical, especially on shared infrastructure, so design to tolerate it.
• Averaging percentiles. Do not average p95 values across servers; add the histograms first, then compute the percentile. This is standard practice, and it is the reason to store buckets.
• Hardware faults. One sick machine can create a slow group that looks like a request property; segment by instance too. The GPU failure page covers that case.

How to answer it in an interview

The spoken structure: say what the number means, then segment, then check saturation, then fix by priority, then monitor.

Common mistakes

• "The average latency is fine, so users are fine." (wrong) A 100 millisecond average can hide 1% of requests at 5 seconds. (right) "I look at the p95 and p99 and the median, never the mean alone."
• "The model got slower, so I would scale the model." (wrong) That skips the evidence. (right) "I segment by input length, route and instance before I blame the model."
• "I include errors in the latency chart." (wrong) Fast failures drag the percentile down. (right) "I track successful and failed requests separately."
• "I average the p95 across hosts." (wrong) Percentiles do not average. (right) "I add histogram buckets across hosts, then compute the percentile."
• "I will remove all latency variance." (wrong) That is impractical on shared infrastructure. (right) "I tolerate it and design for the tail."

Follow-up questions

How would you debug a latency spike that appears only when the conversation history passes 4,000 tokens?

Plot p95 against input token count in buckets and confirm a step at that size (the 4,000 figure is as candidates report it). Then test hypotheses: a context-window or memory limit, a batch cap, a truncation or retrieval branch, or longer prompts filling the KV-cache pool. Reproduce with a synthetic prompt just under and just over the threshold.

Why did the p95 move but the median stay flat?

Only a subgroup of requests changed. The p95 sits at the edge of the slowest 5%, so once a slow group above 5% of traffic exists, the p95 lands inside it while the median stays in the fast mode.

How would you handle traffic spikes without overwhelming the model provider?

A strong answer limits concurrent calls to the provider, puts a bounded queue in front, sheds or defers low-priority work when it fills, and retries with spacing so retries do not add to the surge. Say which requests you would drop first.

How do you think about cost and capacity planning for an LLM app at scale?

Plan from the traffic shape and the token counts per request, then set a utilization target below full, since systems degrade before 100% utilization. The Kubernetes for machine learning (ML) infrastructure page covers the scheduling side.

What strategies would you use to optimize the cost of large-scale LLM inference while keeping performance acceptable?

Reported strategies include routing simple queries to cheaper or distilled models, batching requests, and scaling capacity to traffic. Measure the p95 and answer quality after each change, because a cost saving that raises the tail has moved the cost to users.

Sources

1. Chapter 6: Monitoring Distributed Systems, Site Reliability Engineering — Google (SRE book), 1 Jan 2016
2. The Tail at Scale — Communications of the ACM (Dean and Barroso), 1 Feb 2013
3. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al.), 12 Sep 2023