In this article
Before you start
Tail latency is part of system design.
Why interviewers ask this
Tail latency is where a design that looks fast on paper feels slow to users, so a system design round often reaches it when you describe a service that calls many other services. The interviewer wants to see whether you reach for the average, which hides the problem, or for percentiles, and whether you can explain why more servers can make the slowest requests more common.
A weak answer says "we will make it faster" and quotes an average. A strong answer names the percentile you would track, explains fan-out, and lists causes such as queueing and background work. A Staff-level answer adds a mitigation with its cost, such as hedged requests, and says how it limits the extra load. The same reasoning appears in back-end interview questions and in any round where you size a service.
Tail latency from first principles
- Percentile latency
- Percentile latency is the response time that a stated share of requests stay under; in an illustrative case, a p99 of one second means 99 of every 100 requests finish within one second and the slowest one in 100 takes longer.
- Fan-out
- Fan-out is the pattern in which one incoming request is split into many sub-requests sent to different servers, with the final answer built only after the needed replies come back, so the user waits for the slowest reply.
- Hedged request
- A hedged request is a duplicate of a request sent to another replica, often after a chosen delay when the first copy has not answered, with the client using whichever reply arrives first and ignoring the other.
In plain words
Picture a table of 100 guests, with one cook for each plate (illustrative numbers). The waiter brings the food only when every plate is ready. Each cook is late one time in a hundred. With 100 cooks, someone is late on most nights. So the whole table is usually waiting on the slowest cook. The analogy breaks in one place. The sum assumes each cook is late by chance, alone. Real cooks share one oven. Servers share queues, switches and machines. So they are often slow together, and the real numbers can differ from the simple sum.
In one precise sentence, the more components a request depends on, the more often the slowest component decides how long the user waits.
Step 1: Percentiles show the slow end that the average hides
Percentile latency shows the slow end of the response-time distribution, which an average flattens. The Google site reliability engineering (SRE) book gives a service with an average of 100 milliseconds at 1,000 requests per second in which 1% of requests might take 5 seconds. That is 10 slow requests every second, and they pull the average up by 50 milliseconds, so the other requests average only about 50.5 milliseconds.1
To see the tail, collect a histogram: a table of request counts per latency bucket, not a list of raw times. The same chapter suggests bucket boundaries spaced roughly exponentially, by factors of about three. Do not average percentiles across servers; the Prometheus guidance (as of October 2026) calls the average of precomputed quantiles "statistically nonsensical", and recommends adding bucket counts first. You now have a way to measure the tail; the next step shows why it grows with scale.
Step 2: Fan-out turns a rare slow server into a common slow request
Fan-out turns a rare slow server into a common slow request because the user waits for the slowest of many replies. If each server is slow with probability p and a request waits on N servers whose slowness is independent, the request is slow with probability 1 − (1 − p)^N.1
Dean and Barroso (2013) give the example of servers that usually answer in 10 milliseconds but take one second at the 99th percentile. A request on one server is slow one time in 100. A request that collects replies from 100 such servers is slow 63% of the time, and the formula agrees: 63.4%.1 The paper also reports a production case in which one random request had a 99th-percentile latency of 10 milliseconds, while all requests finishing together reached 140 milliseconds. This is the payoff of the lead: the slow-server rate you can ignore alone dominates once requests fan out.
Step 3: Queueing and background work create the slow tail
Queueing and background work create the slow tail. Queueing is waiting in line for a resource that is busy, and the paper notes that several layers of queues in servers and network switches amplify the variability of response times. Background tasks, such as log compaction in storage systems and garbage collection in languages that use it, load the central processing unit (CPU), disk or network and slow the requests that arrive at the wrong time. Contention for shared resources adds a third cause.
The practical lesson is to measure at a fine grain. The SRE book warns that watching CPU load over a minute does not reveal even fairly long spikes that drive high tail latency, and that the 99th percentile over a short window, such as a minute, can be an early signal of saturation. Caches help by removing work; the distributed cache design page covers how.
Step 4: Hedged requests and other techniques tolerate the tail
Hedged requests tolerate the tail instead of trying to remove it. Dean and Barroso argue that large services should build a predictably responsive whole from less predictable parts, just as fault-tolerant systems build a reliable whole from unreliable parts. The simple version sends the same request to several replicas and uses whichever replies first.
Sending every request twice costs load, so the paper waits first. In its example, a hedge sent after a 10 millisecond delay cut the 99.9th-percentile latency of retrieving 1,000 values from 1,800 milliseconds to 74 milliseconds, about 24 times lower, while sending only 2% more requests.1 Dean and Barroso add that waiting until a request has run past the 95th-percentile expected latency limits the extra load to about 5% while still cutting the tail sharply; the price is that a hedged request still waits that long before its second copy is sent.
Their refinement is the tied request: the client sends the request to two servers, each copy names the other, and the first server to start executing tells the other to cancel. That removes most duplicate work while still avoiding queueing delay. Other techniques in the paper: many small partitions per machine, so load can shift in roughly 5% steps with 20 partitions, and good-enough results once most of the data has been searched.
Step-by-step description
- Percentiles show the slow end.
- A rare slow server becomes a common slow request.
- Queueing and background work.
- Hedged requests shrink the tail.
- Percentiles show the slow end that the average hides.
- Fan-out turns a rare slow server into a common slow request.
- Queueing and background work create the slow tail.
- Hedged requests tolerate the tail.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
A search page asks 100 servers for results and cannot render until all reply. Each server is slow, illustratively, one time in a hundred. The table gives the share of user requests that are slow as fan-out grows, plus the other quantities computed from the same formula.1
| Quantity | Value (computed) |
|---|---|
| Slow requests, fan-out 1, server slow 1% | 1.0% |
| Slow requests, fan-out 10, server slow 1% | 9.6% |
| Slow requests, fan-out 100, server slow 1% | 63.4% |
| Slow requests, fan-out 1,000, server slow 1% | about 100% (99.996%) |
| Slow requests, 2,000 servers, each slow 0.01% | 18.1% |
| Per-server slow rate needed to keep fan-out 100 at 1% slow | about 0.01% |
| Server percentile that the median fan-out 100 request experiences | 99.3 |
| Hedging speed-up at the 99.9th percentile (1,800 to 74 milliseconds) | 24.3 times |
The 2,000-server row matches the paper's statement that a one-in-10,000 slow rate at that scale gives almost one request in five. The two middle results change how you set targets. To keep a fan-out of 100 at 1% slow, each server must be slow only about one time in 10,000, which is a 99.99th-percentile target, not a 99th. And the median user request sees the 99.3rd percentile of a single server, which is what the SRE book means when it says one backend's 99th percentile can become the frontend's median.1
Text description of the diagram
- With each server slow 1% of the time, 63.4% of requests are slow at a fan-out of 100.
- The slow share is 1% at a fan-out of 1.
You can redo this with a calculator: raise 0.99 to the power 100, which gives 0.366, and subtract it from one. For sizing arithmetic of this kind, see back-of-the-envelope estimation.
Trade-offs and edge cases
Every tail technique trades something. Hedging spends extra load and fits requests that are safe to run twice, such as reads; a delayed hedge keeps that load small, but each hedged request still waits out the delay first. Small partitions add bookkeeping. Good-enough results return a partial answer, which a payment or a ledger cannot accept, so it suits search and recommendations, not money movement as in a payment system design.
Text description of the diagram
- Hedged requests send a second copy and use the first reply, at the cost of extra load.
- Good-enough results may be partial.
Edge cases: the formula assumes independent slow events. Real slowness is often correlated, because servers that share a switch, a deployment or a noisy neighbor slow together, so treat the formula as a model, not a forecast. Retries and hedges add load to a system that may already be saturated, so pair them with a limit, as in a distributed rate limiter. A slow database query often sets the tail of the whole service; the database interview questions cover how to find it. Some domains push the idea to the limit; see low-latency trading system design. Reported practice on how far to push percentile targets differs by team and by product.
How to answer it in an interview
What is tail latency, and why does it matter?
Tail latency is how slow the slowest requests are, such as the p99, which an average hides. It matters most with fan-out: if a request waits on a hundred servers that are each slow one time in a hundred, about 63 percent of requests are slow. The causes are queueing, background work and shared resources. I measure with histograms, never averaged percentiles, and I shrink the tail with hedged requests sent after a delay, which cost little extra load, plus smaller partitions and good-enough partial results.
The spoken structure: define the tail, give the fan-out calculation, name the causes, then the mitigation with its cost.
Common mistakes
- "Our average latency is 100 milliseconds, so we are fine." (wrong) An average can sit beside a slow 1% at 5 seconds. (right) "I track the p99 and p99.9 from histograms, not only the average."
- "A 1% slow rate per server is negligible." (wrong) Fan-out multiplies it. (right) "At a fan-out of 100 it makes 63% of requests slow."
- "Hedging doubles the load." (wrong) A delayed hedge added about 2% in the paper's example. (right) "I hedge after roughly the p95 delay, so only the slowest few percent are duplicated."
- "I will average the p99 of each server." (wrong) Averaging quantiles gives meaningless values. (right) "I add histogram buckets across servers, then compute the percentile."
Follow-up questions
What is a good p99 latency target?
Set it from what users need, not from a universal number. A service level objective (SLO) is a target for a measured quantity, such as a percentile of latency over a period. The Dean and Barroso paper notes that responses within about 100 milliseconds feel fluid to users, a useful anchor for interactive systems.
Why not just use the average latency?
The average mixes the fast majority with the slow tail and hides it. Step 1 shows the same average hiding 10 requests a second that take 5 seconds.
Do hedged requests double the load?
No, if you delay them: hedging only requests still outstanding after about the p95 latency adds about 5% load in the paper; a shorter delay helps sooner but duplicates more requests.
How do you stop hedged requests wasting work?
Use tied requests. The client sends the request to two servers, each copy names the other, and the first server to start executing tells the other to cancel its copy, as Dean and Barroso describe. Most duplicate work disappears while queueing delay is still avoided.
Why can you not average p99 across servers?
A percentile of a combined set cannot be recovered from percentiles of its parts. Keep histogram buckets per server and add them, as the Prometheus guidance describes, then take the percentile of the sum.
How does tail latency change when you add servers?
If a request must wait on every server, more servers means a higher chance that one is slow, so the slow share grows. If you add replicas behind a load balancer without more fan-out, it need not grow. See the system design learning path for related trade-offs.
- Tail latency is the response time of the slowest requests, tracked with percentiles, not averages.
- Fan-out turns a rare slow server into a common slow request: 1 − (1 − p)^N.
- Queueing, background work and shared resources create the tail.
- Hedged requests sent after a delay cut the tail for a small share of extra load.
- Aggregate histogram buckets, never percentiles.
Sources
- The Tail at Scale — Communications of the ACM (Dean and Barroso), 1 Feb 2013
- Monitoring Distributed Systems (Site Reliability Engineering book) — Google, 1 Jan 2016
- Histograms and summaries — Prometheus documentation, 26 Oct 2026
