Distributed systems interview questions: how the round works

In this article

1. The vocabulary of distributed design
2. Start here: the learning path
3. The core ideas behind the path
4. Questions interviewers ask on distributed systems
5. All 49 guides in System design
6. Follow-up questions
7. Sources

The vocabulary of distributed design

Distributed system

A distributed system is a set of computers that communicate over a network and coordinate their work so that users experience a single service, while any individual machine or network link can fail independently.

Service level objective

A service level objective (SLO) is a target for a measured service quality, such as the share of requests that succeed or a latency percentile, that a team commits to meeting over a period.

Consistent hashing

Consistent hashing is a way of assigning keys to servers on a hash ring so that adding or removing a server moves only a small share of the keys instead of nearly all of them.

Idempotency key

An idempotency key is a unique token a client sends with a request so that the server can recognise a retry and return the saved result instead of performing the operation a second time.

Start here: the learning path

Study the round and the shared ideas first; cache and payment designs are where they apply most directly.

Text description of the diagram

• Caches and storage teach consistent hashing and replication, which appear in almost every later design.
• Money and correctness teach idempotency and strong consistency, the cleanest examples of making a consistency choice.
• Messaging and real time, pipelines and search, protection and control planes, and feeds and collaboration each apply the same method to a different kind of system.

Caches and storage teach consistent hashing and replication, which appear in almost every later design. Money and correctness teach idempotency and strong consistency, the cleanest examples of making a consistency choice. Messaging and real time, pipelines and search, protection and control planes, and feeds and collaboration each apply the same method to a different kind of system.

For the machine learning version of the round, see machine learning system design; for the production side of models, machine learning operations (MLOps); and for the pipelines under both, data engineering.

1. Consistent hashing: how to design a distributed cache
2. How do you design a payment system?
3. Key value store system design: object, blob and file storage
4. How do you design a collaborative editing system like Google Docs?
5. Database interview questions on keys, indexes, transactions and consistency
6. Backend interview questions on HTTP, APIs, caching, retries and security
7. How to design a distributed rate limiter for an API gateway
8. How to design a message queue / pub-sub system
9. Design Slack: a chat system with live comments
10. Design an ad click aggregator: a metrics and logging pipeline
11. How do you design a distributed job scheduler / task execution system?
12. Design a web crawler
13. How do you design Ticketmaster, a flash sale or hotel booking?
14. How do you design the whole product in a system design interview?
15. How do you design abuse and access-control infrastructure?
16. Low latency trading system design
17. How do you design a notification / push system at scale?
18. How do you design dispatch and matching for a ride or delivery app?
19. How do you design a top-k trending system with time windows?
20. How do you design a news feed and a social graph?
21. How do you design a search backend?
22. How do you design a GPU cluster scheduler and control plane?
23. How do you design multi-tenant code sandboxes?
24. Concurrency interview questions on race conditions, locks, deadlock and asyncio
25. Classic system design interview questions: LeetCode, Tic-Tac-Toe, Yelp and more
26. API design interview questions on pagination, idempotency, versioning and errors
27. How would you build a simple version of Identity Access Management?
28. How to answer message queue and event-driven questions
29. Design distributed training for a 10B to 1T parameter model

The core ideas behind the path

How does the distributed design round differ from an ML design round?

A distributed-systems round starts from requirements about availability, consistency and scale, while a machine learning round starts from a prediction task, data and a metric. One published delivery framework (Hello Interview, as of October 2026) orders the round as requirements, core entities, an application programming interface (API), an optional data flow, a high-level design and deep dives, with roughly a quarter of the time on deep dives (Hello Interview delivery framework). The same page advises doing capacity estimation only when it changes the design. Other orders exist; our system design interview framework teaches a four-layer version (requirements, interface, data model, scale and trade-offs) of the same round.

What order does the round follow?

The round on this page runs in six steps: requirements, entities, interfaces, a first design, scale barriers and deep dives. The diagram follows the published order with two changes: it leaves out the optional data-flow step and adds a step of our own, scale barriers, where you stress the first design with numbers. Requirements come first, functional and non-functional, with scale numbers only where they change the design.

Entities and interfaces fix what the system stores and exposes before any boxes are drawn. The first design is simple; scale barriers are found by stressing it with numbers; deep dives go where the design is weakest. Interviewers use the structure to see whether you can lead: you propose the next step, check the interviewer follows, and spend time where the risk is.

1 · Requirements2 · Entities3 · Interfaces4 · First design5 · Scale barriers6 · Deep divesReplay

Step-by-step description

1. Agree what the system must do and how fast, available and consistent it must be.
2. Name the things the system stores and the relationships between them.
3. Write the API calls clients make; they fix the data flow.
4. Draw the simplest set of boxes that meets the requirements.
5. Stress the design with numbers to find what breaks first.
6. Pick two or three weak spots and design them in detail.

• Requirements come first, functional and non-functional, with scale numbers only where they change the design.
• Entities and interfaces fix what the system stores and exposes before any boxes are drawn.
• The first design is simple; scale barriers are found by stressing it with numbers; deep dives go where the design is weakest.

What do you ask first, and when do numbers matter?

Ask who the users are, what the system must do, and how available, consistent and fast it must be, then estimate scale only to decide whether one machine suffices. Take an illustrative service with 10 million daily active users making 20 requests each. That is 200 million requests a day, an average of 2,315 queries per second (QPS), and with a peak three times the average about 6,944 QPS.

If one request in ten is a write at 1,000 bytes, that is about 20 gigabytes (GB) of new data a day and 7.3 terabytes (TB) a year.¹ Those figures decide the design: they are large but not exotic, so a candidate who answers "for scale" without computing them can over-build. A candidate who computes them can say which parts need distribution and why.

What are SLOs and "nines", and why do dependencies matter?

A service level objective (SLO) is a target for an indicator such as success rate or a latency percentile, and the Google Site Reliability Engineering (SRE) book defines indicators, objectives and agreements as separate things, noting that an agreement carries consequences for missing the objective (Google SRE book, chapter 4). Availability is quoted in "nines". One service at 99.9 percent is down about 526 minutes a year, at 99.99 percent about 53 minutes and at 99.999 percent about 5.¹

The less obvious point is that availability multiplies along a chain of required services. Three in series are down about 1,575 minutes and ten about 5,232 minutes a year, so ten services each 99.9% available give about 99.0% together. The calculation assumes independent failures.¹ The downtime an objective allows is an error budget that each required dependency spends. Averages hide the tail, the slowest few percent of requests: the SRE chapter on service level objectives gives a typical request near 50 milliseconds while 5% of requests are 20 times slower.

Text description of the diagram

• One service at 99.9 percent is down about 526 minutes a year.
• Three in series are down about 1,575 minutes and ten about 5,232 minutes a year.
• The calculation assumes independent failures.

What does the CAP theorem say, and what do real systems do?

The Consistency, Availability and Partition tolerance (CAP) theorem says that when a network partition splits a system, it cannot keep both consistency (every response is correct) and availability (every request gets a response); you choose which to give up during the partition. Gilbert and Lynch review the theorem and the techniques for coping with its trade-offs (Perspectives on the CAP Theorem, 2012). Many systems let you tune the choice, and some fix it system-wide.

Amazon's Dynamo key-value store states that it sacrifices consistency under certain failure scenarios to provide an "always-on" experience, using object versioning and application-assisted conflict resolution (DeCandia et al., Symposium on Operating Systems Principles (SOSP) 2007). In an interview, say which operations may be stale (a like count) and which may not (a payment). As advice, also say that outside a partition the same trade-off shows up as latency against consistency, because waiting for more replicas to agree costs time.

What is consistent hashing, and why is it the answer for caches?

Consistent hashing places both servers and keys on a ring, so that a key belongs to the next server clockwise and a new server takes over only the keys between itself and its predecessor. The original paper describes a consistent hash function as one that "changes minimally as the range of the function changes" (Karger et al., 1997). Dynamo uses the ring with virtual nodes, several positions per physical server, to spread load evenly.

In our simulation of 100,000 keys going from 10 to 11 servers, hash mod N moved 90.88 percent while a ring with 200 virtual nodes per server moved 9.75 percent; the ideal is 1 in 11, 9.09 percent.¹ In general, adding a server to N moves about 1/(N+1) of the keys on a ring and about N/(N+1) with hash mod N: about 1 percent against 99 percent at 100 servers. A cache that loses most of its entries sends a flood of requests to the database.

Text description of the diagram

• With hash mod N, going from 10 to 11 servers moved 90.88 percent of 100,000 keys.
• Consistent hashing with 200 virtual nodes per server moved 9.75 percent.
• The ideal is 1 in 11, 9.09 percent.

How do you make a payment safe to retry?

Make the operation idempotent with an idempotency key: the client generates a unique key per logical operation and the server stores the result of the first attempt and returns it on retries.

Stripe's version 1 API documentation (as of October 2026) describes exactly this: it saves the status code and body of the first request made with a key and returns the saved response on retries, accepts keys up to 255 characters, may prune keys at least 24 hours old, and compares incoming parameters with the original request, returning an error if they differ (Stripe idempotent requests reference); those rules are stated for version 1, and Stripe's version 2 handles retries differently. The design question behind it is what a client should do after a timeout: it cannot know whether the server acted, so it retries with the same key.

How should you handle queues, real-time and pipeline questions?

Treat each as a trade-off between delivery semantics, ordering and cost, and state the semantics you choose before drawing boxes. At-most-once delivery can lose a message, at-least-once can deliver it twice so the consumer must tolerate repeats, and exactly-once in practice means at-least-once plus removal of duplicates. Order is cheap inside one partition of a queue and expensive across many. The method on this page applies to message queues, chat systems, notification pipelines and metrics aggregators alike: write the requirements, name the entities, fix the interface, then find the scale barrier with numbers like those above.

How would you design a traffic management load balancer?

A load balancer is a server that spreads incoming requests across several identical servers, so that no single one is overloaded and a failed one can be skipped. Name the health check that removes a dead server and what the balancer does when every server looks unhealthy. Say whether the balancer is itself a single point of failure and how a standby takes over, because the component that protects the others must not be the weakest one.

What is replication, and what does it cost?

Replication keeps copies, called replicas, of the same data on several machines, so one failure loses nothing and reads can be spread out. The cost is that copies can disagree for a moment, which is the consistency choice above. Ask early whether users sit in one region or many: users spread across regions are served from nearby copies to cut network delay, at the price of keeping those copies in step.

How do rate limiters, gateways and schedulers fit the pattern?

These are control-plane designs: they decide who may do how much, and what runs where and when, rather than moving user data. A distributed rate limiter counts requests per caller across many servers, so the first question is where the count lives and what happens when the counter store is unreachable: refuse everything or allow everything. A gateway sits in front of services and applies such rules once. A job scheduler assigns work to machines and must survive a machine dying mid-job, which brings back the idempotency idea above, because a retried job must not do its work twice. For all three, say which failure you accept, and what the caller sees when it happens.

How do feeds and collaborative editing fit the pattern?

Both combine the earlier parts, and both turn on one choice. A news feed decides whether to build each reader's list when a post is written or when the reader opens the app, trading write cost against read latency, and a social graph with very popular accounts makes the first option expensive for them. Collaborative documents decide how simultaneous edits merge so that every user ends with the same text, which is a consistency question with no payment-style need for strictness but a hard need for convergence. State the read-write balance first, as in the follow-up below, and the rest of the design follows from it.

Questions interviewers ask on distributed systems

Expect three shapes of question: a named system (design a cache, a chat service, a ticketing system), a property (how do you keep this consistent, what happens when a server does not respond), and a number (what scale, which latency target). Draw an elaborate diagram only after the requirements justify it. For the wider set of questions across engineering loops, see the AI engineer question list and how interview loops are structured; for serving cost, see large language model (LLM) inference.

What is a quorum in a distributed database?

A quorum is the minimum number of replicas that must take part in a read or a write for it to count. With N replicas, W the number that must confirm a write and R the number consulted on a read, choosing R + W > N makes every read overlap the latest write; the Dynamo paper describes this setting as a quorum-like system (DeCandia et al., 2007). Smaller R or W is faster and more available, but reads can be stale.

What is eventual consistency?

Eventual consistency means that if no new updates arrive, all replicas converge to the same value over time, though a read in the meantime may return an older one. Dynamo takes this route, propagating updates to replicas asynchronously and resolving conflicting versions in the application (DeCandia et al., 2007). Use it for data such as like counts, not for payments.

How does leader election and consensus work?

Consensus is the problem of getting a group of machines to agree on one value, or one order of operations, even when some of them fail. Raft solves it by electing a single leader that orders commands in a replicated log; a command is committed once a majority of servers store it, and a new election starts if the leader goes silent (Ongaro and Ousterhout, 2014). Because a majority must agree, a cluster of five servers keeps working with two down.

What is sharding?

Sharding is partitioning a data set across machines so that each machine holds only a part of it, which lets storage and traffic scale beyond one server. The main design choice is how a key maps to a shard: hash mod N is simple but moves nearly every key when N changes, while the consistent hashing section above moves only about 1/(N+1) of them (Karger et al., 1997). Name the hot-shard risk too.

Two-phase commit or saga for a transaction across services?

Two-phase commit has a coordinator ask every participant to prepare and then commit together, which keeps the transaction atomic but ties the services to one coordinator. A saga is a sequence of local transactions in which each step has a compensating action that undoes it, as proposed by Garcia-Molina and Salem (1987) for long-lived transactions. As advice: use a database transaction inside one service, and a saga across services, accepting that intermediate states are visible.

Follow-up questions

What is the concept of distributed systems?

Several machines cooperate over a network to behave like one service, which gives scale and fault tolerance at the price of partial failure, network delay and the need to coordinate.

Are we optimizing for read-heavy or write-heavy workloads?

It is the first design question after requirements, because reads can be cached and replicated cheaply while writes need ordering and durability. The capacity example above uses a write share of 10%, so reads dominate.

What's the expected scale in daily users, peak QPS and data volume?

Estimate peak QPS as daily requests divided by 86,400 seconds times a peak factor, then storage as writes times record size. Our example gives about 6,944 peak QPS and 20 GB a day.

Are there specific geographic regions or latency targets?

Ask it right after scale: one region keeps the design simple, while several regions need copies of the data near users and a decision on which writes must cross regions. A latency target is a percentile, such as a 99th-percentile response time, not an average.

What happens when a server doesn't respond?

Treat it as failed after a timeout, retry on another replica with an idempotency key if the operation is not safe to repeat, and accept that a slow server and a dead server look the same to the caller.

Which would you give up in a partition, consistency or availability?

Decide per operation: give up availability for payments and inventory, give up consistency for feeds and counters, and say what the user sees in each case.

Sources

1. Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web — STOC 1997 (Karger, Lehman, Leighton, Panigrahy, Levine, Lewin), 4 May 1997
2. Dynamo: Amazon's Highly Available Key-value Store — SOSP 2007 (DeCandia et al.), 14 Oct 2007
3. Perspectives on the CAP Theorem — IEEE Computer, 2012 (Gilbert and Lynch), 1 Feb 2012
4. Service Level Objectives (Site Reliability Engineering, chapter 4) — Google SRE book, O'Reilly 2016, 1 Apr 2016
5. Idempotent requests (API reference) — Stripe documentation (read 2026-10-24), 24 Oct 2026
6. System design delivery framework — Hello Interview (page undated; read 2026-10-24), 24 Oct 2026
7. In Search of an Understandable Consensus Algorithm — USENIX ATC 2014 (Ongaro and Ousterhout), 19 Jun 2014
8. Sagas — SIGMOD 1987 (Garcia-Molina and Salem), 1 Dec 1987