Design Slack: a chat system with live comments

In this article

1. Before you start
2. Why interviewers ask this
3. A chat system from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step on the system design learning path.

Why interviewers ask this

A chat system is a standard test of real-time design because every message must reach the right people quickly, in order, even when connections drop or users are disconnected.

It is reported in interviews at AI labs and large tech companies as design Slack, design WhatsApp or design live comments, with follow-ups on scaling 100 times and 1000 times and on users across the world. A weak answer draws clients, a server and a database and says "WebSockets". A strong answer explains how connections are held and found, how order is kept, what happens when delivery fails, and why a stream with a million viewers needs a different design from a chat room. The order to practise is: size, connections, store and order, fan-out, recovery, then the big-audience case.

A chat system from first principles

WebSocket

A WebSocket is a long-lived two-way connection between a client and a server, opened with an ordinary web request that is then upgraded, over which either side can send messages at any time without a new request.

Sequence number

A sequence number is a counter, assigned by the server per channel, that gives every message in that channel one position in a single agreed order.

In plain words

Think of a post office with a wall of pigeonholes. Each channel is a pigeonhole, and every message gets a number as it goes in. Many people have a mail slot at home that the post office can push letters through. If a letter is dropped on the way, the person sees a missing number on the next one and asks for it. The analogy breaks in one place. A post office delivers each letter to one address, while a chat message may need to reach thousands of open slots at once.

In one precise sentence: a chat system holds a connection per client, gives each message a per-channel position, stores it, and pushes it to every connected member. Clients recover anything the push missed. Slack's April 2023 post describes the connection and fan-out half of this shape; the sequence numbers and the store-first rule are this page's design. In Slack's description, gateway servers hold each client's WebSocket connections. Channel servers hold channel state. A message travels from the sender through the channel server to every gateway that has subscribers.

Step 1: Clarify what the system does and size it

Fix the features and the load first, because the number of open connections drives the design more than the number of messages. Ask whether it is one-to-one chat, channels or both, how large a channel can be, whether history is searchable, whether a user may have several devices, and what presence (online or away) must show. Then state numbers. This page uses illustrative inputs: 10 million users connected at once, 100,000 connections per gateway server and 10 kilobytes of memory per open connection.

Those inputs give 100 gateway servers at full load, or 200 with room for failures, and about 1 gigabyte of connection memory on each.¹ The point is that connections, not requests, set the server count. Slack's post reports tens of millions of connected clients in its own system, which is the order of magnitude this exercise assumes.

1 · Size the load2 · Hold connections3 · Stamp and store4 · Fan out5 · Fill the gaps6 · Thin live commentsReplay

Step-by-step description

1. Clients are the unit: 10 million at once.
2. Gateways keep each client's WebSocket open.
3. A sequence number, then a durable write.
4. Pub/sub carries each stored message to the gateways.
5. Clients fetch missing numbers from the store, via a gateway.
6. A sampler keeps a huge audience readable.

• Clients keep a WebSocket connection to a gateway server.
• The chat service stamps a sequence number on each message and writes it to the message store before anything is published.
• Pub/sub carries each message to the gateways that hold subscribers, directly or through the sampler.
• A client that notices a gap in the sequence fetches the missing range from the message store, through its gateway.
• A sampler sits between pub/sub and the gateways and thins the live comments for a very large audience.

Step 2: Hold the connections on gateway servers

Gateway servers keep each client's WebSocket open and know which channels that client has subscribed to. Request for Comments (RFC) 6455 defines the protocol: the client sends an HTTP GET request asking to upgrade, the server replies with status 101, and the same connection then carries messages both ways. Either side can send a Ping frame to check the other is alive, and the receiver must answer with a Pong, which is how a gateway notices dead clients.

Clients connect to the nearest region. Slack runs its gateway servers in several geographic regions and moves users from a failed region to healthy ones. The channels themselves are spread over servers by consistent hashing, so a channel id always leads to the same channel server. In Slack's description, the admin servers find a channel's server through the consistent-hash ring when a message is sent, and each gateway uses the same ring to subscribe to the channel servers that hold its users' channels.

Step 3: Stamp a sequence number, then store the message

On a send, the chat service assigns the next sequence number for that channel, writes the message to a durable store, and only then acknowledges the sender. A single writer per channel, chosen by hashing the channel id, makes the numbers gapless and cheap, because no two servers ever race for the next number. The client attaches its own message id, so a resend after a lost acknowledgement is recognised and not stored twice; see payment system design for the same idempotency idea.

Store first, publish second. The store is the truth: a message that is in the store can always be fetched again, and the sequence number tells a reader exactly which ones it has. Publishing is the fast copy. The order of the two matters, because a message published before it is stored could be seen by users and then vanish after a crash.

Step 4: Fan out to the gateways, not to the users

After the write, the message is published so that every gateway that holds a subscriber of the channel gets one copy, and each gateway pushes it to its own connected members. Fan-out at the gateway level keeps server-to-server traffic small. In Slack's design, the channel server forwards a message to every subscribed gateway server worldwide, and each gateway pushes it to its connected clients.

For a channel with 100,000 seats of which 30% are connected, sending to each connected seat from the chat service is 30,000 sends. Sending once per gateway is at most 200, a 150-fold reduction.¹ Pub/sub systems are often fast and simple but weak on delivery: the Redis documentation says its pub/sub has at-most-once semantics, so if a subscriber cannot handle a message the message is lost for good. That is why Step 3 stored it first.

Step 5: Fill gaps and handle disconnected devices

Clients detect loss by the sequence number and fetch what is missing from the store. A client that last saw number 41 and receives 44 knows 42 and 43 are missing and buffers 44. If 42 and 43 do not arrive within a short hole timeout (say 1 second), or more than 5 messages pile up in the buffer, the client asks the store for the range 42 to 43, then shows 42, 43 and 44 in order. A late duplicate of 43 is ignored. A lost last message is caught because the heartbeat and every reconnect carry each channel's latest sequence number.

In a 10,000-message simulation with 2% of messages dropped, 3% duplicated and jittered arrival, a client that rendered messages in arrival order missed 212, showed 305 repeats and had 2,336 neighbouring swaps. A sequence-aware client showed all 10,000, once each and in order, after 206 gap fetches.¹ For an away user, the store keeps the messages and a push notification service nudges the device; unread counts are two numbers per channel.

Step 6: Thin live comments for a huge audience

A live video with a million viewers cannot send every comment to every viewer, so a sampler caps what each viewer receives. Facebook's 2011 engineering post describes the first version of this problem: polling every few seconds would have overloaded the servers, so the team moved to push, which needs a record of which viewers are looking at which content. The post reports about 650,000 comments a minute and about 16 million new viewer-to-content associations a second, and a rule of writing the association locally and reading it globally when a comment is posted.

Take 1 million viewers and 10,000 comments posted a second. Sending every comment to every viewer is 10 billion deliveries a second. Showing each viewer at most 10 comments a second is 10 million, a thousandfold cut. The sampler sends each gateway only the sampled stream, so 200 gateways each receive 10 comments a second and push them to about 5,000 viewers each, 50,000 outbound messages a second per gateway.¹

Worked example

The numbers of the six steps fit in one table, each following from the illustrative inputs above.¹

Text description of the diagram

• With 1 million viewers and 10,000 comments posted a second, sending every comment to every viewer means 10 billion deliveries a second.
• Showing each viewer at most 10 comments a second means 10 million deliveries a second.

The client's recovery rule is small enough to write out:

next_seq, buffer = 1, {}
def on_message(seq, msg):
    if seq < next_seq or seq in buffer:
        return                                  # a repeat: ignore
    buffer[seq] = msg
    drain()
    if len(buffer) > 5:                         # more than five buffered: fetch now
        fill_gap()
def drain():
    global next_seq
    while next_seq in buffer:                   # show every message that is now in order
        show(buffer.pop(next_seq)); next_seq += 1
def fill_gap():                                 # also called by a 1-second hole timer
    if buffer:
        for m in fetch_from_store(next_seq, min(buffer) - 1):
            buffer[m.seq] = m                   # fill the hole from the durable store
        drain()
def on_heartbeat(latest_seq):                   # also run on every reconnect
    if latest_seq >= next_seq and not buffer:
        for m in fetch_from_store(next_seq, latest_seq):
            buffer[m.seq] = m                   # a lost last message: fetch the tail
        drain()

Run on the page's 41, 44 example, the timer fires after 1 second, fetches 42 and 43, and shows 42, 43 and 44; the table below is the result of the same rule, plus the heartbeat tail check, against the simulation.

1 · Shown 412 · 44 arrives3 · Timeout4 · Fetch5 · Show in order6 · RepeatReplay

Step-by-step description

1. Next expected number is 42.
2. A hole: 42 and 43 are missing.
3. The hole has waited 1 second.
4. Ask the store for 42 to 43.
5. Drain the buffer.
6. A late 43 is ignored.

• A client that holds 41 and receives 44 buffers 44 and waits.
• After the hole timeout it fetches 42 to 43 from the store and shows 42, 43, 44 in order.
• A repeat of 43 is ignored because 43 is below the next number.

The numbers come from one seeded run and are illustrative of the effect. In the simulation one tick is the gap between two messages and the hole timeout is 3 ticks. A real client also bounds its buffer and falls back to a full history reload after a long absence.

Trade-offs and edge cases

Every part of this design trades latency, ordering and cost, so name the trade before the number. The hardest ones are ordering against throughput, delivery against cost, and recovery after failure.

Order against throughput. One writer per channel gives a total order and makes a very busy channel a bottleneck. For live comments, a strict order of every comment is not needed; the sampler can order approximately, and chat rooms keep strict order.

Delivery against cost. Pushing every event to every client is expensive. Slack's post notes that presence servers send clients updates only for the users visible on their screen, which is the same idea as sampling: do not deliver what nobody can see.

Reconnect storms. When a gateway fails, all its clients reconnect at once. Clients should retry with growing, randomised delays so the survivors are not overwhelmed, and the sequence-number recovery of Step 5 means a reconnect needs only the missing range, not the whole history. Slack's post describes a draining mechanism that moves users from a failed region to healthy ones, and replacement of an unhealthy channel server in under 20 seconds.

Multiple devices. Each device holds its own connection and its own last-read sequence number, so fan-out is per connection. Slack's gateway holds the subscriptions of a user, and the channel server needs only to know the gateways.

Hot keys and the rate limit. One huge channel or one spammy client can overload a single server; see the distributed rate limiter and the distributed cache for the usual remedies. The counting problem of an audience, with retries and late events, is the one in the ad click aggregator.

How to answer it in an interview

The spoken structure is: connections first, then store with a sequence number, then pub/sub fan-out to gateways, then gap recovery, then the big-audience case.

Common mistakes

"Store messages in the pub/sub system." (wrong) Pub/sub is at-most-once, so a missed message is gone. (right) "I write to a durable store first and publish second."

"Fan out from the chat service to each user." (wrong) That makes the service send tens of thousands of copies per message for one big channel. (right) "I fan out once per gateway, and each gateway pushes to its own clients."

"Order by timestamp." (wrong) Clocks differ and two messages can share a time. (right) "The server assigns a per-channel sequence number, which defines the order."

"Send every comment to every viewer." (wrong) At a million viewers that is billions of deliveries a second. (right) "I sample, so each viewer receives a bounded number a second."

"WebSockets are enough." (wrong) A connection says nothing about delivery, order or recovery. (right) "WebSockets carry messages; sequence numbers and the store make delivery correct."

Follow-up questions

How do you deliver a message to a user with three devices?

Each device has its own connection to some gateway and its own last-read sequence number. The channel's fan-out reaches all three gateways, and a device that was disconnected fetches the missed range from the store when it reconnects.

What ordering do you promise in a group chat?

A total order within a channel, defined by the per-channel sequence number that one writer assigns. Across channels there is no order, and the design makes no promise there.

How do you scale the system by 100 times?

Add gateway servers for connections, spread channels over more channel servers by consistent hashing, and keep gateway-level fan-out so server-to-server traffic grows with the number of gateways, not users. At ten million connections the exercise needs about 100 gateways, so a hundredfold means about 10,000 at the same density, and the hash ring and regional routing become the hard parts.¹

How do you design the API for live commenting?

Offer a call to post a comment on a stream, a subscription that streams sampled comments over the WebSocket, and a call that returns recent comments by sequence range for late joiners. Make the post idempotent with a client-supplied id.

What if the dropped message is the last one?

Then no later number reveals the hole. The heartbeat and every reconnect carry each channel's latest sequence number, so a client that holds 44 and is told the latest is 45 fetches 45 from the store. The same check covers a crash between the store write and the publish.

What happens when a gateway server dies and every client reconnects?

Clients reconnect with randomised, growing delays to other gateways, which re-subscribe them to their channels. Each client sends its last sequence number per channel and receives only what it missed, so the store, not the pub/sub system, absorbs the recovery. See the web crawler design for a similar rule on pacing requests per host.

Sources

1. Real-time Messaging — Slack Engineering, 11 Apr 2023
2. Live Commenting: Behind the Scenes — Facebook Engineering, 7 Feb 2011
3. RFC 6455: The WebSocket Protocol — IETF (Fette, Melnikov), 1 Dec 2011
4. Redis Pub/sub — Redis documentation (date = read date), 24 Oct 2026