In this article
Before you start
This page is one step of the data engineer interview questions: the complete guide. If tables and queries are new to you, start with the Structured Query Language (SQL) interview questions: the complete guide first.
Why interviewers ask this
A weak Kafka answer costs the most when the interviewer asks what breaks. The question shows whether a candidate can explain the storage model, the ordering rule, the replication rule and the consumer model without mixing them up. Kafka questions are reported by candidates in data engineering and system design rounds; no company is named here.
A weak answer calls Kafka "a message queue" and stops. A strong answer names brokers, partitions, replicas and offsets and says what each one is for. A Staff-level answer adds the trade-off: more partitions allow more parallel readers, but order holds only per partition, and acks=all is only as safe as the in-sync set behind it. The streaming vs batch page and the senior data engineering design round show where this answer sits in a longer conversation.
Kafka architecture from first principles
- In-sync replica set
- The in-sync replica set (ISR) is the set of replicas of one partition, the leader included, that are caught up with the leader's log; a write counts as committed only when every member of the set has it, and consumers see only committed events.
In plain words
Picture a newspaper archive. Each day's paper goes on the end of a shelf, and nothing is removed when someone reads it. Every reader keeps their own bookmark. Kafka is that shelf, split into many shelves called partitions, kept in different buildings called brokers, and photocopied so one fire does not lose it.
The analogy breaks at one point: a real archive counts a paper as filed at once, while Kafka counts a write as committed only when enough copies of it exist.
In one precise sentence: Kafka keeps each topic as replicated, append-only partition logs, and every consumer tracks its own position in each log.
Step 1: Brokers store events in append-only logs
A broker is a Kafka server in the storage layer, and Kafka runs as a cluster of one or more servers that can span several datacenters or cloud regions (Apache Kafka introduction). An event is a record with a key, a value, a timestamp and optional headers. A topic is a named folder of events. Producers append events to partition leaders on brokers, and consumers read them later. Events are not deleted after consumption, and each topic has its own retention setting (introduction).
Appending is fast because disks are fast at sequential work. The design documentation reports about 600 megabytes per second for linear writes against about 100 kilobytes per second for random writes on one six-disk array, a difference of over 6000 times, and disk seeks cost 10 milliseconds each. One disk therefore does at most 100 random seeks a second.1 Kafka writes to a persistent log without necessarily flushing each message, so it leans on the operating system's page cache. Input: an event. Output: that event at the log's end, kept after reads.
Step 2: Partitions split a topic and keep order per key
A topic is split into partitions that are spread across brokers, and events with the same key are written to the same partition (Apache Kafka introduction). Order holds inside one partition, not across the topic: two events in different partitions have no fixed order relative to each other. Each partition is an independent log with its own offset counter and its own leader. A single topic-wide order would need one sequencer on every write, which is the one-machine bottleneck partitioning exists to remove. A typical scheme hashes the key and takes the remainder after dividing by the partition count. The hash below, the sum of the characters' codes, is illustrative, not Kafka's real hash.
Take four partitions and six events arriving in order, with illustrative keys.1 The keys cust-17 and cust-8 both land on partition 0, and cust-42 lands on partition 2.1
| Event (arrival order) | Key | Partition | Offset in partition |
|---|---|---|---|
| 1 | cust-17 | 0 | 0 |
| 2 | cust-42 | 2 | 0 |
| 3 | cust-17 | 0 | 1 |
| 4 | cust-8 | 0 | 2 |
| 5 | cust-42 | 2 | 1 |
| 6 | cust-17 | 0 | 3 |
Partition 0 holds four of the six events, a small picture of a hot partition, where one busy key or an unlucky hash overloads one log.1
Step 3: Replication copies each partition, and the in-sync replica set decides what is committed
Each partition has one leader. All writes go to the leader. Reads go there by default, but a consumer can fetch from a nearby in-sync follower, using client.rack and a rack-aware replica selector on the brokers (Kafka Improvement Proposal (KIP) KIP-392). The other replicas are followers that copy the leader's log (Confluent replication documentation, September 2025). The replication factor is the total number of replicas, the leader included, so one means no replication and three is a common production setting (introduction).
A write is committed once every in-sync replica has it; consumers see only committed messages. A replica that falls behind by more than replica.lag.time.max.ms leaves the set. With f plus one replicas a topic tolerates f failures without losing committed messages, while majority voting needs two times f plus one copies: two copies against three for one failure, three against five for two failures.1 Since version 0.11.0.0 Kafka favors consistency over availability by waiting for a consistent replica (Confluent replication documentation).
Step 4: The producer's acks setting chooses how much it waits for
The acks setting tells the producer which acknowledgment to wait for (Confluent replication documentation). With acks=0 the producer does not wait for any response. With acks=1 the leader acknowledges once the message is in its own local log. With acks=all the acknowledgment means the full set of in-sync replicas has the write.
| Setting | Producer waits for | Loss if the leader dies right after the acknowledgment |
|---|---|---|
acks=0 | nothing | possible |
acks=1 | the leader's local write | possible if followers have not copied it |
acks=all | every in-sync replica | not for a committed write while one in-sync copy survives |
The word "all" means all in-sync replicas, not all replicas. If the set has shrunk to the leader alone, an acks=all write is a one-copy write. The topic-level setting min.insync.replicas is the floor that closes this gap: when fewer in-sync replicas exist, the broker rejects an acks=all write instead of accepting a thin one.
Step 5: A consumer group gives each partition to one consumer
A consumer pulls data by sending fetch requests to the brokers that lead (by default) the partitions it wants (design documentation). Each partition is consumed by exactly one consumer within each subscribing consumer group at a time. Producers and consumers are fully decoupled and unaware of each other (introduction), and each group keeps its own offsets, so two groups can read the same topic without affecting each other.
Take a topic with six partitions and an even split of partitions (illustrative). One consumer gets six partitions, two get three each, and three get two each.1 A group of seven consumers has six working and one idle, and nine consumers leave three idle.1 More consumers than partitions adds no speed, so the partition count caps a group's parallelism.
Text description of the diagram
- A group of seven consumers has six working and one idle.
- Nine consumers leave three idle.
Step 6: Offsets are bookmarks stored in an internal topic
An offset is the position of an event inside its partition, and a consumer's position in each partition is a single integer, the offset of the next message to consume; the consumer can rewind it to re-consume data (design documentation). Each partition is read by one consumer per group, which stores its offset in the internal __consumer_offsets topic so a restart resumes where it left off. The committed offset is the next offset to read, not the last one processed (Confluent consumer design).
The high watermark is the offset up to which every in-sync replica has copied the log; a consumer can read only up to it, so it never sees data that could still be lost (Confluent consumer design). Take the documentation's example: position 6, last committed offset 1, high watermark 10. After a crash, the new member starts at offset 1 and reprocesses 5 messages, offsets 1 to 5, and 4 messages still lie between the old position and the high watermark.1 A consumer that can see repeats must handle them.
Step-by-step description
- Producers append events to brokers.
- The key picks one partition.
- Followers copy the leader's log.
- The acks setting decides the wait.
- One consumer per partition.
- Offsets are stored in a topic.
- Producers append events to partition leaders on brokers.
- Followers copy the leader's log, and a write is committed once every in-sync replica has it.
- Each partition is read by one consumer per group, which stores its offset.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How does Kafka work?
Kafka is a cluster of brokers storing events in topics. Each topic is split into partitions, and each partition is an append-only log with one leader and follower replicas. The producer's key picks the partition, so order holds per key, not per topic. A write is committed once every in-sync replica has it, and acks=all waits for that set, so I pair it with min.insync.replicas. Consumers pull from the partition leader by default, a group gives each partition to one member, and the committed offset, the next one to read, lives in an internal topic. I size partitions for consumer parallelism and watch for hot keys.
What makes it Staff-level: each mechanism is tied to its failure mode, the shrinking in-sync set and the hot key, instead of listing parts. Say it in this order: storage, partitions and order, replication and acknowledgment, consumers and offsets, then the trade-off.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Storage model | "A message queue that sends messages." | Append-only partition logs; events stay until retention ends. | Adds why sequential disk writes and the page cache make this fast. |
| Ordering | "Messages are in order." | Order holds per partition; the key chooses the partition. | Adds hot keys, and that partitions can be added but never removed and adding them remaps keys. |
| Durability | "It replicates data." | Leader, followers, and commit when the in-sync set has the write. | Explains acks=all with min.insync.replicas and when the set shrinks. |
| Consumers | "Consumers read messages." | Groups give each partition to one member; offsets are bookmarks. | Covers rewinds, repeats after a crash and the idle-consumer limit. |
| Communication | Lists terms without order. | Walks storage to consumers in a clear sequence. | Ends with the trade-off and what to monitor. |
Three variants of this question
What is the difference between a Kafka topic and a partition?
A topic is the named stream of events; a partition is one ordered log that holds a slice of the topic on one broker. Order holds inside a partition, and a topic with several partitions has no single order.
Why is Kafka fast even though it writes to disk?
Kafka appends to the end of a log, which is sequential, and sequential writes are far faster than random ones; the documentation reports a gap of over 6000 times on one array (design documentation). It also relies on the operating system's page cache instead of flushing every message.
What happens when a Kafka broker dies?
Partitions led by that broker get a new leader from the in-sync followers, and committed messages are not lost while an in-sync replica survives (Confluent replication documentation). Consumers resume from their committed offsets, so some messages may be processed again.
Common mistakes
- "Kafka deletes a message once it is read." (wrong) Events stay until the topic's retention ends and can be read as often as needed. (right) "Retention is set per topic, by time or size, or by compaction, and any group can re-read until then."
- "Order holds across the whole topic." (wrong) It holds only inside one partition. (right) "Order holds per key, because the same key goes to the same partition."
- "More consumers than partitions makes it faster." (wrong) Extra consumers in a group sit idle. (right) "The partition count caps a group's parallelism."
- "
acks=allmeans every replica has the data." (wrong) It means every in-sync replica, which can be the leader alone. (right) "I pairacks=allwithmin.insync.replicas." - "The committed offset is the last message I processed." (wrong) It is the next offset to read. (right) "After a crash the group restarts at the committed offset and may repeat work."
Follow-up questions
Kafka system design interview questions: what should I be ready to explain?
Be ready to choose a key and a partition count, say which acks and min.insync.replicas you would set and why, describe what happens when a broker or a consumer dies, and place Kafka between producers and downstream stores. The senior data engineering design round shows the wider frame.
What do Kafka data engineer interview questions test?
Prepare to explain the log, partition, replica and offset model and to connect it to a pipeline. The data engineer interview questions guide and the page on what a data pipeline is cover the surrounding topics.
Does Kafka still need ZooKeeper?
No. ZooKeeper was removed in Kafka 4.0, where KRaft is the only supported mode (Confluent, December 2025). KRaft is the mode in which Kafka's own controllers, using the Raft protocol, manage metadata: brokers handle data requests and controllers handle metadata requests (Apache Kafka docs).
How many partitions should a topic have?
There is no single right number; it is a trade-off. More partitions allow more consumers in a group to work at once, because each partition has one reader per group, but every partition is also a log that must be stored and replicated. Partitions can be added but not removed, and adding them remaps keys, so per-key order breaks across the change (Apache Kafka documentation); size for expected peak parallelism up front.
Where does Spark fit next to Kafka?
Kafka stores and moves events; a processing engine such as Spark reads them and computes on them. See the Spark interview questions and answers and streaming vs batch.
- Kafka keeps each topic as replicated, append-only partition logs on brokers, and events are not deleted when read.
- The key chooses the partition, so order holds per key, never across a topic.
- A write is committed when every in-sync replica has it;
acks=allneedsmin.insync.replicasto mean more than one copy. - A consumer group gives each partition to one consumer, so extra consumers sit idle.
- The committed offset is the next offset to read, so a crash can repeat work.
Sources
- Introduction to Apache Kafka — Apache Software Foundation, 19 Dec 2025
- Apache Kafka 4.3 documentation: Design — Apache Software Foundation, 22 May 2026
- Kafka Replication and Committed Messages — Confluent Documentation, 26 Sep 2025
- Kafka Consumer Design: Consumers, Consumer Groups, and Offsets — Confluent Documentation, 26 Oct 2026
- Apache Kafka 4.3 documentation: KRaft vs ZooKeeper — Apache Software Foundation, 22 May 2026
- ZooKeeper and Apache Kafka Explained: From Legacy to KRaft — Confluent, 1 Dec 2025
- KIP-392: Allow consumers to fetch from closest replica — Apache Software Foundation, 16 Dec 2019
- Apache Kafka 4.3 documentation: Basic Kafka operations (modifying topics) — Apache Software Foundation, 26 Oct 2026
