In this article
Before you start
This page is one part of the generative AI system design interview guide.
Why interviewers ask this
The question tests whether a candidate sees a voice agent as a real-time system, not as three model calls in a row. Candidates report versions such as keeping the end-to-end delay under 1.5 seconds (an illustrative target from a reported question, not a standard), minimizing the delay before the first audio, and handling a patient who talks over the agent. Reports come from interviews at AI infrastructure and healthcare AI companies; none is tied here to a named asker.
A weak answer lists STT, LLM and TTS and says "make each fast." A strong answer gives every stage a budget, explains how the agent decides the user has finished speaking, and says what happens to the audio and the conversation history when the user interrupts. A Staff-level answer also names the trade-off between a cascade of separate models and one speech-to-speech model, and says which numbers must be measured rather than assumed.
Designing a voice agent from first principles
- Voice agent
- A voice agent is a software system that holds a spoken conversation with a user by turning incoming speech into text, choosing a reply with a language model and turning that reply back into speech, while deciding for itself when the user has finished talking and when to stop talking.
- Endpointing
- Endpointing is the step that decides where a sentence or thought ends in a stream of speech, usually by applying rules to the silence a voice activity detector finds, so the agent knows the user's turn is over and a reply can start.
- Barge-in
- Barge-in is the event in which the user starts speaking while the agent is still speaking, so the agent must stop its audio, drop the rest of its reply and listen, and keep a conversation history that matches what the user actually heard.
In plain words
Picture a phone call with a very fast assistant. It listens and writes down what you said. It thinks of a reply and reads it aloud. It stops talking if you cut in. The analogy breaks in one place. A person judges from tone and context that you have finished a thought. The software has to make that call from audio and a few rules, and it can get it wrong in both directions.
Put precisely, a voice agent is a streaming pipeline of STT, an LLM and TTS. It adds two control decisions: when the user's turn ends and when the agent yields.
The LiveKit documentation shows the cascade shape: a single agent session is wired with separate stt, llm and tts components and a turn_handling option. The next four steps follow one user turn through that loop. You now know the shape: three models and two decisions.
Step 1: Streaming speech-to-text turns audio into words
The first stage receives microphone audio in small frames and returns partial transcripts while the user is still talking, then a final transcript when the stage is sure of the words. Input: a stream of audio frames. Output: growing text plus a final version.
Streaming matters because waiting for a whole recording before transcribing would add the full length of the utterance to the delay. The cost of streaming is that partial text can change, so downstream stages must expect revisions. The delay between the user going quiet and the final transcript arriving is the finalization time, one line of the latency budget below.
The audio frames also go to a second consumer, the voice activity detector, which watches the same audio; a turn-detector model, if used, reads the transcript. You now know where the words come from; the next step decides when to act on them.
Step 2: Turn detection decides when the user has finished
Turn detection is the process of determining when a user begins or ends their turn, so the agent knows when to listen and when to respond (LiveKit, turns overview, accessed 2026-10-26). Most techniques rely on voice activity detection (VAD), a model that marks which stretches of audio contain speech, to find silence. Heuristics applied to that silence perform endpointing.
Two settings bound it. LiveKit's tuning guidance describes a minimum delay after detected silence before the turn closes and a maximum delay so the agent never hangs forever. In an example that configures a realtime model's server-side VAD, LiveKit sets a threshold of 0.7 as less sensitive and better for noisy phone audio, and a 400-millisecond silence window for snappier turn closing (turns overview).
Silence alone misleads, because people pause mid-thought. LiveKit's voice agent guide lists reliable turn detection among the core challenges, and its turns overview says an agent can use endpoints alone or apply a model that understands the meaning of speech to determine when a turn is complete. Latency can also be cut by speculatively starting the LLM request before the end of the turn is confirmed.
Step 3: The LLM and text-to-speech start speaking before the reply is finished
The LLM receives the final transcript plus the conversation history and streams its reply token by token. Time to first token (TTFT) is the delay from sending the request to receiving the first token. The reply is cut into short chunks, such as the first complete clause, and each chunk goes to TTS as soon as it exists.
TTS also streams. Time to first audio (TTFA) is the delay from sending the first text chunk to receiving the first playable audio. Playback begins on that first audio while the rest of the reply is still being written and synthesized. Input: streamed text. Output: streamed audio.
Two design choices follow. A short first chunk lowers TTFA but can sound choppy if later chunks arrive late, so keep a small playback buffer. Tool calls, such as looking up a record, add their own delay; speak a short holding phrase while they run. When the answer needs retrieval or multi-step tools, see agentic retrieval-augmented generation (RAG). You now have audio flowing out.
Step 4: Barge-in stops the audio and trims the history
When the user speaks during playback, the agent yields. In LiveKit's words, turn detection also determines when the user starts speaking mid-response so the agent can yield (turns overview). Yielding means three actions: stop playback, cancel the running LLM and TTS requests, and return to listening.
The history step is the one candidates forget. The TTS stage usually produces audio faster than it plays, so the system has generated more words than the user heard. If the full reply stays in the history, the model will later act as if the user heard it all. Keep only the words actually played. In practice the player reports how much audio it has played, and the system cuts the assistant message at the matching word. Treat that as general design knowledge and check your provider's application programming interface (API) for the exact call.
Not every sound is a real interruption. LiveKit's tuning page covers interruption handling and its thresholds. As general design knowledge, add logic that lets the agent carry on when the sound turns out to be a backchannel such as "mm-hm" or noise, so it is not cut off for good.
Step-by-step description
- Streaming STT turns audio into text.
- The turn detector decides the user is done.
- LLM and TTS stream the reply.
- Stop audio and trim the history.
- Speech-to-text turns streaming audio into words.
- Turn detection decides when the user has finished.
- The LLM and text-to-speech start speaking before the reply is finished.
- Barge-in stops playback and trims the history.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
This example splits an end-to-end budget of 1.5 seconds across five stages. Every stage time is an illustrative assumption, not a measurement; measure your own stack. The delay is counted from the moment the user stops speaking until the first audio is heard.1
The illustrative stages are an endpointing wait of 400 milliseconds (the silence window in LiveKit's server-VAD example; its default minimum endpointing delay is 500 milliseconds, per the tuning page), STT finalization of 150, LLM time to first token of 400, TTS time to first audio of 200 and network of 200. They sum to 1,350 milliseconds, which leaves 150 of the 1,500 as headroom.1
The endpointing wait and the LLM together are 800, or 59% of the total, so they are where to look first. The network line covers audio travelling in both directions: LiveKit's voice agent guide says Web Real-Time Communication (WebRTC) is used between the frontend and the agent, and a phone call uses a telephony link instead. The budget therefore includes the uplink and the downlink, not a single hop.
Now add speculative generation. If the LLM request starts when silence begins, its first token (400) overlaps the endpointing wait and finalization (550 together). On a hit the first token is usable at 550 instead of 950 and the total falls to 950, a saving of 400. With an illustrative hit rate of 70%, the expected saving is 280 and the expected total is 1,070 milliseconds. On a miss the request is wasted and the total stays 1,350.1
| Quantity | Value (illustrative, computed) |
|---|---|
| Endpointing wait | 400 milliseconds |
| STT finalization | 150 milliseconds |
| LLM time to first token | 400 milliseconds |
| TTS time to first audio | 200 milliseconds |
| Network | 200 milliseconds |
| Total, sequential | 1,350 milliseconds |
| Headroom against 1,500 | 150 milliseconds |
| Total on a speculative hit | 950 milliseconds |
| Expected total at a 70% hit rate | 1,070 milliseconds |
Text description of the diagram
- The illustrative endpointing wait is 400 milliseconds.
- The illustrative LLM time to first token is 400 milliseconds.
- The illustrative stages total 1,350 milliseconds.
For barge-in, assume an illustrative reply of 60 words spoken at 150 words per minute and an interruption 6 seconds in. The user heard 15 words, or 25% of the reply, so only those 15 belong in the history.1
Trade-offs and edge cases
The main design choice is a cascade against a single speech-to-speech model. A cascade chains speech-to-text, an LLM and text-to-speech, and you tune each stage. A speech-to-speech model processes audio directly. The OpenAI Agents software development kit (SDK) guide says this keeps latency down and makes interruptions feel more natural. That is the vendor's claim. LiveKit's turns overview shows that a realtime model with server-side turn detection still exposes threshold, prefix_padding_ms and silence_duration_ms, so tuning does not disappear. Whether you can audit or swap a part is not covered by these sources; reason about it from the architecture.
Text description of the diagram
- A cascade chains speech-to-text, an LLM and text-to-speech.
- A speech-to-speech model processes audio directly and the vendor reports lower latency.
- A realtime model with server-side turn detection decides when the user is interrupting.
Edge cases to name aloud:
- Noisy lines. Phone audio needs a less sensitive VAD threshold, such as the 0.7 in LiveKit's server-VAD example, or background noise triggers false barge-in.
- Echo. On speakerphones the agent hears its own voice; cancel echo and suppress background voices before VAD, or the agent interrupts itself. LiveKit's tuning page links a noise and echo cancellation resource.
- Long pauses. A maximum endpointing delay prevents a hang, at the cost of occasionally cutting off a slow speaker.
- Peak load. The sources here do not cover capacity. State assumptions: each call holds a streaming connection, so concurrency limits and graceful rejection matter more than average load. A gateway in front of the models helps; see the LLM gateway questions.
- A 200-millisecond first token on a large model. This target comes from a reported question and is not sourced here. Say what you would measure: model size, prompt length, reuse of a shared prompt prefix and batching. Levers to name: a smaller or quantized model, prefix caching and speculative decoding.
- Grounded answers. If replies must come from documents, retrieval adds delay; see RAG over real documents. For a search-style product that answers from the web, compare the Perplexity design walkthrough.
How to answer it in an interview
How would you design a voice agent that responds in under a second and a half?
I'd build a streaming cascade: speech-to-text emits partial transcripts, a turn detector decides when the user is done, the language model streams tokens, and text-to-speech starts on the first complete clause. I'd give each stage a budget that sums under the target and measure from the end of the user's speech to the first audio. The main trade-off is the endpointing wait: shorter feels snappy but cuts people off, so I'd add a turn-detector model and start the language model speculatively. For barge-in I'd stop playback, cancel generation and keep only the words actually spoken in the history.
The spoken structure: name the loop, give the budget, then the endpointing trade-off, then barge-in and history.
Common mistakes
- "A voice agent is speech-to-text, an LLM and text-to-speech." (wrong) That leaves out turn detection and interruption handling, which the LiveKit documentation treats as core. (right) "It is a streaming cascade plus end-of-turn and barge-in logic."
- "The turn ends after a fixed silence." (wrong) People pause mid-thought. (right) "I use VAD with an endpointing delay and, if needed, a turn-detector model that waits longer when the user seems unfinished."
- "On interruption I stop the audio." (wrong) The history still contains words the user never heard. (right) "I stop audio, cancel generation and trim the history to what was played."
- "Speech-to-speech always wins." (wrong) The speed claim is the vendor's, and turn tuning remains. (right) "I'd compare it with the cascade on latency, control and the need to swap parts."
Follow-up questions
How would you keep end-to-end latency under 1.5 seconds in a streaming audio-to-audio system?
Give each stage a budget, stream every stage, and overlap work: start the LLM speculatively and send the first clause to TTS early. Then measure from end of speech to first audio and tune the largest stage first. If the target drops to 800 milliseconds, the illustrative speculative hit totals 950, so 150 more must go, and the expected total of 1,070 is 270 over.1 Shorten the endpointing wait and speed up the first token and first audio, accepting that slow speakers get cut off more often.
How do you design a streaming text-to-speech pipeline to minimize time to first audio?
Feed TTS the first complete clause instead of the whole reply, play audio as chunks arrive, and keep a small buffer so later chunks do not leave gaps. Measure time to first audio separately from total synthesis time.
How do you handle a user who starts speaking before the agent has finished?
Stop playback, cancel the LLM and TTS requests and return to listening. Add recovery logic so a backchannel or noise does not end the agent's turn for good.
How do you manage the state and prompt history?
Record what was actually played, not what was generated. After a barge-in, truncate the assistant message to the spoken words, then append the user's new turn.
Design a voice assistant for hospital staff. What challenges do you solve first?
Start with accuracy and safety: noisy rooms, specialist vocabulary, what the agent may and may not do, and how patient data is handled under your compliance rules. Then fix the latency budget and the turn logic. The sources on this page do not cover healthcare rules, so state your assumptions.
Should you use a speech-to-speech model or a speech-to-text, LLM and text-to-speech pipeline?
Speech-to-speech removes stages and, per the vendor, feels more natural. The cascade gives per-stage control and replacement. Pick on measured latency and on how much control the product needs.
- A voice agent is a streaming cascade of STT, an LLM and TTS plus two control decisions.
- Turn detection combines VAD, an endpointing delay and, optionally, a turn-detector model.
- Budget latency per stage, and overlap stages with speculative generation.
- On barge-in, stop audio, cancel generation and keep only the words the user heard.
- Compare cascade and speech-to-speech on measured latency and control.
Sources
- Turns overview: turn detection and interruptions — LiveKit Documentation (undated live page, accessed 2026-10-26), 26 Oct 2026
- Voice Agents guide — OpenAI Agents SDK for JavaScript (undated, accessed 2026-10-26), 26 Oct 2026
- AI voice agents: turn detection and the turn-detector model — LiveKit Documentation (undated, accessed 2026-10-26), 26 Oct 2026
- Agent session and turn handling options — LiveKit Documentation (undated, accessed 2026-10-26), 26 Oct 2026
- Turn-taking tuning — LiveKit Documentation (undated, accessed 2026-10-26), 26 Oct 2026
