ASR LLM TTS Pipeline Design: How to Stay Under 1.5 Seconds

In this article

1. Before you start
2. Why interviewers ask this
3. Streaming voice pipeline from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the generative AI system design learning path.

Why interviewers ask this

A voice product feels broken when the reply starts late, and the delay is made of many small waits that add up. Interviewers use this design question to see whether a candidate treats latency as an architecture problem rather than a model-size problem. Candidates report it as: design a streaming audio-to-audio system from a speech recognition model, a language model and a speech synthesis model that keeps end-to-end latency under a second and a half.

A weak answer lists three boxes and says "use fast models." A strong answer explains that every stage must emit output before the previous stage finishes, names the wait that closes the speaker's turn, and says which number to measure. A Staff-level answer adds interruption handling and the choice between a chained pipeline and a single speech-to-speech model.

Streaming voice pipeline from first principles

Chained pipeline

A chained pipeline is a voice system that converts audio to text with speech recognition, writes a reply with a language model, and turns that reply back into audio with speech synthesis, passing text between the stages. Because both sides of the language model are text, the system keeps a transcript and the application can inspect and control every step.

Interim result

An interim result is a draft transcript that a streaming speech recogniser returns while audio is still arriving, which it may revise as it hears more. A final result closes a segment of audio and is not revised.

Time to first audio

Time to first audio is the delay from the end of the user's speech to the first sound of the reply. It is the number a listener actually feels, and the sum of every stage's wait before it.

In plain words

Think of a kitchen that starts plating the starter while the main course cooks. The guest gets food sooner because the work overlaps. The analogy breaks in one place. A plate is never thrown away, but a draft transcript or half a reply sometimes is.

In one precise sentence, a streaming voice pipeline overlaps its stages. The user then hears audio after the sum of the first-output delays, not after the sum of the full run times.

Without streaming, the system waits for the caller to finish, then for recognition, then for the language model, then for speech synthesis. A vendor guide says that spends more than three seconds before any audio goes back. Streaming is therefore the architecture, not a tuning option.

Step 1: Streaming ASR sends interim results, then final results

A streaming recogniser accepts audio in small chunks and returns text as it goes. Google Cloud documents interim results as the current recognition result for a section of audio that may be refined after more audio, and the final result as the last, best guess for that section. Deepgram marks interim events with is_final: false and a finished segment with is_final: true.

Interim results are drafts; a final result closes a segment. Input: audio chunks. Output: revisable text. Because recognition runs while the caller is still talking, it adds almost nothing on top of the speech itself. You now know where the text comes from; the next question is when to treat it as complete.

Step 2: Endpointing decides when the speaker has finished

A final segment is not the same as a finished turn. Endpointing waits for silence before the turn is closed, and Deepgram describes its endpointing as an audio-based voice activity detector (VAD), a component that tells speech from silence. Its speech_final: true flag marks a detected pause.

The silence threshold is the main turn-taking trade-off. Deepgram lists 10 milliseconds as the default for chatbots expecting short utterances and 300 to 500 milliseconds for conversations where speakers pause mid-thought. A short threshold answers fast but interrupts people who are thinking; a long one is polite but adds its whole length to the gap. One vendor guide says most latency sits in turn-taking and the language model's first token, not in recognition or synthesis. Input: silence after speech. Output: a closed turn.

Step 3: The LLM streams tokens, and the first token starts the clock for speech

Once the turn closes, the transcript goes to the language model, which returns tokens as it writes them. The delay before the first token arrives is called time to first token (TTFT), and it matters more than total generation time because speech can begin long before the reply ends. The LLM streams tokens; the first token starts the clock for speech.

Published targets differ. Hamming lists 200 to 500 milliseconds for the language model, and an arXiv paper treats a median first token of roughly 500 milliseconds as a practical design target. One vendor guide notes that speculative prefill on early transcripts can cut the critical path: the model starts reading the likely transcript before the turn is certain. Input: the closed transcript. Output: a token stream.

Step 4: TTS starts on the first sentence

Speech synthesis does not wait for the whole reply. TTS starts on the first sentence, or the first clause, because most synthesis voices sound natural only when they see a phrase of context, and a buffer holds the tokens until one is complete. The wait before the first sound is the time to first audio of the synthesiser, called time to first byte in vendor material.

The buffer has a cost that is easy to forget. Twelve tokens at fifty tokens per second (both illustrative) add 240 milliseconds, and the 150 and 300 in the table come from the same twelve tokens at 80 and 40 tokens per second (illustrative) before synthesis can begin.¹ Splitting at the first comma instead of the first full stop trades a little naturalness for a shorter wait. Hamming lists 40 to 200 milliseconds for synthesis time to first byte. Input: the first sentence. Output: streaming audio.

Step 5: Barge-in cancels playback and generation

Barge-in is the user speaking over the agent, and the system must stop talking, not finish its sentence. The design stops audio playback, cancels the language model and synthesis requests that are still running, and cuts the stored conversation back to what the user actually heard. If the history keeps text that was generated but never played, the next turn answers a conversation that did not happen.

Barge-in cancels playback and generation. Input: speech detected while the agent is talking. Output: a stopped pipeline and a corrected history, with the new turn starting at Step 1 again. The voice agent design page goes deeper on echo and interruption handling.

1 · Streaming ASR2 · Endpointing3 · LLM stream4 · TTS5 · Barge-inReplay

Step-by-step description

1. Interim results are drafts; a final result closes a segment.
2. Silence closes the turn.
3. The first token starts the clock for speech.
4. Synthesis starts on the first sentence.
5. An interruption cancels playback and generation.

• Interim results are drafts; a final result closes a segment.
• Endpointing waits for silence before the turn is closed.
• The LLM streams tokens; the first token starts the clock for speech.
• TTS starts on the first sentence.
• Barge-in cancels playback and generation.

Worked example

Take a target of a second and a half from the end of speech to the first sound. Each stage gets a low, midpoint and high value taken from the sourced ranges above, plus the illustrative buffer and network values.¹ Retell lists 50 to 100 milliseconds for the final transcript and 100 to 200 for synthesis first audio, and the first-token range runs from Retell's 250 budget to the arXiv paper's 500 target. Endpointing runs 300 to 500 milliseconds, the final transcript 50 to 100, the language model's first token 250 to 500, synthesis first audio 100 to 200, and network and jitter 50 to 150 (illustrative).¹

For contrast, a sequential design (illustrative values) waits 400 milliseconds for the end of speech, 300 for full recognition, 375 for the first token, 1,600 more to finish writing 80 tokens at 50 per second, 600 to synthesise the whole reply and 100 for the network. That is 3,375 milliseconds before any sound plays.¹ Streaming at the midpoint of each range finishes inside the target; the sequential design does not.

Two published budgets sit beside this one. A vendor guide adds network 50, turn-taking 200, language model first token 250 and synthesis 100 to reach roughly 600 milliseconds. A second vendor guide reports Twilio's guidance of 350, 375 and 100 for the three stages, 825 in total, against a median mouth-to-ear target of 1,115, which leaves 290 for other hops.¹ The ranges differ, so present them as ranges.

Text description of the diagram

• Streaming at the midpoint of each range finishes inside the target.
• A sequential design does not.

Trade-offs and edge cases

The biggest design choice is whether to chain at all. OpenAI's voice agents guide frames the key choice as how speech connects to reasoning and tools: a continuous conversation with a separate backend, a single voice model, or a pipeline you control stage by stage. A chained pipeline gives a transcript and control over each step, and it is a natural way to add voice to an existing text application. A single speech-to-speech model typically gives lower latency and keeps tone and prosody, but offers less direct control.

Text description of the diagram

• A chained pipeline gives a transcript and control over each step.
• A speech-to-speech model typically gives lower latency and keeps tone.

Edge cases follow from streaming. A speculative model call started on an interim result is wasted if the transcript changes, so it costs compute in exchange for latency. A slow tool call leaves silence, and a filler phrase such as "one moment while I check that" is a documented mitigation. Averages hide the tail: Introl suggests tracking median, 95th and 99th percentiles, and flags a 95th percentile above 800 milliseconds as a warning.

Do not borrow a telephony standard as a voice-AI target. The International Telecommunication Union (ITU) recommendation G.114 rates one-way delays of 0 to 150 milliseconds acceptable for most applications, but it governs mouth-to-ear transmission time. No standard sets a conversational voice-AI gap, so the budget is an engineering choice. The same stage-by-stage thinking applies to the LLM gateway that sits in front of the model and to making retrieval-augmented generation (RAG) fast and cheap when the reply needs retrieved context.

How to answer it in an interview

The spoken structure: say that every stage streams, name the three waits that dominate, give the metric, then close with interruption handling. For a full end-to-end design with a worked scenario, see how to design ChatGPT or Claude.ai.

Common mistakes

• "Speech recognition and synthesis are the slow parts." (wrong) Recognition runs while the caller talks. (right) "Most of the gap is the endpointing wait and the language model's first token, so I measure them first."
• "I wait for the full reply before synthesis." (wrong) That puts the whole generation time into the gap. (right) "I start synthesis on the first sentence or clause."
• "A final transcript means the turn is over." (wrong) A final result closes a segment, and a pause flag signals the turn. (right) "I close the turn on endpointing, not on the first final segment."
• "The average latency is fine." (wrong) Users feel the slow tail. (right) "I alert on the 95th and 99th percentiles per stage."
• "Interrupted text stays in history." (wrong) The model then answers words the user never heard. (right) "I truncate history to what was played."

Follow-up questions

How do you keep a voice pipeline's end-to-end latency under a second and a half?

Stream every stage, keep the endpointing threshold as short as the use case allows, choose a language model with a low first-token delay, and start synthesis on the first sentence or clause. The worked example shows the midpoint budget fits with little spare room, so track each stage separately.

Where does the latency in a voice pipeline actually go?

Mostly to turn-taking and the language model's first token, according to a vendor guide. One practitioner source says to optimise the slowest component first, usually language model inference.

Should you use a chained pipeline or a speech-to-speech model?

Use a chained pipeline when you need a transcript, policy checks, a choice of text model or an easy path from an existing text application. Use speech-to-speech when interactivity and tone matter most and you accept less control.

What happens when the user interrupts the agent?

Barge-in stops audio playback, cancels the in-flight model and synthesis requests, and truncates history to the text actually played. Echo from the agent's own speech must not trigger a false interruption.

What should you measure in production?

Measure the delay from the end of user speech to the first audio byte, the first-byte time of each stage, and the median, 95th and 99th percentiles, as Introl recommends. Pair it with quality checks as in evaluating a RAG system stage by stage.

Sources

1. One-way transmission time (Recommendation G.114) — ITU-T, 1 Feb 1996
2. Voice agents guide — OpenAI developer documentation, 26 Oct 2026
3. Configure Endpointing and Interim Results — Deepgram documentation, 26 Oct 2026
4. Understanding end of speech detection — Deepgram documentation, 26 Oct 2026
5. Cloud Speech-to-Text streaming requests — Google Cloud documentation, 26 Oct 2026
6. Voice agent architecture: STT, LLM, TTS pipeline design — Deepgram (vendor blog), 14 Oct 2026
7. How Real-Time Voice AI Works: STT, LLM and TTS Explained — Retell AI (vendor blog), 25 Sep 2026
8. Perfecting Human-AI Interaction at Clinical Scale (arXiv 2603.29893) — arXiv, 1 Mar 2026
9. Voice AI Latency: What's Fast, What's Slow, and How to Fix It — Hamming AI (practitioner blog), 12 Jan 2026
10. Voice AI Infrastructure: real-time speech agents — Introl, 1 Dec 2025