Get Started

Capstone 03, Real-Time Voice Assistant (ASR to LLM to TTS)

A voice agent that feels right has end-to-end latency under 800ms, knows when you have stopped talking, handles barge-in, and can call a tool without stalling. Retell, Vapi, LiveKit Agents, and Pipecat all hit this bar in 2026. They do it with the same shape: a streaming ASR, a turn-detector, a streaming LLM, and a streaming TTS, all wired through WebRTC with aggressive latency budgets at every hop. Build one, measure WER and MOS and false-cutoff rate, and run it under packet loss.

30 hoursCapstonePython (agent + pipeline), TypeScript (web client)Free lesson

Problem

Voice has been the fastest-moving AI UX category of 2025-2026. The technical ceiling dropped each quarter. OpenAI Realtime API, Gemini 2.5 Live, Cartesia Sonic-2, ElevenLabs Flash v3, LiveKit Agents 1.0, and Pipecat 0.0.70 all put sub-800ms first-audio-out within reach. The bar is not latency alone. It is the interaction feel: not cutting the user off, not getting cut off, recovering from a mid-sentence interruption, calling a tool mid-conversation without stalling the audio, surviving jittery mobile networks.

You cannot get there by stitching three REST calls. The architecture is pipelined streaming end to end. Build it and the failure modes become visible: a VAD tuned for phone audio firing on background TV, a turn-detector waiting for punctuation that never comes, a TTS that buffers 400ms before emitting. The capstone is to fix these one at a time under load and publish a latency-and-quality report.

Concept

The pipeline has five streaming stages: audio in (WebRTC from browser or PSTN), ASR (streaming partial transcripts from Deepgram Nova-3 or faster-whisper), turn detection (VAD plus a small turn-detector model that reads partial transcripts for completion cues), LLM (streaming tokens as soon as the turn is judged complete), TTS (streaming audio out within ~200ms of the first LLM token).

Three cross-cutting concerns. Barge-in: when the user starts speaking while the agent is speaking, the TTS cancels and the ASR picks up immediately. Tool use: mid-conversation function calls (weather, calendar) must run on a side channel without stalling the audio; the agent pre-fills an acknowledgement token ("one second...") if latency exceeds 300ms. Backpressure: under packet loss, partial transcripts are held, VAD raises the speech-gate threshold, and the agent avoids speaking over an unacknowledged message.

The measurement bar is quantitative. WER under 8% on the Hamming VAD benchmark at 15 dB SNR. First-audio-out p50 under 800ms on 100 measured calls. False-cutoff rate under 3%. MOS above 4.2 on TTS. 50 concurrent calls on a single g5.xlarge. These numbers are the deliverable.

Architecture

Code
browser / Twilio PSTN
        |
        v
   WebRTC / SIP edge
        |
        v
  LiveKit Agents 1.0  (or Pipecat 0.0.70)
        |
   +----+--------------+--------------+-----------------+
   |                   |              |                 |
   v                   v              v                 v
  ASR              VAD v5         turn-detector     side-channel
(Deepgram         (Silero)          (LiveKit)        tools
 Nova-3 /         speech-gate    completion score    (weather,
 Whisper-v3)      per 20ms        on partials        calendar)
   |                   |              |
   +--------+----------+--------------+
            v
        LLM (streaming)
     GPT-4o-realtime / Gemini 2.5 Flash /
     cascaded Claude Haiku 4.5
            |
            v
        TTS streaming
     Cartesia Sonic-2 / ElevenLabs Flash v3
            |
            v
     audio back to caller
            |
            v
   OpenTelemetry voice traces -> Langfuse

Stack

  • Transport: LiveKit Agents 1.0 (WebRTC) plus Twilio PSTN gateway; Pipecat 0.0.70 as the alternate framework
  • ASR: Deepgram Nova-3 (streaming, sub-300ms first partial) or faster-whisper Whisper-v3-turbo self-hosted
  • VAD: Silero VAD v5 plus the LiveKit turn-detector (small transformer that reads partial transcripts)
  • LLM: OpenAI GPT-4o-realtime for tight integration, Gemini 2.5 Flash Live, or cascaded Claude Haiku 4.5 (streaming completions, separate audio path)
  • TTS: Cartesia Sonic-2 (lowest first-byte), ElevenLabs Flash v3, or open-source Orpheus for self-host
  • Tools: FastMCP side-channel for weather/calendar/booking; agent pre-emits filler if tool takes >300ms
  • Observability: OpenTelemetry voice spans, Langfuse voice traces with audio replay
  • Deployment: single g5.xlarge (24GB VRAM) for self-hosted Whisper + Orpheus; hosted APIs for lowest latency

Build It

  1. WebRTC session. Stand up a LiveKit room and a web client that streams microphone audio. On the server, attach an agent worker that joins the room.
  2. ASR streaming. Feed 20ms PCM frames to Deepgram Nova-3 (or faster-whisper on GPU). Subscribe to partial and final transcripts. Log per-partial latency.
  3. VAD and turn detector. Run Silero VAD v5 on the frame stream. On speech-end event, fire the LiveKit turn-detector against the latest partial transcript. Only commit to "turn complete" when VAD says silence for 500ms and the turn-detector scores completion > 0.6.
  4. LLM stream. On turn complete, start the LLM call with the running conversation plus the final transcript. Stream tokens out. At the first token, hand off to TTS.
  5. TTS stream. Cartesia Sonic-2 streams audio chunks back. The first chunk must leave the server within 200ms of the first LLM token. Emit chunks to LiveKit room; client plays through WebRTC jitter buffer.
  6. Barge-in. When VAD detects new user speech while TTS is playing, cancel the TTS stream immediately, drop the remaining LLM output, and re-arm the ASR. Publish a tts_canceled span.
  7. Tool side channel. Register weather and calendar as function-calling tools. When invoked, fire the call concurrently; if it does not resolve within 300ms, have the LLM emit "one second, let me check" as a filler; resume once the tool returns.
  8. Eval harness. Record 100 calls. Compute WER (against a held-out transcript), false-cutoff rate (TTS cancelled while user was mid-sentence), first-audio-out p50, TTS MOS (human or NISQA), and a jitter-loss test (drop 3% of packets).
  9. Load test. Drive 50 concurrent calls on a single g5.xlarge with a synthetic caller. Measure sustained first-audio-out p95.

Use It

Code
caller: "what is the weather in tokyo tomorrow"
[asr  ] partial @280ms: "what is the"
[asr  ] partial @540ms: "what is the weather"
[turn ] completion score 0.82 at @820ms; commit
[llm  ] first token @960ms
[tool ] weather.tokyo tomorrow -> 68/52 partly cloudy @1140ms
[tts  ] first audio-out @1040ms: "Tokyo tomorrow will be partly cloudy..."
turn latency: 1040ms user-stop -> audio-out

Ship It

outputs/skill-voice-agent.md is the deliverable. Given a domain (customer support, scheduling, or kiosk), it stands up a LiveKit agent with the ASR/VAD/LLM/TTS pipeline tuned to the measurement bar. Rubric:

WeightCriterionHow it is measured
25End-to-end latencyp50 first-audio-out under 800ms across 100 recorded calls
20Turn-taking qualityFalse-cutoff rate under 3% on the Hamming VAD benchmark
20Tool-use correctnessMid-conversation tool calls that return the right data without stalling audio
20Reliability under packet lossWER and turn-taking stability with 3% packet drop injected
15Eval harness completenessReproducible measurements with public config
100

Exercises

  1. Swap Deepgram Nova-3 for faster-whisper v3 turbo on a g5.xlarge. Measure the latency and WER gap. Identify where CPU-vs-GPU decisions matter.
  2. Add an interruption-arbitration policy: what does the agent do when the user barges in during a tool call? Compare three policies (hard cancel, finish-tool-then-stop, queue next turn).
  3. Run an adversarial turn-detector test: give the user long pauses mid-sentence. Tune the VAD silence threshold and the turn-detector score threshold for lowest false-cutoff without blowing past 900ms.
  4. Deploy the same agent on PSTN via Twilio. Compare PSTN first-audio-out to WebRTC. Explain the jitter-buffer and codec differences.
  5. Add voice activity detection for non-English languages (Japanese, Spanish). Measure the Silero VAD v5 false-trigger rate versus language-specific fine-tunes.

Key Terms

TermWhat people sayWhat it actually means
Turn detection"End of utterance"Classifier that, given VAD silence and a partial transcript, decides the user is done speaking
Barge-in"Interruption handling"Canceling TTS mid-playback when VAD detects new user speech
First-audio-out"Latency"Time from user stops speaking to the first audio packet leaving the server
VAD"Speech gate"Model classifying audio frames as speech vs silence; Silero VAD v5 is the 2026 default
Jitter buffer"Audio smoothing"Client-side buffer that holds packets briefly to absorb network variance
Filler"Acknowledgment token"Short phrase the agent emits to avoid silence when a tool is slow
MOS"Mean opinion score"Perceptual speech quality rating; NISQA is the automated proxy

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

Why can a voice agent not be built by stitching three blocking REST calls?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Real-time voice pipeline — VAD + turn-detection + barge-in scheduler.

The hard architectural primitive in a 2026 voice agent is not the ASR or the
TTS. It is the streaming scheduler that arbitrates between VAD events, ASR
partials, turn-completion scores, LLM streaming, TTS streaming, and user
barge-in, all with bounded latency. This scaffold simulates audio frames and
implements the scheduler in full: state machine, barge-in cancellation, tool
side-channel with filler injection, latency accounting.

Run:  python main.py
"""

from __future__ import annotations

import random
import time
from dataclasses import dataclass, field
from enum import Enum, auto


# ---------------------------------------------------------------------------
# frame stream  --  simulated 20ms audio frames
# ---------------------------------------------------------------------------

@dataclass
class Frame:
    t_ms: int              # timestamp ms since session start
    is_speech: bool        # VAD verdict (Silero v5 stand-in)
    partial: str = ""      # ASR cumulative partial (Deepgram Nova-3 stand-in)


def synth_call(script: str, start_ms: int = 0, noise: float = 0.0) -> list[Frame]:
    """Generate a frame stream for a simulated caller utterance."""
    words = script.split()
    frames: list[Frame] = []
    t = start_ms
    # 120ms silence before speech
    for _ in range(6):
        frames.append(Frame(t_ms=t, is_speech=random.random() < noise))
        t += 20
    partial = ""
    for w in words:
        partial = (partial + " " + w).strip()
        # each word ~320ms of speech
        for _ in range(16):
            frames.append(Frame(t_ms=t, is_speech=True, partial=partial))
            t += 20
    # trailing silence, 2200ms (enough to cover tool + LLM + TTS)
    for _ in range(110):
        frames.append(Frame(t_ms=t, is_speech=False, partial=partial))
        t += 20
    return frames


# ---------------------------------------------------------------------------
# turn detector  --  combines VAD silence duration and completion score
# ---------------------------------------------------------------------------

def turn_completion_score(partial: str) -> float:
    """Tiny stand-in for the LiveKit turn-detector model."""
    if not partial:
        return 0.0
    if partial.rstrip().endswith(("?", ".", "!")):
        return 0.95
    # heuristic: more words, more confidence the turn is done
    n = len(partial.split())
    if n < 3:
        return 0.2
    if n < 6:
        return 0.55
    return 0.75


# ---------------------------------------------------------------------------
# state machine  --  IDLE -> LISTENING -> THINKING -> SPEAKING -> (barge-in)
# ---------------------------------------------------------------------------

class State(Enum):
    IDLE = auto()
    LISTENING = auto()   # user is mid-utterance
    WAITING = auto()     # VAD says silence, checking turn score
    THINKING = auto()    # LLM streaming but no TTS yet
    SPEAKING = auto()    # TTS streaming out
    TOOL = auto()        # side-channel tool in flight


@dataclass
class Metrics:
    events: list[str] = field(default_factory=list)
    turn_complete_ms: int = 0
    first_llm_token_ms: int = 0
    first_audio_out_ms: int = 0
    false_cutoffs: int = 0
    barge_ins: int = 0

    def log(self, msg: str) -> None:
        self.events.append(msg)

    def latency_ms(self) -> int:
        if self.turn_complete_ms and self.first_audio_out_ms:
            return self.first_audio_out_ms - self.turn_complete_ms
        return -1


# ---------------------------------------------------------------------------
# tool side channel  --  async weather/calendar with filler injection
# ---------------------------------------------------------------------------

@dataclass
class Tool:
    name: str
    latency_ms: int
    result: str


WEATHER = Tool("weather.tokyo_tomorrow", latency_ms=420, result="68/52 partly cloudy")


# ---------------------------------------------------------------------------
# scheduler  --  the full pipeline, streamed frame by frame
# ---------------------------------------------------------------------------

def run_session(frames: list[Frame], use_tool: bool = True,
                barge_in_at_ms: int | None = None) -> Metrics:
    m = Metrics()
    state = State.IDLE
    silence_run_ms = 0
    final_partial = ""
    llm_stream_started_at = -1
    tts_stream_started_at = -1
    tool_started_at = -1
    tool_done_at = -1
    filler_emitted = False

    for f in frames:
        # barge-in: user starts speaking while we are SPEAKING or THINKING
        if (barge_in_at_ms is not None and f.t_ms >= barge_in_at_ms
                and state in (State.SPEAKING, State.THINKING)
                and f.is_speech):
            m.barge_ins += 1
            m.log(f"{f.t_ms}ms BARGE-IN: cancel TTS, re-arm ASR")
            state = State.LISTENING
            tts_stream_started_at = -1
            llm_stream_started_at = -1
            continue

        if state == State.IDLE:
            if f.is_speech:
                state = State.LISTENING
                m.log(f"{f.t_ms}ms LISTENING")

        elif state == State.LISTENING:
            if f.is_speech:
                silence_run_ms = 0
                final_partial = f.partial or final_partial
            else:
                silence_run_ms += 20
                if silence_run_ms >= 500:
                    score = turn_completion_score(final_partial)
                    if score >= 0.6:
                        state = State.WAITING
                        m.turn_complete_ms = f.t_ms
                        m.log(f"{f.t_ms}ms TURN COMPLETE (score={score:.2f})"
                              f" partial='{final_partial}'")
                    else:
                        m.log(f"{f.t_ms}ms SILENCE but score={score:.2f}, waiting")

        if state == State.WAITING:
            # kick off LLM
            llm_stream_started_at = f.t_ms + 140  # simulated time-to-first-token
            state = State.THINKING
            m.log(f"{f.t_ms}ms LLM call fired")
            if use_tool:
                tool_started_at = f.t_ms
                state = State.TOOL

        elif state == State.TOOL:
            if tool_started_at >= 0 and not filler_emitted:
                if f.t_ms - tool_started_at >= 300:
                    filler_emitted = True
                    m.log(f"{f.t_ms}ms filler 'one second, let me check'")
            if tool_started_at >= 0 and f.t_ms - tool_started_at >= WEATHER.latency_ms:
                tool_done_at = f.t_ms
                m.log(f"{f.t_ms}ms tool result: {WEATHER.result}")
                llm_stream_started_at = f.t_ms + 140
                state = State.THINKING

        elif state == State.THINKING:
            if llm_stream_started_at > 0 and f.t_ms >= llm_stream_started_at:
                if m.first_llm_token_ms == 0:
                    m.first_llm_token_ms = f.t_ms
                    m.log(f"{f.t_ms}ms LLM first token")
                tts_stream_started_at = f.t_ms + 180
                state = State.SPEAKING

        elif state == State.SPEAKING:
            if tts_stream_started_at > 0 and f.t_ms >= tts_stream_started_at:
                if m.first_audio_out_ms == 0:
                    m.first_audio_out_ms = f.t_ms
                    m.log(f"{f.t_ms}ms TTS first audio-out")

    return m


# ---------------------------------------------------------------------------
# demo  --  runs two sessions, one clean, one with a barge-in
# ---------------------------------------------------------------------------

def main() -> None:
    random.seed(0)
    print("=== session 1: clean call with tool (weather) ===")
    frames = synth_call("what is the weather in tokyo tomorrow", start_ms=0)
    m = run_session(frames, use_tool=True, barge_in_at_ms=None)
    for line in m.events:
        print(" ", line)
    print(f"  turn_complete  @ {m.turn_complete_ms}ms")
    print(f"  first_llm_tok  @ {m.first_llm_token_ms}ms")
    print(f"  first_audio_out @ {m.first_audio_out_ms}ms")
    print(f"  turn latency   = {m.latency_ms()}ms")

    print()
    print("=== session 2: user barges in mid-response ===")
    frames = synth_call("tell me a long story about", start_ms=0)
    # add a few synthetic speech frames late in the trailing silence
    for i in range(8):
        idx = len(frames) - 20 + i
        if 0 <= idx < len(frames):
            frames[idx] = Frame(t_ms=frames[idx].t_ms, is_speech=True,
                                partial=frames[idx].partial)
    m = run_session(frames, use_tool=False,
                    barge_in_at_ms=frames[-20].t_ms - 60)
    for line in m.events:
        print(" ", line)
    print(f"  barge_ins = {m.barge_ins}")


if __name__ == "__main__":
    main()