Get Started

Voice Agents: Pipecat and LiveKit

Voice agents are a first-class production category in 2026. Pipecat gives you a Python frame-based pipeline (VAD → STT → LLM → TTS → transport). LiveKit Agents bridges AI models to users over WebRTC. Production latency targets land at 450 to 600ms end-to-end for premium stacks.

60 minutesLearnPython (stdlib)Free lesson

Learning Objectives

  • Describe Pipecat's frame-based pipeline: DOWNSTREAM (source→sink) and UPSTREAM (control).
  • Name the canonical voice pipeline stages and which transports Pipecat supports.
  • Explain LiveKit Agents' two voice agent classes (MultimodalAgent, VoicePipelineAgent) and when each fits.
  • Summarize 2026 production latency expectations and how they drive architecture choices.

The Problem

Voice agents are not a text loop with TTS bolted on. Latency budgets are brutal (~600ms), partial audio is the default, turn detection is a model, and transports range from telephony SIP to WebRTC. Either you build a frame-based pipeline (Pipecat) or you lean on a platform (LiveKit).

The Concept

Pipecat (pipecat-ai/pipecat)

  • Python frame-based pipeline framework.
  • Frame → FrameProcessor chain.
  • Two flow directions:

    • DOWNSTREAMsource → sink (audio in, TTS out).
    • UPSTREAMfeedback and control (cancellation, metrics, barge-in).
  • PipelineTask manages lifecycle with events (on_pipeline_startedon_pipeline_finishedon_idle_timeout) and observers for metrics/tracing/RTVI.

Typical pipeline:

Code
VAD (Silero) → STT → LLM (context alternates user/assistant) → TTS → transport

Transports: Daily, LiveKit, SmallWebRTCTransport, FastAPI WebSocket, WhatsApp.

Pipecat Flows adds structured conversations (state machines). Pipecat Cloud is the managed runtime.

LiveKit Agents (livekit/agents)

  • Bridges AI models to users over WebRTC.
  • Key concepts: AgentAgentSessionentrypointAgentServer.
  • Two voice agent classes:

    • MultimodalAgentdirect audio via OpenAI Realtime or equivalent.
    • VoicePipelineAgentSTT → LLM → TTS cascade; gives text-level control.
  • Semantic turn detection via a transformer model.
  • Native MCP integration.
  • Telephony via SIP.
  • 50+ models with no API keys via LiveKit Inference; 200+ more via plugins.

Commercial platforms

Vapi (~450 to 600ms on an optimized premium stack) and Retell (~600ms end-to-end across 180 test calls) build on top of these. Pick a platform when you want a managed voice stack without a WebRTC team.

Where this pattern goes wrong

  • No barge-in handling. User interrupts; agent keeps talking. Requires UPSTREAM cancel frames in Pipecat, equivalent in LiveKit.
  • STT confidence ignored. Low-confidence transcripts fed to the LLM as if gospel. Gate on confidence or request confirmation.
  • TTS mid-sentence cutoff. When the pipeline cancels mid-utterance, TTS needs to know or cut audio.
  • Latency budget ignored. Every component adds 50 to 200ms. Sum your chain before shipping.

Typical 2026 latencies

  • VAD: 20 to 60ms
  • STT partial: 100 to 250ms
  • LLM first token: 150 to 400ms
  • TTS first audio: 100 to 200ms
  • Transport RTT: 30 to 80ms

End-to-end 450 to 600ms is premium. 800 to 1200ms is common. Anything > 1500ms feels broken.

Build It

code/main.py is a frame-based toy pipeline with:

  • Frame types (audio, transcript, text, tts_audio, control).
  • Processor interface with process(frame).
  • A five-stage pipeline (VAD → STT → LLM → TTS → transport) as scripted processors.
  • An UPSTREAM cancel frame to demonstrate barge-in.

Run it:

Code
python3 code/main.py

The trace shows normal flow and a barge-in cancel that stops TTS mid-utterance.

Use It

  • Pipecat for full control, custom processors, Python-first, pluggable providers.
  • LiveKit Agents for WebRTC-first deployments and telephony.
  • Vapi / Retell for hosted voice agents without a WebRTC team.
  • OpenAI Realtime / Gemini Live for direct audio-in/audio-out (MultimodalAgent).

Ship It

outputs/skill-voice-pipeline.md scaffolds a Pipecat-shaped voice pipeline with VAD + STT + LLM + TTS + transport plus barge-in handling.

Exercises

  1. Add a metrics observer to your toy pipeline: count frames per stage per second. Where does latency accumulate?
  2. Implement confidence-gated STT: below threshold, request "could you repeat that?"
  3. Add semantic turn detection: simple rule, if transcript ends with "?", end of turn.
  4. Read Pipecat's transport docs. Swap the stdlib transport for the SmallWebRTCTransport config (stub).
  5. Measure an OpenAI Realtime vs STT+LLM+TTS cascade on the same query. What latency cost does text-level control carry?

Key Terms

TermWhat people sayWhat it actually means
Frame"Event"Typed unit of data in the pipeline (audio, transcript, text, control)
Processor"Pipeline stage"Handler with process(frame)
DOWNSTREAM"Forward flow"Source to sink: audio in, speech out
UPSTREAM"Feedback flow"Control: cancel, metrics, barge-in
VAD"Voice activity detection"Detects when user is speaking
Semantic turn detection"Smart end-of-turn"Model-based decision that the user is done
MultimodalAgent"Direct audio agent"Audio in, audio out; no text in the middle
VoicePipelineAgent"Cascade agent"STT + LLM + TTS; text-level control

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

Which two flow directions does a Pipecat pipeline use?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Toy Pipecat-style voice pipeline: VAD  STT  LLM  TTS  transport.

Frames travel DOWNSTREAM (source to sink) and UPSTREAM (cancel/control).
A scripted input shows normal flow plus a barge-in cancel that stops TTS.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Any, Callable


@dataclass
class Frame:
    kind: str
    payload: Any
    direction: str = "downstream"


class Processor:
    def __init__(self, name: str) -> None:
        self.name = name
        self.next: Processor | None = None
        self.prev: Processor | None = None
        self.trace: list[str] = []

    def process(self, frame: Frame) -> None:
        self.trace.append(f"{self.name} saw {frame.kind}")
        if self.next is not None and frame.direction == "downstream":
            self.next.process(frame)
        elif self.prev is not None and frame.direction == "upstream":
            self.prev.process(frame)


class VAD(Processor):
    def process(self, frame: Frame) -> None:
        if frame.kind == "audio_chunk":
            is_speech = bool(frame.payload)
            self.trace.append(f"VAD: speech={is_speech}")
            if is_speech:
                super().process(Frame("vad_speech", frame.payload))
        else:
            super().process(frame)


class STT(Processor):
    def process(self, frame: Frame) -> None:
        if frame.kind == "vad_speech":
            transcript = str(frame.payload)
            self.trace.append(f"STT: -> {transcript!r}")
            super().process(Frame("transcript", transcript))
        else:
            super().process(frame)


class LLM(Processor):
    def __init__(self, name: str, replies: dict[str, str]) -> None:
        super().__init__(name)
        self.replies = replies

    def process(self, frame: Frame) -> None:
        if frame.kind == "cancel":
            self.trace.append("LLM: cancelled")
            super().process(frame)
            return
        if frame.kind == "transcript":
            text = str(frame.payload)
            reply = self.replies.get(text, "[no canned reply]")
            self.trace.append(f"LLM: {text!r}  -> {reply!r}")
            super().process(Frame("text", reply))
        else:
            super().process(frame)


class TTS(Processor):
    def __init__(self, name: str) -> None:
        super().__init__(name)
        self.cancelled = False

    def process(self, frame: Frame) -> None:
        if frame.kind == "cancel":
            self.cancelled = True
            self.trace.append("TTS: cancel received; drop pending audio")
            super().process(frame)
            return
        if frame.kind == "text":
            self.cancelled = False
            words = str(frame.payload).split()
            emitted: list[str] = []
            for w in words:
                if self.cancelled:
                    self.trace.append(f"TTS: cut mid-word after {emitted}")
                    break
                emitted.append(w)
            self.trace.append(f"TTS: emitted {emitted}")
            super().process(Frame("tts_audio", emitted))
        else:
            super().process(frame)


class Transport(Processor):
    def __init__(self, name: str) -> None:
        super().__init__(name)
        self.delivered: list[list[str]] = []

    def process(self, frame: Frame) -> None:
        if frame.kind == "tts_audio":
            self.delivered.append(list(frame.payload))
            self.trace.append(f"transport: sent {len(frame.payload)} words")
        else:
            super().process(frame)


def link(*processors: Processor) -> None:
    for a, b in zip(processors, processors[1:]):
        a.next = b
        b.prev = a


def main() -> None:
    print("=" * 70)
    print("VOICE PIPELINE (PIPECAT-SHAPED) — Phase 14, Lesson 22")
    print("=" * 70)

    vad = VAD("vad")
    stt = STT("stt")
    llm = LLM("llm", replies={
        "hello": "hi there, how can I help today?",
        "refund please": (
            "sure, I can help with a refund; what order number should I look up?"
        ),
    })
    tts = TTS("tts")
    transport = Transport("transport")
    link(vad, stt, llm, tts, transport)

    print("\nscenario 1: normal flow")
    vad.process(Frame("audio_chunk", "hello"))
    print(f"  transport delivered: {transport.delivered[-1]}")

    print("\nscenario 2: barge-in mid-utterance")
    tts.cancelled = False
    vad.process(Frame("audio_chunk", "refund please"))
    transport.process(Frame("cancel", None, direction="upstream"))

    print("  trace across pipeline")
    for proc in (vad, stt, llm, tts, transport):
        for line in proc.trace:
            print(f"    {proc.name}: {line}")

    print()
    print("barge-in needs UPSTREAM cancel frames that propagate back to TTS+LLM.")
    print("sum latency per stage; premium stack lands at 450-600ms end-to-end.")


if __name__ == "__main__":
    main()