Voice Agents: Pipecat and LiveKit
Voice agents are a first-class production category in 2026. Pipecat gives you a Python frame-based pipeline (VAD → STT → LLM → TTS → transport). LiveKit Agents bridges AI models to users over WebRTC. Production latency targets land at 450 to 600ms end-to-end for premium stacks.
Learning Objectives
- Describe Pipecat's frame-based pipeline: DOWNSTREAM (source→sink) and UPSTREAM (control).
- Name the canonical voice pipeline stages and which transports Pipecat supports.
- Explain LiveKit Agents' two voice agent classes (MultimodalAgent, VoicePipelineAgent) and when each fits.
- Summarize 2026 production latency expectations and how they drive architecture choices.
The Problem
Voice agents are not a text loop with TTS bolted on. Latency budgets are brutal (~600ms), partial audio is the default, turn detection is a model, and transports range from telephony SIP to WebRTC. Either you build a frame-based pipeline (Pipecat) or you lean on a platform (LiveKit).
The Concept
Pipecat (pipecat-ai/pipecat)
- Python frame-based pipeline framework.
Frame→FrameProcessorchain.Two flow directions:
- DOWNSTREAMsource → sink (audio in, TTS out).
- UPSTREAMfeedback and control (cancellation, metrics, barge-in).
PipelineTaskmanages lifecycle with events (on_pipeline_startedon_pipeline_finishedon_idle_timeout) and observers for metrics/tracing/RTVI.
Typical pipeline:
VAD (Silero) → STT → LLM (context alternates user/assistant) → TTS → transportTransports: Daily, LiveKit, SmallWebRTCTransport, FastAPI WebSocket, WhatsApp.
Pipecat Flows adds structured conversations (state machines). Pipecat Cloud is the managed runtime.
LiveKit Agents (livekit/agents)
- Bridges AI models to users over WebRTC.
- Key concepts:
AgentAgentSessionentrypointAgentServer. Two voice agent classes:
- MultimodalAgentdirect audio via OpenAI Realtime or equivalent.
- VoicePipelineAgentSTT → LLM → TTS cascade; gives text-level control.
- Semantic turn detection via a transformer model.
- Native MCP integration.
- Telephony via SIP.
- 50+ models with no API keys via LiveKit Inference; 200+ more via plugins.
Commercial platforms
Vapi (~450 to 600ms on an optimized premium stack) and Retell (~600ms end-to-end across 180 test calls) build on top of these. Pick a platform when you want a managed voice stack without a WebRTC team.
Where this pattern goes wrong
- No barge-in handling. User interrupts; agent keeps talking. Requires UPSTREAM cancel frames in Pipecat, equivalent in LiveKit.
- STT confidence ignored. Low-confidence transcripts fed to the LLM as if gospel. Gate on confidence or request confirmation.
- TTS mid-sentence cutoff. When the pipeline cancels mid-utterance, TTS needs to know or cut audio.
- Latency budget ignored. Every component adds 50 to 200ms. Sum your chain before shipping.
Typical 2026 latencies
- VAD: 20 to 60ms
- STT partial: 100 to 250ms
- LLM first token: 150 to 400ms
- TTS first audio: 100 to 200ms
- Transport RTT: 30 to 80ms
End-to-end 450 to 600ms is premium. 800 to 1200ms is common. Anything > 1500ms feels broken.
Build It
code/main.py is a frame-based toy pipeline with:
Frametypes (audio, transcript, text, tts_audio, control).Processorinterface withprocess(frame).- A five-stage pipeline (VAD → STT → LLM → TTS → transport) as scripted processors.
- An UPSTREAM cancel frame to demonstrate barge-in.
Run it:
python3 code/main.pyThe trace shows normal flow and a barge-in cancel that stops TTS mid-utterance.
Use It
- Pipecat for full control, custom processors, Python-first, pluggable providers.
- LiveKit Agents for WebRTC-first deployments and telephony.
- Vapi / Retell for hosted voice agents without a WebRTC team.
- OpenAI Realtime / Gemini Live for direct audio-in/audio-out (MultimodalAgent).
Ship It
outputs/skill-voice-pipeline.md scaffolds a Pipecat-shaped voice pipeline with VAD + STT + LLM + TTS + transport plus barge-in handling.
Exercises
- Add a metrics observer to your toy pipeline: count frames per stage per second. Where does latency accumulate?
- Implement confidence-gated STT: below threshold, request "could you repeat that?"
- Add semantic turn detection: simple rule, if transcript ends with "?", end of turn.
- Read Pipecat's transport docs. Swap the stdlib transport for the SmallWebRTCTransport config (stub).
- Measure an OpenAI Realtime vs STT+LLM+TTS cascade on the same query. What latency cost does text-level control carry?
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Frame | "Event" | Typed unit of data in the pipeline (audio, transcript, text, control) |
| Processor | "Pipeline stage" | Handler with process(frame) |
| DOWNSTREAM | "Forward flow" | Source to sink: audio in, speech out |
| UPSTREAM | "Feedback flow" | Control: cancel, metrics, barge-in |
| VAD | "Voice activity detection" | Detects when user is speaking |
| Semantic turn detection | "Smart end-of-turn" | Model-based decision that the user is done |
| MultimodalAgent | "Direct audio agent" | Audio in, audio out; no text in the middle |
| VoicePipelineAgent | "Cascade agent" | STT + LLM + TTS; text-level control |
Further Reading
- Pipecat docsframe-based pipeline, processors, transports
- LiveKit Agents docsWebRTC + voice primitives
- Vapimanaged voice platform
- Retell AImanaged voice, latency-benchmarked
Check what you learned
7 questions. Answer each one to see why.
Which two flow directions does a Pipecat pipeline use?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Toy Pipecat-style voice pipeline: VAD STT LLM TTS transport.
Frames travel DOWNSTREAM (source to sink) and UPSTREAM (cancel/control).
A scripted input shows normal flow plus a barge-in cancel that stops TTS.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Callable
@dataclass
class Frame:
kind: str
payload: Any
direction: str = "downstream"
class Processor:
def __init__(self, name: str) -> None:
self.name = name
self.next: Processor | None = None
self.prev: Processor | None = None
self.trace: list[str] = []
def process(self, frame: Frame) -> None:
self.trace.append(f"{self.name} saw {frame.kind}")
if self.next is not None and frame.direction == "downstream":
self.next.process(frame)
elif self.prev is not None and frame.direction == "upstream":
self.prev.process(frame)
class VAD(Processor):
def process(self, frame: Frame) -> None:
if frame.kind == "audio_chunk":
is_speech = bool(frame.payload)
self.trace.append(f"VAD: speech={is_speech}")
if is_speech:
super().process(Frame("vad_speech", frame.payload))
else:
super().process(frame)
class STT(Processor):
def process(self, frame: Frame) -> None:
if frame.kind == "vad_speech":
transcript = str(frame.payload)
self.trace.append(f"STT: -> {transcript!r}")
super().process(Frame("transcript", transcript))
else:
super().process(frame)
class LLM(Processor):
def __init__(self, name: str, replies: dict[str, str]) -> None:
super().__init__(name)
self.replies = replies
def process(self, frame: Frame) -> None:
if frame.kind == "cancel":
self.trace.append("LLM: cancelled")
super().process(frame)
return
if frame.kind == "transcript":
text = str(frame.payload)
reply = self.replies.get(text, "[no canned reply]")
self.trace.append(f"LLM: {text!r} -> {reply!r}")
super().process(Frame("text", reply))
else:
super().process(frame)
class TTS(Processor):
def __init__(self, name: str) -> None:
super().__init__(name)
self.cancelled = False
def process(self, frame: Frame) -> None:
if frame.kind == "cancel":
self.cancelled = True
self.trace.append("TTS: cancel received; drop pending audio")
super().process(frame)
return
if frame.kind == "text":
self.cancelled = False
words = str(frame.payload).split()
emitted: list[str] = []
for w in words:
if self.cancelled:
self.trace.append(f"TTS: cut mid-word after {emitted}")
break
emitted.append(w)
self.trace.append(f"TTS: emitted {emitted}")
super().process(Frame("tts_audio", emitted))
else:
super().process(frame)
class Transport(Processor):
def __init__(self, name: str) -> None:
super().__init__(name)
self.delivered: list[list[str]] = []
def process(self, frame: Frame) -> None:
if frame.kind == "tts_audio":
self.delivered.append(list(frame.payload))
self.trace.append(f"transport: sent {len(frame.payload)} words")
else:
super().process(frame)
def link(*processors: Processor) -> None:
for a, b in zip(processors, processors[1:]):
a.next = b
b.prev = a
def main() -> None:
print("=" * 70)
print("VOICE PIPELINE (PIPECAT-SHAPED) — Phase 14, Lesson 22")
print("=" * 70)
vad = VAD("vad")
stt = STT("stt")
llm = LLM("llm", replies={
"hello": "hi there, how can I help today?",
"refund please": (
"sure, I can help with a refund; what order number should I look up?"
),
})
tts = TTS("tts")
transport = Transport("transport")
link(vad, stt, llm, tts, transport)
print("\nscenario 1: normal flow")
vad.process(Frame("audio_chunk", "hello"))
print(f" transport delivered: {transport.delivered[-1]}")
print("\nscenario 2: barge-in mid-utterance")
tts.cancelled = False
vad.process(Frame("audio_chunk", "refund please"))
transport.process(Frame("cancel", None, direction="upstream"))
print(" trace across pipeline")
for proc in (vad, stt, llm, tts, transport):
for line in proc.trace:
print(f" {proc.name}: {line}")
print()
print("barge-in needs UPSTREAM cancel frames that propagate back to TTS+LLM.")
print("sum latency per stage; premium stack lands at 450-600ms end-to-end.")
if __name__ == "__main__":
main()