Agent Observability: Langfuse, Phoenix, Opik
Three open-source agent observability platforms dominate 2026. Langfuse (MIT), 6M+ installs/month, tracing + prompt management + evals + session replay. Arize Phoenix (Elastic 2.0), deep agent-specific evals, RAG relevancy, OpenInference auto-instrumentation. Comet Opik (Apache 2.0), automated prompt optimization, guardrails, LLM-judge hallucination detection.
Learning Objectives
- Name the three top open-source agent observability platforms and their licenses.
- Distinguish what each one is strongest at: Langfuse (prompt mgmt + sessions), Phoenix (RAG + auto-instrumentation), Opik (optimization + guardrails).
- Explain why 89% of organizations report having agent observability in place by 2026.
- Implement a stdlib trace-to-dashboard pipeline with LLM-judge evaluation.
The Problem
OTel GenAI (Lesson 23) gives you the schema. You still need the platform that ingests spans, runs evaluations, stores prompt versions, and surfaces regressions. The three contenders each emphasize different parts of the lifecycle.
The Concept
Langfuse (MIT)
- 6M+ SDK installs/month, 19k+ GitHub stars.
- Features: tracing, prompt management with versioning + playground, evaluations (LLM-as-judge, user feedback, custom), session replays.
- June 2025: formerly commercial modules (LLM-as-a-judge, annotation queues, prompt experiments, Playground) open-sourced under MIT.
- Strongest for: end-to-end observability with tight prompt-management loop.
Arize Phoenix (Elastic License 2.0)
- Deeper agent-specific evaluation: trace clustering, anomaly detection, retrieval relevancy for RAG.
- Native OpenInference auto-instrumentation.
- Pairs with managed Arize AX for production.
- No prompt versioning, positioned as a drift/behavioral-regression tool alongside broader platforms.
- Strongest for: RAG relevancy, behavioral drift, anomaly detection.
Comet Opik (Apache 2.0)
- Automated prompt optimization through A/B experiments.
- Guardrails (PII redaction, topical constraints).
- LLM-judge hallucination detection.
- Benchmark from Comet's own measurement: Opik logs + evals in 23.44s vs Langfuse 327.15s (~14x gap), take vendor benchmarks as directional.
- Strongest for: optimization loop, automated experimentation, guardrail enforcement.
Industry data
Per Maxim (2026 field analysis): 89% of organizations have agent observability in place; quality issues are the top production barrier (32% of respondents cite them).
Picking one
| Need | Pick |
|---|---|
| All-in-one with prompt management | Langfuse |
| Deep RAG evaluation + drift | Phoenix |
| Automated optimization + guardrails | Opik |
| Open licensing, no ELv2 | Langfuse (MIT) or Opik (Apache 2.0) |
| Datadog / New Relic integration | Any, they all export OTel |
Where this pattern goes wrong
- No eval strategy. Tracing without evaluation is just expensive logging.
- Self-rolled LLM-judge without grounding. CRITIC pattern (Lesson 05) applies, judges need external tools for factual verification.
- Prompt versions not tied to traces. When prod regresses, you cannot bisect to the prompt that caused it.
Build It
code/main.py implements a stdlib trace collector + LLM-judge evaluator:
- Ingest GenAI-shaped spans.
- Group by session, tag failed runs (guardrail trips, low-confidence evals).
- A scripted LLM-judge that scores agent responses on a rubric.
- A dashboard-like summary: failure rate, top failure reasons, eval score distribution.
Run it:
python3 code/main.pyOutput: per-session eval scores and failure categorization matching what Langfuse/Phoenix/Opik would show.
Use It
- Langfuse self-hosted or cloud; wire via OTel or their SDK.
- Arize Phoenix self-hosted; auto-instrument OpenInference.
- Comet Opik self-hosted or cloud; automated optimization loop.
- Datadog LLM Observability for mixed ops+ML teams that already run Datadog.
Ship It
outputs/skill-obs-platform-wiring.md picks a platform and wires traces + evals + prompt versions into an existing agent.
Exercises
- Export a week of OTel traces to Langfuse cloud (free tier). Which sessions failed? Why?
- Write an LLM-judge rubric for your domain (factual correctness, tone, scope adherence). Test on 50 traces.
- Compare Langfuse prompt versioning against Phoenix's trace clustering. Which tells you what broke faster?
- Read Opik's guardrail docs. Wire a PII redaction guardrail to one of your agent runs.
- Benchmark the three on your corpus. Ignore vendor-published numbers; measure your own.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Tracing | "Spans collector" | Ingest OTel / SDK spans; index by session |
| Prompt management | "Prompt CMS" | Versioned prompts tied to traces |
| LLM-as-judge | "Automated eval" | Separate LLM scores agent output against a rubric |
| Session replay | "Trace playback" | Step through past runs for debugging |
| RAG relevancy | "Retrieval quality" | Does the retrieved context match the query |
| Trace clustering | "Behavioral grouping" | Cluster similar runs for drift detection |
| Guardrail enforcement | "Policy at log time" | PII/toxicity/scope checks on logged content |
Further Reading
- Langfuse docstracing, evals, prompt mgmt
- Arize Phoenix docsauto-instrumentation, drift
- Comet Opikoptimization + guardrails
- OpenTelemetry GenAI semantic conventionsthe schema all three consume
Check what you learned
7 questions. Answer each one to see why.
What license does Langfuse ship under after the June 2025 open-sourcing of formerly commercial modules?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Stdlib trace collector + LLM-judge evaluator.
Mirrors what Langfuse / Phoenix / Opik do with richer UIs: ingest spans,
group by session, score with an LLM judge, surface failure categories.
"""
from __future__ import annotations
from collections import Counter
from dataclasses import dataclass, field
from typing import Any, Callable
@dataclass
class SpanEvent:
trace_id: str
session_id: str
name: str
status: str = "ok"
attributes: dict[str, Any] = field(default_factory=dict)
@dataclass
class SessionSummary:
session_id: str
trace_count: int
error_count: int
eval_score_mean: float
failure_reasons: Counter
class TraceCollector:
def __init__(self) -> None:
self.spans: list[SpanEvent] = []
def ingest(self, span: SpanEvent) -> None:
self.spans.append(span)
def by_session(self) -> dict[str, list[SpanEvent]]:
result: dict[str, list[SpanEvent]] = {}
for span in self.spans:
result.setdefault(span.session_id, []).append(span)
return result
def scripted_llm_judge(session_spans: list[SpanEvent]) -> tuple[float, str]:
errors = sum(1 for s in session_spans if s.status == "error")
has_tool = any(s.name.startswith("tool_call") for s in session_spans)
has_final = any(s.attributes.get("gen_ai.output.reference_id")
for s in session_spans)
tokens_over = any(s.attributes.get("tokens", 0) > 2000 for s in session_spans)
score = 1.0
if errors:
score -= 0.4
if not has_final:
score -= 0.3
if not has_tool:
score -= 0.1
if tokens_over:
score -= 0.1
score = max(0.0, score)
if score >= 0.8:
verdict = "PASS"
elif score >= 0.5:
verdict = "WARN"
else:
verdict = "FAIL"
return score, verdict
def categorize_failures(session_spans: list[SpanEvent]) -> Counter:
reasons: Counter = Counter()
for span in session_spans:
if span.status != "error":
continue
reason = span.attributes.get("error.reason", "unknown")
reasons[reason] += 1
return reasons
def summarize(collector: TraceCollector) -> list[SessionSummary]:
summaries: list[SessionSummary] = []
for session_id, spans in collector.by_session().items():
score, _ = scripted_llm_judge(spans)
summaries.append(SessionSummary(
session_id=session_id,
trace_count=len(spans),
error_count=sum(1 for s in spans if s.status == "error"),
eval_score_mean=score,
failure_reasons=categorize_failures(spans),
))
summaries.sort(key=lambda s: s.eval_score_mean)
return summaries
def main() -> None:
print("=" * 70)
print("AGENT OBSERVABILITY PLATFORMS — Phase 14, Lesson 24")
print("=" * 70)
collector = TraceCollector()
ok_spans = [
SpanEvent("t001", "s001", "invoke_agent",
attributes={"gen_ai.provider.name": "anthropic"}),
SpanEvent("t001", "s001", "chat",
attributes={"gen_ai.output.reference_id": "c001",
"tokens": 800}),
SpanEvent("t001", "s001", "tool_call search_tool",
attributes={"gen_ai.tool.name": "search_tool"}),
SpanEvent("t001", "s001", "chat",
attributes={"gen_ai.output.reference_id": "c002",
"tokens": 400}),
]
err_spans = [
SpanEvent("t002", "s002", "invoke_agent",
attributes={"gen_ai.provider.name": "anthropic"}),
SpanEvent("t002", "s002", "chat", status="error",
attributes={"error.reason": "rate_limited",
"tokens": 0}),
]
slow_spans = [
SpanEvent("t003", "s003", "invoke_agent",
attributes={"gen_ai.provider.name": "openai"}),
SpanEvent("t003", "s003", "chat",
attributes={"gen_ai.output.reference_id": "c003",
"tokens": 2500}),
]
for span in ok_spans + err_spans + slow_spans:
collector.ingest(span)
print("\nsummary per session (what Langfuse/Phoenix/Opik show)")
for summary in summarize(collector):
score, verdict = scripted_llm_judge(collector.by_session()[summary.session_id])
print(f" {summary.session_id} verdict={verdict} score={score:.2f} "
f"spans={summary.trace_count} errors={summary.error_count}")
if summary.failure_reasons:
for reason, count in summary.failure_reasons.most_common():
print(f" failure: {reason} x{count}")
total_errors = sum(s.error_count for s in summarize(collector))
total_sessions = len(collector.by_session())
print(f"\naggregate: {total_errors} errors across {total_sessions} sessions")
print()
print("Langfuse: prompt versions tied to traces.")
print("Phoenix: RAG relevancy + drift/clustering.")
print("Opik: optimization + guardrail enforcement.")
if __name__ == "__main__":
main()