Get Started

Agent Observability: Langfuse, Phoenix, Opik

Three open-source agent observability platforms dominate 2026. Langfuse (MIT), 6M+ installs/month, tracing + prompt management + evals + session replay. Arize Phoenix (Elastic 2.0), deep agent-specific evals, RAG relevancy, OpenInference auto-instrumentation. Comet Opik (Apache 2.0), automated prompt optimization, guardrails, LLM-judge hallucination detection.

45 minutesLearnPython (stdlib)Free lesson

Learning Objectives

  • Name the three top open-source agent observability platforms and their licenses.
  • Distinguish what each one is strongest at: Langfuse (prompt mgmt + sessions), Phoenix (RAG + auto-instrumentation), Opik (optimization + guardrails).
  • Explain why 89% of organizations report having agent observability in place by 2026.
  • Implement a stdlib trace-to-dashboard pipeline with LLM-judge evaluation.

The Problem

OTel GenAI (Lesson 23) gives you the schema. You still need the platform that ingests spans, runs evaluations, stores prompt versions, and surfaces regressions. The three contenders each emphasize different parts of the lifecycle.

The Concept

Langfuse (MIT)

  • 6M+ SDK installs/month, 19k+ GitHub stars.
  • Features: tracing, prompt management with versioning + playground, evaluations (LLM-as-judge, user feedback, custom), session replays.
  • June 2025: formerly commercial modules (LLM-as-a-judge, annotation queues, prompt experiments, Playground) open-sourced under MIT.
  • Strongest for: end-to-end observability with tight prompt-management loop.

Arize Phoenix (Elastic License 2.0)

  • Deeper agent-specific evaluation: trace clustering, anomaly detection, retrieval relevancy for RAG.
  • Native OpenInference auto-instrumentation.
  • Pairs with managed Arize AX for production.
  • No prompt versioning, positioned as a drift/behavioral-regression tool alongside broader platforms.
  • Strongest for: RAG relevancy, behavioral drift, anomaly detection.

Comet Opik (Apache 2.0)

  • Automated prompt optimization through A/B experiments.
  • Guardrails (PII redaction, topical constraints).
  • LLM-judge hallucination detection.
  • Benchmark from Comet's own measurement: Opik logs + evals in 23.44s vs Langfuse 327.15s (~14x gap), take vendor benchmarks as directional.
  • Strongest for: optimization loop, automated experimentation, guardrail enforcement.

Industry data

Per Maxim (2026 field analysis): 89% of organizations have agent observability in place; quality issues are the top production barrier (32% of respondents cite them).

Picking one

NeedPick
All-in-one with prompt managementLangfuse
Deep RAG evaluation + driftPhoenix
Automated optimization + guardrailsOpik
Open licensing, no ELv2Langfuse (MIT) or Opik (Apache 2.0)
Datadog / New Relic integrationAny, they all export OTel

Where this pattern goes wrong

  • No eval strategy. Tracing without evaluation is just expensive logging.
  • Self-rolled LLM-judge without grounding. CRITIC pattern (Lesson 05) applies, judges need external tools for factual verification.
  • Prompt versions not tied to traces. When prod regresses, you cannot bisect to the prompt that caused it.

Build It

code/main.py implements a stdlib trace collector + LLM-judge evaluator:

  • Ingest GenAI-shaped spans.
  • Group by session, tag failed runs (guardrail trips, low-confidence evals).
  • A scripted LLM-judge that scores agent responses on a rubric.
  • A dashboard-like summary: failure rate, top failure reasons, eval score distribution.

Run it:

Code
python3 code/main.py

Output: per-session eval scores and failure categorization matching what Langfuse/Phoenix/Opik would show.

Use It

  • Langfuse self-hosted or cloud; wire via OTel or their SDK.
  • Arize Phoenix self-hosted; auto-instrument OpenInference.
  • Comet Opik self-hosted or cloud; automated optimization loop.
  • Datadog LLM Observability for mixed ops+ML teams that already run Datadog.

Ship It

outputs/skill-obs-platform-wiring.md picks a platform and wires traces + evals + prompt versions into an existing agent.

Exercises

  1. Export a week of OTel traces to Langfuse cloud (free tier). Which sessions failed? Why?
  2. Write an LLM-judge rubric for your domain (factual correctness, tone, scope adherence). Test on 50 traces.
  3. Compare Langfuse prompt versioning against Phoenix's trace clustering. Which tells you what broke faster?
  4. Read Opik's guardrail docs. Wire a PII redaction guardrail to one of your agent runs.
  5. Benchmark the three on your corpus. Ignore vendor-published numbers; measure your own.

Key Terms

TermWhat people sayWhat it actually means
Tracing"Spans collector"Ingest OTel / SDK spans; index by session
Prompt management"Prompt CMS"Versioned prompts tied to traces
LLM-as-judge"Automated eval"Separate LLM scores agent output against a rubric
Session replay"Trace playback"Step through past runs for debugging
RAG relevancy"Retrieval quality"Does the retrieved context match the query
Trace clustering"Behavioral grouping"Cluster similar runs for drift detection
Guardrail enforcement"Policy at log time"PII/toxicity/scope checks on logged content

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

What license does Langfuse ship under after the June 2025 open-sourcing of formerly commercial modules?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Stdlib trace collector + LLM-judge evaluator.

Mirrors what Langfuse / Phoenix / Opik do with richer UIs: ingest spans,
group by session, score with an LLM judge, surface failure categories.
"""

from __future__ import annotations

from collections import Counter
from dataclasses import dataclass, field
from typing import Any, Callable


@dataclass
class SpanEvent:
    trace_id: str
    session_id: str
    name: str
    status: str = "ok"
    attributes: dict[str, Any] = field(default_factory=dict)


@dataclass
class SessionSummary:
    session_id: str
    trace_count: int
    error_count: int
    eval_score_mean: float
    failure_reasons: Counter


class TraceCollector:
    def __init__(self) -> None:
        self.spans: list[SpanEvent] = []

    def ingest(self, span: SpanEvent) -> None:
        self.spans.append(span)

    def by_session(self) -> dict[str, list[SpanEvent]]:
        result: dict[str, list[SpanEvent]] = {}
        for span in self.spans:
            result.setdefault(span.session_id, []).append(span)
        return result


def scripted_llm_judge(session_spans: list[SpanEvent]) -> tuple[float, str]:
    errors = sum(1 for s in session_spans if s.status == "error")
    has_tool = any(s.name.startswith("tool_call") for s in session_spans)
    has_final = any(s.attributes.get("gen_ai.output.reference_id")
                    for s in session_spans)
    tokens_over = any(s.attributes.get("tokens", 0) > 2000 for s in session_spans)
    score = 1.0
    if errors:
        score -= 0.4
    if not has_final:
        score -= 0.3
    if not has_tool:
        score -= 0.1
    if tokens_over:
        score -= 0.1
    score = max(0.0, score)
    if score >= 0.8:
        verdict = "PASS"
    elif score >= 0.5:
        verdict = "WARN"
    else:
        verdict = "FAIL"
    return score, verdict


def categorize_failures(session_spans: list[SpanEvent]) -> Counter:
    reasons: Counter = Counter()
    for span in session_spans:
        if span.status != "error":
            continue
        reason = span.attributes.get("error.reason", "unknown")
        reasons[reason] += 1
    return reasons


def summarize(collector: TraceCollector) -> list[SessionSummary]:
    summaries: list[SessionSummary] = []
    for session_id, spans in collector.by_session().items():
        score, _ = scripted_llm_judge(spans)
        summaries.append(SessionSummary(
            session_id=session_id,
            trace_count=len(spans),
            error_count=sum(1 for s in spans if s.status == "error"),
            eval_score_mean=score,
            failure_reasons=categorize_failures(spans),
        ))
    summaries.sort(key=lambda s: s.eval_score_mean)
    return summaries


def main() -> None:
    print("=" * 70)
    print("AGENT OBSERVABILITY PLATFORMS — Phase 14, Lesson 24")
    print("=" * 70)

    collector = TraceCollector()

    ok_spans = [
        SpanEvent("t001", "s001", "invoke_agent",
                  attributes={"gen_ai.provider.name": "anthropic"}),
        SpanEvent("t001", "s001", "chat",
                  attributes={"gen_ai.output.reference_id": "c001",
                              "tokens": 800}),
        SpanEvent("t001", "s001", "tool_call search_tool",
                  attributes={"gen_ai.tool.name": "search_tool"}),
        SpanEvent("t001", "s001", "chat",
                  attributes={"gen_ai.output.reference_id": "c002",
                              "tokens": 400}),
    ]
    err_spans = [
        SpanEvent("t002", "s002", "invoke_agent",
                  attributes={"gen_ai.provider.name": "anthropic"}),
        SpanEvent("t002", "s002", "chat", status="error",
                  attributes={"error.reason": "rate_limited",
                              "tokens": 0}),
    ]
    slow_spans = [
        SpanEvent("t003", "s003", "invoke_agent",
                  attributes={"gen_ai.provider.name": "openai"}),
        SpanEvent("t003", "s003", "chat",
                  attributes={"gen_ai.output.reference_id": "c003",
                              "tokens": 2500}),
    ]

    for span in ok_spans + err_spans + slow_spans:
        collector.ingest(span)

    print("\nsummary per session (what Langfuse/Phoenix/Opik show)")
    for summary in summarize(collector):
        score, verdict = scripted_llm_judge(collector.by_session()[summary.session_id])
        print(f"  {summary.session_id}  verdict={verdict}  score={score:.2f}  "
              f"spans={summary.trace_count}  errors={summary.error_count}")
        if summary.failure_reasons:
            for reason, count in summary.failure_reasons.most_common():
                print(f"    failure: {reason} x{count}")

    total_errors = sum(s.error_count for s in summarize(collector))
    total_sessions = len(collector.by_session())
    print(f"\naggregate: {total_errors} errors across {total_sessions} sessions")
    print()
    print("Langfuse: prompt versions tied to traces.")
    print("Phoenix: RAG relevancy + drift/clustering.")
    print("Opik: optimization + guardrail enforcement.")


if __name__ == "__main__":
    main()