Get Started

OpenTelemetry GenAI Semantic Conventions

OpenTelemetry's GenAI SIG (launched April 2024) defines the standard schema for agent telemetry. Span names, attributes, and content-capture rules converge across vendors so agent traces mean the same thing in Datadog, Grafana, Jaeger, and Honeycomb.

60 minutesLearn + BuildPython (stdlib)Free lesson

Learning Objectives

  • Name the GenAI span categories: model/client, agent, tool.
  • Distinguish invoke_agent CLIENT vs INTERNAL spans and when each applies.
  • List the top-level GenAI attributes: provider name, request model, data-source ID.
  • Explain the content-capture contract: opt-in, OTEL_SEMCONV_STABILITY_OPT_INexternal-reference recommendation.

The Problem

Every vendor invents their own span names. Ops teams end up building per-framework dashboards. OpenTelemetry's GenAI SIG fixes this by defining one standard the whole ecosystem targets.

The Concept

Span categories

  1. Model / client spans. Cover raw LLM calls. Emitted by provider SDKs (Anthropic, OpenAI, Bedrock) and framework model adapters.
  2. Agent spans. create_agent (when the agent is constructed) and invoke_agent (when it runs).
  3. Tool spans. One per tool invocation; connected to the agent span by parent-child relation.

Agent span naming

  • Span name: invoke_agent {gen_ai.agent.name} if named; fallback to invoke_agent.
  • Span kind:

    • CLIENTfor remote agent services (OpenAI Assistants API, Bedrock Agents).
    • INTERNALfor in-process agent frameworks (LangChain, CrewAI, local ReAct).

Key attributes

  • gen_ai.provider.nameanthropicopenaiaws.bedrockgoogle.vertex.
  • gen_ai.request.modelthe model ID.
  • gen_ai.response.modelthe resolved model (may differ from request due to routing).
  • gen_ai.agent.nameagent identifier.
  • gen_ai.operation.namechatcompletioninvoke_agenttool_call.
  • gen_ai.data_source.idfor RAG: which corpus or store was consulted.

Technology-specific conventions exist for Anthropic, Azure AI Inference, AWS Bedrock, OpenAI.

Content capture

The default rule: instrumentations SHOULD NOT capture inputs/outputs by default. Capture is opt-in via:

  • gen_ai.system_instructions
  • gen_ai.input.messages
  • gen_ai.output.messages

Recommended production pattern: store content externally (S3, your log store), record references on spans (pointer IDs, not prose). This is the Lesson 27 content-poisoning defense wired into observability.

Stability

Most conventions are experimental as of March 2026. Opt in to the stable preview with:

Code
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental

Datadog v1.37+ maps GenAI attributes natively into its LLM Observability schema. Other backends (Grafana, Honeycomb, Jaeger) support the raw attributes.

Where this pattern goes wrong

  • Capturing full prompts in spans. PII, secrets, customer data in traces that ops can read. Store externally.
  • No gen_ai.provider.name. Multi-provider dashboards break when attribution is missing.
  • Spans without parent links. Orphaned tool spans. Always propagate context.
  • Not setting stability opt-in. Your attributes may get renamed on backend upgrade.

Build It

code/main.py implements a stdlib span emitter matching GenAI conventions:

  • Span with GenAI attribute schema.
  • Tracer with start_spannested contexts.
  • A scripted agent run that emits: create_agentinvoke_agent (INTERNAL), per-tool spans, chat spans for LLM calls.
  • A content-capture mode that stores prompts externally and records IDs on spans.

Run it:

Code
python3 code/main.py

Output: a span tree with all required GenAI attributes, and an "external store" showing the opt-in content references.

Use It

  • Datadog LLM Observability (v1.37+) maps attributes natively.
  • Langfuse / Phoenix / Opik (Lesson 24), auto-instrument the ecosystem.
  • Jaeger / Honeycomb / Grafana Temporaw OTel traces; build dashboards from GenAI attributes.
  • Self-hostedrun the OTel Collector with a GenAI processor.

Ship It

outputs/skill-otel-genai.md wires OTel GenAI spans into an existing agent with content-capture defaults and external-reference storage.

Exercises

  1. Instrument your Lesson 01 ReAct loop with invoke_agent (INTERNAL) + per-tool spans. Send to a Jaeger instance.
  2. Add content capture in "references only" mode: prompts to SQLite, span attributes carry only row IDs.
  3. Read the spec for gen_ai.data_source.id. Wire it into your Lesson 09 Mem0 search.
  4. Set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental and verify your attributes don't get renamed by the collector.
  5. Build a dashboard: "which tool errors correlate with which models" from GenAI attributes alone.

Key Terms

TermWhat people sayWhat it actually means
GenAI SIG"OpenTelemetry GenAI group"OTel working group defining the schema
invoke_agent"Agent span"Name of the span representing an agent run
CLIENT span"Remote call"Span for a call to a remote agent service
INTERNAL span"In-process"Span for an in-process agent run
gen_ai.provider.name"Provider"anthropic / openai / aws.bedrock / google.vertex
gen_ai.data_source.id"RAG source"Which corpus/store a retrieval hit
Content capture"Prompt logging"Opt-in capture of messages; store externally in prod
Stability opt-in"Preview mode"Env var to pin experimental conventions

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

Which span name represents an agent run in the GenAI conventions?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Stdlib span emitter matching OpenTelemetry GenAI semantic conventions.

Emits invoke_agent INTERNAL spans, per-tool spans, chat spans for LLM calls.
Content capture is opt-in: prompts go to an external store, spans carry IDs.
"""

from __future__ import annotations

import time
from dataclasses import dataclass, field
from typing import Any


@dataclass
class Span:
    name: str
    kind: str = "INTERNAL"
    attributes: dict[str, Any] = field(default_factory=dict)
    children: list["Span"] = field(default_factory=list)
    start_ns: int = 0
    end_ns: int = 0

    @property
    def duration_ms(self) -> float:
        return (self.end_ns - self.start_ns) / 1_000_000


class ExternalContentStore:
    def __init__(self) -> None:
        self._store: dict[str, str] = {}
        self._counter = 0

    def put(self, content: str) -> str:
        self._counter += 1
        cid = f"content_{self._counter:03d}"
        self._store[cid] = content
        return cid

    def get(self, cid: str) -> str:
        return self._store.get(cid, "")

    def items(self) -> list[tuple[str, str]]:
        return sorted(self._store.items())


class Tracer:
    def __init__(self, capture_inline: bool = False,
                 content_store: ExternalContentStore | None = None) -> None:
        self.root = Span(name="__root__")
        self.stack: list[Span] = [self.root]
        self.capture_inline = capture_inline
        self.content_store = content_store or ExternalContentStore()

    def start_span(self, name: str, kind: str = "INTERNAL",
                   attributes: dict[str, Any] | None = None) -> Span:
        span = Span(name=name, kind=kind, attributes=dict(attributes or {}),
                    start_ns=time.perf_counter_ns())
        self.stack[-1].children.append(span)
        self.stack.append(span)
        return span

    def end_span(self) -> None:
        span = self.stack.pop()
        span.end_ns = time.perf_counter_ns()

    def add_content(self, span: Span, key: str, content: str) -> None:
        if self.capture_inline:
            span.attributes[key] = content[:200]
            return
        cid = self.content_store.put(content)
        span.attributes[f"{key}.reference_id"] = cid


def _scripted_llm(prompt: str) -> str:
    if "search" in prompt.lower():
        return "search_tool(\"agent engineering\")"
    if "result" in prompt.lower():
        return "found 3 sources; drafting answer"
    return "final answer: agents in 2026"


def _search_tool(query: str) -> str:
    return f"[3 sources for {query!r}]"


def main() -> None:
    print("=" * 70)
    print("OTEL GENAI SEMANTIC CONVENTIONS — Phase 14, Lesson 23")
    print("=" * 70)

    tracer = Tracer(capture_inline=False)

    create_agent = tracer.start_span(
        "create_agent research_bot",
        attributes={
            "gen_ai.agent.name": "research_bot",
            "gen_ai.operation.name": "create_agent",
            "gen_ai.provider.name": "anthropic",
        },
    )
    tracer.end_span()

    invoke = tracer.start_span(
        "invoke_agent research_bot",
        attributes={
            "gen_ai.agent.name": "research_bot",
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.provider.name": "anthropic",
            "gen_ai.request.model": "claude-opus-4-6",
        },
    )

    for turn in range(3):
        chat = tracer.start_span(
            "chat",
            attributes={
                "gen_ai.operation.name": "chat",
                "gen_ai.provider.name": "anthropic",
                "gen_ai.request.model": "claude-opus-4-6",
                "gen_ai.response.model": "claude-opus-4-6",
            },
        )
        prompt = f"turn {turn}: next action please"
        tracer.add_content(chat, "gen_ai.input.messages", prompt)
        output = _scripted_llm(prompt)
        tracer.add_content(chat, "gen_ai.output.messages", output)
        tracer.end_span()

        if "search_tool" in output:
            tool_span = tracer.start_span(
                "tool_call search_tool",
                attributes={
                    "gen_ai.operation.name": "tool_call",
                    "gen_ai.tool.name": "search_tool",
                    "gen_ai.data_source.id": "corpus://mem0/default",
                },
            )
            result = _search_tool("agent engineering")
            tracer.add_content(tool_span, "gen_ai.tool.result", result)
            tracer.end_span()

    tracer.end_span()

    def render(span: Span, indent: int = 0) -> None:
        if span.name == "__root__":
            for child in span.children:
                render(child, indent)
            return
        pad = "  " * indent
        dur = f"{span.duration_ms:.2f}ms" if span.end_ns else "..."
        print(f"{pad}{span.name}  [{span.kind}]  {dur}")
        for key in sorted(span.attributes):
            val = span.attributes[key]
            if isinstance(val, str) and len(val) > 50:
                val = val[:50] + "..."
            print(f"{pad}  {key} = {val!r}")
        for child in span.children:
            render(child, indent + 1)

    print("\nspan tree (GenAI-shaped)")
    render(tracer.root)

    print("\nexternal content store (opt-in references, not inline)")
    for cid, content in tracer.content_store.items():
        print(f"  {cid}: {content[:60]}")

    print()
    print("content NOT captured inline by default. store externally; span")
    print("attributes carry reference IDs. set OTEL_SEMCONV_STABILITY_OPT_IN")
    print("=gen_ai_latest_experimental to pin experimental attribute names.")


if __name__ == "__main__":
    main()