OpenTelemetry GenAI Semantic Conventions
OpenTelemetry's GenAI SIG (launched April 2024) defines the standard schema for agent telemetry. Span names, attributes, and content-capture rules converge across vendors so agent traces mean the same thing in Datadog, Grafana, Jaeger, and Honeycomb.
Learning Objectives
- Name the GenAI span categories: model/client, agent, tool.
- Distinguish
invoke_agentCLIENT vs INTERNAL spans and when each applies. - List the top-level GenAI attributes: provider name, request model, data-source ID.
- Explain the content-capture contract: opt-in,
OTEL_SEMCONV_STABILITY_OPT_INexternal-reference recommendation.
The Problem
Every vendor invents their own span names. Ops teams end up building per-framework dashboards. OpenTelemetry's GenAI SIG fixes this by defining one standard the whole ecosystem targets.
The Concept
Span categories
- Model / client spans. Cover raw LLM calls. Emitted by provider SDKs (Anthropic, OpenAI, Bedrock) and framework model adapters.
- Agent spans.
create_agent(when the agent is constructed) andinvoke_agent(when it runs). - Tool spans. One per tool invocation; connected to the agent span by parent-child relation.
Agent span naming
- Span name:
invoke_agent {gen_ai.agent.name}if named; fallback toinvoke_agent. Span kind:
- CLIENTfor remote agent services (OpenAI Assistants API, Bedrock Agents).
- INTERNALfor in-process agent frameworks (LangChain, CrewAI, local ReAct).
Key attributes
gen_ai.provider.nameanthropicopenaiaws.bedrockgoogle.vertex.gen_ai.request.modelthe model ID.gen_ai.response.modelthe resolved model (may differ from request due to routing).gen_ai.agent.nameagent identifier.gen_ai.operation.namechatcompletioninvoke_agenttool_call.gen_ai.data_source.idfor RAG: which corpus or store was consulted.
Technology-specific conventions exist for Anthropic, Azure AI Inference, AWS Bedrock, OpenAI.
Content capture
The default rule: instrumentations SHOULD NOT capture inputs/outputs by default. Capture is opt-in via:
gen_ai.system_instructionsgen_ai.input.messagesgen_ai.output.messages
Recommended production pattern: store content externally (S3, your log store), record references on spans (pointer IDs, not prose). This is the Lesson 27 content-poisoning defense wired into observability.
Stability
Most conventions are experimental as of March 2026. Opt in to the stable preview with:
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimentalDatadog v1.37+ maps GenAI attributes natively into its LLM Observability schema. Other backends (Grafana, Honeycomb, Jaeger) support the raw attributes.
Where this pattern goes wrong
- Capturing full prompts in spans. PII, secrets, customer data in traces that ops can read. Store externally.
- No
gen_ai.provider.name. Multi-provider dashboards break when attribution is missing. - Spans without parent links. Orphaned tool spans. Always propagate context.
- Not setting stability opt-in. Your attributes may get renamed on backend upgrade.
Build It
code/main.py implements a stdlib span emitter matching GenAI conventions:
Spanwith GenAI attribute schema.Tracerwithstart_spannested contexts.- A scripted agent run that emits:
create_agentinvoke_agent(INTERNAL), per-tool spans,chatspans for LLM calls. - A content-capture mode that stores prompts externally and records IDs on spans.
Run it:
python3 code/main.pyOutput: a span tree with all required GenAI attributes, and an "external store" showing the opt-in content references.
Use It
- Datadog LLM Observability (v1.37+) maps attributes natively.
- Langfuse / Phoenix / Opik (Lesson 24), auto-instrument the ecosystem.
- Jaeger / Honeycomb / Grafana Temporaw OTel traces; build dashboards from GenAI attributes.
- Self-hostedrun the OTel Collector with a GenAI processor.
Ship It
outputs/skill-otel-genai.md wires OTel GenAI spans into an existing agent with content-capture defaults and external-reference storage.
Exercises
- Instrument your Lesson 01 ReAct loop with
invoke_agent(INTERNAL) + per-tool spans. Send to a Jaeger instance. - Add content capture in "references only" mode: prompts to SQLite, span attributes carry only row IDs.
- Read the spec for
gen_ai.data_source.id. Wire it into your Lesson 09 Mem0 search. - Set
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimentaland verify your attributes don't get renamed by the collector. - Build a dashboard: "which tool errors correlate with which models" from GenAI attributes alone.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| GenAI SIG | "OpenTelemetry GenAI group" | OTel working group defining the schema |
| invoke_agent | "Agent span" | Name of the span representing an agent run |
| CLIENT span | "Remote call" | Span for a call to a remote agent service |
| INTERNAL span | "In-process" | Span for an in-process agent run |
| gen_ai.provider.name | "Provider" | anthropic / openai / aws.bedrock / google.vertex |
| gen_ai.data_source.id | "RAG source" | Which corpus/store a retrieval hit |
| Content capture | "Prompt logging" | Opt-in capture of messages; store externally in prod |
| Stability opt-in | "Preview mode" | Env var to pin experimental conventions |
Further Reading
- OpenTelemetry GenAI semantic conventionsthe spec
- OpenAI Agents SDKGenAI spans by default
- AutoGen v0.4 (Microsoft Research)OTel spans built in
- Claude Agent SDKW3C trace context propagation
Check what you learned
7 questions. Answer each one to see why.
Which span name represents an agent run in the GenAI conventions?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Stdlib span emitter matching OpenTelemetry GenAI semantic conventions.
Emits invoke_agent INTERNAL spans, per-tool spans, chat spans for LLM calls.
Content capture is opt-in: prompts go to an external store, spans carry IDs.
"""
from __future__ import annotations
import time
from dataclasses import dataclass, field
from typing import Any
@dataclass
class Span:
name: str
kind: str = "INTERNAL"
attributes: dict[str, Any] = field(default_factory=dict)
children: list["Span"] = field(default_factory=list)
start_ns: int = 0
end_ns: int = 0
@property
def duration_ms(self) -> float:
return (self.end_ns - self.start_ns) / 1_000_000
class ExternalContentStore:
def __init__(self) -> None:
self._store: dict[str, str] = {}
self._counter = 0
def put(self, content: str) -> str:
self._counter += 1
cid = f"content_{self._counter:03d}"
self._store[cid] = content
return cid
def get(self, cid: str) -> str:
return self._store.get(cid, "")
def items(self) -> list[tuple[str, str]]:
return sorted(self._store.items())
class Tracer:
def __init__(self, capture_inline: bool = False,
content_store: ExternalContentStore | None = None) -> None:
self.root = Span(name="__root__")
self.stack: list[Span] = [self.root]
self.capture_inline = capture_inline
self.content_store = content_store or ExternalContentStore()
def start_span(self, name: str, kind: str = "INTERNAL",
attributes: dict[str, Any] | None = None) -> Span:
span = Span(name=name, kind=kind, attributes=dict(attributes or {}),
start_ns=time.perf_counter_ns())
self.stack[-1].children.append(span)
self.stack.append(span)
return span
def end_span(self) -> None:
span = self.stack.pop()
span.end_ns = time.perf_counter_ns()
def add_content(self, span: Span, key: str, content: str) -> None:
if self.capture_inline:
span.attributes[key] = content[:200]
return
cid = self.content_store.put(content)
span.attributes[f"{key}.reference_id"] = cid
def _scripted_llm(prompt: str) -> str:
if "search" in prompt.lower():
return "search_tool(\"agent engineering\")"
if "result" in prompt.lower():
return "found 3 sources; drafting answer"
return "final answer: agents in 2026"
def _search_tool(query: str) -> str:
return f"[3 sources for {query!r}]"
def main() -> None:
print("=" * 70)
print("OTEL GENAI SEMANTIC CONVENTIONS — Phase 14, Lesson 23")
print("=" * 70)
tracer = Tracer(capture_inline=False)
create_agent = tracer.start_span(
"create_agent research_bot",
attributes={
"gen_ai.agent.name": "research_bot",
"gen_ai.operation.name": "create_agent",
"gen_ai.provider.name": "anthropic",
},
)
tracer.end_span()
invoke = tracer.start_span(
"invoke_agent research_bot",
attributes={
"gen_ai.agent.name": "research_bot",
"gen_ai.operation.name": "invoke_agent",
"gen_ai.provider.name": "anthropic",
"gen_ai.request.model": "claude-opus-4-6",
},
)
for turn in range(3):
chat = tracer.start_span(
"chat",
attributes={
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "anthropic",
"gen_ai.request.model": "claude-opus-4-6",
"gen_ai.response.model": "claude-opus-4-6",
},
)
prompt = f"turn {turn}: next action please"
tracer.add_content(chat, "gen_ai.input.messages", prompt)
output = _scripted_llm(prompt)
tracer.add_content(chat, "gen_ai.output.messages", output)
tracer.end_span()
if "search_tool" in output:
tool_span = tracer.start_span(
"tool_call search_tool",
attributes={
"gen_ai.operation.name": "tool_call",
"gen_ai.tool.name": "search_tool",
"gen_ai.data_source.id": "corpus://mem0/default",
},
)
result = _search_tool("agent engineering")
tracer.add_content(tool_span, "gen_ai.tool.result", result)
tracer.end_span()
tracer.end_span()
def render(span: Span, indent: int = 0) -> None:
if span.name == "__root__":
for child in span.children:
render(child, indent)
return
pad = " " * indent
dur = f"{span.duration_ms:.2f}ms" if span.end_ns else "..."
print(f"{pad}{span.name} [{span.kind}] {dur}")
for key in sorted(span.attributes):
val = span.attributes[key]
if isinstance(val, str) and len(val) > 50:
val = val[:50] + "..."
print(f"{pad} {key} = {val!r}")
for child in span.children:
render(child, indent + 1)
print("\nspan tree (GenAI-shaped)")
render(tracer.root)
print("\nexternal content store (opt-in references, not inline)")
for cid, content in tracer.content_store.items():
print(f" {cid}: {content[:60]}")
print()
print("content NOT captured inline by default. store externally; span")
print("attributes carry reference IDs. set OTEL_SEMCONV_STABILITY_OPT_IN")
print("=gen_ai_latest_experimental to pin experimental attribute names.")
if __name__ == "__main__":
main()