Capstone Lesson 28: Observability with OTel GenAI Spans and Prometheus Metrics
An agent harness without observability is a black box that costs money. This lesson hand-rolls a span builder that emits records compliant with the OpenTelemetry GenAI semantic conventions, writes them to a JSON-Lines file one span per line, and exposes counters and histograms in Prometheus text format. The whole thing is stdlib Python and runs offline.
Learning Objectives
- Build a span data class shaped to the OpenTelemetry GenAI semantic conventions.
- Implement a JSONL exporter that writes one self-contained span per line.
- Build counters and histograms with labels and Prometheus text-format exposition.
- Wrap any callable in a span context manager that records duration, status, and exceptions.
- Verify that the emitted spans roundtrip through
json.loadsand match the spec shape.
The Problem
A coding agent in production produces three classes of artifact every turn: a model call, a tool execution, and a verification gate decision. None of these are useful without structured telemetry.
The first failure mode is the missing trace. Something went wrong on Tuesday but the only record is a 500-line chat log. There is no record of which tool ran, how long it took, how many tokens went into the prompt, or whether the gate refused anything. The agent author has to guess.
The second failure mode is the unparseable trace. The harness wrote spans but used its own ad-hoc field names. Nothing in Grafana, Honeycomb, Jaeger, or the local CLI can read them. Whatever tooling exists in the team's stack is wasted because the spans are non-standard.
The third failure mode is the unaggregated metric. You can see one slow tool call in the trace, but you cannot answer "what is the p95 latency of read_file calls over the last hour?" because there are no metrics, only traces.
The OpenTelemetry GenAI semantic conventions exist exactly for this. They define a small set of standard attributes that span emitters across LLM frameworks share. If your harness writes those attributes, every OTel-compatible backend can read them.
The Concept
Every operation in the harness produces a span. A span has a trace id (the whole agent invocation), a span id (this one operation), a name (e.g. gen_ai.chatgen_ai.tool.execution), attributes that follow the GenAI conventions, a start and end time, and a status.
The GenAI conventions standardise these attribute keys: gen_ai.system (which provider, e.g. anthropicopenai), gen_ai.request.model (the model id), gen_ai.request.max_tokensgen_ai.usage.input_tokensgen_ai.usage.output_tokensgen_ai.response.modelgen_ai.response.idgen_ai.operation.nameplus tool-specific keys gen_ai.tool.name and gen_ai.tool.call.id.
The exporter writes JSONL. One JSON object per line. This is the simplest possible format that downstream tooling can stream, grep, and import. A real OTel exporter would speak OTLP gRPC; the lesson's JSONL exporter is the offline equivalent and exits zero on every workstation.
Metrics live next to traces. A counter increments on each tool call: tools_called_total{tool="read_file"}. A histogram records the observed latency: tool_latency_ms{tool="read_file"}. Both serialise into Prometheus text exposition format, which is the de-facto standard for pull-based metrics.
Architecture
The span builder is a small class with a span(name, attrs) method that returns a context manager. The context manager records start time on enter, records end time on exit, attaches an exception if one was raised, and pushes the finalised span to the exporter.
The metrics registry is two dicts. Counters are {(name, frozen_labels): int}. Histograms keep raw samples in a list and serialise to Prometheus histogram buckets at exposition time.
What you will build
main.py ships:
GenAISpandataclass: trace_id, span_id, parent_span_id, name, attributes, start_unix_nano, end_unix_nano, status, status_message, events.SpanBuilderclass withspan(name, attrs, parent=None)context manager.JSONLExporterclass withexport(span)that appends one line.CounterandHistogramclasses plusMetricsRegistry.prometheus_exposition(registry)that produces text-format output.wrap_tool_call(name)decorator that emits a span and updates metrics.- Demo: synthesises a complete agent invocation (gen_ai.chat span around tool spans), writes traces.jsonl, prints the Prometheus exposition, exits zero.
The span id and trace id are 16-byte hex strings, generated from os.urandom. That matches OTel's W3C trace context. The exporter never throws; IO errors are surfaced but the harness keeps running.
The histogram has a fixed bucket set (the OTel default for latency in milliseconds: 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, +Inf). Samples are stored as a list; exposition computes per-bucket counts on demand.
Why hand-rolled instead of opentelemetry-sdk
The OTel Python SDK is a real dependency. It is also several thousand lines of code, multiple processes for the OTLP exporter, and a runtime cost that swamps a lesson budget. The hand-rolled version teaches the wire format. In production you wire the same attributes into the real SDK and get the OTLP exporter, batching, and resource detection for free.
The conventions are stable. The wire format the lesson emits will keep parsing in 2030 because OTel never breaks GenAI attribute names; they only add new ones.
How this composes with the rest of Track A
Lesson 25 produced the gate chain. Lesson 26 produced the sandbox. Lesson 27 produced the eval harness. Lesson 28 makes all three observable. Lesson 29 wraps every step of the end-to-end demo in spans and prints the Prometheus text at the end.
Running it
cd phases/19-capstone-projects/28-observability-otel-traces
python3 code/main.py
python3 -m pytest code/tests/ -vThe demo emits a traces.jsonl in the lesson's working dir (cleaned up at the end), then prints a sample of three spans, then prints the Prometheus exposition for the counters and histograms. The tests verify that spans serialise round-trip, that the canonical GenAI attributes are present, that counters increment correctly, and that the histogram exposition contains the expected bucket counts.
Check what you learned
6 questions. Answer each one to see why.
Before reading the lesson: what is the practical benefit of writing your harness's spans in the OTel GenAI attribute shape?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""
Observability for an agent harness: GenAI spans + Prometheus metrics.
See: phases/19-capstone-projects/28-observability-otel-traces/docs/en.md
Concept refs:
- OpenTelemetry GenAI semantic conventions (gen_ai.* attribute keys).
- Prometheus text exposition format (counters and histograms).
- W3C Trace Context (16-byte trace_id, 8-byte span_id).
The demo at the bottom emits spans to a temp jsonl, prints the
Prometheus exposition, and exits zero.
"""
from __future__ import annotations
import json
import math
import os
import sys
import tempfile
import time
import uuid
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import Any, Iterator
# ---------------------------------------------------------------------------
# OTel semantic-convention keys
# ---------------------------------------------------------------------------
# Standard GenAI attributes (OpenTelemetry GenAI semantic conventions).
# These keys are stable; only new keys get added, never renamed.
GEN_AI_SYSTEM = "gen_ai.system"
GEN_AI_REQUEST_MODEL = "gen_ai.request.model"
GEN_AI_REQUEST_MAX_TOKENS = "gen_ai.request.max_tokens"
GEN_AI_REQUEST_TEMPERATURE = "gen_ai.request.temperature"
GEN_AI_USAGE_INPUT_TOKENS = "gen_ai.usage.input_tokens"
GEN_AI_USAGE_OUTPUT_TOKENS = "gen_ai.usage.output_tokens"
GEN_AI_RESPONSE_MODEL = "gen_ai.response.model"
GEN_AI_RESPONSE_ID = "gen_ai.response.id"
GEN_AI_OPERATION_NAME = "gen_ai.operation.name"
GEN_AI_TOOL_NAME = "gen_ai.tool.name"
GEN_AI_TOOL_CALL_ID = "gen_ai.tool.call.id"
GEN_AI_TOOL_RESULT_BYTES = "gen_ai.tool.result.bytes"
# Harness-specific attributes (under a non-conflicting prefix).
HARNESS_GATE_DECISION = "agent.harness.gate.decision"
HARNESS_GATE_REASON = "agent.harness.gate.reason"
STATUS_UNSET = "UNSET"
STATUS_OK = "OK"
STATUS_ERROR = "ERROR"
# ---------------------------------------------------------------------------
# Span data shape
# ---------------------------------------------------------------------------
@dataclass
class SpanEvent:
"""Discrete event recorded inside a span (per OTel)."""
name: str
timestamp_unix_nano: int
attributes: dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> dict:
return {
"name": self.name,
"timestamp_unix_nano": self.timestamp_unix_nano,
"attributes": dict(self.attributes),
}
@dataclass
class GenAISpan:
"""A single span shaped to OTel GenAI conventions."""
trace_id: str
span_id: str
name: str
start_unix_nano: int
end_unix_nano: int = 0
parent_span_id: str = ""
attributes: dict[str, Any] = field(default_factory=dict)
events: list[SpanEvent] = field(default_factory=list)
status: str = STATUS_UNSET
status_message: str = ""
kind: str = "INTERNAL"
@property
def duration_ms(self) -> float:
if self.end_unix_nano <= 0:
return 0.0
return (self.end_unix_nano - self.start_unix_nano) / 1_000_000.0
def to_dict(self) -> dict:
return {
"trace_id": self.trace_id,
"span_id": self.span_id,
"parent_span_id": self.parent_span_id,
"name": self.name,
"kind": self.kind,
"start_unix_nano": self.start_unix_nano,
"end_unix_nano": self.end_unix_nano,
"duration_ms": round(self.duration_ms, 4),
"attributes": dict(self.attributes),
"events": [e.to_dict() for e in self.events],
"status": {"code": self.status, "message": self.status_message},
}
# ---------------------------------------------------------------------------
# ID generation
# ---------------------------------------------------------------------------
def new_trace_id() -> str:
"""Random 16-byte hex string. Matches W3C trace context."""
return uuid.uuid4().hex + uuid.uuid4().hex[:0] # uuid4 hex is already 32 chars
def new_span_id() -> str:
"""Random 8-byte hex string. Matches W3C trace context span id."""
return uuid.uuid4().hex[:16]
def now_unix_nano() -> int:
return time.time_ns()
# ---------------------------------------------------------------------------
# Exporter
# ---------------------------------------------------------------------------
@dataclass
class JSONLExporter:
"""Append-only exporter: one span per JSON line."""
path: str
fh: Any = None
closed: bool = False
def _ensure_open(self) -> None:
if self.fh is None:
os.makedirs(os.path.dirname(self.path) or ".", exist_ok=True)
self.fh = open(self.path, "a", encoding="utf-8")
def export(self, span: GenAISpan) -> None:
self._ensure_open()
line = json.dumps(span.to_dict(), separators=(",", ":"))
self.fh.write(line + "\n")
self.fh.flush()
def close(self) -> None:
if self.fh is not None and not self.closed:
self.fh.close()
self.closed = True
class InMemoryExporter:
"""Test-friendly exporter that keeps spans in a list."""
def __init__(self) -> None:
self.spans: list[GenAISpan] = []
def export(self, span: GenAISpan) -> None:
self.spans.append(span)
def close(self) -> None:
return None
# ---------------------------------------------------------------------------
# Metrics primitives
# ---------------------------------------------------------------------------
def _label_key(labels: dict[str, str]) -> tuple[tuple[str, str], ...]:
return tuple(sorted(labels.items()))
def _format_labels(labels: dict[str, str]) -> str:
if not labels:
return ""
parts = []
for k, v in sorted(labels.items()):
escaped = str(v).replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
parts.append(f'{k}="{escaped}"')
return "{" + ",".join(parts) + "}"
@dataclass
class Counter:
"""A simple labelled counter."""
name: str
help: str = ""
values: dict[tuple[tuple[str, str], ...], float] = field(default_factory=dict)
def inc(self, labels: dict[str, str] | None = None, by: float = 1.0) -> None:
key = _label_key(labels or {})
self.values[key] = self.values.get(key, 0.0) + by
def get(self, labels: dict[str, str] | None = None) -> float:
return self.values.get(_label_key(labels or {}), 0.0)
@dataclass
class Histogram:
"""A histogram with explicit buckets, in line with OTel's default ms set."""
name: str
help: str = ""
buckets: tuple[float, ...] = (
5.0,
10.0,
25.0,
50.0,
100.0,
250.0,
500.0,
1000.0,
2500.0,
5000.0,
10000.0,
)
samples: dict[tuple[tuple[str, str], ...], list[float]] = field(
default_factory=dict
)
def observe(self, value: float, labels: dict[str, str] | None = None) -> None:
key = _label_key(labels or {})
self.samples.setdefault(key, []).append(float(value))
def bucket_counts(
self, labels: dict[str, str] | None = None
) -> dict[float, int]:
key = _label_key(labels or {})
values = self.samples.get(key, [])
counts: dict[float, int] = {}
for bound in self.buckets:
counts[bound] = sum(1 for v in values if v <= bound)
counts[math.inf] = len(values)
return counts
def total_count(self, labels: dict[str, str] | None = None) -> int:
return len(self.samples.get(_label_key(labels or {}), []))
def total_sum(self, labels: dict[str, str] | None = None) -> float:
return float(sum(self.samples.get(_label_key(labels or {}), [])))
@dataclass
class MetricsRegistry:
counters: dict[str, Counter] = field(default_factory=dict)
histograms: dict[str, Histogram] = field(default_factory=dict)
def counter(self, name: str, help: str = "") -> Counter:
if name not in self.counters:
self.counters[name] = Counter(name=name, help=help)
return self.counters[name]
def histogram(self, name: str, help: str = "") -> Histogram:
if name not in self.histograms:
self.histograms[name] = Histogram(name=name, help=help)
return self.histograms[name]
def prometheus_exposition(registry: MetricsRegistry) -> str:
"""Render the registry into Prometheus text exposition format."""
lines: list[str] = []
for name in sorted(registry.counters):
counter = registry.counters[name]
if counter.help:
lines.append(f"# HELP {counter.name} {counter.help}")
lines.append(f"# TYPE {counter.name} counter")
if not counter.values:
lines.append(f"{counter.name} 0")
for key, value in sorted(counter.values.items()):
label_str = _format_labels(dict(key))
lines.append(f"{counter.name}{label_str} {value}")
for name in sorted(registry.histograms):
hist = registry.histograms[name]
if hist.help:
lines.append(f"# HELP {hist.name} {hist.help}")
lines.append(f"# TYPE {hist.name} histogram")
keys = list(hist.samples.keys()) or [tuple()]
for key in sorted(keys):
label_dict = dict(key)
counts = hist.bucket_counts(label_dict)
for bound in hist.buckets:
bucket_labels = {**label_dict, "le": _format_le(bound)}
lines.append(
f"{hist.name}_bucket{_format_labels(bucket_labels)} {counts[bound]}"
)
inf_labels = {**label_dict, "le": "+Inf"}
lines.append(
f"{hist.name}_bucket{_format_labels(inf_labels)} {counts[math.inf]}"
)
lines.append(
f"{hist.name}_sum{_format_labels(label_dict)} "
f"{hist.total_sum(label_dict)}"
)
lines.append(
f"{hist.name}_count{_format_labels(label_dict)} "
f"{hist.total_count(label_dict)}"
)
return "\n".join(lines) + "\n"
def _format_le(bound: float) -> str:
if bound == int(bound):
return str(int(bound))
return repr(bound)
# ---------------------------------------------------------------------------
# Span builder
# ---------------------------------------------------------------------------
@dataclass
class SpanBuilder:
"""Owns a trace id and emits spans through one or more exporters."""
trace_id: str = field(default_factory=new_trace_id)
exporters: list[Any] = field(default_factory=list)
metrics: MetricsRegistry | None = None
@contextmanager
def span(
self,
name: str,
attributes: dict[str, Any] | None = None,
parent: GenAISpan | None = None,
kind: str = "INTERNAL",
) -> Iterator[GenAISpan]:
span = GenAISpan(
trace_id=self.trace_id,
span_id=new_span_id(),
parent_span_id=parent.span_id if parent else "",
name=name,
kind=kind,
start_unix_nano=now_unix_nano(),
attributes=dict(attributes or {}),
)
try:
yield span
span.status = STATUS_OK
except BaseException as exc: # noqa: BLE001
span.status = STATUS_ERROR
span.status_message = f"{type(exc).__name__}: {exc}"
span.events.append(
SpanEvent(
name="exception",
timestamp_unix_nano=now_unix_nano(),
attributes={
"exception.type": type(exc).__name__,
"exception.message": str(exc),
},
)
)
raise
finally:
span.end_unix_nano = now_unix_nano()
for exporter in self.exporters:
exporter.export(span)
if self.metrics is not None:
tool = span.attributes.get(GEN_AI_TOOL_NAME)
if tool is not None:
self.metrics.counter(
"tools_called_total", help="Total tool calls"
).inc({"tool": str(tool)})
self.metrics.histogram(
"tool_latency_ms",
help="Tool call latency in milliseconds",
).observe(span.duration_ms, {"tool": str(tool)})
# ---------------------------------------------------------------------------
# Demo
# ---------------------------------------------------------------------------
def run_demo() -> int:
tmp_dir = tempfile.mkdtemp(prefix="otel-demo-")
trace_path = os.path.join(tmp_dir, "traces.jsonl")
jsonl = JSONLExporter(path=trace_path)
metrics = MetricsRegistry()
in_mem = InMemoryExporter()
builder = SpanBuilder(exporters=[jsonl, in_mem], metrics=metrics)
print("OBSERVABILITY DEMO")
print(f"trace_id={builder.trace_id}")
print(f"writing traces to {trace_path}")
# Synthesize an agent turn: a gen_ai.chat span around two tool spans.
with builder.span(
"gen_ai.chat",
attributes={
GEN_AI_SYSTEM: "anthropic",
GEN_AI_REQUEST_MODEL: "claude-track-a",
GEN_AI_REQUEST_MAX_TOKENS: 1024,
GEN_AI_REQUEST_TEMPERATURE: 0.0,
GEN_AI_OPERATION_NAME: "chat",
},
kind="CLIENT",
) as chat_span:
time.sleep(0.01)
chat_span.attributes[GEN_AI_USAGE_INPUT_TOKENS] = 412
chat_span.attributes[GEN_AI_USAGE_OUTPUT_TOKENS] = 96
chat_span.attributes[GEN_AI_RESPONSE_MODEL] = "claude-track-a-2026-05-25"
chat_span.attributes[GEN_AI_RESPONSE_ID] = "msg_" + uuid.uuid4().hex[:8]
with builder.span(
"gen_ai.tool.execution",
parent=chat_span,
attributes={
GEN_AI_TOOL_NAME: "read_file",
GEN_AI_TOOL_CALL_ID: "call_001",
},
) as tool1:
time.sleep(0.005)
tool1.attributes[GEN_AI_TOOL_RESULT_BYTES] = 1024
tool1.events.append(
SpanEvent(
name="agent.harness.gate.decision",
timestamp_unix_nano=now_unix_nano(),
attributes={
HARNESS_GATE_DECISION: "ALLOW",
HARNESS_GATE_REASON: "passed gate chain",
},
)
)
with builder.span(
"gen_ai.tool.execution",
parent=chat_span,
attributes={
GEN_AI_TOOL_NAME: "run_tests",
GEN_AI_TOOL_CALL_ID: "call_002",
},
) as tool2:
time.sleep(0.003)
tool2.attributes[GEN_AI_TOOL_RESULT_BYTES] = 256
# A second turn with an intentionally denied gate to exercise an error span.
try:
with builder.span(
"gen_ai.tool.execution",
attributes={
GEN_AI_TOOL_NAME: "shell",
GEN_AI_TOOL_CALL_ID: "call_003",
},
) as bad_tool:
bad_tool.events.append(
SpanEvent(
name="agent.harness.gate.decision",
timestamp_unix_nano=now_unix_nano(),
attributes={
HARNESS_GATE_DECISION: "DENY",
HARNESS_GATE_REASON: "tool not in allow-set",
},
)
)
raise PermissionError("tool 'shell' not in allow-set")
except PermissionError:
pass
print("")
print(f"emitted {len(in_mem.spans)} span(s):")
for span in in_mem.spans:
attrs_compact = {
k: v
for k, v in span.attributes.items()
if k.startswith("gen_ai.")
}
print(
f" - {span.name:32s} dur={span.duration_ms:7.2f}ms "
f"status={span.status} keys={sorted(attrs_compact)[:3]}"
)
print("")
print("--- prometheus exposition ---")
print(prometheus_exposition(metrics))
jsonl.close()
# Sanity: roundtrip the jsonl file.
with open(trace_path, "r", encoding="utf-8") as fh:
lines = [json.loads(line) for line in fh if line.strip()]
print(f"roundtrip parsed {len(lines)} spans from {trace_path}")
return 0
if __name__ == "__main__":
sys.exit(run_demo())