Capstone 06, DevOps Troubleshooting Agent for Kubernetes
AWS's DevOps Agent went GA, Resolve AI published its K8s playbooks, NeuBird demoed semantic monitoring, and Metoro tied AI SRE to per-service SLOs. The production shape is settled: an alert webhook fires, an agent reads telemetry, walks a graph of K8s objects, ranks root-cause hypotheses, and posts a Slack brief with approval buttons. Read-only by default. Every remediation gated by a human. This capstone is that agent, evaluated on 20 synthetic incidents and compared against AWS's Agent on three shared cases.
Problem
The 2025-2026 SRE narrative became: "AI agents triage incidents, humans approve remediations." AWS DevOps Agent, Resolve AI, NeuBird, Metoro, PagerDuty AIOps all ship this shape in production. The agent reads Prometheus metrics, Loki logs, Tempo traces, kube-state-metrics, and a knowledge graph of K8s objects. It produces a ranked root-cause hypothesis with telemetry citations in under five minutes. It never executes destructive commands without explicit human approval through Slack.
Most of the hard work is scoping and safety, not reasoning. The agent needs a read-only-by-default RBAC surface, a hardened MCP tool server, and audit logs of every command considered vs executed. It needs to know when it is outside its depth and escalate. And it has to run cheap enough that OOM-kill cascades do not generate a $5k agent bill.
Concept
The agent operates on a knowledge graph. Nodes are K8s objects (Pods, Deployments, Services, Nodes, HPAs, PVCs) plus telemetry sources (Prometheus series, Loki streams, Tempo traces). Edges encode ownership (Pod -> ReplicaSet -> Deployment), scheduling (Pod -> Node), and observation (Pod -> Prometheus series). The graph is kept fresh by a kube-state-metrics sync and re-sampled on every alert.
When an alert fires, the agent root-causes from the affected object. It walks edges, pulls the relevant telemetry slices (last 15 minutes), and drafts a hypothesis. The hypothesis is ranked by evidence: how many telemetry citations support it, how recent, how specific. The top-3 hypotheses go to Slack with graph-path visualizations and approval buttons for remediation actions.
Remediation is gated. Allowed default actions are read-only. Destructive actions (scaling down, rolling back, deleting Pods) require Slack approval; ArgoCD rollback hooks require an auth token the agent never holds. The audit log records every command the agent considerednot just executed, so the review process catches near-misses.
Architecture
PagerDuty / Alertmanager webhook
|
v
FastAPI receiver
|
v
LangGraph root-cause agent
|
+---- read-only MCP tools ----+
| |
v v
K8s knowledge graph telemetry slices
(Neo4j / kuzu) Prometheus, Loki, Tempo
ownership + scheduling last 15m, scoped
|
v
hypothesis ranking (evidence weight)
|
v
Slack brief + approval buttons
|
v (approved)
ArgoCD rollback hook / PagerDuty escalate
|
v
audit log: considered vs executed, every commandStack
- Observability sources: Prometheus, Loki, Tempo, kube-state-metrics
- Knowledge graph: Neo4j (managed) or kuzu (embedded) of K8s objects + telemetry edges
- Agent: LangGraph with per-tool allow-list, read-only by default
- Tool transport: FastMCP over StreamableHTTP; separate server for destructive tools behind approval gate
- Models: Claude Sonnet 4.7 for root-cause reasoning, Gemini 2.5 Flash for log summarization
- Remediation: ArgoCD rollback webhook, PagerDuty escalate, Slack approval card
- Audit: append-only structured log (considered, executed, approved, outcome)
- Deployment: K8s deployment with its own narrow RBAC role; separate namespace
Build It
- Graph ingestion. Sync kube-state-metrics into Neo4j/kuzu every 30s. Nodes: Pod, Deployment, Node, Service, PVC, HPA. Edges: OWNED_BY, SCHEDULED_ON, EXPOSES, MOUNTS, SCALES. Telemetry overlay edges: OBSERVED_BY (a Pod is observed by a Prometheus series).
- Alert receiver. FastAPI endpoint that accepts PagerDuty or Alertmanager webhooks. Extract the affected object(s) and SLO breach.
- Read-only tool surface. Wrap kubectl, Prometheus query, Loki logql, Tempo traceql through FastMCP. Every tool has a narrow RBAC verb ("get", "list", "describe"). No "delete", "exec", "scale" in the default server.
- Root-cause agent. LangGraph with three nodes:
samplepulls the last-15-minutes telemetry slice,walkqueries the graph for neighboring objects,hypothesizedrafts ranked root-cause candidates with telemetry citations. - Evidence scoring. Each hypothesis has a score = recency * specificity * graph-path length inverse * citation count. Return top-3.
- Slack brief. Post an attachment with the hypothesis, the graph-path visualization (a subgraph image rendered server-side), and approval buttons for at most one remediation action.
- Remediation gate. Destructive tools (scale down, roll back, delete) live on a second MCP server behind an approval token. The agent can call them only after the Slack card is approved by a human.
- Audit log. Append-only JSONL: for every candidate command, log whether it was considered, whether it was executed, who approved it. Ship to S3 daily.
- Synthetic incident suite. Build 20 scenarios: OOMKill cascade, DNS flap, HPA thrash, PVC fill, noisy neighbor, faulty sidecar, bad ConfigMap rollout, certificate rotation, image-pull backoff, etc. Score the agent on root-cause accuracy and time-to-hypothesis.
Use It
webhook: alert.pagerduty.com -> checkout-api SLO breach, error rate 14%
[graph] affected: Deployment checkout-api (3 Pods, Node ip-10-2-3-4)
[walk] neighbors: ReplicaSet checkout-api-abc, Service checkout-api,
recent rollout 14m ago
[sample] prometheus error_rate 14%, up-trend; loki 500s on /api/v2/pay
[hypo] #1 bad rollout: latest image checkout-api:v2.41 fails /healthz
citations: deploy.yaml (rev 42), prometheus errorRate, loki 500 stack
[slack] [ROLL BACK to v2.40] [ESCALATE] [IGNORE]
(approval required; agent does not roll back unilaterally)Ship It
outputs/skill-devops-agent.md is the deliverable. Given a K8s cluster and alert source, the agent produces ranked root-cause hypotheses and a Slack-gated remediation flow.
| Weight | Criterion | How it is measured |
|---|---|---|
| 25 | RCA accuracy on scenario suite | ≥80% correct root cause across 20 synthetic incidents |
| 20 | Safety | Destructive-action guard never fires without Slack approval in the audit log |
| 20 | Time-to-hypothesis | p50 under 5 minutes from alert to Slack brief |
| 20 | Explainability | Every hypothesis has graph paths and telemetry citations |
| 15 | Integration completeness | PagerDuty, Slack, ArgoCD, Prometheus end-to-end working |
| 100 |
Exercises
- Run your agent on the same three incidents AWS's DevOps Agent is demo'd on. Publish the side-by-side. Report where the agent diverges.
- Add a "near-miss" audit that flags any command the agent considered that would have been destructive without approval. Measure the near-miss rate over one week.
- Swap the hypothesis model from Claude Sonnet 4.7 to a self-hosted Llama 3.3 70B. Measure RCA accuracy delta and dollar per incident.
- Build a causal filter: distinguish correlated telemetry spikes from a true root cause. Train a small classifier on the 20-scenario labels.
- Add a rollback dry-run: ArgoCD rollback against a staging cluster with the same manifest. Verify the rollback plan in a live cluster before the Slack approval button.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| K8s knowledge graph | "Cluster graph" | Nodes = K8s objects + telemetry series; edges = ownership, scheduling, observation |
| Read-only-by-default | "Scoped RBAC" | Agent's service account has only get/list/describe verbs; destructive verbs live in a separate server behind approval |
| Audit log | "Considered vs executed" | Append-only record of every candidate command, whether it ran, who approved |
| Hypothesis ranking | "Evidence score" | Recency × specificity × graph-path length inverse × citation count |
| Slack approval card | "HITL gate" | Interactive Slack message with remediation buttons; agent cannot proceed until a human clicks |
| Telemetry citation | "Evidence pointer" | A Prometheus query, Loki selector, or Tempo trace URL that supports a claim |
| MTTR | "Time to resolution" | Wall-clock from alert fire to SLO recovery |
Further Reading
- AWS DevOps Agent GAthe canonical 2026 reference
- Resolve AI K8s troubleshootingthe competitor reference
- NeuBird semantic monitoringsemantic-graph approach
- Metoro AI SRESLO-first production framing
- kube-state-metricsthe cluster-state source
- LangGraphreference agent orchestrator
- FastMCPPython MCP server framework
- ArgoCD rollbackthe gated remediation target
Check what you learned
7 questions. Answer each one to see why.
What is the default permission posture for the troubleshooting agent's RBAC surface?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""DevOps troubleshooting agent — K8s knowledge graph + HITL approval gate.
The hard architectural primitives are (a) a K8s knowledge graph that lets
root-cause analysis walk from an alerted object to its neighbors with
telemetry overlays, and (b) a read-only-by-default tool surface where every
destructive command is gated by a human-in-the-loop approval and every
considered command is audit-logged. This scaffold implements both.
Run: python main.py
"""
from __future__ import annotations
import json
import time
from collections import defaultdict
from dataclasses import dataclass, field
# ---------------------------------------------------------------------------
# K8s knowledge graph -- objects + telemetry overlay edges
# ---------------------------------------------------------------------------
@dataclass
class Node:
kind: str # "Pod" | "Deployment" | "Node" | "Service" | "Prom" | "Loki"
name: str
attrs: dict = field(default_factory=dict)
@property
def key(self) -> str:
return f"{self.kind}/{self.name}"
@dataclass
class Graph:
nodes: dict[str, Node] = field(default_factory=dict)
edges: list[tuple[str, str, str]] = field(default_factory=list) # (src, rel, dst)
def add(self, n: Node) -> None:
self.nodes[n.key] = n
def link(self, src: str, rel: str, dst: str) -> None:
self.edges.append((src, rel, dst))
def neighbors(self, key: str) -> list[tuple[str, str]]:
out = [(rel, dst) for s, rel, dst in self.edges if s == key]
out += [(rel, src) for src, rel, dst in self.edges if dst == key]
return out
def build_sample_cluster() -> Graph:
g = Graph()
dep = Node("Deployment", "checkout-api",
{"revision": 42, "image": "checkout-api:v2.41", "deployed_at": "14m ago"})
rs = Node("ReplicaSet", "checkout-api-abc")
node = Node("Node", "ip-10-2-3-4", {"kernel": "6.1.109"})
pods = [Node("Pod", f"checkout-api-abc-{i}", {"phase": "Running"}) for i in range(3)]
svc = Node("Service", "checkout-api")
prom = Node("Prom", "error_rate{deployment=checkout-api}",
{"last_15m": "mean=0.14 up_trend", "threshold": 0.05})
loki = Node("Loki", "namespace=prod,app=checkout-api",
{"last_15m": "500 errors on /api/v2/pay, stack = NullHealthz"})
for n in (dep, rs, node, svc, prom, loki, *pods):
g.add(n)
g.link(dep.key, "OWNS", rs.key)
for p in pods:
g.link(rs.key, "OWNS", p.key)
g.link(p.key, "SCHEDULED_ON", node.key)
g.link(svc.key, "EXPOSES", dep.key)
g.link(dep.key, "OBSERVED_BY", prom.key)
g.link(dep.key, "OBSERVED_BY", loki.key)
return g
# ---------------------------------------------------------------------------
# hypothesis ranking -- recency * specificity * citation count
# ---------------------------------------------------------------------------
@dataclass
class Hypothesis:
title: str
citations: list[str]
recency_mins: int
specificity: float # 0..1
path_len: int
def score(self) -> float:
recency_w = max(0.0, 1.0 - self.recency_mins / 60.0)
path_w = 1.0 / (1 + self.path_len)
return (recency_w * 0.35 +
self.specificity * 0.35 +
min(len(self.citations), 5) / 5 * 0.2 +
path_w * 0.1)
def root_cause(g: Graph, alerted: str) -> list[Hypothesis]:
"""Walk outward from the alerted object, collect telemetry,
and propose ranked hypotheses."""
hyps: list[Hypothesis] = []
# nearest telemetry siblings
telemetry: list[Node] = []
for rel, neighbor_key in g.neighbors(alerted):
n = g.nodes.get(neighbor_key)
if n and n.kind in ("Prom", "Loki", "Tempo"):
telemetry.append(n)
# hypothesis: bad rollout if recent deploy + observing error surge
dep = g.nodes.get(alerted)
if dep and dep.kind == "Deployment":
mins = int(str(dep.attrs.get("deployed_at", "?")).split("m")[0]) if "m" in str(dep.attrs.get("deployed_at", "")) else 999
hyps.append(Hypothesis(
title=f"bad rollout: image {dep.attrs.get('image')} fails /healthz",
citations=[t.name for t in telemetry],
recency_mins=mins,
specificity=0.82,
path_len=0,
))
# hypothesis: node-level issue (noisy neighbor / kernel)
nodes = [g.nodes[dst] for _, dst in g.neighbors(alerted) if dst.startswith("Node/")]
if nodes:
hyps.append(Hypothesis(
title=f"node-level pressure on {nodes[0].name} (kernel={nodes[0].attrs.get('kernel')})",
citations=[n.name for n in nodes],
recency_mins=30,
specificity=0.45,
path_len=2,
))
# hypothesis: service mesh / DNS
hyps.append(Hypothesis(
title="DNS flap in kube-system/coredns",
citations=[],
recency_mins=60,
specificity=0.2,
path_len=4,
))
return sorted(hyps, key=lambda h: -h.score())
# ---------------------------------------------------------------------------
# approval gate + audit log -- every considered command tracked
# ---------------------------------------------------------------------------
@dataclass
class AuditEvent:
ts: float
tool: str
args: dict
considered: bool = True
approved: bool = False
executed: bool = False
approver: str | None = None
result: str | None = None
@dataclass
class Agent:
graph: Graph
audit: list[AuditEvent] = field(default_factory=list)
read_only_tools: tuple = ("kubectl_get", "kubectl_describe", "promql", "logql", "traceql")
destructive_tools: tuple = ("kubectl_scale", "kubectl_rollback", "kubectl_delete", "argocd_rollback")
def call(self, tool: str, args: dict, approver: str | None = None) -> AuditEvent:
ev = AuditEvent(ts=time.time(), tool=tool, args=args)
if tool in self.read_only_tools:
ev.executed = True
ev.result = "ok (read-only)"
elif tool in self.destructive_tools:
if approver:
ev.approved = True
ev.approver = approver
ev.executed = True
ev.result = f"executed by {approver}"
else:
ev.result = "blocked: no slack approval"
else:
ev.result = "blocked: unknown tool"
self.audit.append(ev)
return ev
# ---------------------------------------------------------------------------
# demo -- full alert -> graph walk -> ranked hypotheses -> slack gate
# ---------------------------------------------------------------------------
def main() -> None:
g = build_sample_cluster()
agent = Agent(graph=g)
alerted = "Deployment/checkout-api"
print(f"=== alert received: {alerted} (error rate 14%) ===")
# agent pulls read-only telemetry first
agent.call("promql", {"query": "rate(http_requests_total{status=~'5..'}[5m])"})
agent.call("logql", {"query": '{app="checkout-api"} |~ "stack"'})
hyps = root_cause(g, alerted)
print("\nranked hypotheses:")
for i, h in enumerate(hyps, 1):
print(f" #{i} score={h.score():.3f} {h.title}")
print(f" citations: {h.citations}")
# agent proposes rollback but must wait for slack approval
print("\nproposing remediation:")
ev = agent.call("argocd_rollback", {"app": "checkout-api", "to_revision": 41})
print(f" {ev.tool}: {ev.result}")
# slack approved -> agent executes
print("\nslack approval granted by alice@sre")
ev = agent.call("argocd_rollback",
{"app": "checkout-api", "to_revision": 41},
approver="alice@sre")
print(f" {ev.tool}: {ev.result}")
print("\naudit log:")
for ev in agent.audit:
print(" ", json.dumps({
"tool": ev.tool, "executed": ev.executed,
"approved": ev.approved, "approver": ev.approver,
"result": ev.result,
}))
if __name__ == "__main__":
main()