
Mission 10 of 10
Know what happened, cap the damage, and have a plan.
The first nine missions stop things going wrong. This one is for when they do anyway: logs, spend caps, alerts, backups and a written incident plan, then a capstone that checks your whole app.
What you'll do
Write structured logs with personal data stripped out, and put hard spend caps on every user and on the whole app.
Get told when tool calls, denials or errors spike, and keep backups that survive ransomware.
Fill in a one-page plan while you're calm, then run the capstone checklist against your own app.
- Automated dependency updates and uptime monitoring
- A pass or fail score across all ten missions
- An app of your own, ideally the agent and web app you've been hardening through this track
- Python 3.11+ and uv for the logging and alert scripts
- A GitHub repo (for Dependabot/Renovate) and a place your app runs (for uptime monitoring)
- Twenty minutes at the end to run the capstone honestly
Your progress0/6
- 01Structured logs of every tool and model call, PII redacted14 min
- 02Spend caps and rate limits, per user and global12 min
- 03Alert on the three signals that mean trouble10 min
- 04Backups that survive ransomware, auto-updates, and uptime14 min
- 05The one-page incident plan10 min
- 06The capstone: run the whole track against your app15 min
Structured logs of every tool and model call, PII redacted
You can't investigate, alert on, or prove anything without logs. And a log full of PII is a second breach.
Best practice
One structured (JSON) line per meaningful event, queryable later, with personal data and secrets stripped before they're written.
'Structured' is the key word: print('sent email to user') is useless at 3am; {"event":"tool_call","tool":"send_email","decision":"denied","rid":"a1b2"} can be filtered, counted and alerted on.
What to log
For an AI app: every model call (which model, token counts, latency) and every tool call (name, arguments, allowed or denied and why), tied together by a request id so you can follow one user's whole journey.
For any app: auth events (login success/failure, password reset), errors and 5xx responses, and any security-relevant decision. Not: successful reads of public data, or anything so chatty it drowns the signal.
The redaction rule
Logs get shipped to services, read by people, and kept for ages. They must not contain passwords, API keys, full card numbers, or more personal data than you need.
Add this logger to your app. It redacts at the moment of logging, not later:
log.py"""log.py: structured JSON logs, one line per event, with PII and secrets redacted. Structured means machine-queryable: you can later ask "all tool calls that were denied in the last hour".""" import json, re, sys, time, uuid # Redaction: strip the things that must never land in a log file or log service. PATTERNS = [ (re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"), "<email>"), (re.compile(r"\b(?:\d[ -]*?){13,19}\b"), "<card>"), (re.compile(r"\b(sk-[A-Za-z0-9]{10,}|AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{36})\b"), "<secret>"), (re.compile(r"\b\d{4,}\b"), "<num>"), # account numbers, ids: coarse but safe ] def redact(value: str) -> str: for pat, repl in PATTERNS: value = pat.sub(repl, value) return value def log(event: str, **fields) -> None: safe = {k: (redact(v) if isinstance(v, str) else v) for k, v in fields.items()} line = {"ts": round(time.time(), 3), "event": event, **safe} sys.stdout.write(json.dumps(line) + "\n"); sys.stdout.flush() # Give each request an id so you can follow one user's whole journey across many log lines. def request_id() -> str: return uuid.uuid4().hex[:12] # Usage in the tool wrapper from the lock-down-tools mission: # log("tool_call", rid=rid, tool=name, args=json.dumps(args), decision="allowed") # log("tool_call", rid=rid, tool=name, args=json.dumps(args), decision="denied", reason=str(e)) # log("model_call", rid=rid, model=MODEL, input_tokens=usage.input_tokens, output_tokens=usage.output_tokens)Wire it into the tool wrapper from lock down tools and MCP: log every
dispatchas allowed or denied, and log each model call's token usage.Now the denied lines are both your audit trail (who tried what) and, in the next steps, your attack signal (a surge of denials is an attack in progress).
The request id ties a user's model calls and tool calls into one followable story.
What this closes
This closes OWASP A09, logging and monitoring failures, and its LLM-list cousin.
The most common version of that failure isn't 'no logs', it's 'logs that don't contain the one field you need, or contain so much PII you daren't share them'. Structured plus redacted fixes both.
<num> rule over-redacts rather than risk leaking an account number). Prefer logging an id that maps to a user in your database over logging the user's email or name at all. The safest PII in a log is the PII that was never put there.<email> / <secret> instead of the real value.