
Mission 6 of 8
Stop guessing whether your agent got better, measure it.
Agents are non-deterministic and multi-step, so running one once and looking gives you confidence without evidence. You'll build an eval harness that grades whole runs, and a GitHub Actions job that fails a pull request when the pass rate drops.
What you'll do
Turn the failures you've already caused into eight tasks, each with an expected outcome attached.
Build five graders and a runner that logs every trajectory, with cost and seconds per task.
Compare before and after runs, then fail pull requests automatically when the pass rate drops.
- Spot flaky tasks by running each one several times
- Put a framework build under the same harness and compare real numbers
- Missions 3 to 5 done, especially
agent2.pyandtest_tools.pyinagent-lab - Git and a GitHub repo for step 6 (git without memorising if needed). The lab folder should be in a repo by now
- Read test your prompts like code if you have not; this mission builds on its ideas but grades a loop, not a single response
Your progress0/6
- 01What an agent eval measures that a prompt eval cannot8 min
- 02The task suite: pin every failure you have already seen12 min
- 03Five graders, including a judge you can trust a bit12 min
- 04The runner: trajectories, cost and latency per task14 min
- 05Flakiness, regressions and comparing frameworks16 min
- 06Regression runs in CI, with the free tests first13 min
What an agent eval measures that a prompt eval cannot
Four numbers per task, not one. Decide what they are before building anything.
You changed a tool description in mission 3 and the agent seemed better. This step is about replacing "seemed" with numbers, and that starts with knowing what there is to measure.
Why an agent needs more than a prompt eval
A prompt eval checks one response against an expectation. An agent produces a trajectory: a sequence of model calls and tool calls, ending in an answer, costing money and time.
Grading only the answer misses most of what goes wrong. The agent that gets the right total by reading the file four times and doing the arithmetic in its head passed the answer check and failed the agent check.
The four measurements
So each task gets four measurements:
- Outcome. Did the final answer meet the expectation? Exact, numeric, substring or rubric, depending on the task.
- Trajectory. Did it get there the right way? Called the calculator; did not succeed in reading outside the folder; did not call tools when none were needed. The
stepslist from mission 3'srun()makes this gradeable. - Cost. Input and output tokens, priced. Per task, so you see which tasks are expensive and whether a change doubled the bill.
- Latency. Seconds per task. A change that adds a verify step may be worth it; you should know it costs 40 percent more time.
Flakiness, measured across runs
And one measurement across runs: flakiness. If a task passes 2 out of 3 times, it is not passing. It is a coin you have not finished flipping.
The harness runs each task N times and names the tasks that are inconsistent, because those are the ones where a prompt tweak is actually a dice roll.
The contract that makes it possible
It's the dict agent2.run() returns: answer, stop, steps, turns, usage, seconds.
Any agent that returns that shape can be evaluated by the harness. That is why mission 3 built it in.
MODEL in agent2.py, run the suite, compare summaries. Same tasks, different model, real numbers.