Get Started
Agent engineering

Mission 6 of 8

Stop guessing whether your agent got better, measure it.

Agents are non-deterministic and multi-step, so running one once and looking gives you confidence without evidence. You'll build an eval harness that grades whole runs, and a GitHub Actions job that fails a pull request when the pass rate drops.

75 minIntermediateAbout $0.10 per full suite run on Sonnet (8 tasks) and under $0.01 for the Haiku judgesStep 1 free

What you'll do

Write a task suite

Turn the failures you've already caused into eight tasks, each with an expected outcome attached.

Grade and run the suite

Build five graders and a runner that logs every trajectory, with cost and seconds per task.

Gate changes in CI

Compare before and after runs, then fail pull requests automatically when the pass rate drops.

  • Spot flaky tasks by running each one several times
  • Put a framework build under the same harness and compare real numbers
Before you start
  • Missions 3 to 5 done, especially agent2.py and test_tools.py in agent-lab
  • Git and a GitHub repo for step 6 (git without memorising if needed). The lab folder should be in a repo by now
  • Read test your prompts like code if you have not; this mission builds on its ideas but grades a loop, not a single response
Your progress0/6
01/06What an agent eval measures that a prompt eval cannot0 done
STEP 01 of 06~8 min

What an agent eval measures that a prompt eval cannot

Four numbers per task, not one. Decide what they are before building anything.

You changed a tool description in mission 3 and the agent seemed better. This step is about replacing "seemed" with numbers, and that starts with knowing what there is to measure.

Why an agent needs more than a prompt eval

A prompt eval checks one response against an expectation. An agent produces a trajectory: a sequence of model calls and tool calls, ending in an answer, costing money and time.

Grading only the answer misses most of what goes wrong. The agent that gets the right total by reading the file four times and doing the arithmetic in its head passed the answer check and failed the agent check.

The four measurements

So each task gets four measurements:

  • Outcome. Did the final answer meet the expectation? Exact, numeric, substring or rubric, depending on the task.
  • Trajectory. Did it get there the right way? Called the calculator; did not succeed in reading outside the folder; did not call tools when none were needed. The steps list from mission 3's run() makes this gradeable.
  • Cost. Input and output tokens, priced. Per task, so you see which tasks are expensive and whether a change doubled the bill.
  • Latency. Seconds per task. A change that adds a verify step may be worth it; you should know it costs 40 percent more time.

Flakiness, measured across runs

And one measurement across runs: flakiness. If a task passes 2 out of 3 times, it is not passing. It is a coin you have not finished flipping.

The harness runs each task N times and names the tasks that are inconsistent, because those are the ones where a prompt tweak is actually a dice roll.

The contract that makes it possible

It's the dict agent2.run() returns: answer, stop, steps, turns, usage, seconds.

Any agent that returns that shape can be evaluated by the harness. That is why mission 3 built it in.

Tip
Evaluate any AI model is about choosing a model with a test set. This mission's harness does that too: swap MODEL in agent2.py, run the suite, compare summaries. Same tasks, different model, real numbers.
Checkpoint
You can name the four per-task measurements and say which one the mission 1 "did it use the calculator" question falls under.
Stuck? Ask the community
Next mission · Agent engineeringMulti-agent systems that don't collapse: orchestrators, handoffs, shared state, and when one agent is better
→