Get Started

End-to-End Eval Runner

Five lessons of plumbing, one lesson to glue them. The runner reads the task spec from lesson 70, calls a model through an adapter, scores with lessons 71 and 72, attaches the calibration report from lesson 73, and emits the leaderboard from lesson 74. Demo self-terminates.

90 minBuildPythonMembers

Learning objectives

  • Define a ModelAdapter interface that any model (mock, local, API) can satisfy with a small method surface.
  • Run the eval over a fixture JSONL file with parallel task execution across a worker pool.
  • Compose the metric layer (exact_match, F1, BLEU-4, ROUGE-L, code_exec) with the calibration layer in one pass.
  • Emit per-model EvalRun records and feed them straight into the leaderboard aggregator.
  • Output both a JSON report and a markdown table; self-terminate with exit zero on a clean run, non-zero on validation or runtime failure.