End-to-End Eval Runner
Five lessons of plumbing, one lesson to glue them. The runner reads the task spec from lesson 70, calls a model through an adapter, scores with lessons 71 and 72, attaches the calibration report from lesson 73, and emits the leaderboard from lesson 74. Demo self-terminates.
90 minBuildPythonMembers
Learning objectives
- Define a
ModelAdapterinterface that any model (mock, local, API) can satisfy with a small method surface. - Run the eval over a fixture JSONL file with parallel task execution across a worker pool.
- Compose the metric layer (exact_match, F1, BLEU-4, ROUGE-L, code_exec) with the calibration layer in one pass.
- Emit per-model
EvalRunrecords and feed them straight into the leaderboard aggregator. - Output both a JSON report and a markdown table; self-terminate with exit zero on a clean run, non-zero on validation or runtime failure.