Language Model Evaluation Harness
A model that does well on a task you cannot define is a model that does well by accident. The harness is the task definition, the metric, the runner, and the leaderboard, in one short, swappable shape.
90 minutesBuildPythonMembers
Learning Objectives
- Define a task as a JSONL file with
prompttargetsmetricand optionalextrasper example. - Implement five metrics: exact match, rouge-l F1, executable check, multiple choice, and substring contains.
- Build a runner that batches examples per task and dispatches to a swappable model adapter.
- Emit a leaderboard JSON with per-task scores, latency, and an overall average that is reproducible.