Get Started

Language Model Evaluation Harness

A model that does well on a task you cannot define is a model that does well by accident. The harness is the task definition, the metric, the runner, and the leaderboard, in one short, swappable shape.

90 minutesBuildPythonMembers

Learning Objectives

  • Define a task as a JSONL file with prompttargetsmetricand optional extras per example.
  • Implement five metrics: exact match, rouge-l F1, executable check, multiple choice, and substring contains.
  • Build a runner that batches examples per task and dispatches to a swappable model adapter.
  • Emit a leaderboard JSON with per-task scores, latency, and an overall average that is reproducible.