Get Started

Task Spec Format

An eval harness is only as good as the contract its tasks honour. Freeze the JSONL shape and the metric vocabulary before you write a single scoring function.

90 minBuildPythonMembers

Learning objectives

  • Define a JSONL task record schema that covers arithmetic, multiple-choice, code execution, classification, and free-text summarisation in one shape.
  • Pin a closed vocabulary of metric names so downstream lessons (71-73) can dispatch on a single field.
  • Specify few-shot examples and post-processing rules as part of the task, not the runner, so the same prompt produces the same target across models.
  • Implement a strict validator that rejects malformed records before they reach the runner.
  • Ship a 10-task fixture set that exercises every branch of the spec so the validator has something real to chew on.