Get Started

Result Evaluator

The runner produced numbers. The evaluator decides whether those numbers are an improvement, a regression, or noise. Build the verdict path that turns metrics into a one line conclusion.

90 minutesBuildPythonMembers

Learning Objectives

  • Compare a candidate run against a baseline using direction aware improvement and a fixed threshold.
  • Run a paired t test from scratch over per seed metrics and read the resulting p value.
  • Normalise log scaled metrics so a downstream report can blend them with linear metrics.
  • Emit a per hypothesis verdict that the orchestrator can attach to the queue from lesson fifty.
  • Keep every step pure so the same inputs always produce the same verdict.