Get Started

Leaderboard Aggregation

Per-task scores are easy. Per-model rankings across heterogeneous tasks are harder. Statistical significance on a thousand-prediction leaderboard is the part everyone skips. This lesson does not skip it.

90 minBuildPythonMembers

Learning objectives

  • Aggregate per-task scores across multiple models and multiple tasks into a tidy per-model row.
  • Normalise heterogeneous scores so that pass rates and BLEU values do not over-influence the aggregate.
  • Rank models by mean and by win-rate, and explain when each is the right summary.
  • Compute bootstrap confidence intervals on the mean score per model and on pairwise differences.
  • Output the leaderboard as a JSON report and as a markdown table the runner in lesson 75 can paste into a CI comment.