Leaderboard Aggregation
Per-task scores are easy. Per-model rankings across heterogeneous tasks are harder. Statistical significance on a thousand-prediction leaderboard is the part everyone skips. This lesson does not skip it.
90 minBuildPythonMembers
Learning objectives
- Aggregate per-task scores across multiple models and multiple tasks into a tidy per-model row.
- Normalise heterogeneous scores so that pass rates and BLEU values do not over-influence the aggregate.
- Rank models by mean and by win-rate, and explain when each is the right summary.
- Compute bootstrap confidence intervals on the mean score per model and on pairwise differences.
- Output the leaderboard as a JSON report and as a markdown table the runner in lesson 75 can paste into a CI comment.