Perplexity and Calibration
If your model says 90 percent confident on a thousand answers and gets six hundred right, it is not well calibrated. Calibration is half of trustworthy eval. The other half is perplexity, which tells you whether the model thinks the held-out text is plausible at all.
90 minBuildPythonMembers
Learning objectives
- Compute token-level perplexity on a held-out corpus from token negative log-probabilities supplied by the model adapter.
- Compute the expected calibration error (ECE) of a classifier or multiple-choice eval from binned predicted probabilities.
- Compute the Brier score (mean squared error against the indicator of correctness) and explain when it does what ECE does not.
- Build the reliability diagram data needed to plot a confidence-versus-accuracy curve.
- Wire all three into the eval harness so the runner can attach
perplexityeceandbriernumbers to a model report.