Capstone Lesson 41: Full Evaluation Pipeline
Training is the part you can monitor with loss curves. Evaluation is the part you have to design. This lesson builds a unified eval pipeline that takes any trained language model, runs four heterogeneous evals on it, aggregates the results into a per-task report, and ships a local mock LLM-as-judge so the loop runs without a network. The four evals cover the dimensions every shipping model needs: language modelling (perplexity), short-form correctness (exact-match), open-form similarity (token F1), and qualitative scoring (judge).
Learning Objectives
- Compute held-out perplexity with masked-token accounting on a tiny transformer.
- Run an exact-match eval on short-form factual prompts.
- Compute token-level F1 between predicted and reference strings with normalisation.
- Build a local mock LLM-as-judge that scores model outputs on a 1-5 scale.
- Aggregate the four evals into a single weighted report with per-task breakdown.
The Problem
A single metric never describes a language model. Perplexity says how well the model fits the language distribution but says nothing about whether it answers questions. Exact-match says whether the model produces the gold string but punishes correct paraphrases. Token F1 forgives paraphrase but is fooled by lexical overlap with wrong content. LLM-as-judge captures qualitative dimensions but is expensive and stochastic.
The pipeline you actually want has all four. Each eval covers a dimension the others miss. Each runs on a different subset of held-out data shaped for that metric. The final report shows the per-task numbers side by side and an aggregate, so a reviewer can see at a glance which trade-offs the model is making.
This lesson builds that pipeline, end to end, in one file.
The Concept
Each eval is a function from (model, dataset) -> EvalResult. The result carries the metric value, per-example details for inspection, and a name for the aggregate. The pipeline composes them with a config that says which evals to run and how to weight them.
Perplexity, properly counted
Perplexity is exp(mean negative log-likelihood per token). The implementation has two traps:
- The mean must be over actual token positions, not over batch * sequence. Padding tokens have to be excluded from the denominator or perplexity will look better than it is.
- The model predicts the next token, so logits at position
ipredict the token at positioni+1. Off-by-one mistakes here are silent: the loss still trains, but the metric becomes meaningless.
The eval computes per-batch sums of -log p(token) over non-pad positions and a per-batch token count, then divides at the end. This is numerically safer than averaging per-batch perplexities (which under-weights short sequences) and matches the textbook definition.
Exact-match, with normalisation
The harness normalises both the prediction and the reference before comparing:
- Lowercase.
- Strip surrounding whitespace.
- Collapse internal whitespace runs to a single space.
- Drop trailing terminal punctuation (
.!?) if both sides differ only by punctuation.
Normalisation makes exact-match useful in practice. A model that says "Paris" is right; one that says "Paris." is also right; one that says " paris " is also right. The metric still requires the answer to be the same string after normalisation.
Token F1, the right way
Token F1 is the harmonic mean of precision and recall computed over the bag-of-tokens. Steps:
- Normalise prediction and reference (same rules as exact-match).
- Split each into a list of tokens (whitespace tokenisation).
- Count the multiset intersection.
- Precision =
intersection_count / len(pred_tokens). Recall =intersection_count / len(ref_tokens). F1 = harmonic mean.
If both prediction and reference are empty, F1 is 1 (vacuous match). If only one is empty, F1 is 0. This pattern matches the SQuAD evaluation reference and produces stable numbers across paraphrases.
Local Mock LLM-as-Judge
A real judge is a frontier model behind an API. For this lesson the judge has to run offline. The mock judge is a deterministic scorer that takes an instruction, the model's prediction, and the reference, and returns a score in {1, 2, 3, 4, 5} plus a one-line rationale. The scoring rules are explicit:
- 5 if normalised prediction equals normalised reference.
- 4 if token F1 between prediction and reference is at least 0.8.
- 3 if token F1 is in
[0.5, 0.8). - 2 if token F1 is in
[0.2, 0.5). - 1 otherwise.
This is not a real judge, but it has the right interface. Swap in a real model later by changing one function. The pipeline does not care.
Aggregation
The aggregate is a weighted mean of normalised eval scores. Each eval reports its own number in [0, 1]:
- Perplexity: normalise as
1 / (1 + log(perplexity)). A perplexity of 1 maps to 1, infinity maps to 0. - Exact-match: already in
[0, 1]. - Token F1: already in
[0, 1]. - Judge: divide by 5.
Weights are configurable. The default mix is 0.2 perplexity, 0.3 exact-match, 0.3 token F1, 0.2 judge. The choice of weights is a product decision; the lesson exposes the knob so you can experiment.
Architecture
The EvalSuite is a thin orchestrator. Each individual eval is a free function that takes (model, tokenizer, dataset, config) and returns an EvalResult. The Aggregator collects results and produces the final report. The demo prints the table and writes a JSON copy that downstream CI can ingest.
What you will build
The implementation is one main.py plus tests.
TinyGPT: the same decoder-only architecture used in lessons 38-40, included so the lesson stands alone.InstructionTokenizer: byte tokeniser with INST / RESP / PAD specials.- Four fixtures: an LM corpus, an EM set, an F1 set, and a judge set. Twenty examples each, deterministic.
perplexity_eval: returnsEvalResultwith the perplexity value and per-token loss histogram.exact_match_eval: returns mean EM and per-example records.token_f1_eval: returns mean token F1 and per-example records.mock_judgeandjudge_eval: per-example score and rationale, mean score across the set.Aggregator.normalise: per-eval normalisation rule.Aggregator.aggregate: weighted mean and the assembled report.run_demo: trains a tiny model briefly, runs all four evals, prints the report table and writes the JSON, exits zero on success.
Reading the report
The report has three layers. The top is the aggregate score. Below it are the four per-eval numbers. Below those are the per-example breakdowns for diagnostics. A failing CI run typically wants the aggregate, but a reviewer chasing a regression wants the per-example breakdown to see which inputs the model got wrong.
The JSON dump uses stable keys so a CI dashboard can plot trend lines across versions. The pretty-printed table is for humans staring at the terminal after a training run.
Stretch goals
- Add a calibration eval: do the model's softmax probabilities match its accuracy? Bucket predictions by confidence and report the empirical accuracy per bucket.
- Add a robustness eval: tag each example with a perturbation (typo, paraphrase, distractor) and report metric drop per perturbation.
- Replace the mock judge with a real model behind an HTTP call. The function signature does not change.
- Add per-task weight learning: instead of fixed weights, fit weights to a target preference order over models.
The implementation gives you the four evals, the aggregator, and the report. Real evaluation pipelines layer many more dimensions on top; the pattern stays the same: one function per eval, one aggregator, one report.
Check what you learned
6 questions. Answer each one to see why.
Before reading: a colleague reports their model has perplexity 4.2 on the held-out set. They ask if the model is good. What is the most defensible response?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""
Full evaluation pipeline for a tiny language model.
See: phases/19-capstone-projects/41-eval-pipeline/docs/en.md
Implements:
- perplexity_eval (held-out language modelling)
- exact_match_eval (short-form factual)
- token_f1_eval (open-form similarity)
- judge_eval (mock LLM-as-judge with deterministic scoring)
- Aggregator (per-eval normalisation + weighted mean)
- run_demo: trains a tiny TinyGPT briefly, runs all four evals, writes
report.json next to this file, exits 0 on success.
"""
from __future__ import annotations
import json
import math
import os
import random
import sys
from dataclasses import asdict, dataclass, field
from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, Dataset
# ---------------------------------------------------------------------------
# Tokeniser and TinyGPT (shared across lessons 38-41).
# ---------------------------------------------------------------------------
class InstructionTokenizer:
INST_ID = 256
RESP_ID = 257
PAD_ID = 258
VOCAB = 260
IGNORE_INDEX = -100
def encode_pair(self, instruction: str, response: str, max_len: int) -> Tuple[List[int], int]:
ids = [self.INST_ID]
ids.extend(instruction.encode("utf-8", errors="ignore"))
ids.append(self.RESP_ID)
resp_start = len(ids)
ids.extend(response.encode("utf-8", errors="ignore"))
if len(ids) > max_len:
ids = ids[:max_len]
return ids, resp_start
def encode_prefix(self, instruction: str, max_len: int) -> List[int]:
ids = [self.INST_ID]
ids.extend(instruction.encode("utf-8", errors="ignore"))
ids.append(self.RESP_ID)
if len(ids) > max_len:
ids = ids[:max_len]
return ids
def encode_text(self, text: str, max_len: int) -> List[int]:
ids = list(text.encode("utf-8", errors="ignore"))
if len(ids) > max_len:
ids = ids[:max_len]
return ids
def decode_response(self, ids: Sequence[int]) -> str:
return bytes(i for i in ids if i < 256).decode("utf-8", errors="replace")
class CausalSelfAttention(nn.Module):
def __init__(self, hidden: int, heads: int, max_len: int):
super().__init__()
if hidden % heads != 0:
raise ValueError("hidden must divide heads")
self.heads = heads
self.head_dim = hidden // heads
self.qkv = nn.Linear(hidden, hidden * 3, bias=False)
self.out = nn.Linear(hidden, hidden, bias=False)
mask = torch.tril(torch.ones(max_len, max_len, dtype=torch.bool))
self.register_buffer("causal_mask", mask, persistent=False)
def forward(self, x: torch.Tensor, key_pad_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
B, T, D = x.shape
qkv = self.qkv(x).view(B, T, 3, self.heads, self.head_dim).permute(2, 0, 3, 1, 4)
q, k, v = qkv[0], qkv[1], qkv[2]
att = (q @ k.transpose(-2, -1)) / math.sqrt(self.head_dim)
causal = self.causal_mask[:T, :T].view(1, 1, T, T)
att = att.masked_fill(~causal, float("-inf"))
if key_pad_mask is not None:
km = key_pad_mask.view(B, 1, 1, T).to(torch.bool)
att = att.masked_fill(~km, float("-inf"))
weights = F.softmax(att, dim=-1)
weights = torch.nan_to_num(weights, nan=0.0)
ctx = (weights @ v).transpose(1, 2).contiguous().view(B, T, D)
return self.out(ctx)
class Block(nn.Module):
def __init__(self, hidden: int, heads: int, max_len: int):
super().__init__()
self.ln1 = nn.LayerNorm(hidden)
self.attn = CausalSelfAttention(hidden, heads, max_len)
self.ln2 = nn.LayerNorm(hidden)
self.fc1 = nn.Linear(hidden, hidden * 4)
self.fc2 = nn.Linear(hidden * 4, hidden)
def forward(self, x: torch.Tensor, key_pad_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
x = x + self.attn(self.ln1(x), key_pad_mask)
h = self.ln2(x)
return x + self.fc2(F.gelu(self.fc1(h)))
class TinyGPT(nn.Module):
def __init__(self, vocab: int, hidden: int, heads: int, depth: int, max_len: int):
super().__init__()
self.tok = nn.Embedding(vocab, hidden)
self.pos = nn.Embedding(max_len, hidden)
self.blocks = nn.ModuleList([Block(hidden, heads, max_len) for _ in range(depth)])
self.ln_f = nn.LayerNorm(hidden)
self.head = nn.Linear(hidden, vocab, bias=False)
self.max_len = max_len
def forward(self, ids: torch.Tensor, key_pad_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
B, T = ids.shape
positions = torch.arange(T, device=ids.device).unsqueeze(0).expand(B, T)
x = self.tok(ids) + self.pos(positions)
for blk in self.blocks:
x = blk(x, key_pad_mask)
return self.head(self.ln_f(x))
# ---------------------------------------------------------------------------
# Eval result dataclasses
# ---------------------------------------------------------------------------
@dataclass
class ExampleRecord:
instruction: str
prediction: str
reference: str
score: float
detail: Dict[str, Any] = field(default_factory=dict)
@dataclass
class EvalResult:
name: str
metric: float
n_examples: int
records: List[ExampleRecord] = field(default_factory=list)
extras: Dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> Dict[str, Any]:
return {
"name": self.name,
"metric": self.metric,
"n_examples": self.n_examples,
"records": [asdict(r) for r in self.records],
"extras": self.extras,
}
# ---------------------------------------------------------------------------
# Normalisation and string utilities
# ---------------------------------------------------------------------------
_TERMINAL_PUNCT = ".!?"
def normalise(text: str) -> str:
s = text.lower().strip()
while " " in s:
s = s.replace(" ", " ")
return s
def strip_trailing_punctuation(text: str) -> str:
s = text
while s and s[-1] in _TERMINAL_PUNCT:
s = s[:-1]
return s
def normalise_for_em(text: str) -> str:
return strip_trailing_punctuation(normalise(text))
def tokenize_text(text: str) -> List[str]:
return [t for t in normalise(text).split() if t]
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
LM_CORPUS = [
"the quick brown fox jumps over the lazy dog.",
"to be or not to be, that is the question.",
"all that glitters is not gold; some of it is yellow.",
"the rain in spain stays mainly in the plain.",
"an apple a day keeps the doctor away from your door.",
"the early bird catches the worm, but the second mouse gets the cheese.",
"actions speak louder than words in almost every case.",
"rome was not built in a day, but the romans worked at it daily.",
"a stitch in time saves nine future stitches, every tailor agrees.",
"the pen is mightier than the sword in the hands of an editor.",
"every cloud has a silver lining if you wait long enough.",
"birds of a feather flock together near the same trees.",
"do not count your chickens before they hatch under the hen.",
"absence makes the heart grow fonder of letters and small notes.",
"actions taken in haste are regretted at greater length later on.",
"beauty is in the eye of the beholder and the budget of the buyer.",
"fortune favours the bold, the brave, and occasionally the foolish.",
"great minds think alike but fools seldom differ either.",
"honesty is the best policy in most negotiations and contracts.",
"patience is a virtue that takes time to acquire and to keep.",
]
EM_PAIRS = [
("What is the capital of France?", "paris"),
("What is the capital of Japan?", "tokyo"),
("What is the capital of Italy?", "rome"),
("What is the capital of Spain?", "madrid"),
("What is the capital of Brazil?", "brasilia"),
("Compute 2 + 3.", "5"),
("Compute 7 + 8.", "15"),
("Compute 9 - 4.", "5"),
("Compute 6 * 3.", "18"),
("Compute 20 / 5.", "4"),
("Name the largest planet.", "jupiter"),
("Name the smallest planet.", "mercury"),
("Define variable.", "a name bound to a value"),
("Define function.", "a reusable block of code"),
("Define loop.", "a control structure that repeats"),
("Python: print 42.", "print(42)"),
("Python: print hi.", "print('hi')"),
("Python: length of x.", "len(x)"),
("What colour is the sky on a clear day?", "blue"),
("What gas do plants breathe in?", "carbon dioxide"),
]
F1_PAIRS = [
(
"Summarise: the sun rises in the east and sets in the west.",
"the sun moves from east to west across the sky.",
),
(
"Summarise: water boils at one hundred degrees celsius at sea level.",
"water boils at 100 degrees at sea level.",
),
(
"Summarise: the moon orbits the earth roughly every 27 days.",
"the moon takes about 27 days to orbit earth.",
),
(
"Summarise: light travels at three hundred thousand kilometres per second.",
"light moves at 300000 km per second.",
),
(
"Summarise: bees pollinate flowers as they collect nectar.",
"bees pollinate flowers while gathering nectar.",
),
(
"Summarise: dna stores genetic information in cells.",
"dna holds the genetic code inside cells.",
),
(
"Summarise: the heart pumps blood through arteries and veins.",
"the heart moves blood through veins and arteries.",
),
(
"Summarise: the great wall of china stretches thousands of kilometres.",
"the great wall extends thousands of kilometres across china.",
),
(
"Summarise: shakespeare wrote plays and sonnets in elizabethan english.",
"shakespeare authored plays and sonnets in old english.",
),
(
"Summarise: photosynthesis converts sunlight into chemical energy.",
"plants convert sunlight into chemical energy via photosynthesis.",
),
(
"Summarise: electricity flows easily through metals like copper and silver.",
"electricity travels well through copper and silver wires.",
),
(
"Summarise: the human body has 206 bones in adulthood.",
"adults have 206 bones in the human body.",
),
(
"Summarise: gravity pulls objects toward the centre of the earth.",
"gravity attracts objects toward the earth centre.",
),
(
"Summarise: vaccines train the immune system to recognise pathogens.",
"vaccines teach the immune system to identify germs.",
),
(
"Summarise: tectonic plates move slowly across the earth's surface.",
"tectonic plates drift slowly across the surface.",
),
(
"Summarise: the equator is the imaginary line around the middle of earth.",
"the equator is the line around the middle of the planet.",
),
(
"Summarise: octopuses have three hearts and blue blood.",
"octopuses possess three hearts and blue blood.",
),
(
"Summarise: honey never spoils due to its low water content.",
"honey does not spoil because of low water.",
),
(
"Summarise: deserts can be cold or hot but are always dry.",
"deserts may be cold or hot but are dry places.",
),
(
"Summarise: the brain consumes around twenty percent of body energy.",
"the brain uses about twenty percent of the body energy.",
),
]
JUDGE_SET = [
("Name the capital of France.", "the capital of france is paris."),
("Name the capital of Japan.", "the capital of japan is tokyo."),
("Define variable.", "a variable is a name that holds a value."),
("Compute 3 + 4.", "the result is 7."),
("Python: print 100.", "print(100) prints the integer 100."),
("List three colors.", "red, green, and blue are three colors."),
("Define function.", "a function is a reusable block of code."),
("Compute 11 - 4.", "the result is 7."),
("Name the largest planet.", "the largest planet is jupiter."),
("Python: length of items.", "len(items) returns the length of the list."),
("List three vowels.", "a, e, and i are vowels."),
("Compute 6 * 5.", "the result is 30."),
("Define loop.", "a loop is a control structure that repeats."),
("Name the smallest ocean.", "the smallest ocean is the arctic ocean."),
("Compute 8 / 2.", "the result is 4."),
("Python: open f in read mode.", "open('f.txt', 'r') opens a file in read mode."),
("Define string.", "a string is a sequence of characters."),
("Name the longest river.", "the longest river is the nile."),
("List three fruits.", "apple, banana, and cherry are fruits."),
("Compute 9 + 6.", "the result is 15."),
]
# ---------------------------------------------------------------------------
# Datasets
# ---------------------------------------------------------------------------
class LMTextDataset(Dataset):
def __init__(self, texts: Sequence[str], tok: InstructionTokenizer, max_len: int):
self.texts = list(texts)
self.tok = tok
self.max_len = max_len
def __len__(self) -> int:
return len(self.texts)
def __getitem__(self, idx: int):
ids = self.tok.encode_text(self.texts[idx], self.max_len)
attn = [1] * len(ids)
pad = self.max_len - len(ids)
ids = ids + [InstructionTokenizer.PAD_ID] * pad
attn = attn + [0] * pad
return torch.tensor(ids, dtype=torch.long), torch.tensor(attn, dtype=torch.long)
# ---------------------------------------------------------------------------
# Perplexity eval
# ---------------------------------------------------------------------------
@torch.no_grad()
def perplexity_eval(
model: TinyGPT,
tok: InstructionTokenizer,
texts: Sequence[str],
max_len: int,
batch_size: int = 8,
) -> EvalResult:
model.eval()
ds = LMTextDataset(texts, tok, max_len)
dl = DataLoader(ds, batch_size=batch_size, shuffle=False)
total_nll = 0.0
total_tokens = 0
per_example_nll: List[float] = []
for ids, attn in dl:
logits = model(ids, key_pad_mask=attn)
pred = logits[:, :-1, :].contiguous()
target = ids[:, 1:].contiguous()
target_mask = attn[:, 1:].contiguous() # exclude pad targets
loss_per_pos = F.cross_entropy(
pred.view(-1, pred.size(-1)),
target.view(-1),
reduction="none",
).view(target.shape)
loss_per_pos = loss_per_pos * target_mask
per_seq_sum = loss_per_pos.sum(dim=-1)
per_seq_tok = target_mask.sum(dim=-1).clamp(min=1.0)
per_example_nll.extend((per_seq_sum / per_seq_tok).tolist())
total_nll += float(loss_per_pos.sum().item())
total_tokens += int(target_mask.sum().item())
if total_tokens == 0:
ppl = float("inf")
else:
mean_nll = total_nll / total_tokens
ppl = math.exp(mean_nll)
records = [
ExampleRecord(
instruction=texts[i],
prediction="",
reference=texts[i],
score=math.exp(per_example_nll[i]) if per_example_nll[i] < 50 else float("inf"),
detail={"mean_nll": per_example_nll[i]},
)
for i in range(len(texts))
]
return EvalResult(
name="perplexity",
metric=ppl,
n_examples=len(texts),
records=records,
extras={"total_tokens": total_tokens, "mean_nll": total_nll / max(total_tokens, 1)},
)
# ---------------------------------------------------------------------------
# Generative evals (EM, F1, judge)
# ---------------------------------------------------------------------------
@torch.no_grad()
def generate_greedy(
model: TinyGPT,
tok: InstructionTokenizer,
instruction: str,
max_new_tokens: int = 48,
) -> str:
model.eval()
ids = tok.encode_prefix(instruction, max_len=model.max_len)
sentence_ends = {ord("."), ord("!"), ord("?")}
out_chars: List[int] = []
for _ in range(max_new_tokens):
if len(ids) >= model.max_len:
break
x = torch.tensor([ids], dtype=torch.long)
attn = torch.ones_like(x)
logits = model(x, key_pad_mask=attn)
next_id = int(logits[0, -1, :].argmax().item())
if next_id == tok.PAD_ID or next_id == tok.INST_ID or next_id == tok.RESP_ID:
break
ids.append(next_id)
out_chars.append(next_id)
if (
len(out_chars) >= 2
and out_chars[-1] in sentence_ends
and out_chars[-2] in sentence_ends
):
break
return tok.decode_response(out_chars)
def exact_match_score(pred: str, ref: str) -> int:
return 1 if normalise_for_em(pred) == normalise_for_em(ref) else 0
def token_f1_score(pred: str, ref: str) -> float:
p_tokens = tokenize_text(pred)
r_tokens = tokenize_text(ref)
if not p_tokens and not r_tokens:
return 1.0
if not p_tokens or not r_tokens:
return 0.0
p_counts: Dict[str, int] = {}
for tok in p_tokens:
p_counts[tok] = p_counts.get(tok, 0) + 1
r_counts: Dict[str, int] = {}
for tok in r_tokens:
r_counts[tok] = r_counts.get(tok, 0) + 1
intersection = 0
for tok, n in p_counts.items():
intersection += min(n, r_counts.get(tok, 0))
if intersection == 0:
return 0.0
precision = intersection / len(p_tokens)
recall = intersection / len(r_tokens)
return (2 * precision * recall) / (precision + recall)
def exact_match_eval(
model: TinyGPT,
tok: InstructionTokenizer,
pairs: Sequence[Tuple[str, str]],
) -> EvalResult:
records: List[ExampleRecord] = []
hits = 0
for inst, ref in pairs:
pred = generate_greedy(model, tok, inst)
sc = exact_match_score(pred, ref)
hits += sc
records.append(
ExampleRecord(
instruction=inst,
prediction=pred,
reference=ref,
score=float(sc),
)
)
metric = hits / max(len(pairs), 1)
return EvalResult(
name="exact_match", metric=metric, n_examples=len(pairs), records=records
)
def token_f1_eval(
model: TinyGPT,
tok: InstructionTokenizer,
pairs: Sequence[Tuple[str, str]],
) -> EvalResult:
records: List[ExampleRecord] = []
total = 0.0
for inst, ref in pairs:
pred = generate_greedy(model, tok, inst)
sc = token_f1_score(pred, ref)
total += sc
records.append(
ExampleRecord(
instruction=inst,
prediction=pred,
reference=ref,
score=sc,
)
)
metric = total / max(len(pairs), 1)
return EvalResult(name="token_f1", metric=metric, n_examples=len(pairs), records=records)
# ---------------------------------------------------------------------------
# Mock LLM-as-judge
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class JudgeVerdict:
score: int
rationale: str
def mock_judge(instruction: str, prediction: str, reference: str) -> JudgeVerdict:
"""Deterministic judge: scores prediction on a 1-5 scale.
The scoring rules are:
5: normalised prediction equals normalised reference.
4: token-F1 >= 0.8
3: token-F1 in [0.5, 0.8)
2: token-F1 in [0.2, 0.5)
1: otherwise
"""
norm_pred = normalise_for_em(prediction)
norm_ref = normalise_for_em(reference)
if norm_pred == norm_ref:
return JudgeVerdict(score=5, rationale="exact match after normalisation")
f1 = token_f1_score(prediction, reference)
if f1 >= 0.8:
return JudgeVerdict(score=4, rationale=f"high token overlap (F1={f1:.2f})")
if f1 >= 0.5:
return JudgeVerdict(score=3, rationale=f"moderate token overlap (F1={f1:.2f})")
if f1 >= 0.2:
return JudgeVerdict(score=2, rationale=f"weak token overlap (F1={f1:.2f})")
return JudgeVerdict(score=1, rationale=f"low token overlap (F1={f1:.2f})")
def judge_eval(
model: TinyGPT,
tok: InstructionTokenizer,
pairs: Sequence[Tuple[str, str]],
judge_fn: Callable[[str, str, str], JudgeVerdict] = mock_judge,
) -> EvalResult:
records: List[ExampleRecord] = []
total = 0.0
for inst, ref in pairs:
pred = generate_greedy(model, tok, inst)
verdict = judge_fn(inst, pred, ref)
total += verdict.score
records.append(
ExampleRecord(
instruction=inst,
prediction=pred,
reference=ref,
score=float(verdict.score),
detail={"rationale": verdict.rationale},
)
)
metric = total / max(len(pairs), 1)
return EvalResult(name="judge", metric=metric, n_examples=len(pairs), records=records)
# ---------------------------------------------------------------------------
# Aggregator
# ---------------------------------------------------------------------------
DEFAULT_WEIGHTS = {
"perplexity": 0.2,
"exact_match": 0.3,
"token_f1": 0.3,
"judge": 0.2,
}
def normalise_metric(name: str, value: float) -> float:
"""Map a raw metric onto [0, 1]."""
if name == "perplexity":
if value <= 0 or math.isinf(value) or math.isnan(value):
return 0.0
return 1.0 / (1.0 + math.log(max(value, 1.0)))
if name == "exact_match":
return max(0.0, min(1.0, float(value)))
if name == "token_f1":
return max(0.0, min(1.0, float(value)))
if name == "judge":
return max(0.0, min(1.0, float(value) / 5.0))
raise ValueError(f"unknown metric name: {name}")
@dataclass
class FinalReport:
per_eval: Dict[str, float]
normalised: Dict[str, float]
weights: Dict[str, float]
aggregate: float
details: Dict[str, EvalResult] = field(default_factory=dict)
def to_dict(self) -> Dict[str, Any]:
return {
"per_eval": self.per_eval,
"normalised": self.normalised,
"weights": self.weights,
"aggregate": self.aggregate,
"details": {k: v.to_dict() for k, v in self.details.items()},
}
def aggregate(results: Sequence[EvalResult], weights: Optional[Dict[str, float]] = None) -> FinalReport:
weights = dict(weights or DEFAULT_WEIGHTS)
# Normalise weights so they sum to 1.
total_w = sum(weights.get(r.name, 0.0) for r in results)
if total_w <= 0:
raise ValueError("no positive weights match the eval names")
norm_weights = {r.name: weights.get(r.name, 0.0) / total_w for r in results}
per_eval = {r.name: r.metric for r in results}
normalised = {r.name: normalise_metric(r.name, r.metric) for r in results}
agg = sum(normalised[name] * norm_weights[name] for name in normalised)
return FinalReport(
per_eval=per_eval,
normalised=normalised,
weights=norm_weights,
aggregate=agg,
details={r.name: r for r in results},
)
# ---------------------------------------------------------------------------
# Pretty printer
# ---------------------------------------------------------------------------
def render_report(report: FinalReport, log: Callable[[str], None] = print) -> None:
log("EVAL REPORT")
log(" eval raw normalised weight")
log(" ------------ --------- ----------- ------")
for name in sorted(report.per_eval.keys()):
raw = report.per_eval[name]
if name == "perplexity":
raw_repr = f"{raw:>9.3f}"
else:
raw_repr = f"{raw:>9.3f}"
norm = report.normalised[name]
weight = report.weights[name]
log(f" {name:<12} {raw_repr} {norm:>11.3f} {weight:>6.2f}")
log(" ------------ --------- ----------- ------")
log(f" AGGREGATE {report.aggregate:.3f}")
# ---------------------------------------------------------------------------
# Demo: train a tiny model briefly and run all four evals.
# ---------------------------------------------------------------------------
@dataclass(frozen=True)
class EvalConfig:
vocab: int = InstructionTokenizer.VOCAB
hidden: int = 64
heads: int = 4
depth: int = 2
max_len: int = 80
train_epochs: int = 5
lr: float = 3e-3
seed: int = 0
def train_small(model: TinyGPT, tok: InstructionTokenizer, cfg: EvalConfig) -> List[float]:
"""A short pretraining pass on the LM corpus + EM pairs so the model is non-random.
This is enough to make all four metrics non-trivial without locking in a
specific quality bar.
"""
pairs: List[List[int]] = []
for text in LM_CORPUS:
pairs.append(tok.encode_text(text, cfg.max_len))
for inst, ref in EM_PAIRS:
ids, _ = tok.encode_pair(inst, ref, cfg.max_len)
pairs.append(ids)
opt = torch.optim.Adam(model.parameters(), lr=cfg.lr)
losses: List[float] = []
model.train()
for _ in range(cfg.train_epochs):
ep_loss = 0.0
for seq in pairs:
ids = torch.tensor([seq], dtype=torch.long)
attn = torch.ones_like(ids)
logits = model(ids, key_pad_mask=attn)
pred = logits[:, :-1, :].contiguous()
target = ids[:, 1:].contiguous()
loss = F.cross_entropy(pred.view(-1, pred.size(-1)), target.view(-1))
opt.zero_grad()
loss.backward()
opt.step()
ep_loss += float(loss.item())
losses.append(ep_loss / max(len(pairs), 1))
return losses
def run_demo(cfg: Optional[EvalConfig] = None, write_json: bool = True) -> int:
cfg = cfg or EvalConfig()
torch.manual_seed(cfg.seed)
np.random.seed(cfg.seed)
random.seed(cfg.seed)
print("EVAL PIPELINE DEMO")
print(f"fixtures: lm={len(LM_CORPUS)} em={len(EM_PAIRS)} f1={len(F1_PAIRS)} judge={len(JUDGE_SET)}")
print("")
tok = InstructionTokenizer()
model = TinyGPT(cfg.vocab, cfg.hidden, cfg.heads, cfg.depth, cfg.max_len)
print(f"[training] {cfg.train_epochs} epochs over the LM + EM combined corpus...")
losses = train_small(model, tok, cfg)
print(f" final epoch loss = {losses[-1]:.4f}")
print("")
print("[evaluating]")
ppl_res = perplexity_eval(model, tok, LM_CORPUS, cfg.max_len)
em_res = exact_match_eval(model, tok, EM_PAIRS)
f1_res = token_f1_eval(model, tok, F1_PAIRS)
j_res = judge_eval(model, tok, JUDGE_SET)
print(f" perplexity = {ppl_res.metric:.3f}")
print(f" exact_match = {em_res.metric:.3f}")
print(f" token_f1 = {f1_res.metric:.3f}")
print(f" judge (1-5) = {j_res.metric:.3f}")
print("")
report = aggregate([ppl_res, em_res, f1_res, j_res])
render_report(report)
print("")
print("[per-task spot checks]")
for res in [em_res, f1_res, j_res]:
for rec in res.records[:2]:
print(f" {res.name:>11s} inst='{rec.instruction[:35]}' pred='{rec.prediction[:35]}' score={rec.score:.2f}")
if write_json:
here = os.path.dirname(os.path.abspath(__file__))
out_path = os.path.join(here, "report.json")
with open(out_path, "w", encoding="utf-8") as fh:
json.dump(report.to_dict(), fh, indent=2, default=str)
print("")
print(f"wrote {out_path}")
if report.aggregate <= 0.0 or math.isnan(report.aggregate):
print("ERROR: aggregate score is not positive", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(run_demo())