Choose the right AI model for your job, with your own test
Leaderboards tell you which model is best at someone else's exam. Build a small test from your own work, run any mix of local and API models through it, and get one table: how often each is right, how fast it answers and what 1,000 answers cost. Add an AI judge for the open-ended answers, and catch it changing its mind when you swap the order.
- An AI judge with a rubric, plus a position-bias test that shows when not to trust it
- A repeatable way to re-check your choice every time a new model comes out
- Ollama with two models to compare, for example
qwen3.5:4bandgemma4:e2b: Run AI on your PC - uv from docs.astral.sh/uv ↗. The harness needs nothing else
- Optional: an API key to compare a hosted model, with a spending limit set
- Helpful first: The tiny AI model that got basic maths wrong, a quick first comparison by hand
Why your own test beats a leaderboard
The only score that matters is on your inputs, with your definition of a good answer.
Public benchmarks measure maths olympiad problems, coding puzzles and trivia. Your job is answering customers from your own policy, pulling fields out of emails, writing in your house style. A model that tops a leaderboard can still miss your cut-off times, and a model a tenth the size can be good enough for your job at a fraction of the cost and time.
What you'll builda fixed set of test cases from your own work, a harness that sends every case to every model you're considering, and a scorecard with three columns that decide the choice: quality, speed and cost.
Three ways to score an answer, in order of preference:
- Code checks: the answer must contain "£4.95" and "8pm", and must not promise weekend delivery. Cheap, exact, repeatable.
- Exact fields: extract the order number, issue type and refund request as JSON, compare field by field.
- A judge model with a rubric: for open-ended replies where there's no single right string. Useful, and biased in ways you'll measure in step 6.
Get the kitmodel-eval.zip has the harness, 16 sample cases, the sample shop policy and a price file. kit.md has the test-building checklist and the judge caveats.
TipIf you want prompt regression tests that run in CI every time someone edits a prompt, promptfoo ↗ is the established tool for that; this mission is about the earlier decision of which model to use.