Test your AI prompts like code, so they stop silently breaking
You ship an LLM feature, it works, you tweak the prompt a week later to fix one thing, and three other things quietly break. No error, no crash, only wrong answers reaching customers. This mission builds the fix: a tiny pytest harness that runs your prompt against fixed inputs and asserts on the output, so the next time a prompt edit or a model bump breaks something, a red test tells you before a customer does. You build it against a real model running on this computer, watch it catch a planted regression, then wire it into CI.
- Run a ChatGPT-style AI on your own laptop is the gentle way in: it installs Ollama and explains what a local model is. This mission repeats the one install command you need, so you can also start here cold.
- uv, the Python runner used across this track. It fetches the right Python and packages into one project folder. Step 2 has the one-line install.
- You can read a little Python. You will not write any from scratch: every file is complete and in the starter kit.
- About 1.5GB of disk for the model (1.3GB) and the project's packages, 8GB of RAM, and no GPU. No accounts, no API keys, nothing to pay for.
Why a prompt needs tests, and the words for it
See why an LLM feature breaks silently, and learn the handful of terms the rest of the lesson uses, so it reads as plain English.
A prompt is the instruction you send a model: the system message and the user message that together ask it to do a job. In a shipped product the prompt is code. It decides what customers get back, exactly like the function next to it. The difference is that a function fails loudly and a prompt fails quietly. Change one line of a prompt to fix a complaint, and the model can start dropping a field, mislabelling tone or wrapping its answer in chatter, with no error anywhere. The feature still runs. It is wrong now.
That quiet break has a name borrowed from ordinary software: a regression, meaning something that used to work no longer does. Regression testing is how the rest of software stops shipping them, and prompts have gone untested mostly because people assume you cannot test something that does not give the same answer twice. You can. You test the right thing.
The trick is to stop checking for one exact string and start checking properties that must always hold. The reply must contain the order id. The output must be valid JSON with a numeric total. The label must be one of three allowed words. Those hold whatever words the model chooses, and they are what break when a prompt regresses. An assertion is one such check: a line that says "this must be true" and fails the test if it is not.
This is prompt regression testing, and it is a different job from choosing a model. Picking between Llama and Gemma asks "which is best?" once. This asks "did my change break what already worked?" on every edit, forever. The second question is the one that bites teams in production, and it is the one almost nobody automates.
The plan is short. Install a small model and call it from Python. Write your first assertion. Pin the output's shape with a schema. Plant a regression and watch the tests catch it. Add a judge for the fuzzy output you cannot assert on, and handle the flakiness that comes with it. Then make it a gate in CI so nothing merges untested. Every term in that list is defined where it first appears.
TipThe whole harness fits in two short files and runs on a model small enough for a laptop with no graphics card. Nothing here needs a cloud key or a paid API. The point is the method, and the method is the same whether the model is a 1GB local one or a frontier model behind an API.