Stream AI responses so your app feels instant
A reply that takes 30 seconds to appear feels broken. The same reply, typed out word by word as the model thinks, feels alive. Build a small server that streams a local model's tokens to the browser over Server-Sent Events, measure why it feels so much faster, and handle the parts people skip: cancelling cleanly and failing mid-stream without freezing the page.
- Ollama with a small model, from Run AI on your PC. This mission uses
qwen2.5:3b(ollama pull qwen2.5:3b), but any chat model works. No GPU needed. - uv, the Python runner from Level 0's Install anything from GitHub, safely. Check it with
uv --version. - A little Python and HTML. You don't need to know FastAPI or async; every file is complete and explained.
- A code editor (VS Code ↗ is free) and a modern browser.
Why streaming makes an app feel instant
The model takes the same time to finish either way. Streaming changes when the person sees the first word, and that is what they feel.
A language model writes one token at a time (a token is a short piece of text, often a word or part of one). A plain HTTP response waits for the whole answer, then sends it in one block. So the reader stares at a spinner for the full generation time, which for a local model on a laptop can be tens of seconds.
Two different clocks are in play here, and keeping them apart is the whole point of this mission:
- Actual latency is how long the model takes to finish the complete answer. Streaming does not change this at all.
- Perceived latency is how long the person waits before anything useful happens. Streaming cuts this to the time of the first token, because they can start reading while the rest is still being written.
That gap is why every serious chat product streams. The answer is not faster; the wait is. By the end of this mission you will have measured both numbers on your own machine and seen the perceived wait drop to a fraction of the total.
You will build streambox: a small server that sits between the browser and a local model. The model produces tokens, the server relays each one the instant it arrives, and the page prints them as they land. Every file is in the kit: stream-llm-responses.zip (6KB). Unzip it and follow along, or type each file as you go.
TipLocal models on a machine with no GPU are slow to start a reply, so the waits in this mission look large (several seconds to the first word). That makes the point sharper, not weaker: the slower the model, the more streaming matters.