Run background jobs properly with a task queue
Slow work in a web request makes users wait and falls over under load. Move it to a background worker instead. Build a producer and workers with Redis and RQ, run two workers sharing the load, retry failures with backoff, send poison messages to a dead-letter, schedule jobs for later, and measure throughput. Learn why queues give you reliability and scale.
- Docker installed and running. If containers are new to you, do Docker without fear first: this mission assumes you know what an image, a container, a volume and a port are.
- A little Python, and uv, the Python runner from Level 0 (Install anything from GitHub, safely has the one-line install). Check with
uv --version. - A terminal you are comfortable in. New to it? The Terminal survival kit is enough. You will keep two or three terminals open at once.
- Helpful but not required: Build your own API in Python, which is the kind of web app that should hand its slow work to a queue like this one.
Why slow work does not belong in a web request
A web request should answer in milliseconds. The moment it does something slow or unreliable inside the request, your users wait and your server buckles under load. A queue is the fix, and it has a small vocabulary worth learning once.
Picture a signup form that sends a welcome email. If the request sends the email itself, the user stares at a spinner while your code talks to a mail server that might take two seconds, or time out, or rate-limit you. Do that for image processing, PDF generation, calling a slow third-party API or running a report, and a handful of users at once can tie up every web process you have. The site stops answering anyone.
The fix is to not do the slow work in the request at all. The request writes down a note saying what needs doing, drops it in a queue, and returns straight away. Separate processes pick notes off the queue and do the work in the background. This is a task queue (also called a job queue), and it rests on a few words that every tool in this space uses:
- Job (or task): one unit of background work, such as "resize cat.jpg". It is a function name plus its arguments, written down so another process can run it later.
- Broker: the shared place that holds the queue and hands jobs out. It sits between the two sides so neither has to know about the other. Here the broker is Redis.
- Producer: the code that creates jobs and puts them on the queue (your web app). It returns immediately and never does the work itself.
- Consumer, or worker: a long-running process that takes jobs off the queue and runs them. You run as many as you need.
- Concurrency: how many jobs run at once, which you raise by running more workers (or more threads per worker).
- Backpressure: when jobs arrive faster than workers can clear them, the queue grows instead of crashing anything. You can see the backlog and add workers. Nothing is lost, and the website stays fast because it is only ever writing notes.
This split is the whole point. The producer and the workers are decoupled: they run as separate processes, they can be scaled separately, and a slow or failing worker never makes the website slow. A burst of traffic lengthens the queue rather than taking the site down. The rest of this mission builds exactly this, piece by piece, and proves each part with real output.
TipYou will meet two delivery promises in this mission. At-least-once means a job will run, possibly more than once; most queues give you this. Exactly-once (a job runs once and only once) is far harder and most systems do not truly offer it. The practical answer is to make jobs safe to run twice, which step 6 covers under idempotency.