
Mission 1 of 6
Find a high-value dataset before everyone else.
The most useful data in the UK is free and public, but nobody has made it easy to use. Find it, test it and pick one worth building on.
What you'll do
Government portals, public APIs and the places most people never think to check.
Run a simple script that shows how recent each dataset is, its licence and its format.
Score five candidates against five tests, then choose the one to build on.
- Name the person who'd pay before you write any code
- Use a scoring sheet, with three worked examples to copy
- Finish with one chosen dataset, written up and ready for mission 2
- Python 3.10 or newer (
python --version). New to the terminal? Do the survival kit first - A spreadsheet app or a text editor for the scoring sheet
- An hour where you're willing to reject your first idea
Your progress0/5
Where valuable data hides
You can't pick a good dataset if you only know three sources. Here are the ones that keep producing.
Start with the principle. Data is valuable when it's public enough to be legal and awkward enough that nobody has bothered.
Clean, central, well-documented datasets have already been turned into products. The money is one layer down: data that exists but nobody has assembled.
Where to look
Here are the places that keep producing good candidates.
- data.gov.uk. 50,000-odd datasets, most published under the Open Government Licence (OGL), which allows commercial use with attribution. The search is poor, which is why step 2 gives you a script. Look for anything published monthly or weekly.
- Companies House. Every UK company, officer, filing and charge. Free API, 600 requests per 5 minutes. New incorporations, director changes and insolvency filings are all events people pay to hear about. Mission 5 covers the legal lines in detail.
- Government APIs beyond Companies House. Food Standards Agency hygiene ratings (every restaurant inspection, JSON API, OGL), the Charity Commission register, Land Registry Price Paid Data, DVSA MOT history, Environment Agency flood and river data, the Police API (street-level crime by month). Each is free and each has a buyer.
- Local authority portals. Planning applications, licensing (alcohol, taxis, HMOs), business rates, FOI disclosure logs. The value is that every council does it differently. Assembling 50 of them into one table is the product.
- GitHub. Star counts, release cadence, dependency graphs, who's hiring by commit activity. The whole site is a public API. Mission 4 builds a radar on it.
- Forums and job boards. Reddit, Hacker News (free Firebase API), Indeed, LinkedIn job postings (careful: terms of service). Hiring data is the earliest signal of what companies are about to do.
- The 'already in a PDF' category. Tribunal decisions, planning inspectorate appeals, local authority meeting minutes, tender award notices. Nobody scrapes them because they're PDFs. Structured data from any document makes that cheap now.
- Programmatic lists. public-apis on GitHub is a maintained list of free APIs by category. Skim it once; you'll find three you didn't know existed.
A warning about people's data
Run the scout: which datasets are actually fresh
You have a longer list of sources now, so find out which are actually alive. The data.gov.uk search box hides the two things you care about: when it was last updated and whether you can use it commercially. This script shows both.
data.gov.uk runs on CKAN, which has a clean JSON API. package_search takes a query and returns datasets with metadata, including metadata_modified (last touched) and license_id.
Save this as
scout.py.scout.py# scout.py : search data.gov.uk and print what's actually fresh # usage: python scout.py "planning applications" [more terms...] import sys, requests from datetime import datetime, timezone API = "https://data.gov.uk/api/action/package_search" def search(term, rows=25): r = requests.get(API, params={"q": term, "rows": rows}, timeout=30) r.raise_for_status() return r.json()["result"]["results"] def age_days(iso): if not iso: return 9999 dt = datetime.fromisoformat(iso[:19]).replace(tzinfo=timezone.utc) return (datetime.now(timezone.utc) - dt).days for term in sys.argv[1:] or ["companies"]: print(f"\n== {term} ==") for d in search(term): age = age_days(d.get("metadata_modified")) fmts = sorted({(r.get("format") or "?").upper() for r in d.get("resources", [])}) lic = d.get("license_id") or "no licence stated" flag = "FRESH " if age <= 60 else "stale " print(f"{flag}{age:>5}d {lic:<14} {','.join(fmts)[:24]:<24} {d['title'][:60]}") print(f" https://data.gov.uk/dataset/{d['name']}")Install the one library it needs, then run it with a few search terms in quotes.
Terminalpip install requests python scout.py "food hygiene" "planning applications" "business rates"
Reading the results
Each row shows age in days, licence, formats and title, then the dataset URL. FRESH means touched in the last 60 days.
uk-ogl is the Open Government Licence, which is what you want. no licence stated means you email the publisher before building anything; often they'll confirm OGL, sometimes they won't.
Run it for ten different search terms over the next few minutes. Don't search for what you think the product is; search for the raw thing. 'Licensing' not 'pub closures'. 'Incorporations' not 'new company leads'. The product is your idea; the dataset is what exists.
Note what you see about formats. CSV and JSON you can use tonight. XLS is fine. PDF and HTML mean mission 2 (scraping) or pulling tables out of PDFs.
'Stale' with a good licence is not useless: it might be stale on data.gov.uk because the publisher moved to their own API. Click through and check.
TipThemetadata_modifieddate is when the catalogue entry changed, not always when the data did. For anything you shortlist, open the dataset page and look at the resource dates, or download the file and check the newest record inside it. That's the real freshness.
If it breaksModuleNotFoundError: No module named 'requests' + 2 more
ModuleNotFoundError: No module named 'requests'
pip install requests (or python -m pip install requests if pip isn't on your path). On a Mac with Homebrew Python, pip3.It returns results but every licence says 'no licence stated'
The API times out or returns a 5xx
rows to 10.The five tests
Before you score anything, learn what you're scoring against. Every dead data project fails one of these. Learn them now and you'll reject bad ideas in thirty seconds instead of thirty hours.
Read the five tests at a glance. Each one gets a fuller explanation below:
The five testsFRESH Does the value decay? Data that changes weekly is worth paying for repeatedly. Data from 2019 is a one-off download. HARD TO GET Is it behind an awkward portal, split across 300 councils, in PDFs, or only visible one record at a time? Friction you remove is what you sell. SOMEONE PAYS Can you name the job title of the person who'd pay, and what they do with it on a Tuesday morning? LEGAL Is there a licence that allows commercial re-use (OGL, CC BY, an API's terms)? Is there personal data in it? KEEP UPDATED Can a script refresh it unattended? If it needs a human clicking through a CAPTCHA, it's a job, not a product.
Fresh
This is the one that decides whether you have a subscription or a one-off download. A list of all UK charities is a download; anyone can get it, nobody pays twice.
Charities registered this week, with their stated purpose and trustees, is a feed, and a fundraising consultancy pays monthly for it. Prefer data where the value is in the delta.
Hard to get
If a buyer can get the same thing in five minutes from the source, they will. The gap has to be real: assembled from many sources, extracted from a hostile format, or enriched with something the source doesn't have (a classification, a geocode, a match to Companies House). Mission 3 is where you add that.
Someone pays
Name them. Not 'businesses' or 'investors'. 'Lettings agents with more than 50 properties in Manchester' or 'recruiters who place DevOps engineers'. If you can't picture the person and what they do with the data, you have a hobby.
This is the same muscle as finding a SaaS idea people pay for; if you've done that mission, use its buyer interview questions here.
Legal
Three questions. Is there a licence permitting commercial re-use? Does the source's terms of service or robots.txt forbid automated access? Is there personal data?
A 'no' on the first or a 'yes' on the second usually kills it. Personal data doesn't kill it but adds work (mission 5).
Keep updated
Can a script do the refresh on a schedule with no human in the loop? If the source changes its layout every month, or needs a login with 2FA, or blocks after 50 requests, the product will quietly die the first time you're busy.
The question to ask: if I go on holiday for two weeks, does it still update?
Score five candidates
Now apply the five tests. Writing scores down beats arguing with yourself. Three calibrated rows are included so your 2s mean the same thing as ours.
Each test scores 0, 1 or 2, giving a total out of 10.
Save this as
scorecard.csvand open it in a spreadsheet.scorecard.csvdataset,source_url,fresh(0-2),hard_to_get(0-2),someone_pays(0-2),legal(0-2),keep_updated(0-2),total,notes New charity registrations by region,https://register-of-charities.charitycommission.gov.uk,2,1,1,2,2,8,"weekly extract, OGL, buyers = fundraising consultants" Pub closures from licensing registers,council licensing portals,2,2,2,1,1,8,"each council different, personal data in some, hard to keep updated" Awesome-lists repo star counts,api.github.com,2,0,0,2,2,6,"anyone can get it, nobody pays for it raw"
Calibrate on the three examples
Read these before adding yours. The charity one scores 8 and is a plausible product.
The pub-closures one also scores 8 but look at where the points are: it's weak on legal and on keeping updated, which are the two tests that get harder with time, not easier.
The GitHub one scores 6 and is a 0 on 'someone pays', which means it's a 0 overall, whatever the total says.
The rule for rejecting
That's the rule: any 0 in 'someone pays' or 'legal' is a reject, regardless of total.
Totals of 8 or more with no zeros are worth taking to mission 2. Totals of 6 or 7 are worth a second look only if you can see how to raise one score (usually 'hard to get' by combining sources, or 'someone pays' by narrowing the buyer).
Score your own candidates
Add five rows of your own from the scout results and the sources in step 1. Spend two minutes per row, not twenty.
Be harsh on 'someone pays': 2 means you've seen someone pay for something like it (a competitor exists, a job ad mentions it, someone on a forum asked for it). 1 means you can argue it. 0 means you're guessing.
TipCompetitors are evidence, not a problem. If a company already sells this data for £500 a month, the market exists and your version at £49 for a narrower slice is a product. If nobody sells anything like it, be worried, not excited.
If it breaksEverything I score comes out at 5 or 6 + 1 more
Everything I score comes out at 5 or 6
My best idea has a 0 on legal
Choose one and write the brief
Pick one and commit. One paragraph that the next five missions build on. If you can't write it, you haven't chosen yet.
Pick the highest-scoring row with no zeros. If two tie, take the one with the better 'keep updated' score; freshness is what you'll be selling.
Write this in a file called
BRIEF.md, in a new folder for the project.BRIEF.md# Dataset brief Dataset: (name) Source: (URL of the API, download page or site) Licence: (OGL / CC BY / API terms link / unclear, emailed on (date)) Format: (JSON API / CSV download / HTML pages / PDFs) Refresh: (how often the source changes, and how I'll detect a change) Personal data: (none / yes: which fields, and why I need them) Buyer: (job title, company size, what they do with it on a Tuesday) What I add: (the assembly, cleaning, classification or matching that makes it worth paying for) First product shape: (searchable site / CSV subscription / API / newsletter) Score: (n/10 from the scorecard)
Fill in the brief
Fill every line. 'What I add' is the one to spend time on.
The source is free; your product is the source plus what you add.
If the honest answer is 'nothing, I just make it easier to download', the product is a convenience and it will be priced like one. If the answer is 'I match every record to its Companies House entry and flag the ones whose director changed this month', that's an analyst's morning saved and it's priced like one.
Next: scrape it, legally. You'll pull the first thousand records from your chosen source with Firecrawl or Scrapy, respecting robots.txt, rate limits and personal data law, so the thing you build on is a thing you're allowed to sell.
Your first wins
It works. Now do something with it.
Stress-test your brief
Paste the brief and let the model attack it:
Here is a one-paragraph brief for a data product I'm considering: [paste BRIEF.md]. Score it 0 to 2 on each of these tests, with one sentence of reasoning each: fresh (value decays, so people pay repeatedly), hard to get (real friction I remove), someone pays (a nameable buyer), legal (licence permits commercial re-use, no ToS or personal-data problem), keep updated (a script can refresh it unattended). Then tell me the single weakest test and two specific ways to narrow the idea so that score goes up. Don't be encouraging; be accurate.Find the buyers
Turn 'someone pays' from a guess into a list:
I'm building a weekly feed of [dataset] for [buyer type] in the UK. List ten specific places where these buyers already talk about needing this data or pay for something similar: trade bodies, forums, job adverts that mention the task, existing paid products, LinkedIn groups, newsletters. For each, say what I'd search for to find the evidence. UK sources only.Undo it cleanly
How to remove everything this mission installed.
Nothing installed beyond the requests library (pip uninstall requests if you want it gone). Delete the project folder to remove scout.py, scorecard.csv and BRIEF.md.