Get Started
Data hustle

Mission 1 of 6

Find a high-value dataset before everyone else.

The most useful data in the UK is free and public, but nobody has made it easy to use. Find it, test it and pick one worth building on.

50 minBeginnerFree toolsFree mission

What you'll do

Discover hidden data sources

Government portals, public APIs and the places most people never think to check.

Check it's fresh and usable

Run a simple script that shows how recent each dataset is, its licence and its format.

Test and pick a winner

Score five candidates against five tests, then choose the one to build on.

  • Name the person who'd pay before you write any code
  • Use a scoring sheet, with three worked examples to copy
  • Finish with one chosen dataset, written up and ready for mission 2
Before you start
  • Python 3.10 or newer (python --version). New to the terminal? Do the survival kit first
  • A spreadsheet app or a text editor for the scoring sheet
  • An hour where you're willing to reject your first idea
Your progress0/5
01/05Where valuable data hides0 done
STEP 01 of 05~10 min

Where valuable data hides

You can't pick a good dataset if you only know three sources. Here are the ones that keep producing.

Start with the principle. Data is valuable when it's public enough to be legal and awkward enough that nobody has bothered.

Clean, central, well-documented datasets have already been turned into products. The money is one layer down: data that exists but nobody has assembled.

Where to look

Here are the places that keep producing good candidates.

  • data.gov.uk. 50,000-odd datasets, most published under the Open Government Licence (OGL), which allows commercial use with attribution. The search is poor, which is why step 2 gives you a script. Look for anything published monthly or weekly.
  • Companies House. Every UK company, officer, filing and charge. Free API, 600 requests per 5 minutes. New incorporations, director changes and insolvency filings are all events people pay to hear about. Mission 5 covers the legal lines in detail.
  • Government APIs beyond Companies House. Food Standards Agency hygiene ratings (every restaurant inspection, JSON API, OGL), the Charity Commission register, Land Registry Price Paid Data, DVSA MOT history, Environment Agency flood and river data, the Police API (street-level crime by month). Each is free and each has a buyer.
  • Local authority portals. Planning applications, licensing (alcohol, taxis, HMOs), business rates, FOI disclosure logs. The value is that every council does it differently. Assembling 50 of them into one table is the product.
  • GitHub. Star counts, release cadence, dependency graphs, who's hiring by commit activity. The whole site is a public API. Mission 4 builds a radar on it.
  • Forums and job boards. Reddit, Hacker News (free Firebase API), Indeed, LinkedIn job postings (careful: terms of service). Hiring data is the earliest signal of what companies are about to do.
  • The 'already in a PDF' category. Tribunal decisions, planning inspectorate appeals, local authority meeting minutes, tender award notices. Nobody scrapes them because they're PDFs. Structured data from any document makes that cheap now.
  • Programmatic lists. public-apis on GitHub is a maintained list of free APIs by category. Skim it once; you'll find three you didn't know existed.
Tip
The fastest tell for a good source: it has an RSS feed, an API, or a 'download CSV' button, and the last update was within the month. If it has none of those but the data is clearly there, that's a harder but more defensible product, because your competitors will be put off by the same thing.

A warning about people's data

Warning
Data about identifiable people (names, emails, home addresses, directors' dates of birth) is personal data under UK GDPR even when it's public. Public does not mean free to do anything with. You can still build with it; you need a lawful basis and a reason. Missions 2 and 5 cover this properly. For now, prefer datasets about things (companies, properties, repos, premises) over datasets about people.
Checkpoint
You've opened data.gov.uk, the Companies House developer hub and the public-apis list in three tabs, and written down three sources you'd never looked at before.
Stuck? Ask the community
STEP 02 of 05~12 min

Run the scout: which datasets are actually fresh

You have a longer list of sources now, so find out which are actually alive. The data.gov.uk search box hides the two things you care about: when it was last updated and whether you can use it commercially. This script shows both.

data.gov.uk runs on CKAN, which has a clean JSON API. package_search takes a query and returns datasets with metadata, including metadata_modified (last touched) and license_id.

  1. Save this as scout.py.

    scout.py
    # scout.py : search data.gov.uk and print what's actually fresh
    # usage: python scout.py "planning applications" [more terms...]
    import sys, requests
    from datetime import datetime, timezone
    
    API = "https://data.gov.uk/api/action/package_search"
    
    def search(term, rows=25):
        r = requests.get(API, params={"q": term, "rows": rows}, timeout=30)
        r.raise_for_status()
        return r.json()["result"]["results"]
    
    def age_days(iso):
        if not iso:
            return 9999
        dt = datetime.fromisoformat(iso[:19]).replace(tzinfo=timezone.utc)
        return (datetime.now(timezone.utc) - dt).days
    
    for term in sys.argv[1:] or ["companies"]:
        print(f"\n== {term} ==")
        for d in search(term):
            age = age_days(d.get("metadata_modified"))
            fmts = sorted({(r.get("format") or "?").upper() for r in d.get("resources", [])})
            lic = d.get("license_id") or "no licence stated"
            flag = "FRESH " if age <= 60 else "stale "
            print(f"{flag}{age:>5}d  {lic:<14} {','.join(fmts)[:24]:<24} {d['title'][:60]}")
            print(f"        https://data.gov.uk/dataset/{d['name']}")
  2. Install the one library it needs, then run it with a few search terms in quotes.

    Terminal
    pip install requests
    python scout.py "food hygiene" "planning applications" "business rates"

Reading the results

Each row shows age in days, licence, formats and title, then the dataset URL. FRESH means touched in the last 60 days.

uk-ogl is the Open Government Licence, which is what you want. no licence stated means you email the publisher before building anything; often they'll confirm OGL, sometimes they won't.

  1. Run it for ten different search terms over the next few minutes. Don't search for what you think the product is; search for the raw thing. 'Licensing' not 'pub closures'. 'Incorporations' not 'new company leads'. The product is your idea; the dataset is what exists.

    Note what you see about formats. CSV and JSON you can use tonight. XLS is fine. PDF and HTML mean mission 2 (scraping) or pulling tables out of PDFs.

    'Stale' with a good licence is not useless: it might be stale on data.gov.uk because the publisher moved to their own API. Click through and check.

    Tip
    The metadata_modified date is when the catalogue entry changed, not always when the data did. For anything you shortlist, open the dataset page and look at the resource dates, or download the file and check the newest record inside it. That's the real freshness.
Checkpoint
The script runs and you have at least two FRESH, uk-ogl results in CSV or JSON for a search you care about.
If it breaksModuleNotFoundError: No module named 'requests' + 2 more
ModuleNotFoundError: No module named 'requests'
Run pip install requests (or python -m pip install requests if pip isn't on your path). On a Mac with Homebrew Python, pip3.
It returns results but every licence says 'no licence stated'
Common for older council datasets. Open the dataset page: the licence is often in the description text rather than the field. If it's genuinely absent, treat it as not reusable until the publisher says otherwise.
The API times out or returns a 5xx
data.gov.uk has slow hours. Wait a minute and rerun. If it keeps failing, drop rows to 10.
Still stuck? Ask the community
STEP 03 of 05~10 min

The five tests

Before you score anything, learn what you're scoring against. Every dead data project fails one of these. Learn them now and you'll reject bad ideas in thirty seconds instead of thirty hours.

  1. Read the five tests at a glance. Each one gets a fuller explanation below:

    The five tests
    FRESH         Does the value decay? Data that changes weekly is worth
                  paying for repeatedly. Data from 2019 is a one-off download.
    HARD TO GET   Is it behind an awkward portal, split across 300 councils,
                  in PDFs, or only visible one record at a time? Friction you
                  remove is what you sell.
    SOMEONE PAYS  Can you name the job title of the person who'd pay, and what
                  they do with it on a Tuesday morning?
    LEGAL         Is there a licence that allows commercial re-use (OGL, CC BY,
                  an API's terms)? Is there personal data in it?
    KEEP UPDATED  Can a script refresh it unattended? If it needs a human
                  clicking through a CAPTCHA, it's a job, not a product.

Fresh

This is the one that decides whether you have a subscription or a one-off download. A list of all UK charities is a download; anyone can get it, nobody pays twice.

Charities registered this week, with their stated purpose and trustees, is a feed, and a fundraising consultancy pays monthly for it. Prefer data where the value is in the delta.

Hard to get

If a buyer can get the same thing in five minutes from the source, they will. The gap has to be real: assembled from many sources, extracted from a hostile format, or enriched with something the source doesn't have (a classification, a geocode, a match to Companies House). Mission 3 is where you add that.

Someone pays

Name them. Not 'businesses' or 'investors'. 'Lettings agents with more than 50 properties in Manchester' or 'recruiters who place DevOps engineers'. If you can't picture the person and what they do with the data, you have a hobby.

This is the same muscle as finding a SaaS idea people pay for; if you've done that mission, use its buyer interview questions here.

Legal

Three questions. Is there a licence permitting commercial re-use? Does the source's terms of service or robots.txt forbid automated access? Is there personal data?

A 'no' on the first or a 'yes' on the second usually kills it. Personal data doesn't kill it but adds work (mission 5).

Keep updated

Can a script do the refresh on a schedule with no human in the loop? If the source changes its layout every month, or needs a login with 2FA, or blocks after 50 requests, the product will quietly die the first time you're busy.

The question to ask: if I go on holiday for two weeks, does it still update?

Warning
The test people skip is 'someone pays'. Technical people love 'hard to get' because it's a fun problem. A scraper nobody asked for is the most common dead project on this whole site.
Checkpoint
You can say, from memory, why each of the five tests kills a project when it fails.
Stuck? Ask the community
STEP 04 of 05~12 min

Score five candidates

Now apply the five tests. Writing scores down beats arguing with yourself. Three calibrated rows are included so your 2s mean the same thing as ours.

Each test scores 0, 1 or 2, giving a total out of 10.

  1. Save this as scorecard.csv and open it in a spreadsheet.

    scorecard.csv
    dataset,source_url,fresh(0-2),hard_to_get(0-2),someone_pays(0-2),legal(0-2),keep_updated(0-2),total,notes
    New charity registrations by region,https://register-of-charities.charitycommission.gov.uk,2,1,1,2,2,8,"weekly extract, OGL, buyers = fundraising consultants"
    Pub closures from licensing registers,council licensing portals,2,2,2,1,1,8,"each council different, personal data in some, hard to keep updated"
    Awesome-lists repo star counts,api.github.com,2,0,0,2,2,6,"anyone can get it, nobody pays for it raw"

Calibrate on the three examples

Read these before adding yours. The charity one scores 8 and is a plausible product.

The pub-closures one also scores 8 but look at where the points are: it's weak on legal and on keeping updated, which are the two tests that get harder with time, not easier.

The GitHub one scores 6 and is a 0 on 'someone pays', which means it's a 0 overall, whatever the total says.

The rule for rejecting

That's the rule: any 0 in 'someone pays' or 'legal' is a reject, regardless of total.

Totals of 8 or more with no zeros are worth taking to mission 2. Totals of 6 or 7 are worth a second look only if you can see how to raise one score (usually 'hard to get' by combining sources, or 'someone pays' by narrowing the buyer).

Score your own candidates

  1. Add five rows of your own from the scout results and the sources in step 1. Spend two minutes per row, not twenty.

    Be harsh on 'someone pays': 2 means you've seen someone pay for something like it (a competitor exists, a job ad mentions it, someone on a forum asked for it). 1 means you can argue it. 0 means you're guessing.

    Tip
    Competitors are evidence, not a problem. If a company already sells this data for £500 a month, the market exists and your version at £49 for a narrower slice is a product. If nobody sells anything like it, be worried, not excited.
Checkpoint
scorecard.csv has at least five rows of your own with no blank cells, and you've crossed out any with a 0 in 'someone pays' or 'legal'.
If it breaksEverything I score comes out at 5 or 6 + 1 more
Everything I score comes out at 5 or 6
Normal on a first pass. It means your candidates are too broad. Narrow each one: a region, a sector, a time window, a buyer. 'Planning applications' is a 5; 'planning applications for HMOs in Leeds, weekly, for landlords and letting agents' can be an 8.
My best idea has a 0 on legal
Check whether a narrower version passes. Often the problem is one field (home addresses, say) and the product works without it. If the whole dataset is behind terms that forbid re-use, move on. There is more data than you have weekends.
Still stuck? Ask the community
STEP 05 of 05~6 min

Choose one and write the brief

Pick one and commit. One paragraph that the next five missions build on. If you can't write it, you haven't chosen yet.

Pick the highest-scoring row with no zeros. If two tie, take the one with the better 'keep updated' score; freshness is what you'll be selling.

  1. Write this in a file called BRIEF.md, in a new folder for the project.

    BRIEF.md
    # Dataset brief
    
    Dataset: (name)
    Source: (URL of the API, download page or site)
    Licence: (OGL / CC BY / API terms link / unclear, emailed on (date))
    Format: (JSON API / CSV download / HTML pages / PDFs)
    Refresh: (how often the source changes, and how I'll detect a change)
    Personal data: (none / yes: which fields, and why I need them)
    Buyer: (job title, company size, what they do with it on a Tuesday)
    What I add: (the assembly, cleaning, classification or matching that makes it worth paying for)
    First product shape: (searchable site / CSV subscription / API / newsletter)
    Score: (n/10 from the scorecard)

Fill in the brief

  1. Fill every line. 'What I add' is the one to spend time on.

    The source is free; your product is the source plus what you add.

    If the honest answer is 'nothing, I just make it easier to download', the product is a convenience and it will be priced like one. If the answer is 'I match every record to its Companies House entry and flag the ones whose director changed this month', that's an analyst's morning saved and it's priced like one.

    Next: scrape it, legally. You'll pull the first thousand records from your chosen source with Firecrawl or Scrapy, respecting robots.txt, rate limits and personal data law, so the thing you build on is a thing you're allowed to sell.

Checkpoint
BRIEF.md exists with all ten lines filled, in a folder you'll use for the whole track.
Stuck? Ask the community

Your first wins

It works. Now do something with it.

Stress-test your brief

Paste the brief and let the model attack it:

Prompt
Here is a one-paragraph brief for a data product I'm considering: [paste BRIEF.md]. Score it 0 to 2 on each of these tests, with one sentence of reasoning each: fresh (value decays, so people pay repeatedly), hard to get (real friction I remove), someone pays (a nameable buyer), legal (licence permits commercial re-use, no ToS or personal-data problem), keep updated (a script can refresh it unattended). Then tell me the single weakest test and two specific ways to narrow the idea so that score goes up. Don't be encouraging; be accurate.

Find the buyers

Turn 'someone pays' from a guess into a list:

Prompt
I'm building a weekly feed of [dataset] for [buyer type] in the UK. List ten specific places where these buyers already talk about needing this data or pay for something similar: trade bodies, forums, job adverts that mention the task, existing paid products, LinkedIn groups, newsletters. For each, say what I'd search for to find the evidence. UK sources only.

Undo it cleanly

How to remove everything this mission installed.

Nothing installed beyond the requests library (pip uninstall requests if you want it gone). Delete the project folder to remove scout.py, scorecard.csv and BRIEF.md.

Next mission · Data hustleScrape it legally: Firecrawl for AI-ready pages, Scrapy for structured crawls
→