Get Started
Data hustle

Mission 2 of 6

Scrape the web legally and pull your first real batch of data.

Scraping is legal in the UK until you cross one of four lines. Learn them first, then use Firecrawl for clean markdown and Scrapy for structured rows.

60 minIntermediateFreeStep 1 free

What you'll do

Learn the rules first

Run through the four legal lines and the UK GDPR basics before you fetch a single page.

Scrape with Firecrawl and Scrapy

Turn pages into clean markdown, or crawl a site into rows, with polite defaults built in.

Pull your first real batch

Collect the first thousand records of your chosen dataset and keep the raw pages.

  • Run a pre-scrape checklist before every new site
  • Keep raw pages so changing your mind costs nothing
Before you start
  • BRIEF.md from mission 1 with a chosen source
  • Python 3.10+ and pip. Docker only if you want to self-host Firecrawl (Docker without fear)
  • A Firecrawl account for the hosted route (free tier) or Docker for the self-hosted one
  • If your source has an API, you may not need this mission at all: see find the hidden API first
Your progress0/5
01/05The four lines you don't cross0 done
STEP 01 of 05~12 min

The four lines you don't cross

Rules come before tools. Know exactly what makes a scrape legal in the UK, so you can build with confidence and say no to the jobs that aren't.

There are four lines you don't cross. The checklist at the end of this step is what you run before every new site.

Line one: robots.txt

Every site can publish /robots.txt listing paths crawlers shouldn't touch. It isn't law, but ignoring it is the first thing a claimant points to, and it's the clearest signal of what the site owner permits.

  1. Open https://thesite/robots.txt before anything else.

    Disallow: / under User-agent: * means don't. A Crawl-delay line tells you how fast is acceptable. Scrapy obeys it by default; Firecrawl's hosted service does too.

Line two: terms of service

If you've agreed to terms (you made an account, or the site makes a reasonable case that use equals acceptance) and they prohibit automated access, scraping is a breach of contract.

  1. Search the terms page for 'scrap', 'automated', 'crawl', 'bot'.

    Public pages with no account and no click-through terms are the safest ground. Anything behind a login where the terms forbid it: stop.

Line three: access controls

The Computer Misuse Act 1990 makes unauthorised access an offence. Bypassing a login, a paywall, a CAPTCHA, IP blocking or rate limiting to get at data is how a civil matter becomes a criminal one.

The rule is simple: if the site has put something in your way on purpose, you don't go around it. Rotating proxies to beat a ban is going around it.

Line four: rate

Even permitted scraping can be a problem if it degrades the site. One request a second is polite for most sites. Big public services (Companies House, gov.uk) publish limits; use them.

Scrape at night for UK sites. Identify yourself in the User-Agent with a contact email so an admin can tell you to slow down instead of blocking you.

Two more things that help in a dispute

Prefer an official API or bulk download whenever one exists, even if it's clunkier; it's an explicit grant of permission.

And the database right (Copyright and Rights in Databases Regulations 1997) protects substantial investment in collecting data: copying all of a database someone paid to build is a different matter from collecting facts from public pages. Facts aren't copyrightable; someone's curated compilation of them might be.

The checklist

  1. Save this as CHECKLIST.md and run through it for your target site before you scrape it.

    Pre-scrape checklist (save as CHECKLIST.md)
    BEFORE YOU SCRAPE A SITE
    
    [ ] robots.txt read. The paths I want are not Disallowed for * or for my agent.
    [ ] Terms of service read (search the page for "scrap", "automated", "crawl", "robot").
    [ ] There is no login, paywall or CAPTCHA between me and the data.
    [ ] I have identified every field that is personal data, and I either
        don't collect it or have written down why I need it (legitimate interest).
    [ ] Rate: one request per second or slower, in the site's quiet hours,
        with a User-Agent that names me and gives a contact email.
    [ ] I'm storing the raw pages so I never have to re-fetch them.
    [ ] An API or bulk download exists? Then I'm using that instead.
    Warning
    This is a practitioner's summary, not legal advice. If the dataset is going to be your main income, spend an hour with a solicitor who knows data law before launch. It's cheaper than the alternative and it'll be the best-spent money on the whole product.
Checkpoint
You've opened your target site's robots.txt and terms, and ticked or crossed every line of the checklist for it.
If it breaksrobots.txt Disallows the exact pages I need + 1 more
robots.txt Disallows the exact pages I need
Then that source is out for scraping. Check for an API, a bulk download, or an FOI route (mission 5). Or find a different source for the same data; the same records are often published in more than one place.
There's no robots.txt at all
That means no restriction has been expressed, not that anything goes. The other three lines still apply. Go slower than you otherwise would.
Still stuck? Ask the community
Next mission · Data hustleClean and enrich it: from a messy dump to a table an LLM has labelled
→