Scrapy
A Python framework for crawling websites and pulling structured data out of them at scale.
What you can do with it
- Collect public prices from a set of shops into one sheet
- Build a dataset from public listings
- Monitor a page for changes
Worth knowingNeeds Python. Respect each site's terms and robots.txt, go slowly, and don't collect personal data about people.
The full tutorial
01The four lines you don't crossFREE02Personal data: the UK GDPR basics for scrapersMEMBERS03Firecrawl: any page to clean markdownMEMBERS04Scrapy: a polite spider that yields rowsMEMBERS05Pull the first real batch of your datasetMEMBERS
Scrape it legally: Firecrawl for AI-ready pages, Scrapy for structured crawls · ~60 min · tested
Open missionMore in Agents & automation
MiroFish
Agents & automation
Simulates thousands of AI agents with memory and personalities to forecast how a scenario could play out.
OpenClaw
Agents & automation
A personal AI assistant that runs on your devices and actually does things across your apps.
Espanso
Agents & automation
A free text expander: type a short code like :addr and it becomes your full address, an email template or today's date, in any app.