Web scraping: a program that reads web pages and turns them into data
Price lists, product pages, job posts and reviews are scraped every day. A scraper sees what any visitor sees, at the moment it runs.
Web scraping is using a program to fetch web pages and pull their content out as data, such as prices, product names or the page’s text, instead of copying it by hand. A scraper reads the page in front of it at the moment it runs. A page behind a login needs an account and steps someone writes, and nothing is kept unless you store it.
On this page
What a web scraper reads
A scraper requests a page, gets back its HTML and keeps the parts it was told to keep. Many tools also run the page’s JavaScript in a headless browser, so pages that build themselves in the browser can be read too.
Firecrawl’s scrape feature, for example, turns a URL into Markdown, cleaned HTML, a screenshot or structured JSON. It can also click, type and wait on a page before reading it. A pricing team might scrape 200 competitors’ product pages each morning and load the prices into a sheet.
How robots.txt applies to scraping
Sites publish a robots.txt file to tell crawlers which paths they may fetch. The rules are standardised in RFC 9309, the Robots Exclusion Protocol, which says these rules are not a form of access authorisation. A site’s terms of use and the law in each country are separate questions, and a page that sits behind a sign-in is a different matter again.
Where web scraping stops
- Logins: a scraper gets past a sign-in only with an account someone created and steps someone wrote, and the steps break when the page changes.
- Time: a scraper reads the page as it is now. A price that changed on Tuesday is gone by Wednesday unless an earlier run stored it.
- Inboxes: emails and texts a company sends after a sign-up never appear on a page, so no scraper sees them.
Each of those sits behind a login wall: the welcome email, the signed-in price, the trial’s limits and the offer in a cancel flow.
How what companies show customers is kept over time
Obsession collects what companies show only to customers, step by step: their screens, emails and texts, member prices, trial limits, quotes and cancel offers, each dated with a screenshot. Teams and AI agents look up any company through the API or MCP, and a company not yet covered is added on request. Each account stays open for as long as you follow the company, so every new email, price and offer lands dated the day it arrives.
A scraper reads one page on the day it runs, so teams often use both. Obsession vs Firecrawl sets out which job each one does.
A crawler follows links to discover pages, as a search engine does. A scraper extracts content from pages it was pointed at. Many tools do both: crawl to find the pages, then scrape to read them.
When someone gives it an account and writes the steps to sign in, and the site doesn’t stop it with a CAPTCHA or a bot check. It still reads only what that account shows on the day it runs.
Login wall, definedOnly what you store. Each run returns the page as it is now, so a history means saving every run with its date and comparing them.
Sources
In the order the page cites them. Every page was read on 11 Nov 2026.