How to scrape data behind a login: sessions, cookies, MFA, and when to use an index

To scrape data behind a login, sign in once with your own account, save the session the site sets, and send it with every request until it expires. Use a Python requests session for a plain form login or Playwright’s saved storage state for a JavaScript site, and plan for CSRF tokens, one-time codes and rate limits. For what many companies show their customers over weeks, Obsession keeps each email, price and offer dated with its screenshot, by API or MCP (how AI agents call tools).

Developers and data teams scrape behind a login to pull reports, orders, prices or messages from accounts they hold, when the site offers no export that fits. Every tool follows the same method. Sign in once, keep the cookie or token the site hands back, send it with each request, and stop when the site stops accepting it.

How a login keeps you signed in

A login is an exchange. You send a username and password once, and the site answers with something your browser sends back on every later request to prove it’s still you. On most sites that is a session cookie. RFC 6265, the standard for cookies, has the server set a cookie, a short named string, in a Set-Cookie header. The browser returns it in a Cookie header on each later request to the same site. A session cookie usually holds a long random ID that points to your session on the server. It is often marked HttpOnly, which RFC 6265 says “limits the scope of the cookie to HTTP requests”, so the page’s own scripts can’t read it.

Sites built as JavaScript apps often keep a token in local storage or IndexedDB, the browser’s own storage for each site, instead of a cookie or beside one. Forms add a CSRF token, short for cross-site request forgery token. In OWASP’s guidance, the server generates it once per session or per request, sends it inside the page, and rejects a form post where the token is missing or doesn’t match.

MFA (multi-factor authentication) asks for a second check after the password. It can be a code from an authenticator app or a text message, a push to a phone, or a passkey. Authenticator codes follow TOTP, the time-based one-time password standard, which derives a fresh code from a secret shared at setup and the clock, every 30 seconds by default. A scraper can’t answer any of these without the phone, the secret or the passkey, which is the point of MFA. When you set up your own account’s authenticator, you hold the secret and your code can compute the codes, but a secret stored beside the password turns two factors back into one.

A scraper behind a login therefore has three jobs: get the session once, carry it on every request, and notice when the site stops accepting it.

Before you write any code

• Use your own account, or one your company holds for this job. Someone else’s password, a shared login the terms forbid, or an account the site has closed is where the legal risk gathers.
• Look for an export or an API first. A CSV export or a documented API stays inside the site’s terms and survives a redesign. A scraper breaks every time the page layout changes.
• Read the terms you accepted at sign-up. Check whether they forbid automated access or scraping. Breaking them can be a contract claim even where computer misuse law doesn’t apply.
• Read robots.txt as crawling preferences. RFC 9309, the standard for robots.txt, says its rules “are not a form of access authorization”. The login is the site’s access control.
• Store the session like a password. A saved cookie lets anyone who holds it act as you until it expires. Keep it in a secrets store or an ignored folder, never in a repository.

Scraping a website with login in Python

For a site that renders its pages on the server and signs in through a plain HTML form, an HTTP client is enough. A requests Session keeps cookies across every request made from it and reuses the connection to the same host, so you sign in once and the cookie travels with each later call.

1. Create the session: s = requests.Session().
2. Fetch the login page with s.get(login_url, timeout=10) and read the hidden CSRF field from the form with an HTML parser.
3. Post the form with s.post(login_url, data=form). The form holds the username, the password read from an environment variable, the token and any other hidden fields.
4. Request a page only a signed-in user sees and look for a marker, such as your account name or a Log out link.
5. Fetch the pages you need from the same session, with a timeout on every call and a pause between pages.

A pass: step 4 finds the marker. If the response is the login page again, the post was rejected. Sign in by hand once with the browser’s developer tools open on the Network tab. Compare the request the form sent with yours: the field names, the hidden values and the redirect after it. A site that signs in through JavaScript calls, or keeps its token in local storage, needs a browser.

Playwright: sign in once, save the storage state

Playwright is an open source library that drives Chromium, Firefox and WebKit through one API. Its storage state is a JSON file of the cookies and stored data a signed-in browser holds. A new browser context, Playwright’s name for a clean, separate browser profile, loads it and starts signed in. In Python, the steps are these:

1. Sign in once in a visible browser, entering any one-time code by hand.
2. Save the state with the context’s storage_state, giving it a path such as state.json.
3. Start every later run from it: open a new_context with storage_state set to that file.
4. Open a signed-in page first and check the marker. If it’s gone, stop and sign in again.

Playwright’s docs say saved state covers cookies, local storage and passkey logins. IndexedDB is saved only when you turn on the indexed_db option, which apps that keep their token there need; Firebase Authentication is Playwright’s example. Session storage isn’t saved at all, and the docs give a short script for the rare site that keeps sign-in data there. They also warn that the file can hold cookies and headers that let anyone with it act as you. Keep it out of version control, and delete it once it expires.

A pass: a fresh run opens a signed-in page without ever seeing the login form.

Hosted scrapers and browsers that keep a login

When the job runs in the cloud, on a schedule or across many browsers at once, hosted tools keep the same saved session for you. Each is described from its own pages as of 11 November 2026.

Firecrawl: actions, Interact and profiles

Firecrawl turns web pages into clean markdown. Its scrape endpoint accepts actions, such as write, press, click and wait, which run in order before the page is read. Its docs show actions filling in a login form. For signed-in work its docs recommend Interact, a browser session driven by plain-language prompts or Playwright code. Interact can save cookies and local storage to a named profile when the session stops, and later scrapes load that profile already signed in. One session at a time can write to a profile. Sessions end after 10 minutes by default, or after 5 minutes idle. A browser minute costs 2 credits with code or 7 with prompts.

Browserbase: Contexts

Browserbase runs browsers in the cloud. A Context carries a browser’s user data from one session to the next: cookies, local storage, IndexedDB and session storage. Browserbase encrypts it at rest. A session started with persist set to true writes its changes back when it closes. Its authentication guide advises signing in once by hand through its live view of the browser, then reusing the Context. It says two-step codes and magic links usually need a person in the loop. The Contexts page adds that Contexts don’t expire but a site’s cookies still do, after 30 days in its example, and advises one Context per site and login, with no two sessions on one Context at once.

Browser Use: your Chrome or a cloud profile

Browser Use is an open source library in which an AI model drives the browser from a task written in plain words. It can attach to the Chrome you already have open, with your logins, or start a new Chrome on a temporary copy of one of your Chrome profiles. Its docs warn that code the model writes can read the cookies in those profiles, and advise running untrusted tasks in a container or virtual machine. Cloud browsers keep logins in a profile of cookies and local storage, which the docs say to sync again when a site’s login expires. Cloud browsers also solve CAPTCHAs automatically unless the solver is turned off.

n8n web scraping with login

n8n builds scheduled workflows from nodes. Its HTTP Request node sends any request with your own headers, can return the response headers and status code with the body, and can import a request pasted as a cURL command. Its batching option sets how many items go in each batch and how long to wait between batches.

1. An HTTP Request node posts the login form, with the response headers turned on.
2. The next step reads the session cookie from the set-cookie header.
3. Every later HTTP Request node sends it as a Cookie header, held in a Header Auth credential so the cookie stays out of the workflow itself.
4. The HTML node extracts each field from the page by CSS selector.

n8n has no browser of its own, so for a JavaScript site the HTTP Request node calls a hosted browser’s API. A request copied from the browser’s developer tools as cURL carries your live session cookie, so treat the pasted command like a password.

Teams building agents that sign in for a person face the same cookies and codes at a larger scale. Can AI agents log in to websites covers browser agents, signed agents and the cost of each run.

Why logged-in scraping breaks, and the fix

Scrapers behind a login fail quietly. The most common failure is a run that saves the login page as data, because the session expired overnight. Check a signed-in marker before every run, and stop the run on any of these signs.

RFC 6585 defines 429 Too Many Requests for a client that “has sent too many requests in a given amount of time”. The server may add a Retry-After header saying how long to wait. Honour it, keep a pause between pages, and keep one session per account, because two runs on one login can sign each other out.

The legal lines, in the US and the UK

Scraping behind a login is a different legal question from scraping public pages, because the login is the site’s permission system. Each point names the place it applies, and all of it is general information, never legal advice.

United States: the Computer Fraud and Abuse Act

The Computer Fraud and Abuse Act covers anyone who accesses a computer “without authorization” or “exceeds authorized access” and obtains information (18 U.S.C. § 1030). It is a criminal law, and it also lets anyone who suffers damage or loss sue. In Van Buren v. United States, decided on 3 June 2021, the Supreme Court read it as a “gates-up-or-down inquiry”. A person exceeds authorized access by obtaining information from areas of a computer, such as files, folders or databases, that are off limits to them. The Court left open whether limits written into contracts or policies count, or only technical ones.

In hiQ Labs v. LinkedIn, filed on 18 April 2022, the Ninth Circuit said the Act is likely violated when a person circumvents “generally applicable rules regarding access permissions, such as username and password requirements”. The public parts of a website, it said, have no gate to lift. Earlier cases govern situations where permission is required and was never given or has been revoked. Sites also keep other claims where the Act doesn’t apply, breach of contract among them. On those lines, your own credentials on your own account, used within the terms you accepted, are the safest ground.

United Kingdom: the Computer Misuse Act

Section 1 of the Computer Misuse Act 1990 makes it an offence to cause a computer to perform a function intending to access a program or data. The access must be unauthorised, and the person must know it. Under section 17(5), access is unauthorised when the person isn’t entitled to control it and has no consent from someone who is.

Personal data on the page

Pages behind a login often show other people’s details, such as a member directory or a customer list. The UK GDPR’s Article 4 counts collection, recording, storage and retrieval of personal data as processing, so the law’s duties apply to a scraper as much as to a form. Watching a competitor as a customer raises its own questions, which is it legal to mystery shop competitors answers by jurisdiction.

When an index is the better answer

A logged-in scraper answers one question well: what does this account show right now. Three jobs sit outside it.

• What arrives, rather than what sits on a page. Welcome emails, texts and renewal notices reach an inbox or a phone. The offer a company makes when a customer clicks Cancel appears once, at that moment. No page holds any of them for a scraper to read.
• Many companies at once. Following 30 competitors means 30 accounts, each with its own email address and often a phone number. It also means 30 sign-in routines to keep alive through expiries and codes, and 30 parsers to mend after each redesign.
• A dated history. A scraper’s record starts the day it starts. A team that has to show when an offer appeared, or that a reminder never came, needs each email kept as it arrived and each page captured with its time.

Obsession collects what companies show only to customers, step by step: their screens, emails and texts, member prices, trial limits, quotes and cancel offers, each dated with a screenshot. Teams and AI agents look up any company through the API or MCP, and a company not yet covered is added on request.

Exa, Tavily and Firecrawl give AI agents the public web, the pages any visitor can read, and Exa vs Tavily vs Firecrawl compares them. Playwright, Browserbase and Firecrawl’s profiles keep your own logins working. Obsession keeps what each company shows only to its customers, past the sign-up and the login, dated, for as long as anyone follows the company. An agent can call all of them: search for what a company publishes, a scraper for what your own account holds, and Obsession for what each company sends and shows its customers. The web behind the login explains what sits past each sign-up, login, form, checkout and cancel flow.

Agents reach it with 5 verbs: lookup, facts, evidence, index and notify. They work the same over MCP, the standard way AI agents call tools, the API, and SDKs in TypeScript and Python. The evidence verb returns the raw email with its DKIM result, a signature checked on arrival to prove which company sent it, or the page as captured, with its time. Notify posts each new fact to a webhook, an address on your server, the moment it lands. Each Obsession agent uses accounts of its own and tells any person it talks to that it’s an AI. No Obsession agent gets round a CAPTCHA or a bot block, as the rules every Obsession agent keeps set out. Obsession’s agents have run 1,500 sign-ups, baskets and checkouts at real companies, and counting.

Tools for scraping behind a login, compared

The options run from the site’s own export to your own code, hosted browsers, AI agents and an index. We make Obsession, so every other option is described from its provider’s own pages as of 11 November 2026.

For what many companies send and show their customers over weeks, with each email and page kept as it arrived, use Obsession. For your own account’s data, start with the site’s export or API. Use a requests session for a server-rendered form login, and Playwright with saved storage state for a JavaScript site. Firecrawl Interact fits signed-in pages as markdown, and Browserbase Contexts fit many cloud browsers on saved logins. Browser Use fits tasks too varied to script, and n8n runs any of them on a schedule.

When your own script is enough

• Your own account on one site, with a report you pull each week and a layout that rarely changes.
• A one-off backfill of your own data before you move to another tool.
• A site with an awkward export, where a script tidies a file the site already gives you.
• Data that sits on a page whenever you look, such as an order history or a usage dashboard, rather than messages that arrive over time.

Start with one signed-in page

Sign in by hand once, save the session, and write the marker check before anything else; it catches most failures before they become bad data. For what other companies send and show their customers, across as many companies as your agent asks about, connect Obsession through the API or MCP.

Questions about scraping behind a login