The web behind the login: what search engines can’t see, and why AI agents need it

The web behind the login is what companies show only to people who sign up, log in, fill a form, check out or try to cancel: the emails and texts that follow, member prices, trial limits, quotes and the offer made to stay. Search engines can’t index it because their crawlers never sign in, so AI agents that rely on search can’t see it either. Obsession collects it at any company, each fact dated with its screenshot, and teams and AI agents look it up through the API or over MCP, the standard way AI agents call tools.

Most of what a company does to its customers never appears on a page a search engine can open. The welcome email and its code, the member price and the limits of a free trial all sit behind a step that asks the visitor who they are. So do the quote a form returns, the delivery fee shown at the last step and the offer made to someone who tries to cancel.

A large share of revenue travels through that private channel. Klaviyo’s own 2026 benchmarks show that automated flows, such as welcome and basket reminder emails, generate nearly 41% of email revenue from 5.3% of sends. Only people who signed up or left a basket ever receive them.

What the web behind the login is

The web behind the login is everything a website shows only to people who have signed up, logged in, filled in a form, started a checkout or tried to cancel, together with everything it sends them afterwards by email, text or chat.

“The login” is shorthand. The same line falls wherever a website asks for an email address, an account, details for a quote or a payment page before it shows the next screen. On the near side is the public web: the pages anyone can open, which search engines crawl and AI search tools retrieve. On the far side, what a person sees depends on who they are, when they joined and what they did.

Three properties set it apart from a public page. It needs an identity, such as an email address, a phone number or an account. It unfolds over time: an offer sent on day 12 of a free trial exists only for accounts that were there on day 12. It also differs by person, so a member, a new subscriber and a visitor from another country can each see a different price for the same product.

What sits behind each step

Each step a website puts in front of its customers hides different information, and competitors, prospects, investors and regulators each want a different part of it.

A silence counts too. The reminder that never came after a basket was left says as much about a company as the one that did. Only an account that was there, waiting, when it should have arrived can show it.

How much of the web search engines can’t see

In August 2001 the Journal of Electronic Publishing published Michael K. Bergman’s white paper The Deep Web: Surfacing Hidden Value, from data collected in March 2000. Bergman wrote that search engines build their indices by crawling pages that are static and linked. Deep web pages, he wrote, ‘do not exist until they are created dynamically as the result of a specific search’.

He estimated that public information on the deep web was 400 to 550 times larger than the surface web, at about 7,500 terabytes against 19. He counted nearly 550 billion documents against 1 billion, across more than 200,000 deep web sites.

The most telling later work came from inside Google. In Google’s Deep-Web Crawl, presented at the VLDB conference in Auckland in August 2008, Jayant Madhavan, Alon Halevy and four colleagues described how Google surfaced the deep web. It pre-computed submissions for HTML forms and added the result pages to its index. They put the number of high-quality HTML forms on the order of 10 million, and their surfaced content appeared in the first 10 results on Google.com for over 1,000 queries a second.

The same paper drew the line that still holds. Google filtered out ‘forms with password inputs and any keywords typically associated with personal information, e.g., username, login’. It also left aside forms that submit changes, which under the HTML specification are meant for actions the paper lists as ‘shopping carts, travel reservations, and logins’. The system Google built to index the hidden web excluded logins and shopping baskets by design.

Most of what Bergman measured was public. He found that 95% of the deep web was publicly accessible, ‘not subject to fees or subscriptions’, and that more than half of it sat in topic-specific databases. Neither study measured what companies show and send their own customers after a sign-up. That can’t be measured from outside, because it differs for every account and arrives over time.

The two problems need different machines. Bergman’s deep web is a retrieval problem: send the right query and the page appears, which is why Google could surface millions of forms automatically. The web behind the login is a problem of identity and time. Someone has to become a customer, stay one and keep what arrives, dated, because none of it can be fetched again later.

None of this is the dark web. The dark web means sites reachable only through anonymity networks such as Tor. The web behind the login is ordinary commerce: a bank’s rates for existing customers, a shop’s member prices, the emails a software company sends during a trial.

Why search engines stop at the login

A search engine builds its index by crawling: a program called a crawler fetches a page, follows its links and stores what it finds. Google says its crawlers ‘automatically visit publicly accessible webpages’, and that its index covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size. Five gaps keep content out of that index.

1. No account. Google’s Search Console help says that ‘Googlebot never provides credentials’. A page that asks for a login is reported as blocked due to an unauthorized request, and Google can’t reach a page that ‘requires a login’.
2. No session. A login lasts because the website stores a small token in the browser. That token is a cookie, which RFC 6265 describes as the way servers ‘maintain a stateful session over the mostly stateless HTTP protocol’. Without an account there is no session to keep, so every page is served as it would be to a stranger.
3. No address. An email or a text message lives in one person’s inbox, never at a web address a crawler can fetch. The welcome series, the basket reminder and the renewal notice are delivered to each customer separately.
4. No single version. A signed-in price or a trial screen changes with the account viewing it, so there is no one page to store.
5. No past. An index stores a page as it was when crawled. The offer sent on day 12 of a trial, or the price rise notice before a renewal, exists only for accounts that were there when it went out.

Pages a site asks crawlers to skip are a separate case. The robots.txt standard, RFC 9309 from September 2022, lets a site list the addresses crawlers should leave alone, and it says that ‘these rules are not a form of access authorization’. Anyone can still open a page listed in robots.txt; nobody can open a page behind a login without an account.

Why AI agents need it now

AI agents now answer questions and take actions for people: comparing plans, booking, buying and cancelling. Many of them reach the web through search APIs built for them. Exa calls itself ‘The world’s data for AI agents’ and says it searches ‘the largest index of public web and private information’. Its search page describes its own index of 100 billion documents and names the proprietary data beside it, such as contact data, job postings, earnings call transcripts and investor portfolios.

Tavily offers ‘one secure API for real-time web access’. Parallel describes ‘web infrastructure for AI to search, extract, monitor, and reason over the world’s information’. Firecrawl calls itself an open-source web data API for AI agents and apps built on large language models. Its Interact feature runs ‘a stateful browser session that stays alive across calls’, with profiles, for pages that need a login. How the four compare on index, freshness, extraction and price is in Exa vs Tavily vs Firecrawl vs Parallel.

Each is strong at its job, and each answers from what a page shows when the agent asks. None of their pages describes an account kept at a company for weeks, an inbox that receives what the company sends, or a dated history of it. That history can’t be fetched after the fact: an email sent last month to new subscribers exists only in the inboxes of people who had subscribed before it went out.

Agents that drive a browser can sign in to a person’s own account for one task. Anthropic’s computer use tool gives Claude ‘screenshot capabilities and mouse/keyboard control’. Its documentation advises against giving the model ‘access to sensitive data, such as account login information’. It also warns that using the tool inside applications that require a login raises the risk of prompt injection, where instructions hidden in a page take over the agent. An agent in a session can read what one account shows today. It can’t have been a subscriber for the last 30 days.

Many of the questions people ask agents about companies sit on the far side of the login. They ask about a company’s offer to new subscribers this month, where its free plan stops and its offer to someone who cancels. An agent grounded only in the public web answers those from the marketing page, or not at all. Can AI agents log in to websites covers sessions, one-time codes and the law, and how to scrape data behind a login covers doing it in your own account.

Who uses the web behind the login

Any team whose decisions depend on what other companies do to their customers: the teams that sell to other companies, compete with them or invest in them.

• Agencies open a pitch with the prospect’s own emails, and the ones that never came.
• Sales teams see what each account sends its own customers before the call.
• Marketing and product teams follow every competitor’s emails, onboarding and paywall without signing up themselves.
• Pricing teams compare competitors’ member prices with their public ones, date by date (member-only pricing).
• Banks, fintechs and their conduct teams keep dated screenshots and emails of what customers are shown and sent, their own and competitors’ (Consumer Duty fair value assessment).
• Compliance teams at subscription companies test that cancel, unsubscribe and STOP work. The US Federal Trade Commission and two international enforcement networks, ICPEN and GPEN, reviewed 642 subscription websites and apps with officials from 27 authorities in 26 countries. Nearly 76% used at least one possible dark pattern.
• Funds and research firms read offers, prices and rates across a market, month after month, because that history can’t be collected later.
• Teams building AI agents give their agent a competitor’s real price, trial limit or offer in one call (the API and MCP).

How the web behind the login is collected

Collecting it means doing what any customer does and staying long enough to see what follows. That means signing up with a real inbox and phone number, starting the trial, asking for the quote, leaving the basket, opening the chat and starting to cancel. Every email, text and screen is then kept exactly as it arrived, with the time.

Obsession collects what companies show only to customers, step by step: their screens, emails and texts, member prices, trial limits, quotes and cancel offers, each dated with a screenshot. Teams and AI agents look up any company through the API or MCP, and a company not yet covered is added on request.

An Obsession agent signs up the way any customer can, with its own inbox, phone number and browser, and says it’s an AI when it talks to a person. It never gets round a CAPTCHA or a bot block, and it never invents identity details (who visits a site, and the lines every agent keeps). Each new email, price and offer is added on the day it arrives, so a company’s history builds from the first day it is followed. One question can be asked across many companies at once, a row each. Obsession’s agents have run 1,500 sign-ups, baskets and checkouts at real companies, and counting.

Where every other source stops

We make Obsession, so every other option is described from its provider’s own pages as of 11 November 2026.

Each source is good at its own job, and the sources differ in how far past the public page they reach and for which companies. Exa’s pricing, Tavily’s, Parallel’s and Firecrawl’s are priced per call, per credit or per page. Panoramata follows the competitors on a plan across emails, flows, SMS, ads and landing pages. MailCharts says it signs up to, triggers and classifies thousands of ecommerce email journeys. Competiscan draws on long-running consumer panels, and Klue turns what a sales team collects into battlecards and win-loss stories. The full comparison, step by step, is on the comparison page.

Search engines index what websites publish. Obsession indexes what they show only past the login.

For what a company shows and sends its customers after the login, at any company you name, dated and kept for as long as you follow it, use Obsession. For what a company publishes, use a search engine or a web search API. Use a crawler to read pages you can open, an archive or panel for the brands it already follows, and a competitive intelligence tool to share what a sales team learns. They work together: an agent can call a search API for the public page and Obsession for what sits behind it.

When the public web is enough

When the question is about what a company publishes, the public web answers it: a press release, a pricing page, documentation, job posts, reviews. A team that follows two or three competitors whose offers rarely change can sign up by hand once a month and keep screenshots in a shared folder. The web behind the login matters when the answer changes with who is asking, arrives after the first visit, or has to be shown later with its date.

Questions about the web behind the login