How AI Companies Collect Web Data: Crawlers, Tools and the Fight Over Scraping
Every major AI company now runs its own web crawlers, sells tools that search and read the web, and is fighting over who gets to collect what. We read each company's own documentation and the main court rulings to separate what they actually do from what's assumed.
AI bot activity for training
~80%
Share of AI crawling done for training, July 2025[26]
AI companies run three different kinds of bot. One collects training data, one builds a search index, and one fetches pages when a user asks. Blocking one doesn't block the others.
The bots that fetch for a user often skip robots.txt by design. OpenAI, Google, Meta, Amazon and Perplexity all say so in their own documentation. Anthropic says its bots follow robots.txt and won't try to get past CAPTCHAs[2].
You can opt out of AI training without leaving search at Google (Google-Extended) and Apple (Applebot-Extended). Neither token crawls anything; it only tells the company how it may use what its search crawler collects[3][6].
AI bots take far more than they send back. In July 2025, Cloudflare counted 38,065 Anthropic crawls and 1,091 OpenAI crawls for every visitor those platforms referred[26].
Compliance is disputed. Cloudflare accused Perplexity of using undeclared crawlers to get around blocks, and Perplexity disputed it[28][29]. TollBit data put about 30% of AI scrapes in late 2025 against sites' robots.txt instructions[27].
AI web tools are built to answer questions, not to extract data at scale. They charge per search, follow robots.txt in some cases, and can't always read JavaScript-built pages[10].
Courts are drawing lines. Ignoring robots.txt alone wasn't treated as breaking anti-circumvention law, but getting past JavaScript challenges and CAPTCHAs can be[16][20].
Quick facts
Training share: training drove nearly 80% of AI bot activity in July 2025, up from 72% a year earlier[26].
Sites blocking AI bots: in mid-2025, 546 of 3,816 top domains checked (about 14%) had robots.txt rules aimed at AI bots, most often GPTBot[30].
Googlebot vs. everyone else: in 2025 Googlebot made 4.5% of HTML requests on Cloudflare's network, more than all other AI bots combined at 4.2%[31].
Fastest growth: AI crawling triggered by user requests grew more than 21 times during 2025[31].
Three kinds of AI bot
Training crawlers collect pages that may be used to train future models, for example GPTBot, ClaudeBot and Meta-ExternalAgent.
Search crawlers build the index behind AI search answers, for example OAI-SearchBot, Claude-SearchBot and PerplexityBot.
User-triggered fetchers load a specific page because a person asked the assistant to, for example ChatGPT-User, Claude-User and Perplexity-User. Most companies treat these like a person's browser rather than an automated crawler.
Crawlers by company
From each company's own crawler documentation. A dash means we didn't find it in their docs.
We found no official crawler documentation from xAI, DeepSeek or ByteDance. Common Crawl, the open web archive used widely for AI training, says it honors robots.txt and doesn't go behind paywalls[32].
Who follows robots.txt
Automatic crawling: the companies above say their training and search crawlers follow robots.txt. Google says its common crawlers "always respect robots.txt rules for automatic crawls"[33].
User requests: fetches triggered by a person are treated differently. OpenAI says robots.txt "may not apply" to ChatGPT-User[1], and Google says its user-triggered fetchers "generally ignore robots.txt rules"[4].
Anthropic is the exception among the major labs: it says all its bots honor robots.txt, support Crawl-delay, and "will not attempt to bypass CAPTCHAs"[2].
Testing it: when Cloudflare blocked ChatGPT in robots.txt, it found ChatGPT-User fetched the robots file and stopped[28].
How much they crawl
Pages crawled for every visitor sent back
July 2025, Cloudflare data[26]. Bars are to scale; values are on the right.
Anthropic
38,065
OpenAI
1,091
Perplexity
195
Anthropic's ratio fell sharply during 2025, from 286,930 crawls per visitor in January[26]. Perplexity's rose over the same period.
Search and fetch tools for developers
Every major lab now sells API tools that let its models search or read the web. Prices are each company's published list prices.
$5.00 per 1,000 requests; Fast Search $1.00 per 1,000[14][15]
These tools are built to ground an answer in a few pages, not to collect data in bulk. Anthropic's fetch tool, for example, can only open URLs that already appear in the conversation, skips pages blocked by robots.txt, and doesn't run JavaScript[10].
Browsing agents
Google: Chrome "auto browse" launched in January 2026 as a US-only preview for Google AI Pro and Ultra subscribers[34].
Microsoft: retired Edge's Copilot Mode in May 2026[35].
Amazon and Perplexity: Amazon's fight with Perplexity's Comet shopping agent became a court test of AI agents acting inside logged-in accounts[22][23].
Court cases
Case
When
What happened
Ziff Davis v. OpenAI (S.D.N.Y.)
Dec 2025
Judge dismissed a claim that ignoring robots.txt broke anti-circumvention law; robots.txt doesn't control access to content[16]
Bartz v. Anthropic (N.D. Cal.)
Jun 2025; settled
Training on the books was ruled fair use, but claims over pirated copies went toward trial. A $1.5 billion settlement got final approval in July 2026[17][18]
Reddit v. Anthropic
Mar 2026
Federal court sent the case back to California state court; Reddit's contract and tort claims over scraping are not pre-empted by copyright law[19]
Reddit v. SerpApi, Perplexity and others (S.D.N.Y.)
Jul 2026
Court treated Google's SearchGuard (JavaScript challenges and CAPTCHAs) as an access control, letting circumvention claims proceed[20]
Thomson Reuters v. ROSS (3rd Cir.)
Sep 2026
Appeals court affirmed the ruling against ROSS's AI legal research tool, in a sealed opinion[21]
Amazon v. Perplexity
Mar–Aug 2026
Amazon won an injunction against Perplexity's Comet shopping agent; an appeals court later overturned it[22][23]
Getty Images v. Stability AI (UK)
Appeal pending, Nov 2026
High Court rejected Getty's secondary infringement claim; Getty's appeal was pending when we checked[24]
GEMA v. OpenAI (Munich)
Nov 2025
Court held that ChatGPT memorizing song lyrics was a reproduction of protected works[25]
Licensing deals
Reddit and Google: Reuters reported the deal at about $60 million a year (one source; not confirmed by the companies)[36].
News Corp and OpenAI: a multi-year deal reported at $250 million over five years in cash and credits[37].
Standards and blocking
Cloudflare blocks AI crawlers by default since July 1, 2025, unless they pay[38]. Its Pay Per Crawl answers crawlers with HTTP 402 Payment Required and a price[39].
Content signals: Cloudflare's robots.txt extension separates search, AI input and AI training permissions, and was turned on for over 3.8 million domains[40].
AWS WAF has a rule that blocks AI bots by default, supports cryptographic bot identity (Web Bot Auth), and can answer AI bots with HTTP 402[41][42].
IETF: a working group is drafting a shared vocabulary for AI preferences such as "train-ai". The draft says it doesn't yet reflect group consensus[43].
Really Simple Licensing (RSL): a machine-readable way for sites to state AI licensing terms; version 1.0 was published in December 2025[44].
EU: companies signing the EU's AI code of practice, including OpenAI, Google, Anthropic, Microsoft, Amazon and Mistral, commit to crawlers that follow robots.txt[45][46]. xAI signed only the safety chapter[45].
What it means if you need web data
AI assistants aren't a data pipeline. Their web tools are priced per search, return a handful of pages, and some won't read JavaScript-built sites or pages blocked by robots.txt.
Sites are blocking more. Default blocking at Cloudflare and AWS means automated collection increasingly needs clear identity, permission or payment.
The legal lines are about access. Getting past technical barriers such as CAPTCHAs carries more legal risk than ignoring a robots.txt file, based on the 2025–2026 rulings above.
Methodology
We gathered these facts on November 15, 2026 from the companies' own documentation, pricing pages and blogs, court documents and law-firm summaries, and Cloudflare's published traffic data. Each fact links to its source, and we rechecked the key wording against the live pages. Where a page was unavailable or unclear, we left the fact out. Court cases and prices change often, so check the source before relying on any one detail.
Marquee Data has built and managed custom web data collection since 2017 — Fortune 500 and hedge-fund clients, 100% US-based senior engineers, fixed fees. You don't pay until you're satisfied with the data.