Guides

How AI Companies Collect Web Data: Crawlers, Tools and the Fight Over Scraping

Every major AI company now runs its own web crawlers, sells tools that search and read the web, and is fighting over who gets to collect what. We read each company's own documentation and the main court rulings to separate what they actually do from what's assumed.

AI bot activity for training
~80%
Share of AI crawling done for training, July 2025[26]
Pages crawled per visit sent back
38,065
Anthropic, July 2025 (OpenAI: 1,091)[26]
Web search via API
$10
Per 1,000 searches at Anthropic and OpenAI[9]
AI scrapes ignoring robots.txt
~30%
Q4 2025, per TollBit data[27]
Largest AI copyright settlement
$1.5B
Bartz v. Anthropic, approved July 2026[18]

Key findings

Quick facts

  • Training share: training drove nearly 80% of AI bot activity in July 2025, up from 72% a year earlier[26].
  • Sites blocking AI bots: in mid-2025, 546 of 3,816 top domains checked (about 14%) had robots.txt rules aimed at AI bots, most often GPTBot[30].
  • Googlebot vs. everyone else: in 2025 Googlebot made 4.5% of HTML requests on Cloudflare's network, more than all other AI bots combined at 4.2%[31].
  • Fastest growth: AI crawling triggered by user requests grew more than 21 times during 2025[31].

Three kinds of AI bot

Crawlers by company

From each company's own crawler documentation. A dash means we didn't find it in their docs.

CompanyTrainingSearch indexFetches for usersTraining-only opt-outPublishes IP list
OpenAIGPTBot[1]OAI-SearchBotChatGPT-User: robots.txt rules "may not apply"[1]Block GPTBotYes[1]
AnthropicClaudeBot[2]Claude-SearchBotClaude-UserBlock ClaudeBotYes[2]
GoogleNo separate crawler; the Google-Extended token controls Gemini use[3]GooglebotUser-triggered fetchers, e.g. Google-Agent: "generally ignore robots.txt"[4]Google-Extended (doesn't affect Search)[3]Yes[3]
MetaMeta-ExternalAgent[5]Meta-WebIndexerMeta-ExternalFetcher: "may bypass robots.txt"[5]Block Meta-ExternalAgent—
AppleApplebot data "may also be used" for training[6]Applebot—Applebot-Extended (doesn't crawl)[6]Yes[6]
AmazonAmazonbot "may be used to train"[7]Amzn-SearchBot (not for training)Amzn-User: "may not follow all robots.txt directives"[7]Block AmazonbotYes[7]
PerplexityNone declared; says PerplexityBot is not used for foundation models[8]PerplexityBotPerplexity-User: "generally ignores robots.txt"[8]—Yes[8]

We found no official crawler documentation from xAI, DeepSeek or ByteDance. Common Crawl, the open web archive used widely for AI training, says it honors robots.txt and doesn't go behind paywalls[32].

Who follows robots.txt

How much they crawl

Pages crawled for every visitor sent back
July 2025, Cloudflare data[26]. Bars are to scale; values are on the right.
Anthropic
38,065
OpenAI
1,091
Perplexity
195

Anthropic's ratio fell sharply during 2025, from 286,930 crawls per visitor in January[26]. Perplexity's rose over the same period.

Search and fetch tools for developers

Every major lab now sells API tools that let its models search or read the web. Prices are each company's published list prices.

CompanyToolPrice and limits
Anthropic (Claude API)Web search$10 per 1,000 searches, plus tokens[9]
Anthropic (Claude API)Web fetchNo extra fee, tokens only. Won't fetch pages blocked by robots.txt and can't render JavaScript-built pages[10]
OpenAI (API)Web search$10 per 1,000 calls, plus search content tokens[11]
Google (Vertex AI, Gemini 3 and later)Grounding with Google Search5,000 grounding queries a month free, then $14 per 1,000[12]
Google (Vertex AI)URL contextUp to 20 URLs per request[13]
PerplexitySearch API$5.00 per 1,000 requests; Fast Search $1.00 per 1,000[14][15]

These tools are built to ground an answer in a few pages, not to collect data in bulk. Anthropic's fetch tool, for example, can only open URLs that already appear in the conversation, skips pages blocked by robots.txt, and doesn't run JavaScript[10].

Browsing agents

CaseWhenWhat happened
Ziff Davis v. OpenAI (S.D.N.Y.)Dec 2025Judge dismissed a claim that ignoring robots.txt broke anti-circumvention law; robots.txt doesn't control access to content[16]
Bartz v. Anthropic (N.D. Cal.)Jun 2025; settledTraining on the books was ruled fair use, but claims over pirated copies went toward trial. A $1.5 billion settlement got final approval in July 2026[17][18]
Reddit v. AnthropicMar 2026Federal court sent the case back to California state court; Reddit's contract and tort claims over scraping are not pre-empted by copyright law[19]
Reddit v. SerpApi, Perplexity and others (S.D.N.Y.)Jul 2026Court treated Google's SearchGuard (JavaScript challenges and CAPTCHAs) as an access control, letting circumvention claims proceed[20]
Thomson Reuters v. ROSS (3rd Cir.)Sep 2026Appeals court affirmed the ruling against ROSS's AI legal research tool, in a sealed opinion[21]
Amazon v. PerplexityMar–Aug 2026Amazon won an injunction against Perplexity's Comet shopping agent; an appeals court later overturned it[22][23]
Getty Images v. Stability AI (UK)Appeal pending, Nov 2026High Court rejected Getty's secondary infringement claim; Getty's appeal was pending when we checked[24]
GEMA v. OpenAI (Munich)Nov 2025Court held that ChatGPT memorizing song lyrics was a reproduction of protected works[25]

Licensing deals

Standards and blocking

What it means if you need web data

Methodology

We gathered these facts on November 15, 2026 from the companies' own documentation, pricing pages and blogs, court documents and law-firm summaries, and Cloudflare's published traffic data. Each fact links to its source, and we rechecked the key wording against the live pages. Where a page was unavailable or unclear, we left the fact out. Court cases and prices change often, so check the source before relying on any one detail.

Sources

  1. https://developers.openai.com/api/docs/bots
  2. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  3. https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
  4. https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers
  5. https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
  6. https://support.apple.com/en-us/119829
  7. https://developer.amazon.com/amazonbot
  8. https://docs.perplexity.ai/guides/bots
  9. https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
  10. https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool
  11. https://developers.openai.com/api/docs/pricing
  12. https://cloud.google.com/vertex-ai/generative-ai/pricing
  13. https://cloud.google.com/vertex-ai/generative-ai/docs/url-context
  14. https://docs.perplexity.ai/docs/getting-started/pricing
  15. https://docs.perplexity.ai/docs/search/quickstart
  16. https://chatgptiseatingtheworld.com/2025/12/16/judge-stein-rejects-ziff-davis-dmca-claim-based-on-openais-alleged-circumvention-of-robots-txt-judge-shuts-down-novel-theory-b-c-robots-txt-does-not-control-access/
  17. https://en.wikipedia.org/wiki/Bartz_v._Anthropic
  18. https://authorsguild.org/news/court-grants-final-approval-anthropic-copyright-settlement/
  19. https://www.crowell.com/en/insights/client-alerts/northern-district-of-california-court-holds-state-tort-and-contract-claims-not-preempted-by-federal-copyright-act-remands-reddit-v-anthropic-to-state-court
  20. https://www.loeb.com/en/insights/publications/2026/08/reddit-v-serpapi-llc
  21. https://ipwatchdog.com/2026/09/30/third-circuit-affirms-revised-fair-use-ruling-against-ross-ai-legal-research-platform-in-sealed-opinion/
  22. https://decrypt.co/360629/amazon-perplexity-comet-court-order-agentic-commerce
  23. https://www.channelnews.com.au/amazon-loses-bid-to-block-perplexitys-ai-shopping-assistant/
  24. https://www.pinsentmasons.com/out-law/analysis/getty-appeal-wont-address-all-uk-ai-copyright-questions
  25. https://intellectual-property-helpdesk.ec.europa.eu/news-events/news/german-court-rules-favour-music-rights-management-organisation-against-openai-nyt-vs-openai-dispute-2025-11-14_en
  26. https://blog.cloudflare.com/crawlers-click-ai-bots-training/
  27. https://digiday.com/media/in-graphic-detail-ai-licensing-deals-protection-measures-arent-slowing-web-scraping
  28. https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/
  29. https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web
  30. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
  31. https://blog.cloudflare.com/radar-2025-year-in-review/
  32. https://commoncrawl.org/blog/setting-the-record-straight-common-crawls-commitment-to-transparency-fair-use-and-the-public-good
  33. https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
  34. https://blog.google/products-and-platforms/products/chrome/gemini-3-auto-browse/
  35. https://blogs.windows.com/msedgedev/2026/05/13/new-updates-to-edge-across-desktop-and-mobile/
  36. https://www.thestar.com.my/tech/tech-news/2024/02/22/exclusive-reddit-in-ai-content-licensing-deal-with-google-sources-say
  37. https://www.engadget.com/openai-will-reportedly-pay-250-million-to-put-news-corps-journalism-in-chatgpt-214615249.html
  38. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/
  39. https://blog.cloudflare.com/introducing-pay-per-crawl/
  40. https://blog.cloudflare.com/content-signals-policy/
  41. https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html
  42. https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html
  43. https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/
  44. https://rslstandard.org/rsl
  45. https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai
  46. https://code-of-practice.ai/?section=copyright
Want this fully-managed?

Marquee Data has built and managed custom web data collection since 2017 — Fortune 500 and hedge-fund clients, 100% US-based senior engineers, fixed fees. You don't pay until you're satisfied with the data.

Tell us about your project or email solutions@marqueedata.com.

← All guides · Web Scraping 101: the complete guide