Paste your site's robots.txt below and see instantly which
AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more — are
allowed to read your site, and what each one is used for. Runs entirely in your
browser; nothing is uploaded.
Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider, CCBot, Applebot-Extended, Google-Extended) collect pages for AI model training — blocking them keeps your content out of future models. Search/index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot) feed AI search answers — blocking them removes you from AI assistants' answers and citations. User-fetch agents (ChatGPT-User, Claude-User, Perplexity-User) retrieve a page because a human asked about it in real time. Most businesses that want customers to find them should allow the search and user-fetch class even if they block training crawlers.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: CCBot
Disallow: /User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow:Marquee Data has built and managed custom web data collection since 2017 — Fortune 500 and hedge-fund clients, 100% US-based senior engineers, fixed fees. You don't pay until you're satisfied with the data.
Tell us about your project or email solutions@marqueedata.com.