Guides · Updated October 2026

Anti-Bot Protection in 2026: We Tested 200 Major Websites

In October 2026 we sent one ordinary page request to each of 200 of the most-visited US consumer websites and recorded what came back. The result: 64% run detectable anti-bot protection, and 39% refused or challenged even a single polite request carrying a normal browser user-agent. Another 10% never responded to our test client at all. This is the market study we wished existed, so we ran it.

Key findings

• 114 of the 179 sites that responded (64%) showed signals of a commercial anti-bot or bot-management product.
• 70 of 179 (39%) blocked, challenged, or rate-limited a single request — before any actual scraping behavior could exist. 21 more sites (10% of the panel) timed out or refused the connection entirely.
• Akamai (44 sites) and Cloudflare (41) dominate, followed by DataDome (10), Imperva (9), and HUMAN/PerimeterX (9). Several sites run more than one layer.
• Ticketing is the most hostile industry (8 of 10 sites blocked a plain request), followed by real estate portals and hotels & airlines (7 of 10 each), then big-box retail, department stores, jobs boards, and C2C marketplaces (6 of 10).
• Finance/market-data sites and sports leagues were the most open — ironically, the sources whose data feeds the most trading strategies.
• Social media looks undefended in this test but isn't: the major platforms gate nearly everything behind login walls and app APIs instead of edge anti-bot vendors, so a homepage probe sees little. Their real defenses engage deeper in.

Industry breakdown

*Includes sites that never responded to the test client (counted as blocked, since they are unreachable to simple collection). Social media's low numbers reflect login walls rather than openness — see key findings.

What this means if you need web data

The practical takeaway isn't that scraping is impossible — every one of these pages loads fine in a real browser, and public-page collection remains lawful under the US case law (hiQ v. LinkedIn; Meta v. Bright Data, 2024). The takeaway is economic: for roughly half of major consumer sites, naive collection fails on request one, and the engineering required to collect reliably — and the cost of doing it wrong — varies by orders of magnitude from site to site. This is why the same dataset can cost 100x more under one approach than another, and why anti-bot handling has become the core competency of professional collection firms. For the options when you're blocked, see our guide to scraping Cloudflare-protected websites.

Methodology

October 29, 2026. One GET request per site homepage, sent through a US ISP connection with an ordinary Chrome desktop user-agent, 18-second timeout, no retries, no challenge solving, no evasion of any kind — deliberately, since the study measures what a simple client experiences. Vendor identification from response headers, cookies, and page markers (e.g., cf-ray, _abck/ak_bmsc, x-datadome, _px*, visid_incap, x-kpsdk, captcha script references). Detection is conservative: a vendor's CDN-only products can trigger its signals, and server-side or login-gated defenses are invisible to this probe, so true protection adoption is likely higher than reported. Panel: 200 widely-visited US consumer sites across 20 industries, 10 per industry. The full domain list and raw classifications are free to download and reuse with attribution: antibot-survey-2026-data.csv.

← All guides · Web Scraping 101: the complete guide