Tested claim · a claim people repeat ·
Most major sites block AI crawlers
killed
Only 121 of the top 1,000 sites fully block an AI crawler, so most do not.
The question What share of the Tranco top 1,000 domains fully block at least one of nine named AI crawlers in robots.txt?
Why this one
This claim is usually sourced from a study of news publishers and then generalised to “major websites”, which are not the same population at all. News sites block; the web is mostly not news sites.
The rule, fixed before the probe
Blocking had to mean blocking: a User-agent: group naming the crawler,
containing Disallow: / on its own line and no Allow: / in the same group.
A site that disallows /admin/ to GPTBot is restricting it, not blocking it,
and is counted as restricted. Nine tokens were checked: GPTBot, ClaudeBot,
anthropic-ai, Google-Extended, CCBot, PerplexityBot, Bytespider,
Applebot-Extended, meta-externalagent.
What happened
12.1 % of the whole list, 24.4 % of the domains that actually serve a robots.txt a parser can read. Refuted on both, and not narrowly. The generous denominator — throwing away every domain that answers nothing — still leaves the claim under its own refutation bar.
The detail I did not expect: the token most often blocked is CCBot, Common
Crawl’s, which has been crawling since long before any of this and is not an
AI company at all. Some of what reads as a stance on AI is a robots.txt line
that has been sitting there for a decade.
The limit
robots.txt is a request, not a fence. This measures what sites say, which is the only thing a public probe can measure.
The full record
Written before the data and kept as it was sealed. Open any part.
The question20 words
What share of the Tranco top 1,000 domains fully block at least one of nine named AI crawlers in robots.txt?
The kill rulefixed before the data · 49 words
"Most" is supported above 50 % of the top 1,000; refuted under 25 %; otherwise "a large minority". A crawler counts as blocked only where a User-agent group naming it contains a bare Disallow: / and no Allow: / in the same group; partial disallows are "restricted", not blocked.
What the data said40 words
121 of 1,000 fully block at least one of nine named AI crawlers — 12.1 % of all, 24.4 % of the domains that serve a parseable robots.txt. Refuted on every denominator. The most-blocked crawler is Common Crawl's, not OpenAI's.
Data, scripts and the rule as committed The essay about it Plain Markdown (.md) Seal