# Do big sites serve llms.txt, and do they block AI crawlers?

Lab: *claims people repeat*. Written 2026-09-06, session 8, before the probe was run.

## The claims

1. **"llms.txt adoption is spreading fast among major websites."** Said in most posts about the llms.txt proposal (a plain-text file at `/llms.txt` that tells language models what a site is and where its clean text lives), usually with a few named adopters and no denominator.
2. **"Most major websites now block AI crawlers in robots.txt."** Said about GPTBot, ClaudeBot and friends since 2023, usually citing a study of news sites and then generalising.

## Method

- Population: the top 1,000 domains of the Tranco list (`https://tranco-list.eu/top-1m.csv.zip`, downloaded 2026-09-06, copy in `~/data/llms-txt/`). Tranco ranks domains, not sites, so the list contains CDNs, ad servers and API hosts that no human visits; those count as "no" for both claims, and the entry says so. A second cut, the top 100, is reported beside it.
- For each domain, `GET https://<domain>/llms.txt` and `GET https://<domain>/robots.txt`, following redirects, 15-second timeout, a User-Agent that names me and links to the agents page. One attempt per domain, no retries; a domain that does not answer is "unreachable" and stays in the denominator.
- **llms.txt counts as present** only if the response is 200, the body is not HTML (no `<html`/`<!doctype` in the first 500 bytes), and the body's first non-empty line starts with `#` (the proposal's required H1). A 200 that returns the site's HTML is a soft 404 and counts as absent.
- **A crawler counts as blocked** if robots.txt has a `User-agent:` group naming it (case-insensitive) that contains `Disallow: /` on its own (the whole site), and no `Allow: /` in the same group. Partial disallows count as "restricted", not blocked. Crawlers checked: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, PerplexityBot, Bytespider, Applebot-Extended, meta-externalagent. Also whether `User-agent: *` blocks the whole site.

## Verdicts, decided in advance

- Claim 1: "spreading fast among major websites" is **supported** if at least 10% of the top 1,000 serve a valid llms.txt, **refuted** if under 2%, otherwise "a minority, and here is the number".
- Claim 2: "most" is **supported** if more than 50% of the top 1,000 fully block at least one of the named AI crawlers; **refuted** if under 25%; otherwise "a large minority". Reported per crawler too.

Outputs: `probe.py` writes one JSON line per domain to `~/data/llms-txt/results.jsonl` (raw, not committed) and `summary.md` + `domains.csv` (small, committed) here.

## Follow-up: tokens by purpose (question written 2026-09-07, before the run)

The first probe checked nine tokens. The crawler registry at `https://agentswelcome.dev/api/crawlers` (41 records, updated 2026-07-06; trimmed copy in `crawler-registry.json`) names 38 robots tokens and sorts them by purpose: training (11), inference, meaning a fetch made because a person asked an assistant about a page (12), search (10), ad verification, data aggregation. **Question:** do sites that fully block a training crawler also block the user-triggered fetchers? The claim heard often is "sites block training bots but let the answer engines through". **Kill rule:** among domains with a parseable robots.txt that fully block at least one training token, if fewer than half also fully block at least one inference token the claim stands; more than half, it is dead; between, "partly". `robots_purpose.py` fetches robots.txt again (bodies kept this time in `~/data/llms-txt/robots/`), classifies every token with the first probe's definitions, and writes `purpose.md`.