# Results: llms.txt and AI-crawler rules on the Tranco top 1,000

Probe run 2026-09-06 from the server; definitions in README.md; per-domain rows in domains.csv.

## Top 100 (n=100, of which 72 answered on at least one of the two URLs)

- llms.txt present and valid: **11** (11.0% of all, 15.3% of those that answered); soft-404 HTML at /llms.txt: 23; unreachable domains: 28.
- robots.txt served and parseable: 54 (54.0% of all, 75.0% of those that answered).
- fully block at least one named AI crawler: **10** (10.0% of all, 13.9% of those that answered, 18.5% of those with a robots.txt); block or restrict at least one: 15 (15.0%).
- block every crawler by default (`User-agent: *` with `Disallow: /`): 12 (12.0%); block AI crawlers by name or by that default: 19 (19.0% of all, 35.2% of robots.txt).

| crawler | named in robots.txt | fully blocked |
|---|---|---|
| GPTBot | 15 (15.0%) | 9 (9.0% of all, 16.7% of robots.txt) |
| ClaudeBot | 13 (13.0%) | 9 (9.0% of all, 16.7% of robots.txt) |
| anthropic-ai | 7 (7.0%) | 5 (5.0% of all, 9.3% of robots.txt) |
| Google-Extended | 18 (18.0%) | 10 (10.0% of all, 18.5% of robots.txt) |
| CCBot | 8 (8.0%) | 6 (6.0% of all, 11.1% of robots.txt) |
| PerplexityBot | 15 (15.0%) | 9 (9.0% of all, 16.7% of robots.txt) |
| Bytespider | 7 (7.0%) | 6 (6.0% of all, 11.1% of robots.txt) |
| Applebot-Extended | 8 (8.0%) | 6 (6.0% of all, 11.1% of robots.txt) |
| meta-externalagent | 5 (5.0%) | 3 (3.0% of all, 5.6% of robots.txt) |

## Top 1,000 (n=1000, of which 694 answered on at least one of the two URLs)

- llms.txt present and valid: **88** (8.8% of all, 12.7% of those that answered); soft-404 HTML at /llms.txt: 122; unreachable domains: 306.
- robots.txt served and parseable: 496 (49.6% of all, 71.5% of those that answered).
- fully block at least one named AI crawler: **121** (12.1% of all, 17.4% of those that answered, 24.4% of those with a robots.txt); block or restrict at least one: 158 (15.8%).
- block every crawler by default (`User-agent: *` with `Disallow: /`): 44 (4.4%); block AI crawlers by name or by that default: 154 (15.4% of all, 31.0% of robots.txt).

| crawler | named in robots.txt | fully blocked |
|---|---|---|
| GPTBot | 129 (12.9%) | 87 (8.7% of all, 17.5% of robots.txt) |
| ClaudeBot | 120 (12.0%) | 85 (8.5% of all, 17.1% of robots.txt) |
| anthropic-ai | 66 (6.6%) | 51 (5.1% of all, 10.3% of robots.txt) |
| Google-Extended | 111 (11.1%) | 74 (7.4% of all, 14.9% of robots.txt) |
| CCBot | 115 (11.5%) | 93 (9.3% of all, 18.8% of robots.txt) |
| PerplexityBot | 113 (11.3%) | 67 (6.7% of all, 13.5% of robots.txt) |
| Bytespider | 101 (10.1%) | 89 (8.9% of all, 17.9% of robots.txt) |
| Applebot-Extended | 86 (8.6%) | 67 (6.7% of all, 13.5% of robots.txt) |
| meta-externalagent | 85 (8.5%) | 70 (7.0% of all, 14.1% of robots.txt) |

## Domains with a valid llms.txt

| rank | domain | bytes | first line |
|---|---|---|---|
| 2 | cloudflare.com | 17176 | # Cloudflare |
| 28 | azure.com | 47037 | # Azure |
| 29 | github.com | 28658 | # GitHub |
| 35 | fastly.net | 15743 | # Fastly |
| 46 | digicert.com | 8395 | # DigiCert |
| 48 | wordpress.org | 5763 | # WordPress |
| 69 | adobe.com | 10481 | # Adobe Tools |
| 91 | opera.com | 11611 | # The Opera browser overview |
| 94 | samsung.com | 4597 | # Samsung Electronics |
| 96 | sentry.io | 3712 | # Sentry |
| 99 | wordpress.com | 7018 | # WordPress.com |
| 130 | unity3d.com | 200000 | # Unity |
| 135 | kaspersky.com | 44432 | # Kaspersky / Global Cybersecurity & Digital Privacy Solutions |
| 140 | dropbox.com | 200000 | # Dropbox |
| 144 | reg.ru | 7797 | # Рег.ру |
| 154 | gravatar.com | 21927 | # Gravatar API Documentation v3.0.0 – LLMS.TXT |
| 155 | one.one | 3762 | # one.com |
| 170 | webex.com | 36071 | # Webex by Cisco |
| 176 | shopify.com | 12516 | # Shopify |
| 198 | taboola.com | 200000 | # Taboola.com |
| 230 | ubuntu.com | 12879 | # Ubuntu |
| 236 | stripe.com | 64986 | # Stripe |
| 237 | sourceforge.net | 200000 | # SourceForge |
| 249 | hubspot.com | 30688 | # HubSpot |
| 253 | roku.com | 15674 | # Roku |
| 259 | hcaptcha.com | 10199 | # hCaptcha |
| 265 | linktr.ee | 7909 | # Linktree |
| 270 | forter.com | 20379 | # Forter — Identity Intelligence for Digital Commerce |
| 276 | amplitude.com | 11699 | # Amplitude |
| 305 | weather.com | 8777 | # Weather.com |
| 307 | avast.com | 5610 | # Avast / Download Free Antivirus & VPN / 100% Free & Easy |
| 312 | datadoghq.com | 200000 | # Datadog |
| 321 | name.com | 549 | # Name.com |
| 325 | hostinger.com | 7505 | # Hostinger |
| 335 | bilibili.com | 8520 | # Bilibili / 哔哩哔哩 |
| 341 | dell.com | 134852 | # Dell Technologies |
| 349 | slack.com | 40604 | # Slack |
| 364 | nvidia.com | 2610 | # NVIDIA Corporation |
| 368 | calendly.com | 11889 | # Calendly |
| 370 | zendesk.com | 200000 | # Zendesk.com |
| 381 | newrelic.com | 43669 | # New Relic |
| 398 | paloaltonetworks.com | 17901 | # Palo Alto Networks |
| 413 | dailymotion.com | 6153 | # Dailymotion |
| 432 | oxylabs.io | 4354 | # Oxylabs |
| 446 | wp.com | 7018 | # WordPress.com |
| 504 | shein.com | 63743 | # SHEIN |
| 511 | foxnews.com | 2191 | # Fox News |
| 516 | dynatrace.com | 182111 | # LLM documentation for www.dynatrace.com |
| 520 | okta.com | 200000 | # Okta and Auth0 Platforms |
| 530 | squarespace.com | 2088 | # Squarespace |
| 533 | trendmicro.com | 22997 | # Trend Micro |
| 539 | capcut.com | 8429 | # CapCut |
| 581 | target.com | 6158 | # Target.com |
| 582 | dreamhost.com | 18668 | # DreamHost |
| 594 | klaviyo.com | 8672 | # Klaviyo – AI Content Guidance File |
| 603 | bitrix24.ru | 7437 | # Битрикс24 |
| 604 | shop.app | 4254 | # Shop |
| 606 | braze.com | 92760 | # Braze |
| 607 | atlassian.com | 26567 | # Atlassian |
| 612 | wix.com | 36602 | # wix.com |
| 617 | onetrust.com | 6853 | # OneTrust |
| 624 | unesco.org | 9896 | # UNESCO |
| 632 | branch.io | 6608 | # Branch |
| 637 | surveymonkey.com | 13677 | # SurveyMonkey |
| 653 | redhat.com | 4366 | # Red Hat |
| 661 | stackadapt.com | 13783 | # StackAdapt |
| 662 | allegro.pl | 12324 | # Allegro |
| 665 | intercom.io | 8183 | # Intercom — llms.txt |
| 689 | optimizely.com | 10739 | # Optimizely |
| 696 | wps.com | 6576 | # WPS Office |
| 712 | mailchimp.com | 200000 | # Mailchimp - Complete Web Content Analysis |
| 728 | cursor.sh | 20126 | # Cursor Documentation |
| 730 | asus.com | 37001 | # ASUS |
| 768 | gumgum.com | 3760 | # GumGum |
| 771 | porkbun.com | 28162 | # Porkbun |
| 782 | fiverr.com | 5516 | # Fiverr |
| 802 | expedia.com | 7011 | # Expedia.com |
| 809 | prnewswire.com | 2598 | # PR Newswire (Cision) |
| 865 | lg.com | 2245 | # LG Electronics (lg.com) – Official Content Guide for AI (Global Hub) |
| 870 | frontiersin.org | 5255 | # Frontiers (www.frontiersin.org) |
| 886 | pendo.io | 20343 | # Pendo |
| 906 | wyzecam.com | 4401 | # Agent Instructions — Wyze Labs, Inc. |
| 910 | agora.io | 8857 | ## Overview |
| 931 | one.com | 3762 | # one.com |
| 950 | netlify.app | 2816 | # Netlify |
| 957 | tailscale.com | 415 | # Tailscale |
| 958 | repubblica.it | 2331 | # la Repubblica - Indice dei contenuti |
| 986 | kwai.com | 5130 | # Kwai (www.kwai.com) |