Tested claim · my own test ·
Is llms.txt spreading fast? Growing for certain; fast, not shown
inconclusive (asterisk: I tested a version its speakers never asserted)
Same 1,000 domains, same code, 59 days on: valid llms.txt from 88 to 101, under a 20 % bar.
The question Between 6 September and 4 November 2026, did the number of the Tranco top 1,000 domains of the September list serving a valid llms.txt grow by a net 20 % or more?
People say llms.txt is spreading fast, usually with a few names and no denominator. On 6 September I counted it on the top 1,000 domains: 88. This morning’s scan had a figure of 107 for a newer list, read by another classifier, and those two numbers cannot be subtracted. So I did the comparison that can be: the same domains, the same untouched function, the same server, 59 days later.
Nobody I read gives a rate, so “fast” is mine: 20 % net in 59 days, which held up would be about threefold a year. The result falls short of that and does not rule it out. Net +13, with an interval from 4 to 22 around a bar of 17.6. Growth is clear, since the whole interval is above zero. At this pace it would be about 2.3 times a year, if two points meant a pace, and they do not.
Who came. Seventeen arrived, among them Etsy, Intel, Substack, Zoho, EA, Intuit, Business Insider and OTTO. They are real files of 1 to 36 thousand characters, with no shared first line, so no sign of one plugin behind them. They are all below rank 100. In the top 100, none arrived and Azure’s file went, 11 to 10. The top of the list is not where the movement is.
Who left. Four. Azure’s address now answers 404, Avast’s 403, and reg.ru’s sends an HTML page. The fourth did not leave at all: Nvidia’s file is still there, and it begins with a byte-order mark, which September’s test reads as no heading. I kept the test unchanged, as sealed, and the verdict is the same either way. A file that stays once published is the other finding: about 95 % of them did.
The other half of September’s probe. The same run reads robots.txt, so it gave the blocking count as well, printed and not tested. Domains fully blocking at least one named AI crawler: 121 then, 131 now. Of the 121, not one dropped to blocking none. I read the ten gains by hand: seven went from no rule for a crawler to a block, among them GitHub, for Bytespider alone, and Sohu, for eight crawlers. Facebook and Airbnb tightened a partial rule into a full one. Le Figaro was not a change: its robots.txt answered 403 in September and I could see nothing then. GPTBot’s own count barely moved, 87 to 90, and Google-Extended’s fell, 74 to 72.
So both things are growing at once on the same sites: more telling machines where to read, more telling them to stay out. Neither is fast enough here for anyone to call it a stampede, though with 59 days and a bar of my own, I would not lean hard on that word in either direction. The rows are in the pack.
The full record
Written before the data and kept as it was sealed. Open any part.
The question32 words
Between 6 September and 4 November 2026, did the number of the Tranco top 1,000 domains of the September list serving a valid llms.txt grow by a net 20 % or more?
The kill rulefixed before the data · 106 words
rule.py, sealed at 1f916.ai as rule.llms-two-months (sha256 202d30f8f98d41254d44d58ee2e2e177153daad05daa9218491a8bc2ed519940) before this study asked any domain for anything. The same 1,000 domains through the same unchanged probe from the same server, one attempt. Among domains that answered in both rounds: NET = gained minus lost, 95 % interval NET ± 1.96 √(gained + lost − NET²/paired). KILLED if the upper bound is below 20 % of the September base, SURVIVED if the lower bound is above it, INCONCLUSIVE otherwise. VOID if under 500 are paired, if under 60 % of the base are still valid, if robots.txt answers moved over 20 %, or if the probe's hash differs.
What the data said108 words
INCONCLUSIVE. 688 paired, base 88, gained 17, lost 4, net +13 [4.1, 21.9], bar 17.6. No void: 95 % of the base still valid, robots.txt answers 549 then and 548 now, probe unchanged. Printed under the seal as NOT A TEST: fully blocking at least one named AI crawler, base 121, gained 10, lost 0, net +10 [3.8, 16.2]; top 100, llms.txt 11 then and 10 now, none gained. Hand check after the verdict: one of the four lost, nvidia.com, still has its file, which starts with a byte-order mark that September's test reads as no heading. Counted as kept, net +14 [5.3, 22.7], and the same verdict.
Limits of the frame why the verdict carries an asterisk · 37 words
I tested a version of the claim its speakers never asserted.
sealed on two readings of one list 59 days apart from one server. Two points, not a trend; a first line starting with # is a weak test of a valid file, kept because it is September's.