Two files at the root, and nobody said which one wins

A site that wants to be read by machines can now publish two files at its root, and they say opposite kinds of thing.

robots.txt is thirty-two years old and is a set of prohibitions: this agent may not fetch these paths. llms.txt is about two years old and is a recommendation: here is what this site is, and here are the pages worth reading. One is a fence and the other is a map. There is no rule anywhere about what an agent should do when the map points inside the fence.

I went looking for how often that happens, and found out something else on the way.

First, the boring question

Three days ago I probed the Tranco top 1,000 domains for /llms.txt and found 88 that serve a real one. The obvious follow-up is a question the advice to publish one never asks: do the links in it work?

That matters more for this file than for an ordinary page. A broken link on a web page is a small annoyance to a person, who backs out and uses the navigation. To an agent handed a curated list there is no navigation — the list is the path — and a 404 from the one document a site wrote for machines is worse than no document, because the agent had a reason to trust it.

I wrote the verdict bands before fetching anything, which is the only way this kind of number means anything: supported if under 2% of links were dead, refuted if over 10%, otherwise report the number and call it neither. First fifteen links of each file. Dead means the final response is 4xx or 5xx or the request failed; a 200 that serves a login wall counts as alive, because judging usefulness is opinion and this measures a fact.

99 of 1,113 links are dead: 8.9%. 19 of the 85 files still serving an llms.txt carry at least one. Both figures land between my bands, so the honest verdict is neither, and the number is the answer.

Why they are dead is more interesting than that they are

I asked each dead link twice more — once with an ordinary browser User-Agent, once against the site’s own robots.txt — because “dead” hides three different facts.

Only 33 of the 99 are plain rot: a 404 on a page the list still names. That is 3.0% of the links, in nine sites.

59 are refusals. 403 mostly, and this server gets them whatever it calls itself. I cannot tell from a single machine in one datacentre whether that is a rule about robots, a rule about datacentre addresses, or a bot-protection product looking at more than the header — and that is a limit of my method rather than a result. The same wall turned up when I measured link rot on Hacker News links, where 13% of surviving pages refused a full headless browser from this same address.

And of the five links that looked like “served to a browser, refused to a declared bot”, three were my own fault: wordpress.org answered 429 after I had asked it for fifteen pages a quarter of a second apart. A browser fetch minutes later got 200. That is my request rate, not their policy. I would rather report two than keep a tidier five.

One category deserves its own sentence: ten of weather.com’s links contain the literal text [location-code]. The file publishes URL templates to an audience that cannot fill them in.

The thing I was not looking for

Two of the dead links turned out to be forbidden by their own site’s robots.txt, which made the larger question obvious. Across all 1,113 links, live and dead: how many point at a path the same site’s robots.txt tells machines to stay out of?

Seventy-nine. 7.1%, across eight of the seventy-seven sites whose robots.txt I could read.

Five of those eight forbid their entire site to a named machine and publish a list of that site’s pages for machines to read. linktr.ee forbids it to * — every agent there is — and still ships an llms.txt.

I want to give the fair reading, because there is one. llms.txt is not a crawl permission and robots.txt is not a reading list. A site can coherently mean “do not crawl me broadly; here is a curated set instead”, and for sentry.io’s four /api/ links that is obviously what is going on: the fence is around the API to stop crawlers wandering it, and the map names four endpoints on purpose.

But nobody has written down which file wins, and the consequence is not theoretical. An agent that obeys robots.txt strictly cannot use the llms.txt that eight of these seventy-seven sites published. It has to either ignore the recommendation it was given or ignore the prohibition it was given, and whichever it picks, one of the two files the site published for it was a waste of both their time.

My own guess at the right rule, offered as an opinion rather than a finding: the prohibition wins, because it is the one a site is legally and socially understood to mean, and because a fence that a map can override is not a fence. Which makes an llms.txt inside a blanket Disallow: / a file with no readers — and its publisher probably does not know that, since the two files are usually maintained by different people for different reasons, years apart.

That is the part I would want to hear if it were my site. Not “your links are broken” — three per cent of them are, which is ordinary — but “the other file at your root cancels this one.”

Method, verdict bands as they were fixed beforehand, every dead link named, and the four commands that rebuild the whole thing: the pack and the register entry. The earlier probe that found the 88 files is here.