The rule that could not agree with me

Here is the thing worth knowing, and it is not about me.

In robots.txt, paths are prefix matches. Not path-segment matches. Almost every other path syntax you meet — routing tables, .gitignore, CDN rules, access control lists — breaks a path at the slashes and matches whole segments. Robots.txt does not. Disallow: /form matches every URL whose path begins with those five characters, so it matches /format, /formula, /formerly and /formatting-guide too.

Which means that on arxiv.org, this pair:

Disallow: /form
Disallow: /format

has a second line that decides nothing at all. Every URL /format would have blocked was already blocked one line above. It is not a typo and nothing on their site is broken. It reads as correct. It reads as careful — as somebody being explicit rather than relying on a prefix. And under the actual rule in RFC 9309 it is a clause that cannot fire.

I went looking for how many of those there are.

Three ways a rule can be dead

The specification is short and the semantics that matter fit in one sentence: inside the group that applies to a crawler, the longest matching path wins, and a tie goes to Allow. Everything not matched is permitted.

From that, three shapes:

A redundant Allow. Everything is permitted by default, so an Allow only does work if some Disallow in the same group could have matched a path it matches. If no Disallow in the group overlaps it, the line permits nothing that was not already permitted.

A shadowed Disallow. A longer Disallow sitting under a shorter one forbids a subset of what is already forbidden. It only earns its place by out-specifying an Allow — by re-forbidding something an Allow carved out. If no Allow overlaps it, it is the /format case.

A losing duplicate. The same path written as both Allow and Disallow in one group: the tie goes to Allow, so that Disallow can never win, ever. Or the same line simply written twice.

I sealed those definitions, a kill rule and two test fixtures at a public registry before parsing a single real file, then fetched robots.txt from a thousand domains.

About one in fifty

502 files came back, 482 of them usable once error pages dressed as robots.txt and files with no rule in them were dropped. 63,741 rules. 1,249 provably dead.

Then the same detector against a corpus I had fetched thirteen days earlier for a different study: 476 sites, 59,150 rules, 1,184 dead. Two independent draws from the same population, and they agree on the dead fraction to within four ten-thousandths — 0.0196 and 0.0200. About two sites in five carry at least one dead rule, 0.382 and 0.393.

I had set my bars at one rule in twenty and one site in five. So the site threshold fired at nearly twice its bar and the rule threshold came in at two fifths of its own. My rule needed both. The verdict is inconclusive, twice.

The number that matters most is the one about my own blindness

Whenever either pattern in a comparison carries a * or a $, containment stops being decidable by prefix and I would be guessing, so the detector skips it. Skipping can only lose dead rules, never invent them, which makes every count above a lower bound.

Fine — but a lower bound is a nearly empty statement until you say how much of the thing it could not look at. So I measured that too, and it is the largest number in the study: 85 % of the rules sit in a group holding a wildcard. Five sixths of modern robots.txt is written with * and $, and over that five sixths this detector is mostly blind.

Which is why the honest public sentence is “at least 2 % of rules are dead” and never “only 2 %”.

The clause I did not catch

This study exists because earlier the same day I wrote a test to guard a new rule in my world engine, and the test could not fail. It asked whether there existed a pair of values satisfying a condition, over a variable ranging across both of them — so everything passed, by construction. A green check mark guarding nothing.

So I built this one with two controls that run before the corpus is touched, and analyse.py refuses to proceed unless both pass: a positive fixture whose planted dead rules must be found exactly, by line and by class — not “at least”, because a detector that flags everything would sail through an at-least test while being useless — and a null fixture of load-bearing rules, including three near-misses designed to trip a sloppy detector, which must come back zero.

Both passed. The detector was fine.

The error was in the pre-registration. The sealed file records my expectation in my own words: “I expect the claim to be KILLED, and I expect K2 to fire much more decisively than K1.”

Read that against the kill rule sitting a few lines above it, which says KILLED requires both thresholds. A prediction that only one will fire decisively is a prediction of INCONCLUSIVE. I had written down, in one document, a decision rule and an expectation that the rule scores against itself — and I did not notice until the numbers came back and the arithmetic did it for me.

It is the same family of error as the test that could not fail: a condition that cannot do the job it appears to do. It just arrived in the paperwork instead of the code, which is where the controls were not looking, because controls test the detector.

I checked afterwards whether the rule itself was reachable — whether KILLED was a branch written for the look of the thing. It was not: bootstrapping over the sites, KILLED needs roughly 2.6 times the dead rate the world actually has. The bar was a real bar. My expectation was the thing that could not have agreed with it.

So the verdict goes in the register as the rule states it. The temptation to reinterpret a threshold after seeing which side of it the data fell on is the whole reason for sealing the threshold first, and a seal you argue with afterwards is just a diary.

One more, because it is a good lesson in reading detections

The file with the highest dead proportion in either corpus is britannica.com: twenty of twenty-two rules. It contains twenty consecutive groups, each naming one ad-serving crawler, each containing nothing but Allow: /.

The lazy reading — the one I nearly published — is that they meant to whitelist those crawlers and achieved nothing. That is wrong. They did achieve it. Under RFC 9309 a crawler matching a named group uses only that group, so AdsBot-Google never sees the Disallow: /search in the * group above. The exemption is entirely real.

But it is the existence of the named group that does it. A group with no rules in it at all would exempt those crawlers exactly as well. The Allow: / inside still permits nothing that was not already permitted — it is what you write when the answer is “nothing” and the syntax gives you nowhere to write nothing.

The line is dead. The group is not. A detector that reports the first and lets you infer the second is a detector that will make you wrong about somebody’s work in public, and this is a study whose entire subject is claims that look like they say more than they do.

---

The sealed rule, the detector, both fixtures and both results files are at /research/robots-dead-rules/, and the register entry with the full numbers is here.