How much of robots.txt is a clause that cannot fire?

How much of a real robots.txt is a rule that can never decide anything? Three sealed classes, all under RFC 9309 semantics — longest matching path wins, a tie goes to Allow, paths are prefix matches. D1 redundant Allow: an Allow that no Disallow in its group could ever contest, so it permits nothing that was not already permitted by default. D2 shadowed Disallow: a Disallow sitting under a shorter Disallow with no Allow overlapping it, so it never out-specifies anything. D3 losing duplicate: the same path written as both Allow and Disallow in one group (the tie goes to Allow, so the Disallow can never win), or a rule line repeated verbatim within one group. Measured over ~480 usable robots.txt files drawn from a 1,000-domain list, as a fraction of all rules (K1) and as a fraction of sites carrying at least one (K2).

rule.py, sealed at the 1f916 registry as rule.robots-dead-rules (sha256 560d9ad910cfd9a44a8aca210c300583f90edec00a8def90d6471496b56b13d9) before any robots.txt on this server was parsed for dead rules. Kill rule: the claim that robots.txt files are largely load-bearing is KILLED if both K1 >= 5 % of all rules are dead and K2 >= 20 % of sites carry at least one; INCONCLUSIVE if exactly one fires; SURVIVED if neither; and VOID: UNFAILABLE if an observed value sits at or below the corpus floor of 1/n, the smallest non-zero fraction the corpus could have shown. The headline arm is a fresh fetch of the 1,000 domains on the day of the run; the 2026-09-06 corpus is reported beside it as a replication on a second draw of the same population, and if the two arms disagree in verdict I publish both and call it inconclusive. Two controls must pass before analyse.py will touch a corpus, and it refuses to run otherwise: a POSITIVE fixture whose planted dead rules must be found exactly, by line and by class rather than "at least", because a detector that flags everything would pass an at-least test while being useless; and a NEGATIVE fixture of load-bearing rules including three near-misses — an Allow carving a real exception, a longer Disallow re-forbidding what an Allow permitted, and a group whose wildcard makes its neighbours unjudgeable — which must come back zero. Every comparison where either pattern carries * or $ is skipped as undecidable by prefix, which can only lose dead rules and never invent them, so every count is a lower bound.

computed after the data — conditional on the world as it came out, so it cannot show a branch was unreachable by design

INCONCLUSIVE on both arms — one threshold of two, twice. Fresh (2026-09-19): 1,000 domains attempted, 502 bodies fetched, 482 usable after dropping 64 HTML error pages, 13 files with no rule line, 6 empty and 1 non-200; 63,741 rules; 1,249 dead. K1 = 0.0196 (Wilson 95 % 0.0185-0.0207) against the 0.05 bar, not fired. K2 = 0.3817 (0.3395-0.4259) against the 0.20 bar, fired. Replication (2026-09-06 corpus): 476 sites, 59,150 rules, 1,184 dead; K1 = 0.0200 (0.0189-0.0212) not fired, K2 = 0.3929 (0.3500-0.4374) fired. Both controls passed before either corpus was touched: 8 planted dead rules found exactly by line and class, 11 load-bearing rules returning zero. By class, fresh then 2026-09-06: D2 shadowed Disallow 606 / 552, D1 redundant Allow 393 / 392, D3 duplicate 248 / 238, D3 contradiction 2 / 2. Every figure replicates across thirteen days and two independent fetches, most inside a tenth of a percentage point. Registered as the study's most important number: 54,160 of 63,741 fresh rules (85.0 %, and 83.8 % on the older corpus) sit in a group containing at least one wildcard, where the detector is mostly blind — so the lower bound is a lower bound over a sixth of the corpus. After the data, not sealed: splitting by that blind spot gives a dead rate of 4.98 % in groups the detector can fully read against 1.43 % in opaque ones, and the clear-group rate replicates at 4.95 % on the older corpus, within 0.03 points. That is a hair under the K1 bar and it is not a finding that K1 "really" fired, because opaque groups are not a random sample of groups and a dead rule inside one is harder to see. Two detections read by hand and both genuine: britannica.com writes Allow: / in twenty consecutive single-crawler groups, where the exemption is real but is done by the existence of the named group rather than by the line inside it; and arxiv.org's Disallow: /format is dead because Disallow: /form already prefix-matches every URL beginning /format, which is the most common single shape behind D2 and reads as correct to anyone expecting path-segment matching.

I tested a version of the claim its speakers never asserted.

What is sealed is whether individual rule LINES can decide anything, under RFC 9309, inside the group they are written in. It is not a claim that the files fail at their job: a dead line often marks a group that is doing real work, as britannica.com shows. A reader who wants "is robots.txt broken?" should treat this as evidence about line-level redundancy and nothing more. And because 85 % of rules sit in groups the detector cannot fully read, the honest public sentence is "at least 2 % of rules are dead", never "only 2 %".