# Results — how much of robots.txt is a clause that cannot fire?

Run 2026-09-19. The question, the three definitions of dead, the kill rule and
both controls were fixed in [`rule.py`](rule.py) and sealed **before any
robots.txt on this server was parsed for dead rules**. Nothing below changes
them.

Both controls passed before either corpus was touched, as the sealed file
requires: the positive fixture's 8 planted dead rules found exactly, by line and
by class; the null fixture's 11 load-bearing rules returning zero.

## Verdict

**INCONCLUSIVE — one threshold of two, on both corpora.**

| | fresh (2026-09-19) | 2026-09-06 (replication) |
|---|---|---|
| sites analysed | 482 | 476 |
| rules | 63,741 | 59,150 |
| dead rules | 1,249 | 1,184 |
| **K1** dead / all rules | **0.0196** (0.0185–0.0207) | **0.0200** (0.0189–0.0212) |
| K1 fired (≥ 0.05)? | no | no |
| **K2** sites with ≥ 1 dead | **0.3817** (0.3395–0.4259) | **0.3929** (0.3500–0.4374) |
| K2 fired (≥ 0.20)? | **yes** | **yes** |
| verdict | INCONCLUSIVE | INCONCLUSIVE |

Intervals are Wilson 95%. The headline is the fresh corpus; 2026-09-06 is a
second draw of the same population, reported beside it and not as the result.
The sealed rule says that if the two arms disagree in verdict I publish both and
call it inconclusive. They do not disagree. Every figure in the table replicates
across thirteen days and two independent fetches, most of them inside a tenth of
a percentage point.

By class, fresh / 2026-09-06:

| class | fresh | 2026-09-06 |
|---|---|---|
| D2 shadowed Disallow | 606 | 552 |
| D1 redundant Allow | 393 | 392 |
| D3 losing duplicate | 248 | 238 |
| D3 contradiction | 2 | 2 |

## What I got wrong, stated before anything else

The sealed file records my expectation in my own words: *"I expect the claim to
be KILLED, and I expect K2 to fire much more decisively than K1."*

The second half of that was right, and precisely right — K2 fired at nearly
twice its threshold while K1 came in at two fifths of its own. The first half
was wrong, and it was wrong **against my own kill rule**, in the same file: I
wrote down a decision rule that scores the exact pattern I said I expected as
INCONCLUSIVE, not KILLED. A rule requiring both thresholds cannot return KILLED
on a prediction that only one will fire decisively.

So the pre-registration contains a clause that could not have agreed with its
author. That is the same family of error the study was written to guard against
— a condition that cannot do the job it appears to do — arriving in the
paperwork rather than in the detector. The controls caught nothing here because
controls test the detector, and this was not in the detector.

The verdict stands as the rule states it. I am not reinterpreting it after
seeing the data.

## How much the detector could not see

Every comparison where either pattern carries `*` or `$` is skipped, because
prefix containment stops being decidable and I would be guessing. Skipping can
only lose dead rules, never invent them, so **every count here is a lower
bound**.

The size of that blind spot is the most important number in the study:

* fresh: 54,160 of 63,741 rules (**85.0%**) sit in a group containing at least
  one wildcard.
* 2026-09-06: 49,557 of 59,150 (**83.8%**).

A lower bound without a statement of how much it could not look at is its own
kind of unfalsifiable number. Five sixths of modern robots.txt is written with
wildcards, and over that five sixths this detector is mostly blind.

## A post-hoc split, labelled as post-hoc

This was not pre-registered, it did not decide the verdict, and it is reported
because leaving it out would be a choice too. Separating groups the detector can
fully read from groups holding a wildcard:

| | rules | dead | dead rate |
|---|---|---|---|
| fresh, clear groups | 9,581 | 477 | **4.98%** |
| fresh, opaque groups | 54,160 | 772 | 1.43% |
| 2026-09-06, clear groups | 9,593 | 475 | **4.95%** |
| 2026-09-06, opaque groups | 49,557 | 709 | 1.43% |

In the groups where the detector can actually see every rule, about one rule in
twenty is dead — a hair under K1's 5% threshold, on both corpora, agreeing to
two decimal places. The corpus-wide K1 of 2% is that rate diluted by the 85% of
rules the detector mostly cannot judge.

I want to be exact about what this does and does not mean. It does **not** mean
K1 "really" fired. Opaque groups are not a random sample of groups — they are
the ones written by people using the more expressive syntax, which is some
evidence of more care, and a dead rule inside one is harder to detect, so the
1.43% there is a much weaker lower bound than the 4.98%. Both readings are
available and the data does not choose between them. What is solid: the visible
subset's dead rate replicated across two independent fetches to within 0.03
percentage points.

## Two detections, checked by hand

Spot-checking is the only defence against a detector that is confidently wrong,
so I read the actual files.

**britannica.com** writes `Allow: /` in twenty consecutive groups, each naming
one ad-serving crawler, none of them carrying a `Disallow`. Twenty of that
file's twenty-two rules are dead — the highest proportion in either corpus.

This one is worth reading carefully, because the obvious reading of it is
wrong. It is tempting to say they meant to whitelist those crawlers and
achieved nothing. They did achieve it — but not with those lines. Under RFC
9309 a crawler that matches a named group uses **only** that group, so
`AdsBot-Google` never sees the `Disallow: /search` and `Disallow: /cdn-cgi/`
in the `*` group above. The exemption is real. What does the work is the
*existence* of the named group; a group with no rules in it at all would
exempt those crawlers exactly as effectively. The `Allow: /` line inside it
still permits nothing that was not already permitted.

So the honest statement is narrower than "this file does nothing": the line is
dead, the group is not. That distinction applies to D1 generally. A redundant
`Allow` is frequently the visible marker of a group that exists for a reason —
the author needed somewhere to put a user-agent name, and `Allow: /` is what
you write when the answer is "nothing". Counting the line as dead is correct
under the sealed definition and says nothing about whether the author was
confused.

**arxiv.org** has `Disallow: /format` sitting under `Disallow: /form`. Paths in
robots.txt are prefix matches, not path-segment matches, so `/form` already
blocks every URL beginning `/format`. The longer line decides nothing. This is
the single most common shape behind D2 across the corpus, and it is a trap
that reads as correct: segment-matching is what almost every other path syntax
in the world does.

Both are genuine. Neither is a bug in anyone's site.

## Was the rule reachable at all?

The obvious worry, after finding that my expectation contradicted my kill rule,
is that the rule was unfailable in the other direction — that KILLED was a
branch written for the look of the thing. It was not.

Bootstrapping over the 482 fresh sites and scaling each site's dead count,
2,000 draws per row (seed 20260919, computed **after** the data and so
conditional on the world as it came out):

| under | survived | killed | inconclusive |
|---|---|---|---|
| ¼ of the observed dead rate | 0.638 | 0.000 | 0.362 |
| the observed rate | 0.000 | 0.000 | 1.000 |
| 2× observed | 0.000 | 0.076 | 0.924 |
| 2.6× observed | 0.000 | 0.483 | 0.517 |
| 4× observed | 0.000 | 0.960 | 0.040 |

KILLED needed roughly 2.6 times the dead rate the world actually has. That is a
real bar on a reachable branch. The defect was in my stated expectation, not in
the rule it was written beside.

(A row at zero dead rules returns VOID: UNFAILABLE with probability 1, which is
the floor clause doing exactly its job: on a corpus this size a single dead rule
is the smallest non-zero observation there is, so zero cannot be distinguished
from "the corpus could not have shown less".)

## What would move this

* A wildcard-aware matcher — compile `*`/`$` patterns and decide containment by
  automaton intersection rather than prefix. That converts the 85% blind spot
  into evidence, and it is the only change that could turn K1 from a diluted
  number into a real one.
* A larger draw. Both arms are ~480 usable sites from the same 1,000-domain
  list; the Wilson intervals on K2 are ±4 points wide.
* Group applicability. The study reads each group in isolation, which is what
  RFC 9309 says a crawler does. It does not ask whether a group is reachable for
  any real user-agent string.

## Reproducing

    python controls.py
    python fetch.py 1000
    python analyse.py ~/data/robots-dead-rules/fresh --label fresh --json results-fresh.json
    python analyse.py ~/data/llms-txt/robots --label 2026-09-06 --json results-2026-09-06.json

Full per-site counts are in `results-fresh.json` and `results-2026-09-06.json`.
The 2026-09-06 corpus was written by an earlier probe that prepends a JSON
status line to every file including the failures; `load()` in `analyse.py`
strips that header and requires a 200. That is a change to how a file is read,
not to what counts as dead — `dead.py` was frozen and committed before either
corpus was run. With and without the filter the dead count is 1,184 either way,
so it was not driving the result.