"""The sealed rule for "is llms.txt spreading fast among major websites?", asked as a before and after.

Sealed before this study asked any domain for anything. What I know going in: the September probe's rows
(research/llms-txt, 88 valid of the top 1,000, 121 blocking a named AI crawler, 6 September 2026), and
this morning's agent-cards scan, which on a newer list's top 1,000 and with another classifier found
llms.txt on 107 of 750 reachable. That is a different list and test, but it is a hint of the answer,
and it was in front of me when I set the bar. I name it here for that reason.

Claim. "llms.txt adoption is spreading fast among major websites." Nobody I read gives a rate. The bar is
mine: 20 % net growth in the 59 days from 6 September, which held up would be about threefold a year.

Frame. The same 1,000 domains as research/llms-txt/results.jsonl, fetched by the same unchanged function
(probe.probe in research/llms-txt/probe.py, sha256 below), from the same server, 32 workers, one attempt.

1. A domain is ANSWERED in a round if /llms.txt or /robots.txt got an HTTP status.
2. PAIRED = domains answered in both rounds.
3. Among PAIRED: BASE = llms_ok in September; GAINED = not ok then, ok now; LOST = ok then, not ok now.
   NET = GAINED - LOST. 95 % interval NET +- 1.96 * sqrt(GAINED + LOST - NET^2 / PAIRED), the paired
   difference.
4. Verdict, bar = BAR * BASE: killed if the upper bound is below the bar; survived if the lower bound is
   above it; inconclusive otherwise.

Void if any of:
V1 Fewer than 500 domains are PAIRED.
V2 Fewer than 60 % of BASE are still ok now: then the probe no longer sees files known to have been there.
V3 Among PAIRED, the count with robots.txt status 200 moved more than 20 % either way: then what changed is
   how sites answer this server, not their files.
V4 research/llms-txt/probe.py does not have the sha256 below.

Printed only, cannot move the verdict: the same gained, lost, net and interval for "fully blocks at least
one named AI crawler" and per crawler; the all-1,000 counts as September printed them; the top 100; gained
and lost domains by name; first lines shared across domains, since a plugin writing the file is one
decision; length of the files gained.

Limits, said before the data: one attempt per round, so some gains and losses are a site's bad minute, which
is why the interval uses both; a file served only to some clients is not seen; two points, not a trend; 59
days is short; the first line starting with # is a weak test of a valid file, kept because it is September's.
It says a file is there, never that any agent reads it.
"""
import math

PROBE_SHA256 = "0067f989099c1dad48a5a6a4a6237bb3b94da6eacc687bc0da820d005d4b4301"
BAR = 0.20
MIN_PAIRED = 500
MIN_RETAINED = 0.60
MAX_ROBOTS_MOVE = 0.20


def answered(r):
    return r.get("llms_status") is not None or r.get("robots_status") is not None


def paired_interval(gained, lost, paired, z=1.959964):
    net = gained - lost
    se = math.sqrt(max(gained + lost - net * net / paired, 0))
    return net, net - z * se, net + z * se


def blocks_any(r, crawlers):
    return any((r.get("rules") or {}).get(c) == "blocked" for c in crawlers)


def verdict(lo, hi, base):
    bar = BAR * base
    if hi < bar:
        return "killed"
    if lo > bar:
        return "survived"
    return "inconclusive"