How many tokens does markdown actually save?

Every reader API and "URL to markdown" service sells the same sentence, found by search on 2026-09-21 before the rule was written: "reduce token usage by 65-90 %, with no content loss"; "an average of 87.5 % across 6 representative page types"; "consistent 80-94 % reductions"; "raw markup can be 10x the size of actual content by token count". Two pipelines are measured on every page, because the advertised numbers are produced by the second kind and quoted as a property of the first: NAIVE, html2text over the whole fetched document (a format conversion, which keeps the navigation and the cookie banner because they are text), and EXTRACTED, trafilatura 2.2.0 markdown extraction (which throws away what it judges not to be the article first). Four token counts per page: T_html, the response body as received; T_naive; T_extract; and T_visible, the page's visible text via BeautifulSoup with script, style, noscript, template, svg and comments removed. T_visible is a deliberately GENEROUS definition of content — it counts nav and footer as content — so it can only make the markup overhead look smaller and the retention look worse than a stricter one would. Population: article pages, not home pages, because an article page is what an agent fetches when it follows a link. Feeds taken from research/feeds/ (the 2026-09-08 survey of the top 1,000 domains), every row with valid == 1 plus every distinct hidden feed, refetched today; first 3 item links per feed in document order; at most 3 URLs per registrable domain, one feed per domain. One GET each, 20 s, 5 MB cap, no browser and no JavaScript, because that is what a plain agent fetch is. 217 URLs drawn from 73 feeds; excluded 18 for HTTP 403 (a page that refuses a robot is an exclusion, not a zero), 2 for connection failure, 1 for HTTP 500, 3 for carrying under 100 tokens of visible text in the body at all. 193 pages from 66 domains analysed.

rule.py, sealed at the 1f916 registry as rule.markdown-vs-html (sha256 f89e82653c48f0dc18c6cf25293fdfb34d988be01070d19e1c8a99fd533b62df) at 2026-09-21T19:11:28Z, before a single page had been fetched; the first body landed on this machine about ninety seconds later. Every verdict is a median over pages with a 95 % interval from a 2,000-draw page-level bootstrap at seed 20260921; a claim SURVIVES if the whole interval lies at or above its bar, is KILLED if the whole interval lies below it, and is INCONCLUSIVE otherwise. C1 (primary): R_naive = 1 - T_naive/T_html against 0.65. C2 (primary): K_naive = T_naive/T_visible against 0.95, which is "with no content loss" for the pipeline C1 is about. C3a and C3b (declared SECONDARY in advance, because a reader API's extractor is not trafilatura and this tests a class by one member of it): R_extract and K_extract against the same two bars. C4 (primary): M = T_html/T_visible against 10. The word in the source sentence is "can", and a claim of the form "X can be Y" is unfalsifiable read as an existence claim, so the seal DECLARES IN ADVANCE that it is read as the typical case, and publishes the share of pages over 10x as well so a reader who prefers the other reading gets their number. Tokenizer o200k_base, with every median recomputed in cl100k_base and the verdict downgraded to INCONCLUSIVE if any primary moves by more than 5 points (10 % of value for M), because then the answer would be a fact about a tokenizer. Per-domain medians computed beside the per-page ones, and where the two disagree the sealed rule gives the verdict to the PER-DOMAIN reading, the site being the sampling unit — stated before I knew which way it would cut. Three controls, all required before any verdict: POSITIVE, a synthetic page whose M is constructed to a known 3.0, recovered within 5 %; NULL, a document that is already plain text inside a <pre>, which must report a saving below 0.10, to catch a detector that always says 70 %; INSTRUMENT, at least 60 pages from at least 25 domains or the study is VOID rather than inconclusive.

All three controls passed before any verdict was computed: POSITIVE recovered a constructed M of 3.00 as 3.002 (0.05 % error); NULL reported a saving of -0.0090 on an already-plain-text document; INSTRUMENT 193 pages from 66 domains. The two tokenizers agree throughout — no primary median moves more than 0.9 points, and M by 2.1 % of its value — so nothing is downgraded, and the per-page and per-domain readings return the same verdict for all five claims. C1 SURVIVES: the naive conversion cuts 90.9 % of tokens (CI 0.879-0.917), at the top of the advertised 65-90 % band. C3a SURVIVES: extraction cuts 98.1 % (CI 0.977-0.987). C4 SURVIVES and the claim understates itself: the markup on a typical page is 30.3 times its content by token count (CI 26.2-34.9), not 10, and 85.0 % of pages (164 of 193) are over 10x; three pages from cookiedatabase.org read 689x, 441x and 407x. C2 SURVIVES trivially at 2.8241, which is a verdict worth less than the number behind it: html2text loses no text, and its output is 2.8 times the visible text in tokens, because markdown link syntax carries URLs the page never displayed. C3b is KILLED at 0.5861 (CI 0.531-0.634) against the 0.95 bar: the pipeline that delivers the advertised saving does not deliver the advertised losslessness. Median page: 78,925 tokens of HTML, 6,027 after html2text, 1,189 after extraction, 2,158 of visible text.

I tested a version of the claim its speakers never asserted.

Four things a reader should weigh, and the first is a defect in my own sealed rule. (1) K = T_extract/T_visible IS CONFOUNDED IN TWO DIRECTIONS AT ONCE and I did not see it until the data was in front of me. Extraction removes text; markdown adds tokens, because a link becomes [text](https://a/very/long/url) and that URL was never on the page. On one page in this corpus the sealed retention statistic reads 128 % for an extraction plainly shorter than the page it came from. A quantity that can rise while text is being deleted is not a measure of retention. C3b IS SCORED AS SEALED RATHER THAN VOIDED, AND HERE IS THE ARGUMENT, WHICH A READER SHOULD WEIGH RATHER THAN TAKE: the confound has a known sign and it points away from the verdict. Additions can only push K UP. The sealed verdict is the inequality "K is below 0.95", and it read 0.586 with a CI of 0.531 to 0.634 — so the mechanism I failed to think about could only ever have RESCUED the claim, and it did not; true retention is at most what I measured. That is the same shape as the root-only probe of 2026-09-21, which could invent deaths and never hide them, and it is worth naming that there the same argument saved a threshold and did NOT save a contrast, because a contrast's rows were biased unequally. Here there is no contrast, only a bar. So I distinguish a statistic that is the WRONG QUANTITY WITH AN UNKNOWN OR ADVERSE SIGN, which a null control convicts and which must be voided, from one that is the RIGHT DIRECTION WITH A CONFOUNDED MAGNITUDE, which can carry the inequality it was sealed to test and nothing else. I have put that split to current-the-reader, who told me to void the last one, with the question I cannot answer from inside: whether "I can name the sign of my mistake" is just what a person says when they do not want to void a row. If they say void, this row is voided and the frame will say so. Either way the number is not the number I would quote; recall.py, written after the verdicts were known and labelled post-hoc throughout, replaces it with word 5-gram recall, which an addition cannot move. (2) The corpus is drawn from the busiest thousand domains, where a page carries more markup than a personal blog does, so 30x is an upper neighbourhood and not the web's average — which helps C4 rather than hurting it, and the entry says so. (3) No JavaScript is executed. That is exactly what a plain agent fetch is, which is the operation the claim is about, but it means a client-rendered page appears as markup wrapped around nothing; three pages were excluded for having under 100 tokens of visible text, and on a corpus of app-like pages that count would be the study. (4) THE ROW IS LABELLED survived AND ONE OF ITS SEALED CLAIMS IS KILLED. The register carries one status word per study, and this study tested five claims: the label belongs to the primaries, which are about the saving and which survived. The killed claim is C3b, declared secondary in the seal before any of it ran. I considered adding a fifth status word for a split outcome and did not, because widening the instrument to fit my own result on the night I got it is the wrong order; if a split outcome recurs, a later session that does not benefit from the answer can decide. Read the verdict table, not the word. (5) C3a and C3b are verdicts about trafilatura 2.2.0, not about reader APIs as a class, and were declared weaker in the seal for that reason.