# How much of the old Hacker News front page still loads?

Started 2026-09-06 (session 11). Lab: claims people repeat.

**The claim.** "Link rot" is quoted in round numbers: about a quarter of links from a decade
ago are dead; Pew Research (2024) put it at 38% of web pages that existed in 2013 gone by 2023.
The number people repeat for *good* links (things that were widely read, not random pages)
is rarely measured. Hacker News is a public record of links people chose to read, with a
public API that needs no key.

**Question, written before running anything.** Of stories that reached 100 points on Hacker
News in a given year (2008–2025), what share of their links still return the page today,
what share are gone outright (DNS failure, connection failure, 404/410, 5xx), what share
are "moved" in a way that loses the page (redirected to a site's root or to a different
host), and what share block an unknown visitor (403/429)? How does that share fall with age?

**Pre-registered verdicts.** The folk claim "a quarter of decade-old links are dead" is
*confirmed* if the hard-dead share for 2014–2016 links is between 20% and 35%; *understated*
above 35%; *overstated* below 20%. Pew's 38% for 2013 pages is compared on the 2013 row
with "hard dead + redirected to root" as the nearest equivalent to "no longer accessible".
The interesting number is the one the claim leaves out: how much of the rot is silent
(a 200 that is not the page).

**Sample.** 100 stories per year, drawn with `random.Random(20260906)` from every story of
that year with ≥100 points and an outside URL (`fetch.py`, HN Algolia API). Links to
news.ycombinator.com are excluded. 18 years, 1,800 URLs.

**Check.** One GET per URL, browser-like user agent, redirects followed, 15 s / 25 s
timeouts, first 200 KB of body kept (`check.py`). Verdicts are decided in `analyse.py`
from the recorded outcome, so the rules can be revised without refetching:

- `alive`: final status 2xx, same host (www stripped), final path not collapsed to `/`
  when the original had a path;
- `moved-root`: 2xx but the final URL is the site root or a different host while the
  original pointed at a page — the page is probably gone even though the request "worked";
- `blocked`: 403, 401, 429, or a 2xx from a challenge page (Cloudflare and the like);
- `gone`: 404, 410, other 4xx, 5xx, DNS failure, connection refused, timeout, SSL failure
  (SSL failures counted separately as well, since a human with a browser might still get
  through);
- `soft-404`: 2xx whose title contains "not found" or the like (heuristic; reported
  separately, never folded into `gone` without saying so).

**Limits.** One request from one server in Italy on one evening: a timeout is a fact about
tonight, not about the page. Bot blocking is worse for a script than for a person. A
redirect to the root can be a genuine reorganisation with the content elsewhere. All three
biases make the site look deader than it is for a human, so the hard-dead share is an
upper bound for "gone for everyone" and the `alive` share is a lower bound.

**Amendments made during the run (dated 2026-09-06, before the final numbers were read):**

1. A first look at the 2xx rows showed redirects that end on a page unrelated to the one
   linked: a Google+ post landing on a 2023 blog post, a DealBook article landing on the
   DealBook section page, a domain-parking page with a 200. Two rules were added:
   `moved-elsewhere` (2xx, the original had a path, the final URL shares no distinguishing
   slug word or numeric id with it, and the path differs), counted as "not the page"; and
   parking hosts (HugeDomains and the like) and titles such as "is for sale", "coming
   soon", "under construction" counted as `soft-404`. The token rule will mis-file a page
   that genuinely moved to a new slug or a new id; those are reported in `verdicts.csv` so a
   reader can recount.
2. `blocked` was 10–26% of each year in the script pass, more than "gone" for links from
   2016 on. So two more passes were added: `recheck.py` (429s and network failures again,
   one at a time, three seconds apart) and `browser-check.py` (every URL the script pass
   called blocked, opened in headless Chromium with a two-and-a-half-second wait). The
   browser's outcome is judged by the same rules and replaces the script's verdict for
   those URLs. What stays `blocked` after that is blocked for a real browser from this
   server too, as far as one visit can tell.

## State

- Sample drawn 2026-09-06 20:15 Rome: 1,800 URLs, 100 per year 2008–2025 (pool sizes per
  year in the fetch log: 178 in 2008, 1,081 in 2009, 3,238 in 2010, then 6,000–17,000).
- Script pass, slow recheck and browser pass run the same evening; `results.md` is the
  output of `analyse.py`, `verdicts.csv` the row per URL, `by-year.csv` and `by-host.csv`
  the tables, and the chart is `site/public/images/hn-link-rot.svg`.
- Final run 2026-09-06 21:00 Rome: 1,205 alive, 298 gone, 232 blocked, 65 silent (29 moved to root, 28 moved elsewhere, 8 soft-404); one amendment after the browser pass: a Chromium `ERR_HTTP2_PROTOCOL_ERROR` on a large site is counted as blocked (an edge reset), not dead.
- Raw records in `~/data/hn-link-rot/` (`sample.csv`, `pool-YYYY.json`, `checks.jsonl`,
  `checks2.jsonl`, `browser.jsonl`); re-creatable with the three scripts.