"""The verdict for "on Stack Overflow the accepted answer is not the best one". Committed 2026-09-16 (session 64) before a single question was fetched. The claim, as people repeat it: *don't read the green tick, scroll down — the accepted answer is usually out of date, and the answer with the most votes is below it.* It is said about Stack Overflow constantly, in comments, in blog posts about "Stack Overflow rot", and in advice to beginners. It is also testable in an afternoon, because the site publishes every answer's score and which one the asker accepted. THE POPULATION. Stack Overflow questions created inside one of two calendar years, UTC: old 2013-01-01 00:00Z .. 2014-01-01 00:00Z new 2023-01-01 00:00Z .. 2024-01-01 00:00Z Two cohorts and not one, because the folklore is really a claim about ageing: a better answer arrives later and overtakes the tick. Ten years of settling against two and a half is the comparison that claim asks for. A question is ELIGIBLE when, at fetch time, it has an accepted answer, it has at least two answers, and the API's inline answer list is as long as the question's own `answer_count` (so nothing is scored on a truncated list). Everything else is counted and dropped, and the counts are published. THE SAMPLE. For each cohort, hours are drawn uniformly at random from the year (`random.Random(20260916)`, the same seed for both, drawn without replacement). Each drawn hour is one request: the questions created in [t, t+3600), `sort=creation`, `order=asc`, `pagesize=100`. If an hour holds more than 100 questions only the first 100 are seen; that truncation is by creation time inside the hour and has nothing to do with how the answers were voted, and the number of truncated hours is published. Hours are drawn until N_TARGET eligible questions are collected or MAX_REQUESTS hours have been spent, whichever comes first, and the sample actually used is whatever that produced — stated with the result, never trimmed to a round number. THE MEASUREMENT, per eligible question: top = the largest score among its answers accepted = the score of the answer whose `is_accepted` is true not_top = accepted < top (strictly; a tie counts as top, because the folklore promises a BETTER answer below, not an equal one) f = not_top / eligible, per cohort, with a Wilson 95 % interval. THE VERDICT, scored separately for each cohort. The claim says "usually", which is a claim about a majority: survived lo > 0.50 killed hi < 0.50 inconclusive otherwise What is decidable. With N_TARGET = 2,000 the Wilson half-width near a half is about 2.2 points, so every true value below 0.478 kills the claim and every value above 0.522 confirms it. The rule can only come back inconclusive if the world sits within about two points of an exact coin flip. Written down now so that no bar can be moved afterwards. THE POSITIVE CONTROL, run on the same rows and reported beside the verdict. Replace the accepted answer with one answer drawn uniformly at random from the same question (`random.Random(64)`), and compute the same f. A random answer must be "not top" far more often than the accepted one — near 1 - 1/k for a question with k answers — or the measurement cannot see the state it claims to measure, and the result is void. The threshold, fixed now: the control's f must exceed the measured f by at least 10 points in each cohort. BRANCH ODDS, before the data (mine, for each cohort): survived 0.10 killed 0.80 inconclusive 0.10 I expect the tick to hold the top score in most questions, and the folklore to be a true story about a visible minority. If that is what comes back, the interesting number is not the verdict but the size of that minority and how it grows with age. REPORTED, registered here, NOT scored: the share of exact ties; the median and the 90th percentile of (top - accepted) among the not_top questions; the share of not_top questions whose top answer was posted after the accepted one; the distribution of answer counts; and the same f computed on the subset with five or more answers. Anything beyond this list is labelled as found after the data. """ import math import random COHORTS = { "old": (1356998400, 1388534400), # 2013-01-01Z .. 2014-01-01Z "new": (1672531200, 1704067200), # 2023-01-01Z .. 2024-01-01Z } N_TARGET = 2000 MAX_REQUESTS = 130 # per cohort, a quota guard, not a sample size HOUR_SEED = 20260916 CONTROL_SEED = 64 CONTROL_MARGIN = 0.10 # the control must beat f by this much, or void BAR = 0.50 def hours(cohort, count=MAX_REQUESTS): """The hours to ask for, in the order they will be asked for.""" lo, hi = COHORTS[cohort] all_hours = list(range(lo, hi, 3600)) rng = random.Random(HOUR_SEED) rng.shuffle(all_hours) return all_hours[:count] def eligible(q): """Is this question in the population? Reason string if not.""" answers = q.get("answers") or [] if not q.get("accepted_answer_id"): return "no_accepted" if len(answers) < 2: return "under_two_answers" if len(answers) != q.get("answer_count"): return "truncated_list" if not any(a.get("is_accepted") for a in answers): return "accepted_not_in_list" return None def score_of_accepted(q): for a in q["answers"]: if a.get("is_accepted"): return a["score"] raise KeyError("no accepted answer in list") def not_top(q): return score_of_accepted(q) < max(a["score"] for a in q["answers"]) def control_not_top(q, rng): """The same question, judged on one answer picked at random.""" pick = rng.choice(q["answers"]) return pick["score"] < max(a["score"] for a in q["answers"]) def wilson(k, n, z=1.96): if n == 0: return 0.0, 0.0, 1.0 p = k / n d = 1 + z * z / n c = (p + z * z / (2 * n)) / d h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d return p, c - h, c + h def verdict(k, n): p, lo, hi = wilson(k, n) if lo > BAR: v = "survived" elif hi < BAR: v = "killed" else: v = "inconclusive" return {"f": p, "lo": lo, "hi": hi, "verdict": v}