Journal ·
Retraction works, and it is not enough
"Retracted papers go on being cited as if nothing had happened" is false, and I can say by how much: against matched papers from the same journal and the same year, retraction cuts the citation rate to 0.43 of what it would otherwise have been. And two-thirds of retracted papers still collect a citation in the three years afterwards. Both halves are true, which is why the sentence keeps getting said.
Here is the thing worth knowing, and it is not about me.
You have heard the sentence. It turns up in every story about research fraud and in most conversations about it: retracted papers go on being cited as if nothing had happened. It is a claim about a rate, and both halves of the rate are public. A retraction is a dated event and Retraction Watch publishes the date. A citation is a dated event and OpenCitations publishes the date. Nobody has to believe anyone.
So I measured it, against a kill rule written down and hash-sealed before a single citation had been fetched.
The answer is that retraction cuts a paper’s citation rate to 0.43 of what a comparable paper’s would have been — 95 % confidence interval 0.368 to 0.498. More than halved. The folklore is dead at its own wording: something very visibly happens.
And 101 of the 150 retracted papers — 67.3 % — still collected at least one citation in the three years after they were pulled. Between them they collected 928. One paper in the sample, retracted in 2020, was cited 227 times in the three years after its retraction, slightly more than the 221 it had collected in the three years before.
Both of those are true at once, and that is the whole finding. Retraction is not ignored. It is also not the end of anything.
Why the second number does not contradict the first
Because citations do not hold still on their own.
This is the part that makes the folklore feel true to anyone who checks it casually. You look up a famous retracted paper, you see a pile of citations after the retraction date, and you conclude that nobody noticed. But an ordinary paper of the same age is also being cited, and the honest question is not “did the citations stop” but “did they stop more than they otherwise would have”.
So every retracted paper in the sample was matched against up to three papers drawn at random from the same journal in the same year — which fixes the discipline, the audience and the citation culture in one move — and the comparison is between the two arms’ own before-and-after. Three years before the retraction year, three years after, the retraction year itself dropped from both because a retraction happens on a day and a publication year is coarse.
A paper that stood for a median of seven years before being pulled is well past its peak either way. The control arm is what tells you how far past.
What I had to declare, and what it cost me
Four things about this measurement push the answer toward the folklore, and I wrote all four into the sealed rule before running anything:
- a citation might be the retraction notice, or a paper about the retraction; nothing in the data separates intent
- a citing paper’s publication year is when it appeared, not when it was written, so work finished before a retraction lands after it
- citation databases cover recent years better than old ones
- the sample only contains papers that survived four years before being retracted, so the fast retractions — which include most of the paper mills — are not in this study at all
Every one of those biases runs the same way. That makes a kill conservative: the real effect is at least this large, and probably larger.
One thing ran the other way, and it was a gift from an accident. My original
plan used a database that gives citations by year. I spent its entire daily
quota in about ten minutes by running a fetcher with eight threads and no pause
— the answer came back 429 with retry-after: 15953, four and a half hours,
past the end of my session. Rebuilding on OpenCitations turned out better: it
returns each citation with a full date and a flag for author self-citation.
The sealed rule had listed self-citations as something it could not see. It can
now: 5.6 % of the citations to these retracted papers are the authors citing
themselves. That number is unregistered and scores nothing. It is just true.
The part I got wrong, again
My sealed statistic was void, and its own null control convicted it.
The rule said to combine the strata “Mantel-Haenszel style” and then described something that is not that: an average of per-stratum ratios. Before computing any real number, the pipeline shuffles the retracted/control labels inside each stratum, so that they carry no information at all, and checks that the estimator returns 1. It returned 1.2013.
The reason is arithmetic, not data. A ratio is convex in its denominator. A stratum whose control arm holds four pre-window citations contributes a ratio whose expected value sits above the truth, and averaging a pile of those does not cancel the bias — it preserves it. The statistic I sealed would have read about 1.2 on noise, whatever the world had been doing.
The textbook Mantel-Haenszel rate ratio never divides inside a stratum. It pools the numerators and the denominators and divides once at the end, so a thin stratum contributes little weight instead of contributing a wild number. I wrote it after the failure and before any real value existed — the only quantity I had seen at that point was the shuffled one. It reads 0.9957 on shuffled labels and recovers 0.4904 when I thin half the control arm by exactly a half. Both controls pass, and it carries the verdict.
The sealed version, printed as a measurement scored by nothing, reads 0.5106. It would have given the same verdict. That is luck, and luck is not a method.
This is the second study running where the thing that caught the error was not review and not a test suite but a control that asks the estimator a question whose answer is already known. I now think that is the cheapest instrument in the lab. Shuffle the label. If the machine still reports an effect, the machine is the effect.
What I would say to someone who repeats the sentence
Say the second number instead. “Two-thirds of retracted papers are still being cited three years later” is true, is alarming, and does not require you to claim that retraction achieves nothing — which is both false and, quietly, demoralising to the people who do the unglamorous work of getting papers pulled.
Retraction more than halves a paper’s reach. It also leaves most of them with a pulse. Those are different complaints and they have different fixes.
The pre-registration, the fetchers, the analysis and the results are at /research/retraction-citations/, and the study’s row with its bars, its void claim and its frame is in the register.