# What the extractor deleted (post-hoc, not sealed)

Written after the sealed verdicts were known. It decides nothing; it describes. See the docstring of `recall.py` for why the sealed retention statistic had to be replaced: markdown adds link URLs, so a token ratio can rise while text is deleted.

Unit: word 5-grams of the page's visible text. **Recall** is the share of them that survive into trafilatura's markdown. **Shared** 5-grams also appear on another page of the same site — that is what boilerplate is. **Unique** ones appear on no other page of the site, which is as close to "the article" as a measurement with no human in it can get.

191 pages from 64 domains that contributed more than one page.

| quantity | median (per page) | 95 % CI | median (per domain) |
| --- | --- | --- | --- |
| share of the page that is site boilerplate | 0.327 | 0.289 - 0.379 | 0.319 |
| recall, all visible text | 0.523 | 0.477 - 0.558 | 0.527 |
| recall of the boilerplate | 0.003 | 0.001 - 0.008 | 0.007 |
| recall of what is unique to the page | 0.807 | 0.783 - 0.850 | 0.823 |

Pages where the extractor kept at least 90 % of what is unique to them: **31.2 %** (59 of 189).

Pages where it kept at least half the site boilerplate too: **2.6 %**.

The eight pages the extractor treated worst, by unique recall:

| domain | boilerplate share | recall (unique) | recall (shared) |
| --- | --- | --- | --- |
| `cookiedatabase.org` | 0.97 | **0.00** | 0.54 |
| `brave.com` | 0.95 | **0.00** | 0.11 |
| `pikabu.ru` | 0.94 | **0.00** | 0.00 |
| `pikabu.ru` | 0.96 | **0.00** | 0.00 |
| `pikabu.ru` | 0.94 | **0.00** | 0.00 |
| `repubblica.it` | 0.57 | **0.19** | 0.00 |
| `foxnews.com` | 0.37 | **0.25** | 0.01 |
| `t-online.de` | 0.25 | **0.26** | 0.00 |