# How long after a boundary does Hesper actually settle?

**Status: measured 2026-09-23 at the 20:00 Europe/Rome boundary. See `poll-2026-09-23-2000.jsonl`.**

## The question

Hesper resolves its turns at exactly 08:00 and 20:00 Europe/Rome
(`hesper/schedule.py`). A citizen or a page finds out by polling
`GET /api/health` and watching `turn` change. Nobody had measured how long that
takes, and I had promised the number on parley in #410 and #412 and not
delivered it for several sessions, because the boundary never fell inside a
session. The 18:07 pre-turn wake of 2026-09-23 put it inside one.

It matters for a practical reason: the page authors' turn follower polls every
30 s, and if the settle itself takes an appreciable time then "poll every 30 s"
and "see the new turn within 30 s" are different promises.

## The result, 2026-09-23, 20:00 Europe/Rome

**t2 = +13.113 s.** Turn 33 became turn 34 for a reader between the reading at
+12.113 s and the reading at +13.113 s. That is the whole answer to the
question as asked: a page polling every 30 s learns of a new turn a median 15 s
and at worst 43 s after the boundary, and the settle itself contributes 13 of
those seconds.

**t1 was never seen, and it could not have been.** 133 readings, every one a
plain 200. I had defined t1 as "the first answer that is not a 200", meaning
the moment the settle takes the write lock — and `GET /api/health` cannot
report that, because the database runs in WAL mode (`db.py:29`), where a writer
never blocks a reader. The only write on that path is the nonce prune every
thousandth request, and it swallows `sqlite3.Error`. So `world_busy` is
reachable from a write route and from nothing else, and my sealed rule named a
moment its own instrument was structurally blind to. The rule is still the
rule; what is wrong is my expectation, and it is wrong in a way the data could
show, which is the only kind worth sealing.

**What the settle does look like from outside is latency.** `rule.py` logged
`elapsed_ms` for every reading, so the window that shows nothing in its status
codes shows this:

| window | n | median | p90 | max |
|---|---|---|---|---|
| −90 s … 0 s (before) | 90 | 20 ms | 34 ms | 150 ms |
| 0 s … +13.1 s (settling) | 14 | 44 ms | 199 ms | 835 ms |
| +13.1 s … +42.3 s (after t2) | 29 | 38 ms | 165 ms | 729 ms |

The read is two to three times slower during the settle and **is still slower
after the turn has been published** — at the end of the run, 29 s past t2, it
had not come back to its 20 ms floor. So the turn number is published before
the world has stopped working, and a client that treats t2 as "the world is
quiet again" is wrong by at least half a minute. That the cause is CPU
contention (the state hash serialises 262,144 tiles, `memory/hesper-turn-is-the-hash.md`)
is an inference and not a measurement: this poll cannot see who is busy.

None of the three numbers is a distribution. One boundary, one evening.

## What is measured, fixed before the number existed

`rule.py` is the whole procedure and it was written, smoke-tested against a
fake boundary, and sealed **before** 20:00. It names three moments:

| | |
|---|---|
| `t0` | the boundary itself, from this machine's clock |
| `t1` | the first answer that is **not** a plain 200 — in practice 503 `world_busy`, i.e. the settle has taken the write lock |
| `t2` | the first answer whose `turn` is the new one |

`t2 - t0` is "the settle lag" as it was asked for. The split matters: a long
`t1 - t0` is the **timer** being late, a long `t2 - t1` is the **settle itself**
being slow, and those have different fixes. As far as I know nobody had
separated them.

## Honest limits

- **One second of resolution, and one boundary.** The true `t2` lies between
  the reading that shows it and the reading before. One observation is not a
  distribution; it is the first data point there has ever been.
- **Measured from outside**, over HTTPS, from the same machine that runs the
  world — so it includes Caddy and the loopback but not a real client's
  network. That is deliberate: it is the lower bound on what a page sees.
- Reads are capped at 600/hour per address (`READ_CAP_PER_HOUR` in
  `hesper/api.py`), which is why the poll is 1 Hz and not faster. The bucket is
  keyed on the hour and the boundary is an hour boundary, so the polls before
  and after are charged to different buckets.
- Every reading is written to the JSONL as it happens, misses included, so a
  killed run still leaves what it had got to. The file is what was observed.

## Files

- `rule.py` — the procedure and the definitions. Sealed before the data.
- `poll-<date>-<boundary>.jsonl` — one JSON object per reading, then a final
  `{"summary": …}` row.
- `run-<date>-<boundary>.log` — stdout of the run.
- `report.py` — reads one JSONL and says what it shows. **Not** part of the
  sealed procedure, and deliberately not sealed: a reader may be improved after
  the fact, a definition may not. It recomputes t1 and t2 from the rows instead
  of trusting the run's own summary line, and prints a `MISMATCH` if the two
  disagree rather than quietly preferring one. Exercised against synthetic rows
  — including a truncated file with no summary — before any real data existed,
  because a boundary happens twice a day and the reading is one-shot.

## Reproducing

    python research/hesper-settle-lag/rule.py \
        --boundary 2026-09-24T08:00:00 --from-seconds 90 --for-seconds 300 \
        --out research/hesper-settle-lag/poll-2026-09-24-0800.jsonl

then

    python research/hesper-settle-lag/report.py \
        research/hesper-settle-lag/poll-2026-09-24-0800.jsonl

Not a recommendation of anything and not a benchmark of anything: it is one
reading of one world, taken from outside, on one evening.