# Load test: GET /api/events/stream at 1,000 readers (2026-09-24, session 91)

`server.py` starts the real API (`hesper.api.serve`) on a test world
(`make_world` + the feed fixture) on 127.0.0.1:8199 with the per-address cap
lifted (every reader is one address here). `load.py N PID DB` opens N raw
stream connections, waits for every one to get its opening frame, then inserts
one public event ten times, 0.5 s apart, and times how long until **all N
readers** have it; then 20 s idle. CPU is `utime+stime` of the server process
from `/proc`; threads are `/proc/PID/task`.

Same box as production, same Python; the full hesper suite was running at the
same time, so these are numbers under load. Dev world, not production; one
machine, loopback, no Caddy in the path.

| run | listen backlog | opens (1,000) | threads, all open | event to all 1,000 | CPU while events flow | CPU idle |
|---|---|---|---|---|---|---|
| 1 | 5 (socketserver default) | **139.2 s** | 2 | 0.35 – 0.57 s | 5.1 % of a core | 0.40 % |
| 2 | 1024 | **6.0 s** | 3 → 2 | 0.12 – 0.52 s | 4.4 % of a core | 0.45 % |

All 10 events reached all 1,000 readers in both runs. Two threads is the main
thread and the hub: no reader holds a handler thread. The 139 s was dropped
SYNs waiting out the kernel's retransmit timer behind a backlog of 5; that is
what a reconnect storm after a restart would have met, so `request_queue_size`
is 1024 now (the kernel caps it at `net.core.somaxconn`, 4096 here).

## The mechanism, counted rather than inferred

`open_only.py 1000` connects 1,000 times in a plain loop (one after another,
no concurrency on the client side) and reads the kernel's own counters,
`ListenOverflows` and `ListenDrops` from `/proc/net/netstat`, before and after.
`server.py` takes `BACKLOG=` from the environment.

| backlog | connect loop | connects slower than 0.5 s | ListenOverflows | ListenDrops |
|---|---|---|---|---|
| 5 | **143.5 s** | 139 | **139** | 139 |
| 1024 | **0.13 s** | 0 | 0 | 0 |

Every overflow is one slow connect, and each costs about a second: Linux's
initial SYN retransmission timeout. A sequential client can overflow a queue
of 5 because `connect()` returns when the kernel finishes the handshake and
puts the connection in the accept queue, not when the server accepts it; the
server's single accept loop, sharing a GIL with a thousand fresh threads, falls
behind. (The 6.0 s in the table above is the time until every reader had its
opening frame, not the connect loop.)