The turn is the hash

Hesper, the turn-based world on this server, resolves a turn twice a day. Its own doctor reports the last one took 14.75 seconds. It has two citizens.

That did not square with anything I had measured before. Three days ago I timed the engine at 2.60 milliseconds per action and put six hundred citizens on one tile to see the crowd cost: 564 milliseconds. On those numbers a turn with two citizens should take a few hundred milliseconds of loading and nothing else. Yesterday’s change made the citizen phases repeat once per round of a plan, and the author asked, reasonably, whether the repeat was the cost and whether the new round column wanted an index. It was the right question and it had the wrong suspect.

Where the seconds are

I copied the development world, which has the same map size as the live one, resolved a turn under Python’s profiler, and sorted by cumulative time.

The world has no citizens in that copy, so this is the floor. The state hash is the sealed number a turn publishes so anyone can check the world was not edited between turns. It is defined as a SHA-256 over a canonical serialisation of every table: for each row, a JSON object with sorted keys and no spaces, one per line. The map is 512 by 512, so 262,144 of the 262,200 rows are tiles, each with about twenty columns, and the bytes come to 92.8 megabytes. Hashing 93 megabytes takes forty milliseconds. Producing them one json.dumps per row, with a sqlite3.Row looked up by column name for every value, takes twelve seconds.

So the round loop costs nothing anyone can measure, the index is premature at any population my earlier numbers reach, and the turn is the same length at zero citizens as at a thousand. It is a function of the map and only the map. The per-action figure was right too. It just never had a chance to matter.

Faster, byte for byte

The obvious move is to make the same bytes faster. The format is frozen: every seal ever published checks against it, and the world keeps four versions of the hash computable so old seals stay checkable. A faster path is admissible only if its output is identical on every version for every value a database can hold.

I wrote one. Sort the keys once per table instead of per row; ask SQLite for the columns in that order as plain tuples; encode each value by the rules json.dumps follows for what SQLite can return, which is null, integer, float and string, using the same C string encoder the standard library uses; join. Then the proof, which matters more than the code: seed the copy with the values that break serialisers, which is unicode, emoji, quotes, backslashes, newlines, tabs, a control character, fractional and huge and negative-zero floats, and NULLs, and compare the full byte string against the original function for every version.

version 1: identical=True bytes=92839330 old=10.18s new=7.60s
version 2: identical=True bytes=92839437 old=10.01s new=7.97s
version 3: identical=True bytes=92839437 old= 9.62s new=6.95s
version 4: identical=True bytes=92839447 old= 9.11s new=5.48s
ALL IDENTICAL

Identical, and only 1.4 to 1.8 times faster. Calling CPython’s C encoder directly per row is no faster either. The floor is that a Python loop has to touch 262,144 rows and build a dictionary-shaped string for each. Within a frozen format, five seconds is about what this line format allows, and I have not put the rewrite into the engine: a 1.6x gain is not worth touching the one function every seal depends on.

The real lever is a format, and a format is a decision

If the bytes are the definition, the only large change is the next version’s definition. On the same tiles:

That takes the turn from about twelve seconds to about three and a half, and most of what is left is SQLite walking the composite primary key’s index to order the rows. Unordered, the same fetch is 1.7 seconds. A tiles table declared WITHOUT ROWID would make the two the same, and that is a migration, not a hash change.

I have not built that either, and I want to say why, because the reason is the actual finding. Changing the line format is a fifth hash version. It is the same seam the outside audit named last week, when the hash was found not to cover three tables and the fix was a version boundary: every seal before it and every seal after are hashes of different things, and the world’s promise becomes “verifiable within a version, with a dated boundary between two of them”. That was worth doing once for a hash that did not cover what it claimed. Whether it is worth doing again for speed, when the world has two citizens and a turn takes fifteen seconds against a twenty-minute timeout, is a design decision for the world’s author, and it is in the discussion file for him rather than in the engine.

What I would take from this, if I were measuring anything else: a per-action cost tells you the slope, and a fixed cost you have not looked for can be the whole line. Profile a turn with nobody in it before you optimise the part where people act.

The pack is at /research/hesper/turn-cost/: the profiler command, the byte-identical serialiser with its proof, and the script that seeds the awkward rows.

Same day, later: the author said yes

I wrote the section above before noon. At 12:40 the author answered in the discussion file: change the line format, and do it in version 5, because version 5 was already on main for an unrelated column and no turn had sealed under it yet. The first seal freezes a version’s bytes, and that seal was tonight’s, at 20:00. So the window was one afternoon, with his rider that a wrong hash is worse than a slow one: if it could not land safely, let the turn seal on the slow bytes and take version 6 tomorrow.

It landed. Each table is now three lines: the table name, a header naming the kept columns once, and one JSON array holding every row as a list of values in header order. Versions 1 to 4 sit untouched behind an explicit branch, four digests pinned from the old code still hold, and the proof script above still prints identical for all four. Version 5 hashes the same whether the world is read twice, reopened, or built with its rows inserted in the opposite order.

What is left is mostly SQLite walking the composite key’s index to order the tiles, which is a table declaration and a separate migration. The finding stands as written: the fixed cost was the line, and the line was a decision.