For most of May and June our testnet had a habit. Everything would run fine for a day or two. Then one baked validator would silently drop out of consensus, sit in a corner, and refuse to sign anything until we wiped it and re-synced from a snapshot. Chain kept producing on the other three. Users saw nothing. From our end, it was death by a thousand papercuts.
We tried a lot of things. Round-stale tolerance. Catchup gossip. Restart-safe vote records. Filter tightening on ANT. Deadline tuning. Locked-prevote fixes. A dozen more. Each one made the chain a little more resilient and none of them fixed the root cause, because we didn't understand the root cause.
This is a post about how we found the root cause and what we did about it.
The symptom
When you looked at a "forked" node's logs, it was always the same shape.
[apply-trace h=NNNN] lastCommitBitmap 0x…
[block] applied h=NNNN
[apply-trace h=NNNN+1] parent hash mismatch: local 0xabc…, expected 0xdef…The block the forked node applied at height N had a slightly different hash than what everyone else applied at height N. Not because of the block body — transactions were fine, receipts matched, state root matched. The difference was in one field: lastCommitBitmap, the field that tracks which validators signed the previous block, used for distributing per-block rewards.
If the bitmap differed by even one bit, the block hash differed. If the block hash differed, the next block's parent hash didn't match. Fork.
Why the bitmap could differ
Every node built its own lastCommitBitmap at block-build time by asking its local consensus engine, "which validators do I remember precommitting on the previous block?" The engine's answer depended on the order votes arrived, when the round timer fired, whether a late precommit landed before commit-time or after, and half a dozen other things.
In a synchronous fantasy where every node observed every message at exactly the same time, all engines would agree. In real life across three datacenters — Nuremberg, Helsinki, Ashburn — messages arrive in different orders. Two nodes that observed slightly different vote arrivals produced slightly different bitmaps. Slightly different bitmaps produced slightly different block hashes. Different block hashes produced a fork.
We were rebuilding a piece of the block state from local, non-deterministic engine memory, and using it as if it were canonical chain data.
The fix
Cosmos SDK solved this pattern years ago. It's called a LastCommit cert. The idea is that every block explicitly carries the signatures that formed quorum on the previous block. Not "our local memory of who signed" — the actual signed precommits, embedded in the block body, canonically part of the header hash.
Every validator now computes the reward bitmap from the same bytes. If it's not in the on-chain LastCommit list, it didn't sign. There is no local memory to disagree about.
We shipped this in the last week of June. The block header now carries a lastCommitHash field — a blake3 hash of the SSZ-serialized LastCommit struct. The block body now carries the struct itself: block hash of the parent, round, and up to N signatures with validator index, public key, and Dilithium3 signature. Verifying a block from a peer means:
- Recompute
blake3(SSZ(block.lastCommit))and compare to header. - Check the cert's
blockHashmatches the actual parent hash. - Verify each signature against the appropriate precommit-vote body.
- Check total signed stake ≥ 2/3.
Byte-identical LastCommit on every node produces byte-identical blocks. Byte-identical blocks produce a single canonical chain.
The follow-on
The clean fix has one edge. When a node restarts, its engine memory is wiped. The first proposal after restart has nothing to build a LastCommit from. We shipped an "empty cart" escape hatch that lets these blocks through with a warning and no rewards distribution for the previous block.
That escape hatch got exercised in two ways we didn't love. On 2026-07-02, a scheduled 4-hour snapshot cron restarted all four baked simultaneously. All four produced empty-cart blocks. All four accepted each other's empty carts. Chain kept producing. One block of missed rewards for that height. Fine.
On 2026-07-08, a single node restarted at a different moment than the others. It produced an empty cart. Three peers were still building carts with real signatures. Timing worked out such that the forked node's local latestHeader ended up with a different hash than the canonical chain. Recovery was standard: wipe + fresh snapshot.
The proper fix — the one we shipped today — is restart-safe engine state. After every commit, the engine writes the committed precommits to disk atomically. On boot, it reads them back and seeds itself. The first proposal after restart carries a real cart, not an empty one.
Six weeks of chasing symptoms. One architectural change that removes the class. And a follow-up write-through that removes the last edge case.
The lesson
We were treating a non-determinism bug as if it were a race condition. Race conditions get better with timeouts, retries, and defensive checks. Non-determinism doesn't — you can't retry your way out of two nodes disagreeing about what they're computing.
The moment we admitted that every input to the block hash needs to come from the block or from another deterministic source, the whole class of bugs vanished. Cosmos published this pattern in 2018. Tendermint's paper described it earlier than that. We knew about it. We just thought we could get away without it, because most of the time we could.
Testnet has been running for a week with the new pattern. Around 70,000 blocks. Zero divergences. Zero drift dumps. Zero recovery wipes for reasons related to this bug.
For once, it's actually fixed.