How can two Nethermind databases holding the same state differ 17×, and then agree?

We ran the same EEST bloatnet benchmarks against two Nethermind databases that hold the same state. The one generated from scratch came out up to 17× slower. It wasn't. Five things about how the two stores were built and measured made the gap. Fix them one at a time and the tests that run over a second land at a median of 1.01, 122 of 144 inside ±10%. Here's each one, what it did to the numbers, and what's left.

The problem: the same state, up to 17× apart

Two databases with the same accounts: a mainnet shadowfork snapshot at block 24,402,727 (we call it jochemnet), and the same state written from scratch by state-actor. Same fixtures, same machine, page cache dropped between tests. Every number here is state-actor's throughput divided by jochemnet's: 1.00 is parity, 0.10 means the generated store is ten times slower.

Each dot in the figure is one test. The white tick is the category's median, and the numbers on the right are that median and how many of the category's tests land inside ±10%. The same rows and the same axis come back at the end, after the fixes.

The tests that read accounts are 15 to 19 times slower on the generated store. The controls run the same loop and touch no state, and they sit at 0.72. That's the first clue. Whatever makes account reads that slow isn't the whole client being slow.

The second clue is storage. The storage tests that run over a second read far more data than any corner of a store could hold, and they sit at 1.00. Ether transfers, which touch accounts too, sit at 0.38. Whatever this is, it's about reading accounts that already exist.

Two kinds of test. A test that runs for ten seconds and one that runs for a tenth of a second aren't measuring the same thing. Every test boots a fresh client, and for its first fraction of a second the client is still warming up. So every view here is split at one second of measured time. Under a second, one store measured twice lands only 41% to 61% of its tests within ±10% of itself; over a second, 96% to 100% do. Sub-second tests are always shown and never counted as evidence.

Five things caused the gap:

1. the benchmark's accounts were the newest files in the snapshot;
2. every test boots a fresh client, and the two stores warm it up differently;
3. two storage columns nobody had compacted;
4. a contract's code block is shared with whatever the generator packed next to it;
5. we had repacked the top of the snapshot's trie into blocks a quarter of the client's size.

1. The benchmark's accounts were the newest files in the snapshot

Only jochemnet runs a pre-run: 7,736 blocks that create the accounts the benchmark then reads. The harness promotes the result into the image every test restores from. So each of those accounts has its newest copy in a handful of fresh files at the top of the LSM tree, and a lookup stops at the newest file holding the key. On jochemnet the benchmark reads a small, hot corner of the store; on state-actor it reads all of it.

It isn't caching, and it isn't a worse store. The page cache is dropped between tests, so the advantage is on disk. And on keys sampled at random from each store's own accounts, jochemnet is the more expensive one: 2.51 blocks per lookup against 1.99. What the figure shows is that the benchmark's keys are special. The cost per lookup is the same on both stores; what runs out on jochemnet is the supply of blocks it hasn't read yet.

The fix is one compaction of the promoted image, into the bottom level, with the client's own table options. No value changes and the state root doesn't move; only which file holds each key's newest copy. Then the pre-run comes out of the arm's config: its writes are already in the promoted image, and replaying it would rebuild the corner we just flattened. The snapshot's account column went from 364 files on 4 levels to 37 files on the bottom one, rewritten in 185 s. What it did, across the full suite of 1,461 tests: existing EOAs went from 0.059 to 0.979, existing contracts from 0.089 to 0.968, and the share of tests inside the band from 13% to 53%. That's almost all of the 17×.

2. Every test boots a fresh client, and the two stores warm it up differently

The harness restarts Nethermind for every test so tests can't leak into each other. That means every measured block runs in a process a few seconds old, and a young .NET process is still compiling itself: hot methods start on an unoptimised tier and get promoted in the background once they've been called enough.

Why the arms differ. The first block after boot is slow on both arms, whatever it holds. jochemnet's fixtures spend it on a block of real EVM work, so the interpreter is promoted before the measurement starts. state-actor's fixtures spend it on an empty block, a fork-activation block they need because that chain starts at genesis. So state-actor enters the measured block with an interpreter that's still being compiled underneath it. One arm measures warm code, the other measures the JIT catching up.

The compiler isn't a rounding error. Inside the measured steps of 40 control tests, the JIT thread burned 22.6 CPU-seconds on jochemnet and 21.1 on state-actor, more than the threads executing the blocks (8.9 and 10.9).

What it isn't. Turning off the page-cache drops took state-actor's control reads from 96 MB to 0.0 MB and made it no faster (30.9 to 27.8 MGas/s). Turning off parallel execution made the gap wider (0.56). Taking the I/O away didn't remove the gap. Taking the JIT away did.

The test: switch tiered compilation off on both arms, so both run optimised code from the first block. It's a diagnostic, not a production setting. What it did: the controls went from 0.73 to 1.14, and distinct-contract calls from 0.64 to 0.94. That's the tell: state-actor barely moved, and jochemnet got slower (9.7 to 13.6 s). With tiering on it had reached promoted code partway through the test.

The fix that belongs in the harness is a discarded burn-in block before the measured one. The harness doesn't have one yet, so every number after this section is measured with tiered compilation off on both arms. That over-corrects the short tests, which now land above parity, and it's one more reason they aren't counted.

3. Two storage columns nobody had compacted

The snapshot's storage rows sat in 807 files over 6 LSM levels, and its storage trie in 1,961 over 5. The generated store keeps each in one. A lookup for a slot that doesn't exist has to be turned away by every level, so on one test the snapshot read 3,770 blocks where the generated store read 900. This one cut the other way: the generated store was the fast one. It hid in the first comparison because the storage tests over a second sat at parity, and the absent-slot test runs under one. Finding 1 compacted the columns the slow tests read, and storage wasn't one of them.

The fix is the same compaction as in finding 1, one column at a time. What it did: the absent-slot test went from 2.03× to 0.94, and new-value writes from 1.12 to 1.00. Both absent-slot tests run under a second, so their timings are shown, not counted. The reads are what closed: storage-row reads on the absent-slot test are 848 against 896, and storage-trie reads on the new-value test 61,457 against 61,739.

The lesson isn't about storage. A generated store is written once and compacted once. A real one is a stack of levels, and on operations that ask for keys that aren't there that difference was worth up to 2.1×. Compacting the snapshot makes the two stores comparable. It doesn't make the generated one more like mainnet.

4. A contract's code block is shared with whatever the generator packed next to it

After the first three fixes, one kind of test was still slow: calls into a different maximum-size contract every time. Both stores key code by its hash, so a contract's neighbours in a data block are random. What differs is who they are. On the snapshot, 19% of accounts have code, median 45 bytes and a wide spread. On the generated store, 31% do, almost all of it a distinct 23-byte stub that doesn't compress. Every code fetch is one block, and each one is 562 bytes bigger on the generated store, so more of them cross a 4 KB page and pay a second physical read.

Why the obvious answer is wrong. The generated store's code database is 45 GB against 7.6 GB, so the natural guess is that code lookups cost more there. Measured alone, they cost less. One random cold lookup reads 10,513 bytes in 196 µs against 14,208 bytes in 346 µs, and a sweep of 3,000 distinct maximum-size contracts reads 4,186 bytes per contract against 9,705. The two stores even hold about the same number of maximum-size contracts, 442 and 433 per 400,000 accounts. The cost isn't in the lookup. It's in what the lookup drags in with it.

Each code read takes 149 µs longer on average, and 28% of them land in the 1–2 ms bucket against 14%. The traced test runs about a second slower: 14.10–14.49 s against 13.24–13.48 s, five runs each.

The test: rewrite the generated store's code with one contract per block. Every key and value stays the same, and the traced test then ran in 13.18, 13.12, 13.26 s, on jochemnet's side of the gap. Both cells closed (0.938 to 1.008, 0.943 to 1.015) and the reused-contract control stayed put. The fix belongs in the generator's code pool, not in a block size: state-actor#141 slices it from real mainnet bytecode. On a store regenerated with it the two cells read 1.09 and 1.10. That's the gap closed and a little past it.

Besu sees the same split on the store its own study generated: calls into a distinct contract at 0.83 against 0.94 for a reused one. Different client, different block size, same neighbours.

5. We had repacked the top of the snapshot's trie into 4 KB blocks

A transfer writes two accounts, so every transfer re-hashes both paths up to the state root. Nethermind keeps the top of that trie, every node on a path of up to five nibbles, in one column, where a node and its sixteen children sit side by side. The client packs them into 16 KB blocks, 29 nodes each, so a family usually shares one read. The snapshot's column held 7 nodes per 4 KB block: 1.07 top-node reads per touched account against 0.73.

The tries are the same shape. Both tops are the complete sixteen-way tree five nibbles deep, 1,118,481 nodes. The generated trie holds 20% more accounts and is marginally deeper, 8.90 nodes on a cold root-to-leaf walk against 8.82. Shape can't make it read fewer nodes. The packing can.

That layout wasn't the snapshot's. We wrote it. RocksDB numbers files in order, and the 4 KB files (045338–045524) came just before our rebuild of the account column (045615+). The tool that did it opened the store with the client's options for that one column and RocksDB's defaults for the rest, and a pending compaction rewrote the top-of-trie column under those defaults. The test: give each store the other's packing. Across 94 transfer test pairs the cell went from 1.074 to 1.014, the read counts swapped, and each arm moved about half the gap (jochemnet ×1.034, state-actor ×0.975). The snapshot now carries the client's packing.

Also found and fixed, and moving nothing: the generator's closing compaction left files the client rewrote on every boot, 1,962 MB in 60 idle seconds (state-actor#139). The dead ends and our own measurement mistakes are in the README.

Where it stands

Over a second, 122 of 144 tests now sit inside the band (median 1.012, and 1.012 in the second pair), against 138 of 144 when state-actor is measured twice. The account reads that started at 0.05 are at 0.975 to 0.999. Calls into an EOA, a minimal contract or a reused one sit at 0.997 to 1.002, and storage at 1.00. Three rows are still further from parity than the floor explains: ether transfers at 1.05, and the distinct-contract and jumpdest calls at 1.09 and 1.10. They're the first two items below.

Pair B repeats the whole comparison with fresh runs of both stores, so neither arm's number is shared between the two pairs. Over a second it lands at 1.012 against pair A's 1.012. That's how we know the table isn't one lucky run of either store.

Under a second, 40 of 122 tests land in the band, and the same store measured twice manages only 50 (75 for jochemnet). The controls read 1.11 and the warm queries 1.27, above parity because tiering is off. Those rows measure the harness, not the stores.

What's left, and why

The generated state was never 17× slower. Three things are left, and each has a name:

• Distinct-contract code runs 9 to 10% faster on the generated store. #141 samples real mainnet code but keeps a 1 KiB floor on each record, so a fixture contract starts a block of its own more often than on mainnet, where the median contract is 45 bytes. Cheaper than mainnet is as wrong as dearer. The next generator fix is record sizes drawn from mainnet's distribution.
• Ether transfers run about 5% faster on the generated store, and up to 1.5× when the value goes to accounts that don't exist yet. We traced 12 transfer tests on both stores, and the read counts match within 5% on every column. So it's client work, not I/O. The clearest case creates thousands of accounts in every block, and there the snapshot spends 2.14 CPU-seconds in managed thread-pool threads against 1.23. Turning off either warming path doesn't change that. It's open, and the next instrument is a profiler.
• Sub-second tests can't be compared on this harness yet. That's not a store property. Every test boots a fresh client, and under a second the measurement is mostly that client warming up. A discarded burn-in block before the measured one would fix it. Until the harness has one, those numbers are shown and not counted.

None of the three is a reason to distrust the generated store on tests over a second. The first is a generator fix, the second is a question for the client, and the third is a harness fix.

Every number on this page is computed from the collected run data when the page is built, and the build fails if the data stops supporting a sentence.