// REPORT

How can two Nethermind databases holding the same state differ 17× — and then agree?

Ethereum · nethermind · flat DB · benchmarking · 2026 · reproducible pipeline & data →

1,461 tests matched by exact identity across both stores, zero gas mismatches. Nethermind nethermind:glamsterdam-devnet-7, --FlatDb.Enabled=true, rollback_strategy: container-recreate, drop_memory_caches: steps.

The behaviour

Two Nethermind databases. One is a mainnet shadowfork snapshot taken at block 24,402,727; the other is state generated from scratch by state-actor. The same EEST bloatnet fixtures run against both, on the same machine, with the page cache dropped between every test. Where the tests read accounts, the generated store reports up to 17× less throughput.

The two rows that do *not* diverge are the tell. Absent accounts (1.051) never read an account record, and the storage sweep (1.006) is far larger than anything a store can hold in a corner. Whatever is happening, it is specific to reading accounts that exist.
what the test does	n	throughput sa/joc	bytes sa/joc	jochemnet MB	state-actor MB
existing EOA	110	0.059	14.83	231	3907
existing contract	440	0.089	8.81	231	4619
absent account	110	1.051	2.94	78	218
warm query	77	0.876	145.73	1	229
storage slot	88	1.006	1.04	633	626
ether transfer	196	0.385	1.88	4015	7793
control (no state work)	440	0.733	49.54	2	96

Every opcode against every access mode, before any treatment: 41 of 48 cells are outside ±10% of parity, ranging from 0.05 to 1.17. This is not a few bad tests — it is nearly the whole matrix. The single column that stays dim is the one whose lookups never read an account record at all.

Read volume against gas for the contract-reading tests. jochemnet stays flat — 227 MB at 100M gas and 232 MB at 300M, triple the work for the same bytes — while state-actor climbs from 1986 MB to 5772 MB. Flat in gas means the arm is re-reading one bounded set of blocks; rising means every access goes somewhere new.

What we found wrong with the measurement

Three things, in the order they matter. Two are properties of how the baseline was prepared; the third is the harness not having the controls this kind of study needs.

The pre-run is promoted into the baseline. Only one arm replays a pre-run before measuring, and the harness is told to promote the result into the image every test restores from. That leaves the benchmark's own accounts as the newest versions in the youngest files of the LSM tree. Diagnosed below, fixed, and the fix accounts for almost all of the gap.
The measured step inherits its own setup step's cache. The page cache is dropped between the two, but the client is not restarted, so its RocksDB block cache is not. Diagnosed below, not fixed — every number here still carries it.
The harness has no compaction control. BENCHMARKOOR_COMPACT_BETWEEN_STEPS is named in a source comment, never implemented, and BENCHMARKOOR_POST_PRERUN_CMD is implemented on branch state-db-journal-drain only; absent from the binary these runs used. The treatment below had to be applied by hand.

The root cause: the benchmark's keys live in a corner of one store

Only one arm runs a pre-run. Before measuring, the jochemnet arm replays 10.06 GB of blocks — 7,736 of them, 64 transactions each, taking the chain from 24,402,728 to 24,410,463 — which create the accounts the benchmark then reads. The harness is then told promote_post_pre_runs: true, which freezes that post-pre-run layout into the golden image every test is restored from.

This is not a caching story. The harness drops the page cache between every test, and the pre-run happens once, before the loop. What survives the drop is not warm memory — it is which files hold the current copy of each key.

The structure that matters here is not the Merkle trie but the LSM tree underneath it: Nethermind keeps the flat state in RocksDB, whose account column family is keyed by keccak256(address)[0:20]. Those keys are hashes, so they are scattered uniformly — nothing about the benchmark's accounts is adjacent in key order. What is concentrated is the set of files holding their newest versions. The pre-run rewrote every one of those accounts, so their current copies landed together in a handful of recently flushed SSTs near the top of the tree, and a levelled store answers a point lookup from the newest level that has the key and stops. The working set is therefore bounded by the size of those few files, not by the number of keys.

Stated that way the mechanism makes a prediction: merging those files down into the bottom level should destroy the advantage entirely, because afterwards no level holds a privileged copy. That is a compaction, it is testable, and the rest of this page is the test.

Cold lookups against each store's own copy of the fixtures' accounts, fresh process per point, page cache dropped, 100% hits. At 500 lookups the two stores are indistinguishable — 1.88 against 2.00 blocks. By 50,000 jochemnet is at 0.23 because its volume has stopped growing: 45 MB at 20,000 lookups and 47 MB at 50,000. state-actor never saturates, climbing to 404 MB, because there is no corner to exhaust.

The per-lookup cost is not what differs — the number of distinct physical blocks is. Both stores pay about two blocks for a lookup they have not made before. One of them runs out of new blocks to read.

Defect 1: the pre-run is promoted into the baseline — and what compacting it does

The change, concretely. The treatment is a full RocksDB compaction of the affected column families: CompactRange over the whole key range with bottommost_level_compaction=kForce, which is required because a family that already sits entirely in its bottom level is a no-op for an ordinary compaction. The rewrite uses the other store's exact per-family options — filter policy, block size, restart interval, compression — verified knob by knob afterwards, so the only thing that changes is which file holds each key's newest copy. No value is touched and the state root is unchanged.

The pre-run is then removed from the arm's config. That is not a reduction of the workload: its writes are already in the promoted image, so the arm still starts from exactly the state the pre-run produced. Replaying it would simply write those accounts again and rebuild the very thing we just flattened.

The reproducible way to do this is the harness's post-pre-run hook (BENCHMARKOOR_POST_PRERUN_CMD), which runs an operator command after the pre-run and before the snapshot is promoted, aborting the promote if it fails. That hook lives on a branch and is not in the binary these runs used, so the compaction here was applied to the promoted image by hand. Same end state; the hook is the path anyone reproducing this should take.

The files at the top of each tree are the pre-run's. Merging them down is the whole treatment: no value changes, only which file holds the newest copy of a key.
column family	levels before	levels after	GB	rewrite
`flat/Account`	`L3:4 L4:36 L5:304 L6:20`	`L6:37`	17.38	185 s
`flat/StateNodes`	`L0:3 L3:1 L4:4 L5:60 L6:487`	`L6:158`	38.98	708 s
`code`	`L0:3 L5:6 L6:117`	`L6:115`	—	62 s

Then the full suite again, all 1,461 matched tests:

Tests agreeing within ±10% go from 12.8% to 53.0%. The two spread columns are there because a median can flatter a category: *existing EOA* really is tight — middle half 0.970–0.991, 108/110 inside the band — whereas *storage slot* sits on parity at 1.034 while ranging 0.549–2.094 with only 47/88 inside it. Independently reproduced on a 266-test subset that agreed with the uncorrected arm to within a couple of percent, so these ratios are not run-to-run noise.
what the test does	n	with confound	equalised	middle half, equalised	full range	within ±10%
existing EOA	110	0.059	0.979	0.970–0.991	0.883–1.144	108/110
existing contract	440	0.089	0.968	0.916–0.984	0.557–1.138	343/440
absent account	110	1.051	0.904	0.764–1.052	0.386–1.482	34/110
warm query	77	0.876	0.896	0.839–0.971	0.552–1.317	33/77
storage slot	88	1.006	1.034	0.967–1.123	0.549–2.094	47/88
ether transfer	196	0.385	1.013	0.980–1.049	0.681–1.581	160/196
control (no state work)	440	0.733	0.708	0.622–0.820	0.321–1.394	49/440

Each row is one kind of test: amber is the confounded measurement, green the equalised one, and the band is ±10% around parity. The two rows that were already inside the band stay inside it — the control that makes the rest credible.

The same 1,461 tests as individual results rather than medians: one dot per cluster of tests at that ratio, sized by how many, with the middle half drawn as a bar and the median as a tick. The count on the right is how many of that category's tests land inside ±10% of parity. Reading the two panels together is the point — the account clouds do not merely shift, they collapse from a smear across the left of the axis onto the parity band.

Defect 2: the measured step begins with a warm client — and that is not what the control gap is

The harness drops the OS page cache between the setup payload and the measured one. It cannot drop the client's own cache: Nethermind is not restarted inside a test — container-recreate rolls back per test, not per step — so the RocksDB block cache carries whatever setup pulled in straight into the measurement. That is a real gap in the method, and the two arms enter the measurement warmed by very different amounts:

Read the control rows across the two steps: jochemnet pays 31.5 MB in setup and 1.8 MB when measured, state-actor 9.4 MB then 96.1 MB. The arms are inverted between the steps, and the setup figure is identical across categories on each arm, so it is a fixed payload whose only variable effect is how much of each store it happens to leave cached.
tests	arm	setup reads	measured reads	total
control (no account work)	jochemnet	31.5 MB	1.8 MB	33.2 MB
control (no account work)	state-actor	9.4 MB	96.1 MB	105.5 MB
reads accounts	jochemnet	31.5 MB	3,663.0 MB	3,694.4 MB
reads accounts	state-actor	9.4 MB	4,234.0 MB	4,243.4 MB

The same numbers on a log axis. The arms are inverted between the two steps: jochemnet does its reading during setup, state-actor during the measurement. The obvious reading is that one arm enters the measurement warm and the other does not, which is what the next experiment tests.

Before testing which cache, it is worth asking how much any cache could be worth. container-recreate restarts the client for every test, so the setup step starts with nothing warm at all — cold page cache and a client that booted seconds ago. Whatever the measured step then gets for free must have been put into the client's memory by setup, and there are only two ways in: bytes setup read, or bytes setup wrote. Both are recorded. Setup reads 31.5 MB on jochemnet and writes 0.0 MB — the control payload changes no state, so there is no write channel and nothing sitting in a memtable.

That caps the explanation at 30%. The gap to account for is 104.5 MB, and jochemnet's entire carry-over budget is 31.5 MB. Even if every byte setup pulled off disk were retained and served the measurement for free, two thirds of the difference would remain. The bound does not depend on which cache holds the bytes, so it covers the caches we did not test as well as the one we did.

So we starved the cache. Both arms re-ran the same 266 tests with the flat database's block cache cut from 1 GiB to 8 MiB, a factor of 128. Same stores, same tests, same treatment, and the client's own startup log confirms the reduction reached it. If the control gap were the setup step's leftovers, taking the leftovers away had to close it.

Throughput ratio, state-actor over jochemnet, on identical test ids. We had pre-registered the prediction that control would move to 0.85–0.95. It moved +3.0%, to 0.707. Every account category moved by less than 0.9%.
category	1 GiB cache	8 MiB cache	change
CONTROL	0.686	0.707	+3.0%
existing EOA	0.983	0.978	-0.5%
existing contract	0.967	0.970	+0.3%
absent account	0.940	0.931	-0.9%

The read volumes say the same thing from the other side. Starving the cache pushed state-actor's control reads up, from 96.4 MB to 106.3 MB, while jochemnet's stayed at 1.8 MB — unmoved to the decimal. A store being fed by a warm cache reads more when you take the cache away. jochemnet did not, because 1.8 MB is what its control payload actually needs.

So the explanation is dead and the defect is not. The harness still cannot flush the client's cache, and it should be able to — restarting the client between steps is affordable, since a container is already recreated per test in seconds. But the cache is worth a few per cent here, not the gap, and the control asymmetry has to be a property of the two stores rather than an artifact of how we measured them. What property, we do not know.

One lever remains untested: this reduced the flat database's block cache, while Nethermind sizes its trie and database budgets from total machine memory independently. That does not rescue the carry-over story — a cache cannot explain reads that do not happen — but it does mean 'cache' is eliminated as the cause, not as a contributor.

What is left, and how much of it we can account for

Open: a different contract per access still costs 1.55×

What we know

One cell does not come back to parity, and it is the same cell the geth study flagged and left unexplained. Laying every opcode against every access mode shows it is a column, not a scatter — whatever is left does not care which opcode reads the account:

Throughput ratio for each of the 8 opcodes against each access mode, after treatment. Read it by column, not by row: DIFF_MAX sits at 0.67 and JUMPDEST at 0.93, while EOA, MINIMAL and SAME_MAX are all 0.98–0.98. The absent column is the uneven one at 0.88, which is the same dispersion its category showed above. Within the DIFF_MAX column the rows are not uniform, and the split is the whole story: the opcodes that execute the callee's code sit near 0.64 while BALANCE and EXTCODEHASH, which only read the account row, stay at parity.

Sorting the access modes by how much contract code they touch orders it cleanly:

Monotonic in code, and the cause is on disk: the code database is 45 GB on the generated store against 7.6 GB on the snapshot. Touch a different maximal contract every time and you pay for the larger database; reuse one contract, or touch no code at all, and the two stores are within 6%.
access mode	what it touches	jochemnet MB	state-actor MB	bytes sa/joc	throughput sa/joc
`EXISTING_EOA`	no code at all	3704	3907	1.06	0.979
`EXISTING_CONTRACT_MINIMAL`	minimal code	3647	3873	1.06	0.980
`EXISTING_CONTRACT_SAME_MAX`	one max-size contract, reused	3647	3863	1.06	0.980
`EXISTING_CONTRACT_JUMPDEST`	code scanned for jump destinations	4827	5598	1.18	0.931
`EXISTING_CONTRACT_DIFF_MAX`	a different max-size contract per access	4030	5639	1.55	0.655

The ordering is monotonic in how much distinct contract code the mode touches, and it is the incremental step that is lopsided: going from one reused contract to a different one per access costs state-actor 3863→5639 MB (+1775) against jochemnet's 3647→4030 MB (+383) — roughly 4.6× the marginal cost per additional distinct contract.

Where exactly it lives

Splitting the same cells two ways isolates it to a single square. Along one axis, whether the opcode loads the callee's code at all; along the other, whether the contract is a different one each access or the same one reused:

One square out of four. Reusing a contract is fine even for the opcodes that execute it (0.972–0.991 across 6 opcodes), and under DIFF_MAX the two opcodes that never touch code are at parity (0.969, 0.972). Only **loading a distinct contract's code** is expensive, at 0.639 across 6 opcodes (0.626–0.775). So it is not the account row, and it is not distinctness on its own.
	one contract, reused	a different contract each access
opcodes that load the code CALL, CALLCODE, DELEGATECALL, STATICCALL, EXTCODECOPY, EXTCODESIZE	0.981	0.639
opcodes that read only the account row BALANCE, EXTCODEHASH	0.977	0.971

What we ruled out

The obvious answer is the code database: the generated store's is 45 GB against 7.6 GB. Measured three ways, it is wrong in every one of them.

Per 400,000 accounts scanned. The generated store holds more contracts (31.3% of accounts against 19.1%) but far smaller ones (median 23 B against 45 B), and its maximum-size contracts compress to 4,186 bytes on disk against jochemnet's 9,705. Reading code is cheaper on the generated store at every granularity we can measure, which is the opposite of what the benchmark reports.
measurement	state-actor	jochemnet	verdict
cold random code lookup	10,513 B	14,208 B	generated store is cheaper
cold sweep, 3,000 distinct maximum-size contracts	4,186 B/read	9,705 B/read	generated store is cheaper
contracts at or above the 24,576-byte maximum	442	433	same population

What is still open

So the effect is real, it is confined to one square of the table above, and the three explanations that fit that square are each contradicted by a direct measurement. We are leaving it here rather than reaching for a fourth story: the honest statement is that executing a distinct contract costs the generated store 1.56× what it costs the snapshot, and we cannot yet say why. The fixture accounts themselves are not the answer either — all 20,000 addresses in the range the tests use carry no code at all on either arm.

Three clients, one artifact

Three engines, three storage designs, three state sizes for the same logical state — and the same artifact. The Besu and Nethermind columns are computed the same way from each study's own data (48 and 440 account cells respectively, gas matched exactly); treat the geth spread as its published range. Untreated, the generated store looks between 6× and 17× slower. Treated, all three land within a few percent of parity. That is what a methodology artifact looks like, as opposed to a property of any one database.
client	same state costs	generated vs snapshot, untreated	generated vs snapshot, treated
geth	674 GiB	2.01–5.81× on account reads, 0.91 on absent	1.031–1.117×
Besu	532 GiB	6.5× on account reads (10.05× the bytes)	0.908× (1.23× the bytes)
Nethermind	409 GiB	17× on account reads (8.81× the bytes)	0.968–0.979× (1.08× the bytes)

What to do about it

Never compare a promoted-after-pre-run snapshot against a store that did not get one. It is worth an order of magnitude and it does not announce itself.
Neutralise the pre-run's placement effect on the arm that has one — compact after the pre-run, before the snapshot is promoted. That is a single compaction per baseline, and the harness already has the right hook for it (BENCHMARKOOR_POST_PRERUN_CMD), though it is implemented on branch state-db-journal-drain only; absent from the binary these runs used. Note what not to do: compacting between every test's steps, the lever geth could reach over RPC, costs 185 s per column family here — about 3.1 days across 1,463 tests, six times the runtime of the suite it would be preparing.
Give the harness control of the client's cache, not just the page cache. Dropping /proc/sys/vm/drop_caches between steps leaves the client's own RocksDB block cache warm, so a measured step partly reflects what its own setup payload happened to load. Restarting the client between steps is affordable; the harness already recreates a container per test in seconds. Measure before you assume, though: starving that cache here moved the result by 3%, which is worth having and is not an explanation.
Print a locality diagnostic: read bytes per unit of gas, per arm. Flat in gas means a bounded working set and therefore an artifact — jochemnet read 227 MB at 100M and 232 MB at 300M while state-actor climbed from 1986 to 5772 MB. That check is a few lines, and it would have caught this on the first run rather than the thirtieth.

And the answer to the question in the title: the generated state was never 17× slower. With placement equalised, account reads agree within 2–3%, storage within 3%, ether transfers within 1%, and absent-account lookups resolve from a filter in 2.5 against 2.8 microseconds. Synthetic state is a sound substitute for benchmarking state access.

Two caveats stop that being a blanket endorsement. Executing a distinct contract still costs the generated store 1.56× what it costs the snapshot and we cannot say why, so anything code-execution heavy is not yet covered. And the control gap — tests doing no account work at all, where the generated store reads 106 MB against 1.8 MB — survived having its most likely cause tested and eliminated. It is a real difference between the two stores and we cannot yet name it.

What we got wrong on the way

The first compaction used RocksDB's default table options, which silently dropped the ribbon filter and switched to Snappy. Absent-account lookups inverted to a 19× advantage for the generated store, and it took a full re-run with the real per-family options to get the parity figure above.
A rebuild was applied to the mounted volume without promoting it, so the harness restored the untreated image before the first test and a 2.2-hour run measured nothing new. It became an accidental replication, which is the only reason we can quote a run-to-run noise figure.
The probe measured both stores as if they had no filters. It opened every column family with RocksDB's default options, and a reader with no filter_policy configured never consults the filters that are in the files. Absent and present lookups therefore cost the same, which made us retract a correct explanation. Opened properly, the filters work on both arms — 19,821 and 19,786 of 20,000 negatives rejected — and an absent lookup costs 73 against 7,314 bytes.
We published a mechanism for the last cell and it was wrong. The residual was attributed to the generated store's larger code database. Three direct measurements say the opposite: reading code is cheaper on that store per lookup, cheaper on a sweep of distinct maximum-size contracts, and the two stores hold the same number of them. The section above now reports the cell as open.
We had a fourth explanation and it failed too. The control gap looked like cross-step cache carry-over, and we pre-registered the prediction that starving the block cache would pull it to 0.85–0.95. It moved to 0.707. The section above reports the eliminated explanation rather than the one we expected to be writing, which is the only reason this list is worth keeping.

The numbers in this page are computed from the collected run data at build time; the generator refuses to emit the page if the data stops supporting the sentences above.

← all articles source & data →