How can two Besu databases holding the same state differ 9×?

We ran the same EEST benchmarks against a Besu mainnet snapshot and a store that state-actor generated from scratch. On calls into a different contract every time, the generated store came out 9.1× slower; on tests that look up accounts that don't exist, 8.5×. Three things about how the stores were built made the gap. Fixed, absence lands at parity and shared code within 1.5%. Distinct-contract code overshoots, and how far depends on the disk. Here's each finding, what it did, and what's left.

jochemnet block 24,402,727, chain id 1. Snapshot published by hyperledger/besu:25.12.0 under --sync-mode=SNAP; benchmarked with ethpandaops/besu:glamsterdam-devnet-7 (26.8-develop-00d2f04) under --sync-mode=FULL, storage format BONSAI version 3. Every arm ran on one host, from the same NVMe array.

Both stores run byte-identical RocksDB settings on every state column family: compression=kLZ4Compression, block_size=32,768. Account reads go through Bonsai's flat keyspace on both, 99.83% of lookups served from it rather than a trie walk, so what a read costs is the cost of locating one key in one column family, not the depth of a tree.

Every ratio on this page is state-actor's throughput divided by a reference, taken as the median across whatever tests share a row. Above 1× means the generated store is faster; below means it's slower. We use medians rather than a pooled average everywhere, because a handful of huge reads shouldn't get to decide what a typical one costs.

The problem: the same state, 9× apart

Three databases hold the same state: the published mainnet snapshot as it ships, the same snapshot compacted, and state-actor's generated companion. The same 1,100 EEST tests run against each of them, on the same host and the same drive. Gas is identical to six figures, so all three were asked to do the same work. Throughput isn't the same at all.

On the tests that call a different contract every time, the generated store is 9.1× slower than the snapshot as published. On the tests that look up an account which isn't there it's 8.5×. Neither is a client bug. Each is an artifact of how the stores were built, and each has a name below.

Compacted just means flushed and fully merged, RocksDB's own maintenance operation, nothing client-specific. We measure the generated store against both the snapshot as published and the same snapshot compacted, because the two don't agree with each other either, and that disagreement is the first finding below.

Two kinds of test. Every test boots a fresh client, so a test that runs for a tenth of a second measures a young process. The companion Nethermind study splits every view at one second of measured time, and we do the same here, with each test's class fixed once from the compacted arm. On Besu the split matters less. Of the 660 measurement tests only 29 run under a second on the compacted snapshot, all of them absence tests, so the short class is essentially the 440 controls. And Besu's short tests reproduce: measured twice, the compacted snapshot lands 51 of its 52 sub-second tests inside ±10% of itself, and 68 of 68 over a second. We still show the short rows separately and keep the headline to tests over a second, so the two pages read the same way.

Finding 1: the pre-run's rows were the newest versions of their keys

The benchmark's fixtures touch specific accounts that have to exist first. State-actor bakes them straight into the state it generates, so it needs nothing extra. The snapshot is real historical mainnet state, and it doesn't contain the benchmark's own fixture accounts, so the harness has to create them by replaying a pre-run of setup blocks before the measured test ever runs.

That bundle is 10.06 GB of blocks. It's not a big change in file count, only 6,699 SST files against 6,695 shipped. But it's the newest four files, and every account the benchmark reads now has its newest version in one of them.

That's why the lookup is fast on the untreated snapshot. A point lookup checks the youngest files first and stops at the first version it finds. After the pre-run, cf06 holds just 4 files at the top of its tree, against 273 at the bottom. A lookup for a recently written account never has to go further than the top.

The fix is one full-range compaction of the promoted image, 91 minutes, with the client's own table options. No value changes; only which file holds each key's newest copy does. Merging the levels removes 10,833,266 entries from cf06 alone, 365,626,139 down to 354,792,873. Storage drops 51,464,730 entries and trie-branch 85,123,359, the same pattern in every state column.

A store holding one version per key can't lose entries to a compaction, so these were obsolete copies the pre-run had shadowed. Draining the write-ahead log first changes nothing at all; every bucket ratio stays within 1.8% of 1. Compaction does all the work.

What it did: against the snapshot as published, 0 of 48 categories were inside ±10% of parity. Against the compacted one, 24 of 48 are, and the share of individual tests inside the band goes from 0% to 52%. Every category still reads below 1; the closest is CALLCODE EOA at 0.960×. The control, which the treatment must not move, sits at 1.005 drained and 1.010 compacted. What clears the band is every category whose callee has no code or shares it. The 8 absence categories and the 16 distinct-code ones don't, and each has its own cause below.

Finding 2: the generated store carries no bloom filters

Every state column family in the compacted snapshot carries a bloom filter, 10 bits/key, 6.49 GB of them across the four state column families. The generated store's carry none at all. A filter answers “this file cannot contain that key” without reading it. Without one, a lookup that will find nothing still has to read index and data blocks to reach the same conclusion.

The cost lands almost entirely on absence, because that's the one class where the filter would have paid for itself: 50× the bytes, against just 1.21× and 1.32× on reads that find their key. A read that finds its key was going to touch that block anyway; a read that finds nothing didn't have to, and does.

It isn't that one store finds more absent accounts than the other. Besu's own counters put the miss rate at 0.955 on the compacted snapshot and 0.999 on the generated store: near-identical, both close to certain. Same question, same answer, and 50.1× the bytes to arrive at it.

The fix was already merged upstream before this store existed: state-actor#133 (11389bc), merged 2026-08-04. The store measured here was generated 36 days earlier, from a working tree that predates it. So the 50× is real, it's reproducible from the archived store, and it's our own doing. What it did: on a store regenerated at a revision that carries it, the absence class went from 0.111 to 1.002. That store also carries the code-pool change of Finding 3, which doesn't touch absent accounts.

Finding 3: a contract's code shares its data block with its neighbours

Mainnet reuses each bytecode about 28 times: its code column holds 2,416,222 distinct bytecodes behind roughly 68,120,232 contract accounts. The generated store held 134,442,676 behind 135,238,776, almost one each. So a block on the generated store holds 32 distinct small records against mainnet's 2.4, and none of them compress against each other: cf06 records compress 1.182× worse and blocks run 1.180× bigger, close to geth's 1.167 and 1.166 on a different engine.

A code read pays for whatever shares its block. One cold read of the same fixture contract, page cache dropped first, moves 17,597 bytes off disk on the generated store against 901 on the snapshot. Isolated from the account row it rides with, the code read costs 47.3 extra bytes per gas on the generated store against 24.1, 1.97×.

The first fix overshot. #137 and #138 made contracts share bytecode from a fixed pool, but the pool was one runtime, tiled, and it compresses far better than real code. On a store built from it the distinct-code class went from 0.828 to 1.461, past parity the other way. A store that's too fast is as wrong as one that's too slow; it just doesn't look broken.

The fix is a pool of 128 real mainnet contracts, state-actor#141. What it did: the class moved to 1.309, a 33% cut in the overshoot but still outside the band on the fast side, and all 16 distinct-code categories moved toward parity. Throughput follows bytes at every budget: 1.214× at 100M gas on 0.77 of the snapshot's bytes, 1.367 on 0.73 at 300M.

Also found and fixed, and moving nothing: the generator's closing compaction left every bottom-level file with a non-zero sequence number, so the client rewrote the store on every restart (state-actor#139). The dead ends and the probes we had to redo are in the README.

Where it stands

Absence is at parity: 1.002 against 0.117 before, both over every absence test. Tests whose callee has no code or shares it moved from 0.947 to 0.985, inside the band and 1.5% short of parity, and we don't have a mechanism for that last bit. Distinct code went past parity: 1.309, faster than the snapshot and outside the band on the fast side. The control, which does no account-state work, held at 1.016 against 1.019 before, so the harness didn't drift under us.

One caveat, and it's unresolved. The same generated store, same method, same tests, reads 5.67 GB per test on NVMe and 1.48 GB on the HDD array, 3.8× apart. Paired against a reference rebuilt on the same array, the distinct-code class reads 0.979, 8 of 8 categories inside the band, against 1.309 on NVMe. We can't yet say which number belongs to the store and which to the disk. The table quotes NVMe because that's what a node runs on.

What's left, and why

• Which medium the distinct-code number belongs to. 1.309 on NVMe, 0.979 on the same-array pair. A sequential pair with both stores on NVMe answers it: the 1,071 GB reference and the 350 GB generated store on one drive.
• Absence and shared code haven't been re-run against a same-disk reference. The paired run covered only the distinct-code class. The other two still divide by the archived compacted arm on NVMe, whose bytes per test we can't yet pin to a medium. The same paired run would settle all three.
• Besu pays less per extra byte than geth. Before the code fixes, tests on existing accounts ran 6.3% slower than the compacted snapshot; the geth study measured 9.1% on its own pair. Yet Besu read more extra bytes for it, 1.21× to 1.32× against geth's 1.119×. Same sign and same mechanism on a different engine; why the cost per byte differs is open.

Every number on this page is computed from the collected run data when the page is built, and the build fails if the data stops supporting a sentence. The process history, the dead ends, and the probe bugs we found and fixed along the way live in the README, not here.