How can two Besu databases holding the same state differ 8× in performance?

A case study on state-actor versus a Bonsai mainnet-snapshot database, and on why a published snapshot has to be compacted before it can be measured at all.

jochemnet block 24,402,727, chain id 1. Snapshot published by hyperledger/besu:25.12.0 under --sync-mode=SNAP; benchmarked with ethpandaops/besu:glamsterdam-devnet-7 (26.8-develop-00d2f04) under --sync-mode=FULL, storage format BONSAI version 3. Every arm ran on md2 NVMe, both arms, same host.

The behaviour

Three databases hold the same state. The same 1,100 EEST tests run against each of them, on the same host and the same NVMe drive. 660 of those tests do account-state work; the other 440 are controls that run the same loop and deliberately touch no state, and they are kept out of every comparison below and reported on their own. Gas matches across all three to six significant figures, so each one was asked to do the same work. Throughput does not match. On the tests that look up an account which is not there, the generated store is 8.1× slower than the published snapshot.

That is the number in the title. It is also mostly an accident of how the two stores were built, rather than anything about how Besu reads them. There are two separate accidents, and each one shows up on a different kind of read.

The chart below is the shape of the problem. Every dot is one test category, measured against the snapshot after compaction. They fall into two groups about ten times apart. The 8 dots on the left are the categories that have to prove an account is absent; the generated store is 8.5× slower on all of them, and compacting the snapshot does not close the gap. The 40 dots on the right read an account that exists, and they land between 4.0% and 19.0% of the snapshot. Two groups, two causes.

Outside the absent class, the residual comes in two tiers

Against the compacted snapshot, every one of the 40 categories that read an account which exists favours the snapshot, by a median of 5.9%. They do not all favour it equally, and the split is not noise: the 24 categories whose contract code is shared or absent sit at 5.2%, while the 16 that read a distinct contract on every access sit at 17.7%. That gap is the subject of the last section. Everything larger than it is a property of how one of the stores was built.

What Besu's own logs and counters say

Besu states its own configuration at startup, and two of those lines matter.

Existing database at /data. Metadata versionedStorageFormat=BaseVersionedStorageFormat{format=BONSAI, version=3}. Processing WAL...

That one appears on every boot of every arm, including the arm whose write-ahead log is 23 bytes long, so it is boot-time boilerplate rather than evidence of work. The second is the one that shapes the rest:

Flat db mode found FULL

Account reads go to Bonsai's flat keyspace, not down the trie. The metrics agree 99.83% of account lookups are served from the flat database.

That matters because it fixes what a read costs. An account lookup is one key in one column family, so the cost is the cost of locating that key: how many files have to be consulted, and how many bytes each consultation moves. It is not a tree walk whose depth grows with the state, which is the shape most people expect.

What we ruled out first

Before anything else, the two stores have to be shown to be the same kind of object holding different data, rather than differently configured engines.

The databases themselves

Both stores run the same RocksDB settings on every state column family: compression=kLZ4Compression and block_size=32,768, byte-identical. Key-value separation is enabled on exactly two column families, neither of which holds state: the snapshot's 26,018 blob files and 760 GB are block bodies and receipts. The generated store has 1 blob file of 322 bytes, because it has no chain history at all. Whole-store totals are therefore not comparable and are not compared anywhere in this article; the comparison is always per column family.

The extraction is deterministic: three independent unpacks of the published tarball produced the same 6,695 SST files and the same 382,863,226,174 bytes. Trie logs are not a factor either. Besu prints this on every boot:

Forcing --bonsai-limit-trie-logs-enabled=false, since it cannot be enabled with --sync-mode=FULL and --data-storage-format=BONSAI.

The snapshot's trie-log column family holds 2.8 MB, and the generated store's holds no files at all.

What the fixtures actually touch

The suite is 48 opcode/account-mode categories across 11 gas budgets from 100M to 300M. They split into three groups that behave so differently that any single median over them describes none of them:

• absent (110 tests): NON_EXISTING_ACCOUNT targets, where the lookup must prove the account is not there.
• leaf-only (198 tests): BALANCE, or an EXISTING_EOA target, which reads the account record and never the code.
• code-reading (792 tests): EXTCODE* and the CALL family into a contract, which also reads the code blob.

The file that looks like the cause: a 1.2 GB write-ahead log

The published snapshot ships a write-ahead log of 1,258,464,804 bytes across 21 files, and the pre-run replay leaves it that size or larger. A write-ahead log is where a key-value store puts writes that have not yet been folded into its sorted files, so a gigabyte of it sitting inside a snapshot is exactly the shape of a benchmark artifact: state that lives in one place on disk and another in memory, restored on every boot.

The first thing that argues against it is that the size is not even stable. Two pre-run replays of the same extraction left 1.33 GB and 1.22 GB behind. A cause that varies by 9% between runs is a poor explanation for an effect that reproduces.

The second is what happens when you drain it. The whole step takes 3 seconds, leaves 23 bytes, and adds no SST file. The count stays at 6,699. Had the log held writes that were not yet on disk, recovery would have written them out as new files. Promoting the result copies 69 MB in 171 ms. Nothing in that gigabyte was unpersisted; it is a log that had not been garbage-collected yet.

And the explicit flush inside that step reported 0.0 s, because by the time it ran there was nothing in memory to write: RocksDB's open path recovers the log and flushes it itself. On Besu, opening the store is the drain, which means every arm of every suite had already done it before a single test ran.

Measured end to end, draining it changes nothing: 0.992× on absent, 0.986× on leaf-only, 0.982× on code-reading for throughput and 1.026×, 1.019×, 1.015× for bytes read. Throughput holds to within 1.8% and bytes to within 2.6%, against a same-state repeat that reproduces to 0.9%. The one thing in the snapshot that looks like the cause is inert.

The root cause: the pre-run's rows are the newest versions of their keys

Before each test the harness replays a pre-run bundle: 10.06 GB of blocks that create the accounts the benchmark then reads. It takes the store from 6,695 SST files to 6,699. Four files is not much of a change; what matters is that every account the benchmark reads now has its newest version in one of them.

The compaction proves it, by what it deletes. Merging the levels of the account column family removes 10,833,266 entries, 365,626,139 down to 354,792,873, and a merge can only drop an entry if it was an obsolete older version of a key held somewhere else in the tree. The same thing happens in storage and trie-branch:

A point lookup in a levelled store checks the youngest files first and stops at the first version it finds. After the pre-run the account column family holds 4 files at the top of the tree, 18 MB of it, above 273 files holding 16.7 GB at the bottom. The reads the benchmark performs are satisfied in the small, young part of the tree.

Compaction merges all of it into the bottom level: 0 files left at the top, and the same lookup now reads a 32,768-byte block out of a sorted run of 17.9 GB with no locality to whatever was written beside it. The cost of a read goes up while the store gets smaller, which is the signature of removing an accident rather than applying an optimisation.

The fix, and the proof

A full-range compaction takes 91 minutes and leaves the store smaller: 6,699 files and 382.7 GB become 5,454 and 360.3 GB. Most of the time goes to the largest column families: cf09 3104s, cf08 1477s, cf01 414s.

Now measure the generated store against the treated snapshot instead of the published one. Nothing about the generated store changed, so whatever moves is the artifact leaving. Take agreement to within ±10% of parity as the bar: 0 of 48 categories cleared it against the snapshot as published, and 24 of 48 clear it against the compacted one. Per measurement workload rather than per category, 0% becomes 52%.

Read that bar carefully, because it flatters the generated store. Clearing a ±10% band is not the same as agreeing. Not one of the 48 categories reaches parity: the closest is CALLCODE EOA at 0.960×, and not one of the 660 measurement workloads is faster than the snapshot. The best single workload of the 660 manages 0.984×. The control workloads, which run the same loop and touch no account state, sit at 1.019×, so the generated store is 1.9% faster on work that reads nothing. Measured against that as the true zero, every category that reads account state is at least 5.7% slow. The band hides a floor; it does not mean half the suite is clean.

The 24 that converge are the categories whose contract code is shared or absent, and they land at a median of 5.2% off parity, down from 83.8%. Two groups stay outside, with two different causes already named: the 8 absence categories, still 9.1× apart, and the 16 that read a distinct contract on every access, which close to 17.7% and no further. The first is the next section; the second is the last one.

Why absence proofs cost 50× the bytes: the generated store has no bloom filters

The absence categories probe addresses that are missing from both stores. The counters in the second section put the miss rate at 0.955 and 0.999. Identical question, identical answer, and 50.1× the bytes to arrive at it.

The reason is in the files. Every state column family in the snapshot carries a bloom filter; the generated store's carry none at all.

The cost lands almost entirely on absence. A lookup that finds its key was going to read that block anyway, so the filters buy little there, 1.21× on leaf-only and 1.32× on code-reading, against 50.1× when there is nothing to find. This is a property of how the generated store was written, not of what it contains.

What this does not separate

Some part of the 1.21× to 1.32× on reads that find their key may also be filter absence rather than the record geometry of the next section. The two are not separated here.

This one was already fixed upstream before the store was built. state-actor #133 (11389bc) put a full bloom filter at 10 bits/key on every column family and merged on 2026-08-04; the store measured here was generated on 2026-09-09 from a working tree (e4cb205-dirty) that predated it by 36 days. So the 50× is real, reproducible from the archived store, and our own doing: we measured a build that had already been superseded.

The mechanism is fixed rather than argued. Probing each store directly for the cost of one absent-account lookup: the unfiltered store reads 1.002 data blocks per lookup and its filter is consulted never; a filtered store built from the same generator reads 0.010 blocks and rejects 0.990 lookups per lookup at the filter, which is a 100× reduction in exactly the operation this section is about. The snapshot reads 0.038. Whatever remains in the absence class after that is not filter absence.

The residual

What is left after both artifacts is small, consistent, and in two tiers. Against the compacted snapshot every one of the 40 categories that reads an existing account favours the snapshot, but the 24 whose code is shared or absent sit at 5.2% while the 16 that read a distinct contract per access sit at 17.7%. Order the classes by how much unique contract code they touch and you have ordered the residual, which is the whole of this section.

Two ratios carry the argument: compression 1.182× and block size 1.180×. The same two ratios measured on geth were 1.167× and 1.166×, within 1.3% of these, on a different storage engine, with a different compression algorithm and an eight times larger block. Whatever this is, it is not an artifact of one engine.

Both tiers come out of one property of the generated state, and it takes a sentence to measure. Sampling each store's account keyspace, 80.8% of the snapshot's records are plain externally-owned accounts carrying the two empty constants, against 68.6% of the generated store's, which puts 68,120,232 contract accounts in the snapshot against 135,238,776 in the generated store. The snapshot resolves those to 2,416,222 distinct bytecodes and the generated store to 134,442,676: mainnet reuses each bytecode about 28 times, and the generated store reuses none (1.01). That is why the account records compress worse, because a shared code hash becomes a unique one, and it is why a code read pays for block padding.

That property is now fixed upstream, in the two places it came from. state-actor #137 (c0e1162) draws each delegation designator from a fixed pool of 256 authorities instead of minting a unique one per account, so designators repeat the way real ones do, at a delegation rate still matched to mainnet. #138 (95e5a10) gives contracts a shared bytecode pool sized to mainnet's reuse. Measured on a small store built from the result: account records go from 0.513 to 0.447 physical over logical against the snapshot's 0.434, and reuse lands at 32.0 accounts per distinct bytecode against mainnet's 28.2. The record geometry this section measures is therefore gone as a mechanism; whether the throughput follows is a rerun, not an inference, and it has not been done.

The code half is not the code being read

The same chain does not explain the 16 distinct-code categories, and it is worth saying why rather than stretching it. Isolate what the code read itself costs by subtracting the account-only classes: a distinct-contract access moves 24.1 extra bytes per gas on the snapshot against 47.3 on the generated store, a factor of 1.97. The obvious reading is that the generated store's contract code is less compressible. It is not. Sampling the 24,576-byte fixture contracts out of each store's code column family, both compress to about one percent of their size: 0.0109 on the snapshot, 0.0070 on the generated store. The contract being read is nearly free to store in both.

What a lookup pays for is the block the record sits in. RocksDB closes a data block once it passes 32,768 bytes, so a 24,576-byte contract leaves room for whatever comes next in code-hash order, and that is a sample of each store's contract population. The two populations are nothing alike.

So the generated store's code reads are dearer because of the company its contracts keep. It holds 134,442,676 contracts averaging 398 bytes against the snapshot's 2,416,222 averaging 7,671, so 56× as many contracts each a fraction of the size, and 94% of the sampled records are 23 bytes long. The generator reached its size target by contract count rather than by matching mainnet's code-size distribution, and a code read pays for that as block padding.

Which means it is fixable, in two independent places. Drop the code column family's block size to 16,384 bytes and a 24,576-byte contract no longer fits alongside anything: co-tenants go to 0, the block costs 267 bytes on the snapshot against 173 on the generated store, and the penalty inverts. Or give the generator a contract population shaped like mainnet's, including its duplication, since mainnet reuses each bytecode about 28 times and the generated store reuses none. Neither is a change to Besu. It is also why the geth study never met this: pebble defaults to 4 KiB blocks, a quarter of what both of these stores use.

The second of those was taken, and it went too far. #138's pool is one real ERC20 runtime of about 1.7 KB, tiled to reach the sampled code size and rotated per pool entry, so entries are self-similar inside themselves and to each other. It gets reuse and size right and content wrong: on a store built from it, the code column family compresses to 0.056 physical over logical against mainnet's 0.371, individual records deflate to 0.215 against mainnet's 0.443, and a packed block to 0.112 against 0.329. That is 6.6× too compressible, so the expected effect on this class is that it changes sign rather than closes. Real mainnet code compresses because many different contracts repeat across accounts, not because one contract repeats inside itself, and the fix is a corpus. The PR's own comment predicted this failure mode.

Independent of the generator, the cost itself is measurable rather than modelled. One cold read of a 24,576-byte contract, page cache dropped, counting the bytes the block layer actually served: 17,597 bytes on the generated store against 901 on the snapshot, a factor of 20. The population is the same contracts in both stores, and a later rebuild of the same generator lineage gave 17,724, so this is the mechanism of this section weighed on a scale rather than derived. Bytes only: the two stores no longer sit on the same device, and wall time would be measuring that.

Where this stands

Three mechanisms, three states. Every row's store-level column is a measurement on a store built from the merged fix; not one of them is a throughput measurement, because that needs a regenerated store and a refilled payload set.

One thing the rerun will have to explain that this article cannot. The gap is not a fixed cost per read, and not a fixed cost per block: it grows with the gas budget. Across 11 budgets from 100M to 300M the absence class goes 0.147 to 0.099, distinct code 0.852 to 0.790, and shared code 0.952 to 0.940, while the control holds flat at 1.021 to 1.020. A constant per-read penalty predicts a flat ratio and a constant per-block overhead predicts the ratio improving as the budget grows. Neither happens, so any one number for the residual is a number about a gas budget.

The same experiment on two clients

The generator is deterministic across clients: the same seed and spec produced 6,404,913,395 items here against 6,404,913,405 on geth, 10 apart in six billion, and the same state root 0x5b305cc0f85f9ffa…. The stores they produced are not the same size: 532 GiB on Besu against 674.25 GiB on geth, for identical logical state. So the two studies measure the same state through two different engines, and the residual survives the change of engine. Its magnitude does not, and the two are not quite the same measurement: geth published 9.1% across 13 categories with its control sitting inside that figure, where Besu's control is at 1.9% and the classes split 5.2% and 17.7%.