The same EEST benchmarks ran up to 64× slower on a generated reth database than on a mainnet snapshot. None of the causes were in reth. With all six fixed and three refinements on top, 153 of the 154 tests that run half a second or more agree within ±10%.
jochemnet is a mainnet shadowfork snapshot. state-actor writes synthetic state straight into reth's tables. benchmarkoor runs the same 266 EEST bloatnet tests against both, restoring the store before each one. Every ratio below is state-actor's throughput over jochemnet's, so 1.00 is parity, and a test is in band within ±10% on two independent pairs of runs. Tests under half a second are shown but not scored: they barely reproduce against themselves.
The problem: up to 64× apart
Figure 1. The first measurement, before any fix, one dot per test category. Sequential-account lookups ran 39–64× slower, and calls into distinct 24 KB contracts at 0.65–0.79. Each store measured against itself agreed, so the gap was real.Figure 2. The six causes, from the disk up. The numbers are this article's sections.
1. The test rig read the disk one request at a time
Figure 3. Reads per second through a test copy of the same disk setup. In its default mode it stays at 13.7k however many readers there are; with direct I/O it reaches 130.4k.
Both databases lived inside a disk image, attached through Linux's loop device. In its default mode the loop device passes reads to the disk one at a time, so reth's many parallel reads waited in line. It also read ahead, fetching data nobody asked for, which hurt state-actor's layout of contract code most.
Switching it to direct I/O fixed both: every test that reads state got 2–9× faster. Everything below is measured that way. Our earlier geth and Besu runs on this host used the same default.
2. The two stores used different storage layouts
Figure 4. The same six neighbouring accounts in each layout. By address (v1) they share a page; by hash (v2) each lands on a different one. Switching jochemnet to the hashed lookups alone made its sequential tests 7× slower.
jochemnet was a v1 store; state-actor writes v2. reth has two storage layouts: the older v1 looks accounts up by address, the current v2 by hash. EEST's sequential tests read neighbouring addresses, which v1 finds together and v2 scatters. So the two stores weren't doing the same work, and that was most of the gap.
New reth nodes use v2, so we converted jochemnet. From here on, every gap is state-actor's.
3. Rows out of order, and trie rows reth never stores
Figure 5. A full page splits where the new key lands: in the middle for a key inside its range, at the edge for a key past its end.
state-actor wrote accounts in the order it generated them, which is random against the table's hash order, so nearly every insert split a page in half. PR1 writes them in key order.
Figure 6. reth only stores a trie node that has another branch below it; the bottom ones it rebuilds from the leaves it reads anyway.
state-actor stored every node, so state-root updates read rows reth would never have written. PR2 stores only what reth would. Neither PR changes the state root.
4. Page fill is a dial, and mainnet's pages aren't full
Figure 7. libmdbx can leave room free on a page when it splits: none packs it full, the most leaves it half full. The dots are each store's measured account table.
Once writes arrive in key order, how full each page ends up is a choice, and it decides how many pages a lookup touches. jochemnet's account and storage tables carry years of random inserts, so their pages are well short of full.
PR3 puts the rest of state-actor's writes in key order. PR4 lets it choose how full each table's pages end up, with a migrated setting that matches a node converted to v2.
Figure 8. Every store's five state tables. For the two storage tables the fill bar samples the top-level tree only, while the page count includes each contract's own sub-tree. PR7 brings StoragesTrie's bar up to jochemnet's, and value-transfer tests read about 1.8k fewer of its pages.
5. The wrong storage shape, and too many accounts
Figure 9. Slots per storage-holding account. jochemnet: median 1, 55% with exactly one, 97% with at most 28. The PR1-4 store (a 5,000-account sample): median 252. The final store traces jochemnet's curve, tail included: 7 accounts above 16.8M slots, jochemnet 5, each counting the big test contract.
state-actor gave every contract a few hundred slots. Mainnet storage is heavy-tailed: most contracts hold one slot, a few hold millions.
The shape matters because reth stores no storage roots:
A contract with up to about 28 slots keeps them in its own row. A bigger one gets a separate sub-tree.
When a block changes an account, reth re-hashes its path in the account trie, which needs the storage root of every sibling on that path.
For a mainnet-shaped sibling that's one row reth often reads anyway. For one of state-actor's contracts it was a walk through a sub-tree.
Figure 10. One modified account, its siblings, and what reth reads to re-hash them. The sub-tree walks are the pages the tracer labels 'other'.Figure 11. The same traced tests on jochemnet, the PR1-4, PR1-5 and PR1-8 stores, by table. The pink 'other' segment is the sub-tree walk: on the transfer tests, 0.8k pages on jochemnet, 11.6k on PR1-4, 1.1k after PR5, 1.0k on PR1-8.
PR5 draws each contract's size from the distribution measured on jochemnet. We also generated the store at jochemnet's account count (the original had 21% more) and matched the fill of the one big test contract. The new state needed new fixtures, which reth itself filled, and reth checked every trie node of the new store: nothing to repair. PR6 draws the largest contracts too, with no cap, so they grow with the store.
Then: accounts that look like mainnet's
Figure 12. jochemnet, PR1-5 and PR1-8. 87% of PR1-5's balances took 9 bytes, where 50% of jochemnet's are zero, and its median nonce was 497, two bytes on 74% of accounts. Pages are the mean over the 6 traced account-read tests. Account reads ran at 0.970 on PR1-5 and 0.976 on PR1-8.
state-actor's accounts didn't look like mainnet's: big balances and nonces, and almost no contracts with code but no storage. PR8 fixes that, but account reads barely get faster.
A block that reads accounts touches one row page per account it reads, however small the row, so smaller rows don't help. What still differs is the index above the rows: jochemnet's index pages are fuller, so state-actor's index has about 16% more pages.
PR7 writes each contract's trie rows in address order, so the top level of the contract-trie table now fills like jochemnet's (Figure 8, StoragesTrie).
Where it stands
Figure 13. Share of the 154 tests that run half a second or more landing within ±10% of jochemnet, store by store, all against v2 jochemnet under direct-io. PR6-8 refine the store: they fix no cause of the 64×, and the share holds.Figure 14. Every family, store by store. The right column shows original → PR1-8 and how many of the family's tests PR1-8 leaves in band.
Over a second, 113 and 113 of 114 tests are in band on the two pairs; each store against itself manages all 114. state-actor leans 1.4% fast there, under the 3% we set out not to chase.
What's left
sload, bloated contract, 240M. One test stays just outside the band, on the fast side. It reads only the big test contract, whose rows are identical in every store. In PR1-5 it was sstore, no-op write, 160M instead, now back in band. Which test sits just outside changes between regenerations.
Distinct code. Mainnet holds far more of it (Figure 8, Bytecodes), so code-heavy reads still run 2% fast. That's the generator's code pool, not the layout.
The account index. Account reads still run 2% slow, and the gap is index pages (Figure 12). The generator's fill control fills index pages like row pages; jochemnet's index pages are fuller.
HashedStorages' top-level tree takes 11% more pages per contract than jochemnet's at any fill setting. That density is the snapshot's years of history.
Short tests. Under a second, jochemnet run twice agrees with itself on only 69% of tests, so below half a second we don't score.
Four of the six causes change nothing a state root can see, and none was in reth. A comparison between a generated store and a real one should state how each was laid out, not just what it holds.