[experiment] execution/stagedsync: let Normalize read through the tx's own read set

[AskAlexSharov]

Experiment on top of #23027. The question: Normalize runs on the apply loop and reads state the worker already read. Can it take the tx's own read set instead of going back to the domain?

Why it should be possible

The worker cannot credit an account without loading it, and stateObject.SetState performs a versioned read of the previous value before every SSTORE (EIP-2200 gas depends on it). Those reads are recorded per-tx as VersionedRead{ReadHeader, Val}. The write set deliberately does not carry the base — VersionedWrite is WriteHeader + Val — so the base survives only in the read set, which Normalize was never given.

versionedStateReader already layers read set -> versionMap -> domain and is handed to the finalize IBS two functions away. This branch hands it to Normalize too. That is the whole change:

normReader := stateReader
if stateReader != nil { // nil means "no reader"; Normalize guards on it
    normReader = state.NewVersionedStateReader(txVersion.TxIndex, be.blockIO.ReadSet(txVersion.TxIndex), be.versionMap, stateReader)
}

Result

Green locally: execution/state, execution/stagedsync, execution/tests, and go test -count=2 ./rpc/jsonrpc.

Getting there took one wrong turn worth recording, because it is a trap for anyone repeating this. The branch originally wired both Normalize call sites through a CheckedStateReader that answered from the read set and read the domain on every call, panicking on disagreement. That failed widely — 6 rpc/jsonrpc tests, EEST spec tests in CI — while never firing the assert. Three configurations separate the cause:

configuration	go test -count=2 ./rpc/jsonrpc
wrapper, identical readers on both sides	ok
wrapper, versioned + domain	fail
versioned reader alone	ok

The probe was the problem. The domain reader here is NewCurrentCachedReaderV3(..., blockStateCache), and reading through it fills the block state cache; the extra fills perturb it. #22444 bound StateCache fills to transaction views and made stale fills rejectable, so the cache is deliberately sensitive to when fills happen. Unproven as a mechanism, but it is the only hypothesis consistent with all three rows, and it explains why no assert ever fired: there was no read disagreement to find.

checked_reader.go is kept in the branch as documentation of the probe, deliberately not wired in. Anything that double-reads through the cached reader cannot be trusted as a measurement here.

What is not claimed

• No performance number. The point was correctness; Normalize's domain reads were 19% (ReadAccountData) and 13% (ReadAccountCode) of its time in the n5 profile, but this branch has not been benchmarked.
• The one divergence the probe did observe before being abandoned, from TestSelfDestructReceive: ReadAccountStorage returning {0, found:true} from the read set where the domain says {0, found:false}. The no-op filter is provably insensitive to it — its two arms agree whenever the value is zero — but it is the same presence-vs-value class that execution/state: serve warm storage reads from the read set #22993 found load-bearing one layer down, and it is the thing to watch in wider testing.
• This does nothing for the expensive read. domainStorageKeys -> IteratePrefix is bound to applyTx and SharedDomains.mem, cannot come from a worker, and is what produced a 513ms Normalize on a 26-self-destruct tx.

[AskAlexSharov]

Status: experiment, not for merge

Goal is a pure nextResult — an apply loop that reads no DB at all. Worth it beyond any microbenchmark delta: it becomes unit-testable without a domain, deterministic, and immune to disk/RAM variance, which is exactly where a serial apply loop hurts on a cold archive node.

Where it stands

The substitution is correct. NORMALIZE_ASSERT_READS=true runs Normalize twice — once through the read-set-backed reader, once through the domain — and compares the two write sets header-by-header:

suite	comparisons	divergences
rpc/jsonrpc	21,730	0
execution/tests	825	0

That covers the self-destruct / EIP-161 / CREATE2 family, which is where this could plausibly go wrong.

But the current lookup barely fires

Measured read-set hit rate for Normalize's ReadAccountData:

suite	hits	domain	hit rate
execution/tests	51	789	6.1%
rpc/jsonrpc	3,469	34,727	9.1%

Structural, and the code says why:

The BAL pre-population writes only BalancePath/NoncePath/CodePath/StoragePath entries (NOT AddressPath, by design — see validateRead invariant)

The fill loop wants a whole account record and asks reads.GetAddress(...); the read set stores reads per path. The worker's data is there, just not in the shape this lookup wants. So the next step is to resolve the account from GetBalance/GetNonce/GetCodeHash/GetIncarnation rather than GetAddress.

Remaining DB reads to eliminate

1. ReadAccountData (fill loop) — per-path lookup, above.
2. ReadAccountStorage (no-op filter fallback) — the worker already reads the origin before every SSTORE (stateObject.SetState → readStateForSet, required for EIP-2200 gas), so this should be recoverable from the read set.
3. ReadAccountCode (EIP-7702 recovery) — rare.
4. domainStorageKeys → sd.IteratePrefix — not recoverable from any read; nothing enumerates an account's slots during execution. Bound to applyTx + SharedDomains.mem, and it produced a 513ms Normalize on a 26-self-destruct tx. Needs splitting into an invariant files part (enumerable in a worker) plus a small apply-time sd.mem delta.

Notes for anyone reading the history

Three failures on this branch were the instrument, not the hypothesis:

• Wrapping a nil stateReader — Normalize guards on nil, so the wrapper made it take the reader branch and dereference nil. Cause of the original opaque apply loop exited… failures.
• An over-strict predicate — {0, found:true} vs {0, found:false} is unobservable to the no-op filter, whose two arms agree whenever the value is zero.
• Read-level cross-checking is not observation. CachedReaderV3.ReadAccountData calls PutCommittedAccount on a miss, so reading the domain to compare seeds the committed cache with pre-block values and fails blocks. Bisected to ReadAccountData alone. The assert is therefore at output level, comparing write sets, which is neutral.

[AskAlexSharov]

Dig 2: per-path resolution — correct, and worth exactly nothing

Followed the obvious lead from the last comment: the read set stores reads per path, while the fill loop asked reads.GetAddress(...) for a whole record and hit 6-9%. So the fill loop now resolves each missing field from the matching per-path read (GetBalance/GetNonce/GetCodeHash/GetIncarnation) before falling back to the domain, via an optional accountFieldResolver interface — no signature change, no churn in Normalize's callers.

It is correct. With NORMALIZE_ASSERT_READS=true, ~22,500 Normalize outputs still match the domain reader exactly, across rpc/jsonrpc and execution/tests.

It removes no DB reads at all:

	resolved from read set	domain account reads
execution/tests, resolution off	0	1283
execution/tests, resolution on	102	1283
rpc/jsonrpc, resolution off	0	95799
rpc/jsonrpc, resolution on	17795	95799

17,795 fields served from memory, and not one domain read avoided.

Why

The domain read is hoisted per address (fallbackAcc/fallbackLoaded, from #22409) while resolution is per path. One ReadAccountData covers all four fields, so it is skipped only if every missing path resolves without it. Measured coverage of accounts that reach the domain:

	execution/tests	rpc/jsonrpc
any per-path entry	60.5%	74.3%
all four	0.0%	0.0%
balance / nonce / codeHash / incarnation	60 / 8 / 8 / 4%	74 / 45 / 45 / 27%

All-four-present is zero in 35,516 samples. Partial coverage cannot beat a per-address hoist, so this direction is capped at zero by construction.

What that leaves

A pure fill loop needs the whole account record, not more fields. The worker already loads it — ReadAccountData returns all four — and ReadSet.SetAddress exists to record exactly that (read_paths.go:790, intra_block_state.go:2908). It just isn't recorded on the paths that matter, deliberately:

The BAL pre-population writes only BalancePath/NoncePath/CodePath/StoragePath entries (NOT AddressPath, by design — see validateRead invariant)

So the next question is not "how do we squeeze more out of the read set" but "can an AddressPath read be recorded for accounts the worker fully loaded, without breaking the validateRead invariant". That is the only route to zero domain reads in the fill loop, and it is a conflict-detection question, not a performance one.

The per-path resolver is kept on the branch: it is correct, it is free when it hits, and it becomes load-bearing the moment the per-address hoist is reconsidered. But on its own it is a null result, recorded so nobody re-runs it.

[AskAlexSharov]

Dig 3: the read set already knows, and the answer is "nothing there"

The last comment asked whether an AddressPath read could be recorded for accounts the worker fully loaded. Wrong question — it already is. getStateObject ends in accountRead, which does SetAddress(addr, {Source, Version, Val: NewAccountView(&data)}). The "NOT AddressPath, by design" invariant governs BAL write pre-population in the VersionMap; it says nothing about read recording.

So the low hit rate had to come from somewhere else. Classifying every fill-loop account fallback by what the tx's read set held for that address:

read set held	execution/tests	rpc/jsonrpc
AddressPath entry with an account	115 (19.7%)	19235 (47.7%)
AddressPath entry, header-only (Val=nil)	442 (75.7%)	10176 (25.2%)
no AddressPath entry, other paths read	0	0
nothing for this address	27 (4.6%)	10921 (27.1%)

The dominant bucket is the header-only entry, and it is not a gap in the recording — it is an answer. readAccountInternal records the header before the load and accountRead overwrites it with the value the moment a load finds an account, so an entry that stays header-only means the load came back empty. ReadAccountData was throwing that away and re-deriving it from the versionMap and the domain.

The implication holds on every sample. Counting what the read returned for that bucket:

suite	header-only entries	read returned an account	read returned nil
execution/tests	442	0	442
rpc/jsonrpc	10176	0	10176

10618 for 10618, including the self-destruct / EIP-161 / CREATE2 family in execution/tests.

Change

One condition in versionedStateReader.ReadAccountData: don't consult the domain when the tx recorded an AddressPath read for this address. The versionMap steps ahead of it (AccountLifecycle, versionedUpdateAddress) and the BAL synthesis behind it are untouched, so the fall-through is exactly what a nil domain answer produces today.

Result

Apply-loop domain account reads through the versioned reader, rpc/jsonrpc, whole suite:

	reads
NORMALIZE_SKIP_ABSENT=false (before)	18961
NORMALIZE_SKIP_ABSENT=true (after)	8926

−52.9%. Unlike dig 2, this one is not capped at zero by construction: the per-address hoist that made per-path resolution worthless is exactly what makes this work, because one skipped read covers all four missing fields.

Green with NORMALIZE_ASSERT_READS=true (both Normalize outputs compared header-by-header) on execution/tests, rpc/jsonrpc, execution/state, execution/stagedsync.

What is not claimed

• Still no e2e number. NORMALIZE_SKIP_ABSENT is kept so this can be A/B'd on a node without a rebuild.
• The residual noAddrCold bucket is 27.1% of fallbacks in rpc/jsonrpc, but 9501 of its 10921 are answered by the versionMap, not the domain. The tail that genuinely reaches disk is ~1400.
• One reachable case where the skip and the domain could disagree, not observed in 10618 samples: an account self-destructed by an earlier tx and revived by a balance-only credit writes no AddressPath cell, so AccountLifecycle reports revived, versionedUpdateAddress finds nothing, and the old path would return the pre-SD account where this one falls through to BAL synthesis. Normalize's own sdEarlier/sdSet handling covers the fill loop; the finalize IBS reader shares this code and does not.
• domainStorageKeys -> IteratePrefix is still the expensive read and is still not recoverable from any read set.

[AskAlexSharov]

Dig 4: the remaining reads are not cold, they are the same 22 addresses

Two questions settled, and the second one moves the target.

Storage was already solved

Item 2 on the remaining-reads list (ReadAccountStorage, the no-op filter fallback) needs no work. versionedStateReader.ReadAccountStorage returns on any recorded entry — VersionedRead[uint256.Int] has a value-typed Val, so there is no header-only shape to get wrong, unlike the AddressPath case dig 3 found.

Measured on the no-op filter's fallback arm, rpc/jsonrpc:

	count
fallbacks	7789
of those, on the versioned reader	6795
slot found in the tx's read set	6795 (100%)
address recorded but not this slot	0
address not recorded at all	0
reached the domain	48

The worker's pre-SSTORE origin read covers the filter completely. Worth noting what the filter buys for those 7789 reads: 20 writes dropped, 7769 kept.

Items 1 and 3 are also closed. ReadAccountCode and ReadAccountCodeSize carry the same ok && r.Val != nil shape that dig 3 fixed on ReadAccountData — a recorded read with an empty value falls through and re-reads — and so does ReadAccountIncarnation on the AddressPath entry. All three measured 0 domain reads across both suites, so the shape is latent, not costly. Left alone.

Where the remaining reads actually are

Attributing every domain account read through the versioned reader to its call site, rpc/jsonrpc, 8925 reads:

site	reads	distinct addresses
calcFees:1938	6264	4
nextResult:2901 (Normalize fill loop)	1420	3
finalizeSystemTx:1915	1242	15

22 addresses, 8925 reads. The apply loop is not reading cold state — it is re-reading the coinbase, the burnt-fee contract and a handful of system contracts once per transaction.

That closes the read-set line of attack for these. I wired calcFees to result.TxIn to check: no change, 6264 both ways. The worker never reads the coinbase — shouldDelayFeeCalc means the credit is not applied during execution — so there is nothing recorded to reuse. Not a gap in the read set; the read genuinely never happened.

finalizeSystemTx is deliberate: its empty read set is documented as a staleness guard for syscalls that run speculatively.

What this changes

"Give the apply loop the worker's reads" is finished as a direction. Dig 3 took the fill loop's account reads from 18961 to 8926; the rest is a different problem with a different fix — the base account for these 22 addresses is a pre-block value, and applyVersionedUpdates already overlays the in-block deltas from the versionMap on top of it. Reading that base once per block instead of once per transaction is the same substitution one level up.

The open question is invalidation: whether the domain answer for those addresses can change during a block's apply loop, given that applied writes reach SharedDomains as the loop advances. That is what to settle next, and it cannot be settled by double-reading to compare — the cached apply-side reader fills on a miss, so a second read is a mutation, not an observation.

domainStorageKeys -> IteratePrefix is untouched and still the expensive one.

[AskAlexSharov]

Dig 5: chasing the last reads found the cost was never the read

Dig 4 ended on a question — can the base account for those 22 addresses be read once per block instead of once per transaction? The answer is no, twice over:

• pe.rs.ApplyStateWrites lands each published transaction in sd.mem as the loop advances, so the domain moves under the block.
• More decisively, the apply-loop reader is NewCurrentCachedReaderV3, which sets readCurrent=true precisely so the read sees that write buffer. A per-block base is not a missing optimisation, it is the wrong answer.

So the per-transaction read is structural. What is not structural is what a hit costs.

CachedReaderV3.ReadAccountData asks GetCurrentAccount for a blob and decodes it. That helper checks the write buffer, then falls back to committedAccounts — which holds accounts already decoded — and calls SerialiseV3 on one so the caller can DeserialiseV3 it back. The readCurrent=false branch four lines below already takes it decoded.

Measured over rpc/jsonrpc, which of the two paths the apply loop actually takes:

path	count
write buffer (blob is genuine)	1765
committed (re-encoded, then decoded)	7933
not cached, real domain read	17668

The re-encode is the common one, 4.5x over the path the blob format exists for. The apply loop mostly reads addresses no transaction has written yet this block.

benchstat, n=6:

	before	after
committed, sec/op	69.24n	23.81n	-65.6%
committed, B/op	144	96	-33.3%
committed, allocs/op	2	1	-50%
write buffer, sec/op	39.91n	41.48n	~ (p=0.818)

Split out as #23075, off main, since it owes nothing to this branch's wiring.

One thing worth recording for anyone touching that function: the lookup is open-coded rather than extracted into a helper because a helper does not inline — the map[accounts.Address][]byte access alone costs 173 against a budget of 80 — and the call showed up as a reproducible +4.8% on the write-buffer path. Extracting it is the natural-looking change and it is the wrong one.

Where the read-set thesis stands

Finished, and successfully. Dig 3 took the fill loop's account reads from 18961 to 8926. Dig 4 showed storage was already covered (6795/6795) and that ReadAccountCode / ReadAccountCodeSize / ReadAccountIncarnation carry the same latent shape at zero measured cost. The residual is 22 addresses re-read per transaction by design.

domainStorageKeys -> IteratePrefix remains untouched and is now the only item on the original list with real cost behind it.