feat(examples): pluggable grc20 ledger storage + p/nt/hashmap (flat gas for large ledgers)#5965
feat(examples): pluggable grc20 ledger storage + p/nt/hashmap (flat gas for large ledgers)#5965omarsy wants to merge 14 commits into
Conversation
Add gno.land/p/nt/hashmap/v0, a persistent string-keyed map with O(1) persisted-object loads per operation, and a KV storage-backend seam on p/demo/tokens/grc20 (a `WithStorage` option on `NewToken`) so token ledgers can opt into flat storage-access gas as they grow. The default backend stays avl.Tree, so existing tokens are unchanged and grc20 does not import hashmap. Addresses the state-scaling half of gnolang#5906: a cold GRC20 transfer grows from ~6M gas at 1 holder to ~18M at ~1M holders because avl.Tree loads O(log N) objects, each 59k gas cold. Measured cold, hashmap keeps a transfer flat (~-69% vs avl at 1M); the in-tree bptree, via the same seam, preserves ordered iteration at ~-37%. avl at 1M reproduces the real on-chain 18.3M number. Details and full 1..1M measurements in gno.land/adr/prxxxx_hashmap_ledger.md.
🛠 PR Checks SummaryAll Automated Checks passed. ✅ Manual Checks (for Reviewers):
Read More🤖 This bot helps streamline PR reviews by verifying automated checks and providing guidance for contributors and reviewers. ✅ Automated Checks (for Contributors):🟢 Maintainers must be able to edit this pull request (more info) ☑️ Contributor Actions:
☑️ Reviewer Actions:
📚 Resources:Debug
|
…ter merge Master moved avl.Tree.Get (and bptree) to return a single any (nil = absent). Align KV.Get and hashmap.Map.Get to match, so *avl.Tree/*hashmap.Map/*bptree all satisfy KV and grc20's balanceOf/allowance nil-checks compile.
grc20's interface-typed ledger fields grow a token's persisted footprint; gov/dao/v3's treasury imports grc20, so the genesis deposit test1 pays rises ~1.6M ugnot, shifting the two exact-balance assertions. Update to new values.
grc20's larger package (KV seam) raises the gas to deploy/run realms that import it. Two tight-budget txtars hit out-of-gas: bump gas-wanted for the mitm addpkg (wugnot) and change-law proposal run (govdao).
Measured persisted bytes / deposit for avl vs hashmap vs bptree at deploy-base and 20k/100k/1M holders. hashmap is -93% storage vs avl at 100k holders (~144 vs ~2090 bytes/holder), so the backend swap cuts both per-tx gas and storage deposit.
…NV-1a bucketIndex now derives the bucket from crypto/sha256.Sum256 (low 64 bits) rather than an FNV-1a loop written in Gno. Sum256 is a calibrated native binding, so bucket selection no longer runs a per-byte hash loop in the interpreter on every Get/Set/Has/Remove. Measured cold, end-to-end grc20 transfer: ~-0.5M gas, flat from 20k to 1M holders. The interpreted-hash removal saves -0.75M in the VM term, offset by +4 fixed cold reads (+0.24M) for the sha256 package's first-use load — net -0.5M. (A compute-only measurement shows ~-0.9M by omitting that package load.) Ships in this PR, not a follow-up: bucket placement is a function of the digest, so changing the hash after any map is persisted relocates every key.
Store buckets in a two-level directory (√B pages of √B buckets, lazily allocated) rather than one flat B-slot array. A flat array is a single object that every op decodes in full (~190KB at 1024 slots, fixed at any entry count); splitting it means an op decodes only the directory + one page + one bucket, each a √B-slot array. This also makes a high bucket count cheap, so the default rises to 4096 and each leaf bucket stays small as the ledger grows. New()/NewWithBuckets(count) are unchanged in signature; count is now split evenly across the two levels. Placement is fixed (digest + split), so this ships before any map is persisted. Measured cold, end-to-end 1M-holder grc20 transfer: 8.58M -> 6.59M (-1.99M, ~-23% vs the flat design; ~-65% vs today's avl). Nearly flat in N (6.17M at 20k -> 6.59M at 1M). Read categorization: the bucket ArrayValue decode drops 571k -> 118k and the leaf MapValue decode 845k -> 212k; the rest is alloc gas on the same shrunken objects. Unit tests and the full grc20 flow pass.
Independent review found no code defects; the actionable findings were all stale documentation left from before the SHA-256 + two-level rewrite: - ADR Consequences said FNV-1a / default 1024 / "over-sizing counter-productive" — the shipped design is SHA-256, default 4096, and two-level reverses the over-sizing finding. - ADR headline "Measured results" table showed pre-two-level numbers (77 obj / 7.2-7.3M); updated to the shipped 85 / 6.17-6.59M. - Renamed a stale `bucketIndex` reference to `locate`. - Package doc: "√B × √B" is exact only for even exponents (softened to ≈√B and documented the 2:1 split for odd exponents); noted iteration order is preserved across reload and that the persisted image is insertion-order- dependent, not canonical like avl.Tree. Comment/markdown only — no code change.
The ADR's storage-footprint section still showed the flat-1024 numbers. Measured the shipped two-level design (rlm.Storage, same harness): - deploy base 170KB -> 23KB: pages/buckets are allocated lazily, not pre- allocated, so an empty ledger is nearly free. - at scale: 20k 4.42M / 100k 15.98M / 1M 145.6M bytes. Marginal ~144 bytes/ holder (same as a flat map); the 4096-bucket object overhead amortizes by 1M. - vs avl: -89% (20k) / -92% (100k) / -93% (1M); deposit at 100k ~1,598 GNOT vs avl ~20,863 GNOT. Also tightened the NewWithBuckets doc on the bit-split. Comment/markdown only.
…ound A second independent review pass (0 code defects again) caught internal contradictions the first pass missed: - ADR "~200-object package/code floor" was impossible against the ~85-object read floor (conflated total decodes incl. cached stdlib with paying reads) — corrected to "most of those ~85 reads". - "subtract ~5.5M for a full token transfer" had the wrong sign and magnitude; a full transfer ADDS ~0.5-1M of grc20/package overhead over the KV-isolated numbers. - Annotated the intro table's object counts as the initial-investigation harness (avl@20k 182 there vs 191 in the refined tables). - "-0.9M" was mislabeled as the VM term (it's compute-only); "splits n evenly" softened (odd exponents give the directory the extra bit). - Package doc: disambiguated the page/bucket bit-split wording, and dropped an overstated "avl.Tree image is canonical" claim (AVL shape is also history- dependent) while keeping the accurate "not key-sorted" contrast. Comment/markdown only — no code change.
Third (convergence) review round passed all hard checks; these close the two cleanest residual nits: - a leaf bucket is a native map (~N/B entries), not a "√B-slot array" — only the directory and page are; reworded so the √B-slot characterization applies to the two arrays, with the bucket called out separately. - clarified the SHA-256 net saving: −0.5M = −0.75M VM term less the +0.24M package cold-load; the "−0.9M" figure is a higher compute-only estimate. Comment/markdown only — no code change.
…rcount) The original ADR simulated production gas as VM_gas + reads*59k + writes*24k — only 2 of the store's 7 gas dimensions and no depth-gas multiplier — undercounting by ~2x. Re-measured by running the cold transfer through the real cache-wrapped, gas-metered store (DefaultGasConfig + DefaultParams depth params, as app.go configures) and reading GasConsumed(): - cold avl 27.9M (20k) / 29.8M (100k) / 32.0M (200k) / ~35M (1M extrap) - cold two-level hashmap 16.2M / 16.5M / 18.8M -> -42/-45/-46% vs avl (was -65%) - a fixed ~10.7M package-code depth floor dominates every cold transfer (all backends) -> package-code cold-load is now the biggest remaining lever - depth/flat/per-byte are cache-miss-only, so warm transfers (txtar) ~7M, cold (congestion/GC) ~16-28M; on-chain test13 18.3M is a modest-ledger cold number, NOT avl@1M (that "18M@1M" was an undercount artifact) - SHA-256 re-measured: -0.74M vs FNV-1a (clean; crypto/sha256 is stdlib-cached, no package penalty) Independently corroborated by a maintainer's gnodev reproduction. Code and storage-footprint numbers are unaffected; the relative hashmap-vs-avl win holds.
|
@notJoon, the gnodev run caught a real issue: my harness was undercounting by ~2×. It simulated only 2 of the store's 7 gas dimensions and skipped the depth-gas multiplier on cold IAVL reads. Re-measured through the real cache-wrapped, gas-metered store ( Corrected cold transfer:
Itemizing that also surfaced why even hashmap floors at ~16M cold: a fixed ~10.7M IAVL cost, of which ~7.67M (half a cold tx) is the VM re-reading its 13 governance params as 13 separate depth-multiplied keys — backend-independent, on top of the ledger reads. Bundling those into one KV read is a measured flat −7.08M/tx; orthogonal to this PR, so I'll open it separately. Direction and fix are unchanged — only the magnitudes (~2× low) and the old "18M @ 1M" anchor were wrong. |
|
One more correction. I re-measured on current Corrected cold transfer (real cache-store metering,
Two things flip vs my earlier post:
Net: the thesis holds and the fix is stronger than even the corrected numbers suggested. I'd just been measuring on a base that predated the fast-index. |
Summary
Adds a
KVstorage-backend seam top/demo/tokens/grc20and a newp/nt/hashmap/v0package so token ledgers can opt into storage-access gasthat stays flat as the ledger grows, instead of growing with holder count.
This addresses the state-scaling half of #5906. The default backend is
unchanged (
avl.Tree), so this is fully backwards-compatible and opt-in.Why
A cold GRC20 transfer costs ~6M gas at 1 holder but ~18M at ~1,000,000
holders. The cause is the data structure, not the contract:
avl.Treematerializes its search path as one persisted object per node, and the base
store charges a flat ~59k gas per cold object read, so a transfer loads
O(log N)objects. Measured cold (fresh keeper, empty object cache — the statea validator is in after cache GC), same transfer op:
avlat 1M reproduces the real on-chaintest_atonetransfer (18.3M) —independent confirmation of the mechanism.
hashmap(a native-map bucket array; each bucket is a single persistedobject regardless of entry count) stays flat: O(1) object loads per op.
bptree(already in-tree) rides the same seam and keeps orderediteration, which hashmap trades away.
Full analysis and 1..1M measurements are in the ADR.
Changes
New package
examples/gno.land/p/nt/hashmap/v0/—hashmap.gno(Map: Get/Set/Remove/Has/Size/Iterate, FNV-1a bucketing, fixed power-of-two bucket count, sizing guidance),hashmap_test.gno,gnomod.toml.grc20 seam (backwards-compatible, default avl)
examples/gno.land/p/demo/tokens/grc20/types.gno— addKVinterface (satisfied by*avl.Tree,*hashmap.Map,*bptree.BPTree),Option,WithStorage; ledger fields becomeKV.examples/gno.land/p/demo/tokens/grc20/token.gno—NewTokengains variadicopts ...Option; ledger built from the chosen backend (defaultavl.NewTree).examples/gno.land/p/demo/tokens/grc20/storage_option_test.gno— full token flow over all three backends through the one seam.ADR
gno.land/adr/prxxxx_hashmap_ledger.md— context, decision, measured results (incl. the 1M scaling and the bptree comparison), alternatives (native VM storage, block-level warm/cold gas pricing), consequences.Notes / open discussion
is a different workload; block-level warm/cold gas pricing would help it
and every contract automatically — flagged in the ADR as a separate,
protocol-level proposal for maintainers to weigh.
entries, and over-sizing (4096) is counter-productive (every op decodes the
whole bucket-pointer array). Documented in the package.
documented in the package and ADR.
AI-assisted (investigation, implementation, and the cold-cache measurements);
reviewed and owned by the author.
🤖 Generated with Claude Code