stakes: disk backed delegations

[ibhatt-jumptrading]

No description provided.

[github-actions]

 ┌─ ⚡ PERF · 9a600ef vs main@aa3206d ─────────────────────────────────
 │ SUITE                               BASELINE          NEW         Δ
 │ replay tps, mainnet               28,281 tps   28,420 tps  ·  +0.49%
 │ bench tps, localnet              128,646 tps  128,644 tps  ·   0.00%
 │ snapshot load, testnet               15.15 s      15.27 s  ·  +0.78%
+│ mem total, mainnet                171.44 GiB   166.92 GiB  ▼  -2.64%
+│ mem total, testnet                102.31 GiB    97.79 GiB  ▼  -4.42%
 │ clean compile, firedancer        343.4 cpu·s  344.3 cpu·s  ·  +0.24%
 │ binary size, firedancer             85.87 MB     85.93 MB  ·  +0.07%
 ├─────────────────────────────────────────────────────────────────────
@@ 0 REGRESSIONS · 0 WARNINGS · 2 IMPROVED · 5 NOISE @@
 └─────────────────────────────────────────────────────────────────────

history · 11 pushes

 ┌─ HISTORY · Δ vs main, per push, newest first ─────────────────────────
 │ HEAD         TPS    BENCH     SNAP    MEM·M    MEM·T  COMPILE   BINARY
+│ 9a600ef   +0.49%    0.00%   +0.78%   -2.64%   -4.42%   +0.24%   +0.07%
+│ ba165c8   +0.26%    0.00%   -0.66%   -2.64%   -4.42%   +0.52%   +0.07%
+│ b6ca5ba   +0.05%    0.00%   +0.20%   -2.64%   -4.42%   +0.17%   +0.07%
+│ 5a9c16b   +0.83%    0.00%   +0.56%   -2.64%   -4.42%   -1.16%   +0.07%
+│ f5709b6   +0.14%    0.00%   -1.68%   -2.33%   -3.91%   +1.24%   +0.12%
+│ c560332   +0.34%   -0.01%   +1.53%   -2.33%   -3.91%   +0.35%   +0.15%
+│ 787e305   +0.22%    0.00%   +0.73%   -2.33%   -3.91%   +0.17%   +0.04%
!│ 43f1c61   -0.52%    0.00%   +1.02%   -2.33%   -3.91%   -5.54%   +0.04%
+│ fffd9f8   -0.12%    0.00%   +1.54%   -2.33%   -3.91%   +0.50%   +0.04%
+│ b75b7c1   +0.25%    0.00%   +0.76%   -2.33%   -3.91%   -7.74%   +0.04%
+│ dc23109   +0.67%    0.00%   -0.06%   -2.33%   -3.91%   -2.63%   +0.04%
 └───────────────────────────────────────────────────────────────────────

[Copilot]

🔵 Needs a closer look

The change introduces shared disk-backed state and seccomp modifications across several critical validator tiles.

Pull request overview

Adds a disk-backed overflow tier for stake-delegation pubkey fallback entries, reducing locked RAM usage while preserving capacity.

Changes:

• Implements an open-addressed spill file with generation-based reset, refresh, migration, and iterator support.
• Creates and distributes a well-known spill file descriptor to stake-mutating tiles.
• Extends tile seccomp policies and tests for spill-file pread64/pwrite64.

File summaries

File	Description
src/flamenco/stakes/test_stake_delegations.c	Tests spill, reset reuse, and refresh behavior.
src/flamenco/stakes/fd_stake_delegations.h	Defines disk-tier layout, limits, and iterator state.
src/flamenco/stakes/fd_stake_delegations.c	Implements disk-backed pubkey storage and traversal.
src/discof/restore/generated/fd_snapin_tile_seccomp.h	Adds generated snapin spill-file syscall rules.
src/discof/restore/fd_snapin_tile.seccomppolicy	Allows snapin spill-file I/O.
src/discof/restore/fd_snapin_tile.c	Exposes the spill FD to snapin.
src/discof/replay/generated/fd_replay_tile_seccomp.h	Adds generated replay spill-file syscall rules.
src/discof/replay/fd_replay_tile.seccomppolicy	Allows replay spill-file I/O.
src/discof/replay/fd_replay_tile.c	Exposes the spill FD to replay.
src/discof/execrp/test_execrp_tile.c	Updates execrp allowed-FD assertions.
src/discof/execrp/generated/fd_execrp_tile_seccomp.h	Adds generated execrp spill-file syscall rules.
src/discof/execrp/fd_execrp_tile.seccomppolicy	Allows execrp spill-file I/O.
src/discof/execrp/fd_execrp_tile.c	Exposes the spill FD to execrp.
src/discof/execle/test_execle_tile.c	Updates execle allowed-FD assertions.
src/discof/execle/generated/fd_execle_tile_seccomp.h	Adds generated execle spill-file syscall rules.
src/discof/execle/fd_execle_tile.seccomppolicy	Allows execle spill-file I/O.
src/discof/execle/fd_execle_tile.c	Exposes the spill FD to execle.
src/app/shared/commands/run/run.c	Creates, distributes, and closes the spill file.

Review details

• Files reviewed: 14/18 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Full-capacity disk scans create a severe performance risk, and key disk iteration paths remain untested.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 20/24 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Full-capacity scans create severe fallback latency, and zero-byte writes can hang validator tiles.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/flamenco/stakes/fd_stake_delegations.c:459

• With production limits, this allocates 268,435,456 logical slots (about 12 GiB). Once a single key spills, both the fallback iterator and refresh scan disk_slot_cnt_ in full, causing roughly 524k pread calls and 12 GiB of reads per pass. This can stall epoch/reward processing precisely when fallback is needed. Please use a dynamically sized table or maintain a compact index/list of occupied slots so iteration scales with live spill entries rather than maximum capacity.

  stake_delegations->disk_slot_cnt_           = stake_delegations->disk_pubkey_cap_ ? fd_ulong_pow2_up( stake_delegations->disk_pubkey_cap_ + (stake_delegations->disk_pubkey_cap_>>1) ) : 0UL;

• Files reviewed: 20/24 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The fixed maximum-size bucket table makes the first spilled entry trigger roughly 12 GiB scans under the write lock.

Review details

Suppressed comments (1)

src/flamenco/stakes/fd_stake_delegations.c:458

• The bucket table is sized for the maximum fallback capacity, but both refresh and fallback iteration scan every slot. With the production limits (2.15M RAM-root capacity and 100M fallback capacity), this rounds to 268,435,456 slots, so a single spilled key makes each scan read about 12 GiB under the write lock. This turns the first overflow into hundreds of thousands of pread calls and can stall replay/rewards. Please make the on-disk table grow with occupancy or maintain a compact iterable list of live records so scan cost is proportional to live entries.

  stake_delegations->disk_slot_cnt_           = stake_delegations->disk_pubkey_cap_ ? fd_ulong_pow2_up( stake_delegations->disk_pubkey_cap_ + (stake_delegations->disk_pubkey_cap_>>1) ) : 0UL;

• Files reviewed: 25/29 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Disk scans scale with maximum configured capacity and can perform roughly 12.9 GB of I/O for one spilled entry.

Review details

Suppressed comments (2)

src/flamenco/stakes/fd_stake_delegations.c:1142

• The disk iterator scans disk_slot_cnt_, not the number of occupied slots. In the production configuration, the disk capacity is about 95.7M entries and line 458 rounds its table to 268,435,456 slots, so a single overflow entry makes every fallback iteration read roughly 12.9 GB (268,435,456 * 48) from the spill file. Epoch processing invokes this iterator multiple times, making the first overflow impose capacity-sized I/O regardless of occupancy. Please track occupied chunks/records or use an iterable on-disk layout so scan cost scales with live entries.

    if( FD_UNLIKELY( stake_delegations->disk_pubkey_used_ ) ) {
      while( batch_n<FD_STAKE_DELEGATIONS_ITER_BATCH && iter->disk_scan_idx<stake_delegations->disk_slot_cnt_ ) {
        ulong chunk_idx = iter->disk_scan_idx % FD_STAKE_DELEGATIONS_DISK_SCAN_CHUNK;
        if( FD_UNLIKELY( !chunk_idx ) ) {
          ulong chunk_cnt = fd_ulong_min( FD_STAKE_DELEGATIONS_DISK_SCAN_CHUNK, stake_delegations->disk_slot_cnt_-iter->disk_scan_idx );
          disk_read( iter->chunk, iter->disk_scan_idx*sizeof(disk_ref_t), chunk_cnt*sizeof(disk_ref_t) );

src/flamenco/stakes/fd_stake_delegations.c:773

• This refresh sweep also reads every bucket up to disk_slot_cnt_. With the production limits, that is roughly 12.9 GB of reads even when only one pubkey spilled, delaying snapshot boot/refresh in proportion to configured capacity rather than live disk entries. Please make the refresh enumerate occupied chunks or records instead of scanning the entire hash-table address space.

    ulong slot = 0UL;
    while( slot<stake_delegations->disk_slot_cnt_ ) {
      ulong chunk_cnt = fd_ulong_min( FD_STAKE_DELEGATIONS_DISK_SCAN_CHUNK, stake_delegations->disk_slot_cnt_-slot );
      disk_read( chunk, slot*sizeof(disk_ref_t), chunk_cnt*sizeof(disk_ref_t) );

• Files reviewed: 25/29 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Disk refresh and iteration perform repeated 12 GiB full-table scans under the global write lock.

Review details

Suppressed comments (2)

src/flamenco/stakes/fd_stake_delegations.c:773

• Once the disk tier is nonempty, refresh scans every bucket rather than the live records. With the production limits (100,000,000 fallback and 2,150,000 RAM accounts), disk_slot_cnt_ is 268,435,456, so this loop reads 12 GiB through 524,288 pread calls while holding the global write lock—even if only one entry spilled. This can stall epoch-boundary processing precisely when fallback is needed. Please add a compact live-slot index/bitmap or another disk layout that lets refresh visit occupied ranges only.

    ulong slot = 0UL;
    while( slot<stake_delegations->disk_slot_cnt_ ) {
      ulong chunk_cnt = fd_ulong_min( FD_STAKE_DELEGATIONS_DISK_SCAN_CHUNK, stake_delegations->disk_slot_cnt_-slot );
      disk_read( chunk, slot*sizeof(disk_ref_t), chunk_cnt*sizeof(disk_ref_t) );

src/flamenco/stakes/fd_stake_delegations.c:1142

• Every fallback iterator independently scans all 268,435,456 production buckets (12 GiB / 524,288 preads) whenever disk_pubkey_used_ is nonzero. Reward calculation invokes this iterator multiple times, so a single spilled entry adds repeated full-file scans under the stake-delegations write lock. Please make iteration proportional to live disk entries or occupied chunks rather than total configured capacity.

    if( FD_UNLIKELY( stake_delegations->disk_pubkey_used_ ) ) {
      while( batch_n<FD_STAKE_DELEGATIONS_ITER_BATCH && iter->disk_scan_idx<stake_delegations->disk_slot_cnt_ ) {
        ulong chunk_idx = iter->disk_scan_idx % FD_STAKE_DELEGATIONS_DISK_SCAN_CHUNK;
        if( FD_UNLIKELY( !chunk_idx ) ) {
          ulong chunk_cnt = fd_ulong_min( FD_STAKE_DELEGATIONS_DISK_SCAN_CHUNK, stake_delegations->disk_slot_cnt_-iter->disk_scan_idx );
          disk_read( iter->chunk, iter->disk_scan_idx*sizeof(disk_ref_t), chunk_cnt*sizeof(disk_ref_t) );

• Files reviewed: 25/29 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

It substantially changes consensus-critical stake storage, fork projection, disk I/O, and sandbox permissions while remaining marked WIP.

Review details

• Files reviewed: 35/42 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Per-record disk and account-database operations make configured large spill tiers operationally impractical.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 37/44 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The global spill-file layout can corrupt multiple live delegation instances, and disk refresh does not scale to the configured capacity.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 42/49 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Per-record disk reads make epoch-boundary iteration prohibitively expensive once significant spill occurs.

Review details

Suppressed comments (1)

src/flamenco/stakes/fd_stake_delegations.h:214

• Correct the spelling of “Management.”

• Files reviewed: 42/49 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

A fully occupied disk-root tier cannot project or apply newly inserted fork deltas without aborting.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 43/49 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

The cross-process disk-backed consensus state and seccomp changes are high-impact and still contain documentation contract inaccuracies.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/flamenco/runtime/tests/fd_svm_mini.h:68

• This field does not cap “each” disk tier: it caps disk deltas, while disk-root capacity is max_stake_accounts + 2 * max_disk_records. The current wording can cause tests to assume the wrong root limit.
  src/flamenco/stakes/fd_stake_delegations.h:64
• The concurrency note still names the removed fd_stake_delegations_{mark,unmark}_delta API and says the caller holds the lock, while the new frontier_query_begin acquires it internally. Update this note so callers can correctly understand the required begin/iterate/end locking contract.

• Files reviewed: 43/49 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

It changes consensus-critical stake storage, cross-process disk I/O, and multiple seccomp boundaries and is still marked WIP.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/flamenco/runtime/fd_runtime_const.h:35

• This comment says the constant bounds each disk tier, but fd_stake_delegations_new treats it as the delta-tier bound and derives the root-tier bound as max_stake_accounts + 2 * max_disk_records. Please state the actual semantics because this production value drives materially different disk capacities.
  src/flamenco/runtime/tests/fd_svm_mini.h:68
• This describes the limit as applying to each disk tier, but the implementation uses it only as the disk-delta capacity; the disk-root capacity is max_stake_accounts + 2 * max_disk_records. Please document that distinction so test configurations are not interpreted as imposing the wrong root bound.

• Files reviewed: 43/51 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

It changes consensus-critical stake storage, cross-process disk I/O, reward iteration, snapshot loading, and seccomp policy while still marked work in progress.

Review details

• Files reviewed: 45/53 changed files
• Comments generated: 0 new
• Review effort level: Balanced