feat(storage): append topology shards with persistent surrogate lookup

[DecisionNerd]

Parent

Native sub-issue of #901. This issue must block #901 and does not expand its acceptance criteria.

Objective

Make topology construction append-only in bounded immutable shards and provide authenticated disk-backed UUID-to-surrogate lookup without rescanning prior topology.

Requirements

• Flush bounded node, edge, and property shards without decoding or rewriting prior shards.
• Persist surrogate tails so writer open does not enumerate or decode all shards.
• Build/probe an authenticated UUID-to-node-surrogate index using bounded external runs.
• Make every ordinary catalog, adjacency, mutation, query, export, verification, and import reader union canonical shards.
• Expose aggregate work counters for input rows, prior rows decoded, rows encoded, shard count, and bytes.

Acceptance criteria

• Doubling input rows approximately doubles cumulative topology work and physical staged/output bytes, rather than quadrupling them.
• Peak writer state remains within the configured shard window.
• Writer open and endpoint resolution decode zero topology rows per construction batch.
• Duplicate/cross-kind identities and corrupt index/shards fail closed.
• Ordinary reopen, count, query, export, verify, and clean import operate on sharded graphs without a special reader.

Non-goals

The public resumable construction lifecycle and final Graph500 client switch; those are a sibling #901 sub-issue.

[DecisionNerd]

Root-cause audit update

Exact-head review of 6fb72c17f8ecfee58a669d6ba6e4171a375b7deb found four acceptance blockers; the issue is not ready to close:

1. UUID membership publication still rebuilds from every retained topology shard after each topology generation, making repeated bounded construction cumulative N + 2N + ... work and prior-row decoding.
2. Authenticated membership runs reject duplicate UUIDs but do not reject distinct UUIDs sharing one node surrogate.
3. Mutation incident-edge discovery and catalog tail recovery can skip corrupt canonical edge shards instead of failing closed.
4. Writer buffering has no enforced configured window or peak buffered rows/bytes evidence; the existing fresh-graph scaling comparison cannot prove a multi-batch plateau.

The root repair in progress is:

• immutable authenticated UUID/surrogate delta runs published with topology shards, bounded compaction/probing, and linear cumulative work counters;
• surrogate-keyed uniqueness/range validation;
• fatal propagation of canonical shard discovery/read corruption;
• an enforced writer window with peak buffered-state evidence and multi-batch plateau tests;
• an integrated normal-reader lifecycle covering reopen, count/query, export, verify, and clean import on sharded graphs.

No acceptance criterion is being waived. This issue remains the close gate for these outcomes.

[DecisionNerd]

S20 root-cause update (2026-08-24): the remaining ingest slowdown is architectural, not Fly and not the block-spill harness. The current append prototype (1) performs per-incoming-key binary-search seek/read probes across retained UUID/surrogate runs, (2) authenticates and then fully validates historical runs again for each append window, and (3) composite endpoint resolution still scans all node topology. At S20 this amplifies bounded Arrow chunks into millions of random reads plus repeated O(history) scans.

The required repair is now explicit: pin/authenticate retained index handles once per construction session; batch-sort/deduplicate incoming keys and validate them with sequential 1 MiB merge scans (zero per-key seeks); use the UUID index for batched endpoint resolution; write new/compacted runs directly through hashing block writers; and measure actual read/write blocks, random page reads, opens and syncs at the I/O wrappers. S20 evidence is invalid unless bulk ingest reports zero per-record filesystem seeks and historical validation is not repeated per chunk.

[DecisionNerd]

Acceptance audit update (2026-08-24): the clean storage-only stack is not yet close-ready. Independent exact-head review found three production P1s: live callers discard fallible endpoint-budget errors; same-window node/edge cross-kind UUID collisions can bypass pre-admission; and ordinary mutation paths can leave the v3 membership index stale, causing composite endpoint resolution to rebuild/scan historical topology.

The acceptance proof also needs a finite close batch: migrate all ordinary bulk/direct/exec endpoint callers to authenticated fallible lookup; include property/literal buffers and scratch in the writer window; prove cumulative N vs 2N multi-batch topology+index physical work with scoped zero-read counters; strengthen the ordinary lifecycle with real imported scalar/fingerprint assertions, adjacency rebuild/reopen across both shards, and public mutation across old/new shards. These repairs are being implemented separately and will be integrated before the single focused native test pass. Public resumable session/recovery remains correctly excluded for #932.