fix(exec): bound reopen and fixed-hop LIMIT query RSS at scale

[DecisionNerd]

Problem

The exact merged S20 Graph500 ladder on commit eccb6e06726d05cdef9e5242cad885be80565eee completed bounded ingest on a private 2-performance-vCPU / 4 GiB Fly machine, then was kernel-OOM-killed during the ordinary fixed-hop query proof.

Authoritative aggregate evidence:

• generator: 4.80 s, about 80 MB peak RSS
• ingest: 2,602.08 s, 15,698,918 live unique edges, about 688 MB peak RSS
• reopen plus recount: passed in 231.40 s but raised process high-water RSS to 3,328,512,000 bytes
• query phase: MATCH (a)-[r]->(b) RETURN b.node_uuid AS id ORDER BY id LIMIT 1000, followed by the two-hop equivalent if the first completes
• last durable query journal: about 1.90 GB anonymous RSS resident and 3.33 GB process high-water
• kernel victim: scale-g500-ladd, about 3.80 GB anonymous RSS and negligible file RSS
• exit status: 137
• volume: 52% used, so this was not storage exhaustion

The query journal is written before the one-hop request and only persisted again after both requests, so current evidence cannot yet distinguish whether the OOM occurred in the one-hop query or the two-hop query. That attribution gap is part of the required repair.

Objective

Make reopen/recount and fixed-hop ORDER BY ... LIMIT execution remain bounded by explicit batches/frontiers rather than graph cardinality or path-result cardinality. GraphForge is designed to be disk-bound at scale; a larger machine is not the repair.

Requirements

• Measure RSS and operator state independently for project open, node count, edge count, one-hop expand, one-hop order/limit, two-hop first expansion, two-hop second expansion, and two-hop order/limit.
• Identify retained allocations after reopen/recount and prove they are released or intentionally bounded before query execution.
• Push terminal demand through fixed-hop expansion where semantics permit, while preserving deterministic global ORDER BY results.
• Implement bounded top-k/spill for ordered LIMIT rather than materializing all candidate/path rows.
• Keep adjacency/topology access streaming or shard/page selective; do not load the complete edge set into anonymous memory.
• Preserve ordinary Rust facade execution. Do not add a ladder-only query path, skip, relaxed limit, or increased RAM requirement.

Acceptance criteria

• S20 source reopen, recount, one-hop query, and two-hop query complete on the same 4 GiB machine class with phase/operator RSS evidence and reasonable headroom.
• Peak anonymous RSS plateaus across lower rungs and remains bounded by documented operator budgets, not edge/path cardinality.
• Ordered LIMIT results match the existing deterministic semantics.
• Tests fail if a fixed-hop ordered LIMIT materializes the complete candidate set or if query allocations remain retained after completion.
• The journal persists one-hop completion before starting two-hop work, so a host OOM identifies the exact query/operator.
• Full source → export → verify → clean import → reopen proof completes without a benchmark-only fallback.

Root-cause boundaries

This issue is independent of #901. #901 owns superlinear topology publication I/O and retained-generation disk amplification. This issue owns post-ingest reopen/recount and execution memory. Both block the canonical S20 gate #900.

Observability

Record aggregate rows examined/emitted, expansion frontier size, top-k heap/spill bytes, Arrow batch rows, process RSS before/after each operator, and retained memory after each query. Do not record graph identities, values, paths, or credentials.

Testing

• deterministic lower-scale tests with fixed batch/operator budgets across increasing graph cardinality
• one-hop and two-hop ordered LIMIT plan and execution regressions
• allocation-release regression between recount and query phases
• exact changed-surface formatting, clippy, Cargo/Bazel drift, and authoritative CI

[DecisionNerd]

Implementation checkpoint at exact PR head e0b6a63f8988afad0769a293027a6a07999f576c (PR #907):

• Query evidence now attributes RSS separately to each expansion hop and to sort-only lifetime, records returned Arrow batch ownership separately from operator retention, and requires TopK memory to quiesce.
• Reopen, node count, edge recount, one-hop, and two-hop have durable before/after phase journals; the journal persists one-hop success before two-hop begins.
• Lower-rung evidence enforces the configured GraphForge memory budget, declared 4 GiB process envelope, operator headroom, and plateau classification rather than graph-cardinality-derived allowances.
• The S20 entry now executes the complete 17-phase source → export → full verify → clean import → imported reopen/query equivalence lifecycle. The earlier ladder entry did not; that harness defect is fixed rather than waived.
• A checked-in disposable Fly controller pins a linux/amd64 child image digest, requires a clean exact SHA, uses 2 performance CPUs / 4 GiB / 50 GB in DFW with no services and restart disabled, enforces a live-rate $10 ceiling and 4h30 teardown deadline, validates journal/evidence, then destroys Machine → volume → app and verifies absence.
• Current read-only live-rate projection is $1.5363 worst-case including a $1 unpriced-cost reserve.

Local validation on this exact tree:

• cargo test -p graphforge-api --lib -- --test-threads=1: 657 passed
• cargo test -p graphforge-exec --lib: passed
• strict exec/API library clippy and ladder-target clippy: passed
• non-Cypher surface gate + tests: passed (364 methods)
• Python release classification: passed
• Cargo/Bazel drift: passed (f28033349f26385177a9f4bc8324edc4553e64efe616a52d3bb8d59c20543c3c)
• Fly controller tests, Python compile, shell syntax, formatting, and diff checks: passed

Remaining acceptance evidence is the exact-head Fly lower-rung plateau plus full S20 run on the 4 GiB class. No larger-memory substitution and no rung above S20 will be used.

[DecisionNerd]

Exact PR head advanced to 7fc34653 with the finite CI alignment batch: Ruff-formatted controller sources, Node/Python release parity frozen at 249 methods, and the repinned Cargo/Bazel crate-universe lock for Tokio. Local Python format/lint, binding parity policy, Fly controller tests, Cargo/Bazel drift, and //crates/graphforge-exec:graphforge_exec now pass. The implementation/evidence behavior described in the prior checkpoint is unchanged.

[DecisionNerd]

Root-cause update at exact PR head 873fd746:

• This was a GraphForge lifecycle bug, not a Fly sizing/admission problem.
• A pooled in-memory project could retain a disposable private CSR publication across GraphForge::clear(). Small replacement graphs can reuse the same numeric topology generation, so the stale publication could be accepted as current.
• An independent delta bug compared hashed edge relation identities with raw caller stems, dropping appended relation edges in those call paths.
• The repair retains an explicit source-publication identity across ordinary forward-write invalidations, discards only the private derived publication on reset/same-or-older/unreadable source transitions under the rebuild lock, and leaves authoritative project adjacency untouched.
• The regression proves fail-closed deletion while the cleared source identity is unreadable, before the replacement graph is created, then proves correct same-generation rebuild.

Deterministic evidence:

• Bazel API BDD: 118/118
• openCypher TCK: 3897/3897, 0 regressions, 0 xpass
• graphforge-exec lib: 863 passed
• graphforge-storage lib: 605 passed
• focused adjacency integration: 4 delta + 6 expand + 19 persistent passed (1 ignored)
• strict storage/exec library clippy, formatting, and diff hygiene passed
• final fail-closed regression at 873fd746: passed under Bazel

The TCK remains materially slower than its historical baseline (~248s vs ~67s). That is a performance warning to measure in the bounded-RSS ladder; it is not being hidden or rationalized. The Fly S20 run remains gated on exact-head CI and will keep the disk-bound objective and 4 GiB memory envelope.

[DecisionNerd]

Exact-head follow-up at b9c62849:

The first rebased CI run was green everywhere except the pre-matrix multi-ontology parity comparison. This exposed a separate real namespace-isolation issue in #904, not a concurrency or Fly failure: opening a durable facade eagerly created its disposable adjacency cache inside the project root, so the live reopened target had an extra runtime entry after a deliberately failed import.

The root fix places the cache beside the project on the already-admitted data volume (or in explicitly configured spill), outside project/generation authority, and removes it when the facade closes. It now fails closed if an admitted parent volume is unavailable. Existing durable admission proves project and parent native volume identity, so this cannot silently redirect the billion-edge cache to the container root.

Focused exact-tree evidence:

• persistent-open namespace/cleanup regression: passed
• canonical Rust multi-ontology semantic parity report: passed
• before/after authoritative root entries both exactly CURRENT, FORMAT, generations
• before/after authority snapshots equal
• formatting and diff hygiene passed

CI is rerunning at b9c62849; Fly remains gated on the complete exact-head result.

[DecisionNerd]

Fly S20 execution update at exact PR head 781aec165d4815abdf821a2d1552cf1a9de29515:

The disposable DFW 2-performance-CPU / 4 GiB / 50 GB run proved that the #904 memory repair is materially working:

• generate: 5.66 s, 80,420,864 B peak RSS, 134,221,824 B disk
• ingest: 2,476.98 s, 662,941,696 B peak RSS, 22,425,980,928 B disk
• CSR: 466.53 s, 3,042,656,256 B transient peak RSS, then returned to roughly 438–634 MB anonymous RSS
• source reopen/recount: 312.93 s, 651,997,184 B peak RSS, disk flat
• source one-hop ordered LIMIT: 47.56 s, 643,051,520 B peak RSS, exactly 1,000 rows
• one-hop examined 15,698,918 candidates / 141,291,738 edge rows / 13,641,729 node rows; expansion incremental working set was 1,159,168 B against the 512 MiB operator budget; TopK returned 16,112 B and retained 0 operator bytes afterward

The following global two-hop query then ran for more than one hour without disk growth, restart, OOM, or material RSS growth (about 634 MB anonymous RSS with about 3.06–3.09 GB available):

MATCH (a)-[r1]->(b)-[r2]->(c) RETURN c.node_uuid AS id ORDER BY id LIMIT 1000

The run was deliberately stopped rather than spend the remaining envelope enumerating an unbounded number of paths. Cleanup was verified: Machine, volume, and app were destroyed.

Diagnosis: this is no longer a memory-capacity or Fly-admission failure. The global two-hop workload requires evaluating the deterministic global TopK over the complete two-hop path result unless the planner can prove a semantics-preserving demand optimization. Bounded TopK prevents materialization but cannot eliminate traversal work by itself. We are now auditing both the planner contract and the certification query. The root correction must preserve ordered-result semantics and remain a real source/import traversal proof; it will not raise RAM, skip the phase, or weaken evidence. A full exact-head Fly lifecycle will be rerun only after a deterministic regression proves the corrected complexity bound.