Skip to content

fix(exec): bound reopen and fixed-hop query memory - #907

Open
DecisionNerd wants to merge 16 commits into
mainfrom
fix/904-bound-query-rss
Open

fix(exec): bound reopen and fixed-hop query memory#907
DecisionNerd wants to merge 16 commits into
mainfrom
fix/904-bound-query-rss

Conversation

@DecisionNerd

@DecisionNerd DecisionNerd commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • count nodes by streaming bounded Parquet batches instead of materializing complete graph inspection inventories
  • replace ordinary O(E)-memory scan adjacency fallback with single-flight bounded external-sort CSR repair in an isolated admitted-volume cache
  • bind exact relation identities in persistent manifests, including unsafe names and literal _all, and validate concurrent topology generations before serving
  • prove ordered LIMIT uses bounded TopK, expose query-local expansion/sort/memory/RSS evidence, and persist typed per-phase ladder failures

Root-cause evidence

The prior S20 run completed ingest at about 688 MB peak RSS, then reached about 3.33 GB during reopen/recount and was kernel-OOM-killed during fixed-hop execution at about 3.80 GB anonymous RSS. The old node_count materialized node and relationship inventories, and an absent/unreadable adjacency index built a graph-cardinality HashMap in anonymous memory.

This change removes both graph-cardinality allocation paths. Lazy repair never mutates immutable generation contents; it writes a provider-isolated cache on the admitted project volume (or configured spill volume), is single-flight across concurrent queries, and fails closed on build/generation errors.

Verification

  • cargo test -p graphforge-exec --lib — 859 passed, 13 ignored
  • cargo test -p graphforge-api --lib -- --test-threads=1 — 657 passed
  • fixed-hop suite — 7 passed, 2 release-only ignored
  • adjacency focused suite — 13 passed
  • ladder CI rung — passed
  • cargo clippy -p graphforge-exec -p graphforge-api --lib -- -D warnings — passed
  • cargo fmt --all -- --check — passed
  • git diff --check — passed

The exact-head 4 GiB Fly S20 plus lower-rung RSS plateau remains the pre-merge acceptance proof and will be attached to this PR before merge.

Closes #904


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@coderabbitai

coderabbitai Bot commented Aug 23, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6ee148b1-ffdb-407d-a693-ee6a7b920780

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.


Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added executor Changes to query executor core Core source code changes documentation Improvements or additions to documentation labels Aug 23, 2026
@blacksmith-sh

This comment has been minimized.

@github-actions github-actions Bot added testing Test coverage and testing infrastructure tooling Developer tooling and automation labels Aug 23, 2026
@DecisionNerd
DecisionNerd force-pushed the fix/904-bound-query-rss branch 2 times, most recently from b9c6284 to 781aec1 Compare August 23, 2026 17:25
@DecisionNerd
DecisionNerd force-pushed the fix/904-bound-query-rss branch from 781aec1 to 7f9091b Compare August 23, 2026 20:54
@blacksmith-sh

This comment has been minimized.

@github-actions github-actions Bot added the planner Changes to query planner label Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core source code changes documentation Improvements or additions to documentation executor Changes to query executor planner Changes to query planner testing Test coverage and testing infrastructure tooling Developer tooling and automation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(exec): bound reopen and fixed-hop LIMIT query RSS at scale

1 participant