Skip to content

feat(api): stream one-generation graph construction sessions #932

Description

@DecisionNerd

Parent

Native sub-issue of #901. This issue must block #901 and does not expand its acceptance criteria.

Objective

Provide the ordinary Rust construction/import session used by large clients: accept bounded Arrow chunks durably, stage linearly, and publish exactly one immutable project generation.

Requirements

  • Stream node then edge Arrow/Parquet batches through the feat(storage): stage topology append-only with linear ingest I/O #901 shard and UUID-surrogate primitives.
  • Keep memory bounded by configured batch/shard/index-run limits.
  • Persist deterministic resume markers and surrogate/index state.
  • Validate duplicates and cross-kind conflicts with bounded external work.
  • Commit graph, catalog, indexes, counters, and CURRENT once; abort/cancel/corruption leave the prior generation authoritative.
  • Use this public path in the Graph500 ladder; do not loop publish_bulk_edges or add a benchmark-only writer.

Acceptance criteria

  • Any number of accepted chunks produces exactly one final generation publication.
  • Interrupted node and edge staging resumes without duplicate live rows; replay is idempotent.
  • Cancellation, corruption, and validation failures preserve the prior CURRENT.
  • Lower-scale tests prove fixed memory windows and linear work across increasing totals.
  • S20/S22 evidence records per-window RSS, topology work rows, elapsed time, shard count, and physical disk.
  • Full ordinary source to export to verify to clean import to reopen proof passes.

Non-goals

Distributed ingest, increased RAM ceilings, relaxed integrity, or benchmark-specific execution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions