feat(parser): persist generation-bound lexer restart checkpoints (#6982)

[EffortlessSteven]

Claim

The canonical incremental lexer path can now restore a complete behavior-bearing checkpoint without replaying the unchanged old-source prefix.

This PR is stacked on #6759 and targets agent/6704-live-lexer-checkpoints. It preserves #6759’s conservative correctness rule: retain only the proven unchanged token prefix, re-lex the edited generation to EOF, and reuse zero old suffix tokens.

What changed

• added private StoredLexCheckpoint records containing:
  • the compact compatibility summary;
  • complete perl_lexer::LexerCheckpoint state;
  • exact old-source and unchanged-prefix fingerprints;
• persist complete live checkpoints during ordinary lexing rather than retaining them only under #[cfg(test)];
• bind each checkpoint to the exact source generation and reject stale or wrong-source use;
• select the nearest valid stored checkpoint before the edit without invoking the old-prefix replay helper;
• transform only proven unchanged prefix checkpoints into the candidate generation;
• regenerate complete suffix checkpoints while lexing the edited source to EOF;
• commit source, tokens, compact summaries, and complete checkpoints atomically;
• add deterministic sparse capture with a hard 4,096-checkpoint memory bound;
• add StoredCheckpointToEof and explicit receipt fields for:
  • old_prefix_bytes_replayed;
  • fresh bytes lexed;
  • retained prefix/suffix tokens;
  • retained full-checkpoint count;
• retain the deprecated replay strategy token only as a compatibility name; the canonical path reports zero replayed old bytes;
• add public proof for ordinary, method-arrow, quote/regex, Unicode/newline, prototype, and heredoc edits plus sequential-generation checkpoint regeneration.

Correctness boundary

This PR does not reuse an old token suffix. reused_suffix_tokens remains exactly zero. Exact suffix synchronization requires #6986’s complete-state equality, source mapping, and token-window proof.

The current source/prefix fingerprint is an internal stable adapter. #4851 remains the canonical source/content identity owner; migration to its types must not weaken the generation check.

Validation

Hosted CI is authoritative for this connector-built stacked candidate. Focused commands:

cargo fmt --all -- --check
cargo test --locked -p perl-lexer
cargo test --locked -p perl-parser --features incremental --lib incremental::lex
cargo test --locked -p perl-parser --features incremental --test incremental_lexer_restart
cargo test --locked -p perl-parser --features incremental --test incremental_parse_output
python3 scripts/ci/run_parser_integration.py

Refs #6698
Refs #6704
Refs #6982
Refs #2021
Refs #2327

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 44a8597d-72bf-47f5-9f9d-fadec097a6ba

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[EffortlessSteven]

Review verdict: not ready. Persisting complete lexer state is the correct next architecture, but the current identity and lifetime implementation can make ordinary lexing quadratic and can diverge from fresh lexing after an idle editor interval. Those defects sit in the restart authority itself, so token-parity tests run immediately after state construction do not falsify them.

[EffortlessSteven]

[P1] This hashes the complete source—and then the growing prefix—once per retained checkpoint. With up to 4,096 checkpoints, initial lexing and generation carry-forward become O(checkpoints × source_len) and can approach quadratic work on large files before the incremental path starts. Compute the generation digest once and derive prefix identity incrementally or from a precomputed prefix-hash structure.

[EffortlessSteven]

[P1] A 64-bit FNV equality is not sufficient to authorize restoration of behavior-bearing lexer state. A collision makes belongs_to_source accept state from different bytes, which can produce incorrect tokens rather than merely a cache miss. Bind checkpoints to the canonical collision-resistant source/generation identity and use direct unchanged-prefix evidence; do not make a small fingerprint the correctness gate.

[EffortlessSteven]

[P1] Rejecting only checkpoints that are currently timeout-sensitive does not make a persisted start_time safe. A checkpoint captured before a later heredoc has an empty queue, can sit in an editor for minutes, then restores the old Instant; when lexing reaches that heredoc, the wall-clock budget can expire immediately and diverge from a fresh lex. Make timeout/cancellation budget operation-local or deterministically charged, and add a delayed-restart control before persisting this state across generations.

[EffortlessSteven]

Read-only stale-PR preparation (live base fdfe5f1): disposition REPAIRABLE. The incremental checkpoint seam remains current, but unresolved authority risks include source/generation identity binding, delayed-restart timeout behavior, collision-safe identity, and large-source cost. Same-writer current-main repair plus focused falsifiers is required; no mutation was made.

[EffortlessSteven]

Disposition (campaign #11869 stale-queue sweep, 2026-08-21): RESOLVE_CONFLICT

• source basis: PR head vs current main 0fa96eedb; read-only sweep, no proof re-run
• claim checked: generation-bound lexer restart checkpoints (feat(parser-incremental): persist generation-bound full lexer restart checkpoints #6982)
• main interaction: textual conflict against the just-landed fix(parser): restore complete live lexer state and relex to EOF (#7294) #6759 live-checkpoint model — same semantic seam changed on main while candidate open
• stacked-child rule (SPEC-0006): stored-selection delta remains intact and unowned elsewhere; compare models, then rebase/retarget the child-only diff so it is reviewable again
• carries three unresolved P1 review threads that ride the repair rather than dying with a close
• unique value if closed: persisted restart checkpoints + generation binding absent from main in this form
• rationale: sole owner of feat(parser-incremental): persist generation-bound full lexer restart checkpoints #6982; conflict is mechanical topology, not disproof of value
• follow-up owner: feat(parser-incremental): persist generation-bound full lexer restart checkpoints #6982

Idle >3h → reclaimable per umbrella #11869 queue policy.

[EffortlessSteven]

Disposition: REPAIR (campaign #11869 stale-PR sweep, 2026-08-21). Feature-complete lexer-checkpoint slice (+1758/-405) but CONFLICTING with moved incremental-parser code on main; 3 unresolved threads. Repair: rebase against current incremental code, CI replay, human review. Downstream #7332 is sequenced behind this.

[EffortlessSteven]

Disposition (campaign #11869 stale-queue sweep, 2026-08-22): RESOLVE_CONFLICT — reaffirms the 2026-08-21 packet; conflict and stack facts re-verified.

source basis: head dd1afd06 (+1758/−405 across 13 files) vs current main
claim checked: persist generation-bound full lexer restart checkpoints (#6982, open)
current disposition: RESOLVE_CONFLICT
current-main interaction: textual conflict (GitHub mergeable CONFLICTING) against the landed #6759 conservative live-checkpoint model — same semantic seam changed on main while candidate open
unique value: StoredLexCheckpoint persistence of complete lexer state, generation/source-fingerprint stale rejection, receipt fields (old_prefix_bytes_replayed etc.), deterministic sparse capture with 4096 bound, full edit-shape proof suite — none owned elsewhere (#6986 owns suffix reuse only and remains open)
concrete defects/blockers: 3 unresolved P1 review threads; target base agent/6704-live-lexer-checkpoints deleted from origin (verified via ls-remote 2026-08-22) after parent squash landing
proof run: none this sweep (read-only); prior hosted evidence remains attached to its own head
review currentness: standing 2026-08-21 review current for unchanged seams; conflict-affected conclusions must refresh after repair
base-reconciliation reason: stacked parent #6759 landed under squash — child-only delta must be made reviewable against current main (SPEC-0006 stacked-candidates rule)
merge/close rationale: none — controlling issue open, stored-selection delta unowned elsewhere
follow-up: compare models at the seam, retarget/rebase the child-only diff onto current main, resolve threads, replay CI; downstream #7332 stays sequenced behind this