perf: implement session-local runtime reachability and retire engine ref

docs/architecture.md

@@ -163,9 +163,16 @@ Engine lifecycle admission closes session operation and inspection
 registration against shutdown. After admission drops, stable session operation
 entries account effectful foreground work, per-session observer counts account
 standalone diagnostics and progress waits, and mandatory permits account
-accepted caller or internal cleanup work. The crate-private `EngineRef` remains
-an `Arc<EngineInner>` access wrapper for memory reachability; cloning it does
-not create a separate shutdown blocker.
+accepted caller or internal cleanup work. Public session and transaction
+handles retain weak reachability to their exact `SessionState`. Successful
+admission returns a short-lived admitted session wrapper that alone exposes the
+normal weak-state upgrade. Consuming that wrapper produces an admitted
+`SessionRuntime`, retaining the same admission until the stable operation or
+observer proof is registered. Operation and transaction identity then resolve
+directly on the pinned state without a session-registry lookup. The state
+reaches immutable component capabilities through `EngineCore`, whose only
+registry back-reference is weak and used for cold pointer-exact removal after a
+session becomes closed and idle.
 
 ## Logging, Checkpoint and Recovery
 

docs/backlogs/000175-scalable-shared-resource-lifetime-management.md

@@ -23,19 +23,55 @@ Phase 2 accepted this fixed boundary cost as explicit performance debt because
 its cancellation-ownership prerequisite is complete; this backlog owns the
 performance correction rather than treating the original budget as passed.
 
+Task 000255 removed the engine-wide weak upgrade, session-registry operation
+lookup, and attachment `PoolGuards` clone from session-coordinated statement
+checkout. Its paired release measurements against
+`2098cbb70316d383881aa3c05ba6ef56db408cc3` reduced median `stmt-noop` latency
+from 73.524 ns to 47.865 ns at 1 thread/1 session and from 83.893 ns to
+76.425 ns at 4 threads/16 sessions. Median `trx-noop` latency likewise fell
+from 301.035 ns to 223.491 ns at 1/1 and from 264.610 ns to 220.616 ns at
+4/16.
+
+The same matrix exposed a separate contended `index-stream` result. At
+4 threads/16 sessions, the unique-index median increased from 76,799 ns to
+106,144 ns per stream and the non-unique median increased from 82,875 ns to
+105,518 ns. The 1/1 rows were near-neutral by comparison: unique increased
+from 233,689 ns to 240,751 ns and non-unique decreased from 240,770 ns to
+238,612 ns. Independent repeated 4/16 blocks reproduced the unfavorable
+result.
+
+Paired `cargo flamegraph` profiles localized the extra candidate CPU to
+existing buffer and row-page reference-count operations rather than the new
+session runtime path. In warmed unique-index 4/16 profiles, relaxed and release
+`Arc` atomic helpers accounted for about 29.75% of candidate samples versus
+about 4.65% of baseline samples, primarily below row-page lookup, fixed-buffer
+page lookup, and page-frame release. Candidate `SessionRuntime` and attachment
+pool-guard access accounted for about 0.07%; no weak-engine upgrade,
+session-registry lookup, or attachment guard-bundle clone remained in the
+candidate stack. This evidence identifies the contended domain but does not
+yet prove why the ownership-path speedup changes page-frame contention.
+
 ## Deferred From (Optional)
 
-docs/tasks/000247-statement-public-transaction-cancellation-ownership.md; docs/rfcs/0025-session-coordinated-cancellation-cleanup-ownership.md Phase 2
+docs/tasks/000247-statement-public-transaction-cancellation-ownership.md; docs/rfcs/0025-session-coordinated-cancellation-cleanup-ownership.md Phase 2; docs/tasks/000255-session-local-runtime-reachability.md
 
 ## Deferral Context (Optional)
 
-- Defer Reason: Task 000247 is scoped to statement cancellation ownership and its bounded performance work. Redesigning lifetime and destruction policy across the engine, pools, transaction system, and other component resources materially broadens both architecture and shutdown proof obligations, so it should be planned and reviewed independently rather than folded into Phase 2.
-- Findings: The session coordinator is already an authoritative teardown blocker for active public/private transactions and foreground operations, making the custom counted `EngineRef` pin redundant for much of that hot path. It is not yet authoritative for every runtime user: detached observer pins do not occupy the active operation slot, and standalone/internal strong pins rely on `runtime_refs` for efficient shutdown notification. `Arc::strong_count` remains the final ownership backstop, but it has no drop notification, explaining why the separate counter exists. Removing local `TrxInner` copy work exposed contention on shared lifecycle cache lines rather than adding new lifecycle operations.
+- Defer Reason: Task 000247 is scoped to statement cancellation ownership and its bounded performance work. Redesigning lifetime and destruction policy across the engine, pools, transaction system, and other component resources materially broadens both architecture and shutdown proof obligations, so it should be planned and reviewed independently rather than folded into Phase 2. Task 000255 is scoped to session-local runtime reachability; changing buffer-frame ownership or index-stream scheduling to address the newly measured contention would cross that boundary without a proven cause.
+- Findings: The session coordinator is already an authoritative teardown blocker for active public/private transactions and foreground operations, making the custom counted `EngineRef` pin redundant for much of that hot path. It is not yet authoritative for every runtime user: detached observer pins do not occupy the active operation slot, and standalone/internal strong pins rely on `runtime_refs` for efficient shutdown notification. `Arc::strong_count` remains the final ownership backstop, but it has no drop notification, explaining why the separate counter exists. Removing local `TrxInner` copy work exposed contention on shared lifecycle cache lines rather than adding new lifecycle operations. Task 000255 confirmed that session-local weak reachability materially improves statement and transaction no-op paths. Its contended index-stream profiles show increased time in existing page-frame `Arc` increments and decrements, while the new session runtime access is negligible; the relationship between faster statement boundaries and buffer-page contention remains unproven.
 - Direction Hint:
   Start with the narrow performance result: make session-coordinated transaction and foreground-operation access use the session/component lifecycle proof instead of globally counted runtime pins, while retaining admission for the operation-start versus shutdown race. Explicitly account for detached observers, terminal-publication gaps, stale cleanup jobs, and worker-owned work before narrowing or removing `runtime_refs`.
 
   Then evaluate the general resource-lifetime policy. Compare sharded counters with centralized arena/owner destruction rather than assuming one universal mechanism. Prefer centralized destruction when resource lifetime is already bounded by an engine/component owner and individual early reclamation is unnecessary; prefer sharding only where independent lifetime and thread mobility still require counting. Avoid weakening memory ordering or deleting counters without a replacement shutdown and destruction proof.
 
+  Reproduce the task 000255 index-stream profile independently before changing
+  buffer or index code. Separate page-frame atomic operation count from
+  per-operation contention latency, and test whether statement-boundary
+  synchronization, shared root traversal, range overlap, or object/cache-line
+  placement explains the 4/16-only result. Do not optimize index-stream by
+  adding duplicate runtime capabilities or unsafe cached pointers unless a
+  profile-backed design proves that session runtime access is causal.
+
 ## Scope Hint
 
 Inventory high-frequency shared lifetime counters and guard cloning across engine access, buffer pools, transaction-system access, catalog/table runtime access, and related long-lived resources. Separate session-coordinated owners from detached observers and worker-owned pins. Remove or amortize global counter operations where an authoritative session/component owner already proves liveness. Evaluate sharded reference counters and centralized arena/owner-managed destruction against thread mobility, shutdown ordering, reclamation latency, memory safety, and measured contention. Use an RFC if the chosen direction changes ownership architecture across multiple subsystems.

docs/engine-component-lifetime.md

@@ -7,16 +7,22 @@ component-registry migration work.
 ## Terminology
 
 - `Engine`: public owner of top-level teardown state and session creation.
-- `EngineInner`: crate-private shared runtime state held behind the engine
-  owner and internal shared handles.
-- `EngineRef`: crate-private cloneable `Arc<EngineInner>` access wrapper. It
-  provides memory reachability and component access but is not itself a
-  shutdown blocker.
+- `EngineInner`: owner-facing coordination shell containing `EngineCore`, the
+  strong session registry, the lifecycle gate, and the session-id source.
+- `EngineCore`: immutable component-capability set retained by registered
+  session state. It has only a weak back-reference to the session registry.
+- `SessionRuntime`: typed strong reference to one exact `SessionState`.
+- `WeakSessionRef`: weak reference to one exact `SessionState` plus that
+  session's limited lifecycle-admission façade.
+- `AdmittedSessionRef`: short-lived pairing of that exact weak state with the
+  admission acquired through its session façade.
+- `AdmittedSessionRuntime`: the result of consuming an admitted weak reference
+  and upgrading it while retaining the same admission.
 - Public session and transaction handles: weak, non-cloneable capabilities that
-  identify engine-local state and acquire admitted internal access only for one
+  identify exact session-local state and acquire admitted internal access for one
   operation or terminal path.
 - `SessionOperationEntry`: one registry-owned stable operation record keyed by
-  `(SessionID, OperationID)`; it contains no `EngineRef`,
+  `(SessionID, OperationID)`; it contains no engine-wide reference,
   `SessionObserverPin`, or whole operation future.
 - `SessionObserverPin`: non-cloneable standalone observer authority accounted
   by its session lifecycle without consuming the effectful operation slot.
@@ -31,8 +37,12 @@ The runtime uses an explicit owner/runtime split:
 - `Engine` owns:
   - `inner: Arc<EngineInner>`
   - `components: ComponentRegistry`
-- `EngineInner` owns only crate-private shared runtime handles and the
-  lifecycle gate:
+- `EngineInner` owns:
+  - `core: Arc<EngineCore>`
+  - `session_registry: Arc<SessionRegistry>`
+  - `lifecycle: Arc<EngineLifecycle>`
+  - the engine-local session-id source
+- `EngineCore` owns the shared runtime capabilities:
   - engine poisoner
   - mandatory runtime
   - catalog
@@ -41,7 +51,12 @@ The runtime uses an explicit owner/runtime split:
   - fixed and evictable buffer pools
   - table-file subsystem
   - readonly buffer pool
-  - shutdown admission state
+  - a weak session-registry back-reference used only for cold exact removal
+
+The registry owns each `Arc<SessionState>`. Each state retains `Arc<EngineCore>`
+and one `Arc<SessionAdmission>` into the lifecycle gate. `EngineCore` does not
+retain `EngineInner`, the lifecycle gate, or a strong registry reference, so
+the graph has no strong cycle.
 
 `ComponentRegistry` is intentionally not part of `EngineInner`. The registry is
 needed only for explicit reverse-order shutdown and final owner drop. Keeping
@@ -105,7 +120,7 @@ dependency.
 
 `MandatoryRuntime` is registered immediately after the poisoner. Catalog,
 transaction, recovery, and future operation adapters can therefore retain its
-direct `QuiescentGuard` without owning `EngineRef` or another runtime `Arc`.
+direct `QuiescentGuard` without owning the engine owner shell.
 Its build shelves only the runtime guard and configured runner count. The later
 `MandatoryRuntimeWorkers` build starts the fixed runners and registers their
 join-handle owner at the required shutdown position.
@@ -370,37 +385,42 @@ The lazy traversal may hold one DashMap shard read guard during the short
 either inner mutex is held, so there is no reverse lock edge. The iterator is
 dropped before cleanup submission, event waiting, notification, or removal.
 
-The registry owns `Arc<SessionState>`, and an active slot owns
-`Arc<SessionOperationEntry>`. Neither object owns a strong engine runtime
-handle. `EngineRef` exists only in scoped foreground authorities, transaction
-or observer authorities, transaction attachments, claims, and submitted
-cleanup jobs, preventing a registry-to-engine strong reference cycle. Engine
-admission closes every new operation or observer registration against shutdown;
-session entries and observer counts then become the durable shutdown proof
-after admission drops. Mandatory permits provide the corresponding proof for
-accepted caller and internal cleanup work.
+The registry owns `Arc<SessionState>`, each state owns `Arc<EngineCore>`, and an
+active slot owns `Arc<SessionOperationEntry>`. Public `Session` and
+`Transaction` handles own only `WeakSessionRef`. Operation authorities,
+transaction attachments, claims, and cleanup jobs retain `SessionRuntime`, so
+they reach components through the already-pinned exact state without recovering
+`EngineInner` or looking up the registry. Engine admission closes every new
+operation or observer registration against shutdown; session entries and
+observer counts then become the durable shutdown proof after admission drops.
+Mandatory permits provide the corresponding proof for accepted caller and
+internal cleanup work.
 
 The owned-handle inventory follows those authorities:
 
-- `SessionObserverPin` pairs its `EngineRef` with one counted session observer.
+- `SessionObserverPin` pairs `SessionRuntime` with one counted session observer.
 - `SessionOperationPin`, `TrxAttachment`, transaction checkout and completion
-  claims, and DDL or maintenance progress all remain paired with their exact
-  stable `SessionOperationEntry`.
+  claims, DDL or maintenance progress, and cleanup jobs carry `SessionRuntime`
+  and remain paired with their exact stable `SessionOperationEntry`.
 - accepted DDL and maintenance also retain a mandatory caller permit through
   terminal publication.
 - abandoned and terminal-rollback cleanup pair their active session entry with
   a mandatory internal permit; failed-precommit cleanup is covered by mandatory
   internal admission.
-- weak upgrades used for admission rejection, handle drop, or exact terminal
-  resolution either register one of those authorities or stay within a bounded
-  section that cannot use components after rejection.
+- foreground acquisition creates `AdmittedSessionRef` through
+  `SessionAdmission`, consumes it to create `AdmittedSessionRuntime`, validates
+  poison when required, and registers its stable operation or observer before
+  releasing admission and retaining plain `SessionRuntime`.
+- terminal and cleanup paths reuse existing authority, upgrade the exact weak
+  state without new foreground admission, and validate both operation key and
+  transaction id directly on that state.
 - redo, mandatory-runtime, purge, file, and eviction workers are owned and
-  joined by their registered component owners rather than by `EngineRef`.
+  joined by their registered component owners.
 
-An explicit shutdown may finish while a weak public handle's rejected upgrade
-briefly retains an internal `Arc<EngineInner>`. That handle has no admitted
-authority to access components after rejection, so ordinary `Arc` reachability
-is deliberately not a production shutdown condition.
+A surviving public handle retains only a weak state reference and its small
+closed admission façade. Once registry ownership is released it cannot retain
+or recover component capabilities, so explicit shutdown and final owner drop
+do not depend on destruction of public handles.
 
 The final reverse-order shutdown step releases `StorageRootLease`. A later
 engine can therefore acquire the root immediately after explicit shutdown,
@@ -412,11 +432,13 @@ persistent `storage.lock` directory entry is never removed.
 After shutdown succeeds, `Engine` field order makes the final owner-drop
 sequence deterministic:
 
-1. drop `Arc<EngineInner>`
+1. drop `Arc<EngineInner>`, releasing the registry-owned session states and
+   their final `EngineCore` references
 2. drop `ComponentRegistry`
 
-Dropping `EngineInner` first releases the runtime-held quiescent guards before
-registry-owned component owners start their final `QuiescentBox<T>` drains.
+Dropping `EngineInner` first releases `EngineCore` and its runtime-held
+quiescent guards before registry-owned component owners start their final
+`QuiescentBox<T>` drains.
 
 `Engine::drop` invokes the same synchronous drain as `Engine::shutdown()`.
 An unintended owner drop can therefore block indefinitely while
@@ -458,8 +480,11 @@ That provenance rule gives three guarantees:
 - stable owner identity survives cloning because guards keep the owner alive
 - page guards and arena state can rely on one exact pool provenance source
 
-`PoolGuards` is only a named bundle of individually branded guards; it does not
-weaken the single-owner provenance rule.
+`EngineCore` owns one canonical `EnginePools` capability containing the four
+typed pool handles and one prebuilt `PoolGuards` bundle. Session-coordinated
+operations borrow that bundle through `SessionRuntime`; transaction attachments
+do not clone it. `PoolGuards` remains only a named bundle of individually
+branded guards and does not weaken the single-owner provenance rule.
 
 ## Arena And Page-Guard Lifetime Rules
 

docs/lock-system.md

@@ -467,8 +467,9 @@ completion.
 Every terminal user-transaction path drains its owner-local `OwnerLockState`
 before finishing the session transaction lifecycle. Transaction code mints one
 non-cloneable, transaction-id-bound `ReleasedTransactionLocks` proof only after
-the local state is empty. Prepared and precommit paths also consume and drop
-their retained lock-manager guard before minting the proof.
+the local state is empty. Prepared and precommit paths reach the engine lock
+manager through their retained terminal attachment, avoiding a second retained
+component guard.
 
 `TrxAttachment::commit()` and `TrxAttachment::rollback()` consume a matching
 proof before they can make a running session idle or close an abandoned