diff --git a/docs/backlogs/000175-scalable-shared-resource-lifetime-management.md b/docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md similarity index 54% rename from docs/backlogs/000175-scalable-shared-resource-lifetime-management.md rename to docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md index 9257e383..6bb0f049 100644 --- a/docs/backlogs/000175-scalable-shared-resource-lifetime-management.md +++ b/docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md @@ -10,7 +10,7 @@ Task 000247 performance work exposed the issue after boxing `TrxInner` removed l A 50-million-operation 4-thread/16-session perf profile attributed 49.86% of candidate aggregate CPU time to `__aarch64_ldadd8_acq_rel`, versus 25.04% on `origin/main`. Candidate caller attribution was 16.59% through `WeakEngineRef::upgrade -> EngineRef::new -> retain_runtime_ref`, 13.74% through `EngineAdmission::drop -> release_admission`, and 19.54% through `TrxAttachment::drop -> EngineRef::drop -> release_runtime_ref`. At the same time, inline-core `memcpy` fell from 23.06% to 0.67%. -Static review found that session operation entries already block component teardown for active transaction work, while `runtime_refs` remains necessary in the current design for detached pins such as `SessionObserverPin` and as a waitable notification layer over `Arc` ownership. The current counter is therefore broader than the hot-path lifetime proof requires. +At the task 000247 revision, static review found that session operation entries already blocked component teardown for active transaction work, while `runtime_refs` still covered detached pins such as `SessionObserverPin` and supplied waitable notification over `Arc` ownership. That counter was therefore broader than the hot-path lifetime proof required; task 000254 later removed it after moving observer authority into session lifecycle state. The fresh task-resolution matrix reproduced the contended result. Against `origin/main` `768842e8e8c1`, the candidate reduced median `stmt-noop` latency @@ -51,6 +51,70 @@ session-registry lookup, or attachment guard-bundle clone remained in the candidate stack. This evidence identifies the contended domain but does not yet prove why the ownership-path speedup changes page-frame contention. +An exact-revision reproduction on 2026-08-08 isolated the cause. Task 000255 +moved `PoolGuards` construction from `SessionState::new` to one canonical +`EnginePools` bundle. `PoolGuard` does not retain one `Arc` per page frame; it +retains a `SyncQuiescentGuard<()>`, and every page lookup clones that wrapper +into `PageLatchGuard`. The canonical bundle therefore made metadata- and +row-pool page accesses from every session update the same two `Arc` strong-count +cache lines. The underlying arena `QuiescentGuardCount` remained pool-global +but was not touched by those clones. + +Seven alternating release samples reproduced unique `index-stream` medians of +73,046 ns for the pre-task baseline and 105,969 ns for task 000255 at 4/16. A +controlled candidate that restored fresh guard roots per session reduced the +median to 77,109 ns without changing cache-hit or row counts; its 1/1 median +was unchanged within noise. Relaxed and release `Arc` helpers fell from 28.96% +of profiled samples to 5.90%, matching the pre-task profile. Non-unique streams +showed the same result. This proves the regression is cross-session contention +on the canonical `PoolGuard` roots, not additional page operations, statement +synchronization, or index scheduling. + +The implemented session-root correction was then measured against exact +`HEAD` `916471d9c3cb`. Seven alternating release samples reduced the unique +4/16 median from 101,072 ns to 73,938 ns (-26.85%) and the non-unique median +from 101,336 ns to 77,525 ns (-23.50%). Unique and non-unique 1/1 medians +changed by -0.05% and +0.16%, respectively. A final candidate profile +attributed 5.83% of samples to relaxed/release atomic helpers, below the 10% +acceptance ceiling and consistent with session-local rather than cross-session +refcount traffic. + +The follow-up API audit renamed the misleading `BufferPool::pool_guard()` +accessor to `create_base_guard()` and made its construction cost explicit. +Production root creation is now limited to engine/session owners, catalog +bootstrap, and detached eviction workers. Readonly cache misses and +invalidation, CoW writes, DDL, recovery, and checkpoint work receive the +session- or operation-scoped guard instead, so those paths cannot silently +acquire another pool-global quiescent keepalive. + +After rebasing onto `e5152e8`, a fresh-root current-working-tree smoke used +100 streams of 1,000 rows at 4 threads/16 sessions after one warmup. Unique +and non-unique average latency was 77,752.510 ns and 77,550.420 ns per stream, +respectively, with 100,000 rows returned and zero failures in each run. This +bounded check is consistent with the earlier seven-sample medians; it is not a +replacement for that paired matrix. + +## Resolution + +The implemented result uses a measured hybrid lifetime policy rather than one +universal counter. Task 000254 removed engine-global runtime reference +accounting in favor of registered session operations, session-local observers, +mandatory permits, and component-worker ownership. Task 000255 removed the +engine weak upgrade, registry lookup, and guard-bundle clone from transaction +checkout. The final buffer follow-up shards the high-frequency outer +`PoolGuard` `Arc` roots per session while retaining the pool-global +`QuiescentGuardCount` only for deliberate base-root acquisition. The packed +`EngineAdmission` counter remains solely as the operation-start versus shutdown +race gate and is released before effectful work. + +The repository-wide ownership inventory is maintained in +`docs/engine-component-lifetime.md`: it records admission and shutdown +authorities, session and observer handles, mandatory work, component workers, +quiescent ownership, pool-root provenance, and teardown order. Production +`create_base_guard()` sites were audited down to engine/session owners, catalog +bootstrap, and detached eviction workers; page access, invalidation, COW, DDL, +recovery, and checkpoint paths receive an existing owner-scoped guard. + ## Deferred From (Optional) docs/tasks/000247-statement-public-transaction-cancellation-ownership.md; docs/rfcs/0025-session-coordinated-cancellation-cleanup-ownership.md Phase 2; docs/tasks/000255-session-local-runtime-reachability.md @@ -58,19 +122,20 @@ docs/tasks/000247-statement-public-transaction-cancellation-ownership.md; docs/r ## Deferral Context (Optional) - Defer Reason: Task 000247 is scoped to statement cancellation ownership and its bounded performance work. Redesigning lifetime and destruction policy across the engine, pools, transaction system, and other component resources materially broadens both architecture and shutdown proof obligations, so it should be planned and reviewed independently rather than folded into Phase 2. Task 000255 is scoped to session-local runtime reachability; changing buffer-frame ownership or index-stream scheduling to address the newly measured contention would cross that boundary without a proven cause. -- Findings: The session coordinator is already an authoritative teardown blocker for active public/private transactions and foreground operations, making the custom counted `EngineRef` pin redundant for much of that hot path. It is not yet authoritative for every runtime user: detached observer pins do not occupy the active operation slot, and standalone/internal strong pins rely on `runtime_refs` for efficient shutdown notification. `Arc::strong_count` remains the final ownership backstop, but it has no drop notification, explaining why the separate counter exists. Removing local `TrxInner` copy work exposed contention on shared lifecycle cache lines rather than adding new lifecycle operations. Task 000255 confirmed that session-local weak reachability materially improves statement and transaction no-op paths. Its contended index-stream profiles show increased time in existing page-frame `Arc` increments and decrements, while the new session runtime access is negligible; the relationship between faster statement boundaries and buffer-page contention remains unproven. +- Findings: At deferral time, the session coordinator was already an authoritative teardown blocker for active public/private transactions and foreground operations, making the custom counted `EngineRef` pin redundant for much of that hot path. Detached observer pins did not occupy the active operation slot, so the then-current design still relied on `runtime_refs` for efficient shutdown notification. Task 000254 subsequently replaced that dependency with session-local observer accounting. Removing local `TrxInner` copy work exposed contention on shared lifecycle cache lines rather than adding new lifecycle operations. Task 000255 confirmed that session-local weak reachability materially improves statement and transaction no-op paths. Its canonical pool-guard bundle accidentally changed the sharing domain of the outer `SyncQuiescentGuard` `Arc` from one root per session to one root per engine pool. Restoring one fresh bundle per `SessionState` preserves the existing lifetime proof while sharding page-guard clone/drop traffic. - Direction Hint: Start with the narrow performance result: make session-coordinated transaction and foreground-operation access use the session/component lifecycle proof instead of globally counted runtime pins, while retaining admission for the operation-start versus shutdown race. Explicitly account for detached observers, terminal-publication gaps, stale cleanup jobs, and worker-owned work before narrowing or removing `runtime_refs`. Then evaluate the general resource-lifetime policy. Compare sharded counters with centralized arena/owner destruction rather than assuming one universal mechanism. Prefer centralized destruction when resource lifetime is already bounded by an engine/component owner and individual early reclamation is unnecessary; prefer sharding only where independent lifetime and thread mobility still require counting. Avoid weakening memory ordering or deleting counters without a replacement shutdown and destruction proof. - Reproduce the task 000255 index-stream profile independently before changing - buffer or index code. Separate page-frame atomic operation count from - per-operation contention latency, and test whether statement-boundary - synchronization, shared root traversal, range overlap, or object/cache-line - placement explains the 4/16-only result. Do not optimize index-stream by - adding duplicate runtime capabilities or unsafe cached pointers unless a - profile-backed design proves that session runtime access is causal. + Preserve one fresh `PoolGuards` root bundle per session while continuing to + borrow it from transaction attachments and operation pins. Keep canonical + engine roots for non-session work, and test both pool identity and outer + `Arc` root identity so provenance-preserving centralization cannot silently + recreate the contention. Removing the remaining per-page `Arc` operations + would require scoped page guards plus owned promotion for detached I/O and + should remain separate unless its measured incremental benefit justifies the + broader lifetime/API change. ## Scope Hint @@ -97,3 +162,11 @@ When a backlog item is moved to `docs/backlogs/closed/`, append: - Reference: - Closed At: ``` + +## Close Reason + +- Type: implemented +- Detail: Tasks 000254 and 000255 removed engine-global runtime accounting and statement lookup traffic. The follow-up sharded pool-guard roots per session, audited base-root creation boundaries, preserved lifecycle authority, and restored contended index-stream performance without a repeatable 1/1 regression. +- Closed By: backlog close +- Reference: docs/tasks/000254-remove-engine-runtime-reference-accounting.md; docs/tasks/000255-session-local-runtime-reachability.md; docs/engine-component-lifetime.md; doradb-storage/src/session.rs; doradb-storage/src/buffer/mod.rs +- Closed At: 2026-08-09 diff --git a/docs/engine-component-lifetime.md b/docs/engine-component-lifetime.md index 72b789a2..d1067486 100644 --- a/docs/engine-component-lifetime.md +++ b/docs/engine-component-lifetime.md @@ -578,11 +578,28 @@ That provenance rule gives three guarantees: - stable owner identity survives cloning because guards keep the owner alive - page guards and arena state can rely on one exact pool provenance source -`EngineCore` owns one canonical `EnginePools` capability containing the four -typed pool handles and one prebuilt `PoolGuards` bundle. Session-coordinated -operations borrow that bundle through `SessionRuntime`; transaction attachments -do not clone it. `PoolGuards` remains only a named bundle of individually -branded guards and does not weaken the single-owner provenance rule. +`EngineCore` owns one `EnginePools` capability containing the four typed pool +handles and a prebuilt `PoolGuards` bundle for engine-owned work. Each +`SessionState` constructs one fresh bundle from those same handles and +session-coordinated operations borrow it through `SessionRuntime`; transaction +attachments do not clone it. The engine and session bundles carry identical +`PoolIdentity` values but distinct outer `Arc` roots. + +This distinction is intentional. The arena's `QuiescentGuardCount` remains +pool-global and is acquired once for each long-lived root. Page guards clone +the outer `SyncQuiescentGuard` on every retained page access, so allocating one +root per session keeps that high-frequency `Arc` traffic on session-local cache +lines. Moving session paths back to the canonical engine bundle would preserve +provenance and memory safety but reintroduce cross-session refcount contention. +`PoolGuards` remains only a named bundle of individually branded guards and +does not weaken the single-owner provenance rule. + +`BufferPool::create_base_guard()` names the exceptional root-construction +operation explicitly. It is appropriate at engine, session, component +bootstrap, detached-worker, and isolated test-fixture ownership boundaries. +Page access, cache-miss retries, invalidation, CoW mutation, DDL, recovery, and +checkpoint helpers must instead accept an owner-scoped guard and clone it only +when an owned sub-operation can outlive the borrow. ## Arena And Page-Guard Lifetime Rules diff --git a/docs/rfcs/0025-session-coordinated-cancellation-cleanup-ownership.md b/docs/rfcs/0025-session-coordinated-cancellation-cleanup-ownership.md index 115c5421..876ee80f 100644 --- a/docs/rfcs/0025-session-coordinated-cancellation-cleanup-ownership.md +++ b/docs/rfcs/0025-session-coordinated-cancellation-cleanup-ownership.md @@ -119,7 +119,7 @@ RFC-0026. [D5] [D6] [D7] - [B1] `docs/backlogs/closed/000170-session-coordinated-cancellation-cleanup.md` - [B2] `docs/backlogs/closed/000124-statement-execution-cancellation-safety.md` -- [B3] `docs/backlogs/000175-scalable-shared-resource-lifetime-management.md` +- [B3] `docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md` - [B4] `docs/backlogs/000171-exact-family-lock-system-redesign.md` ## Decision @@ -223,7 +223,8 @@ work. Lifecycle events are created only for an actual close or shutdown waiter. Phase 2 improved uncontended statement and transaction boundaries but exposed repeatable contended statement-boundary and stream regressions. The cancellation result was accepted with that fixed overhead recorded as explicit -debt in backlog 000175; detailed samples and flamegraphs remain in task 000247. +debt in backlog 000175; the debt was subsequently resolved, while detailed +original samples and flamegraphs remain in task 000247. [D6] [B3] [U4] ### 5. RFC-0026 owns all post-Phase-2 execution design @@ -319,7 +320,8 @@ numbered phase. [D8] - Implementation Summary: Implemented cancellation-safe public statement ownership, synchronous residual-effect settlement, whole-transaction cleanup, boxed transaction cores, and a reusable public-session core cache; - accepted measured contention debt is tracked by backlog 000175. + accepted measured contention debt was later resolved through backlog + 000175. ### Superseded Remainder @@ -348,8 +350,8 @@ RFC-0026 defines the replacement five-phase runtime-first program. [D7] [U5] 000246 and 000247 or the current code. - Existing transitional state names may remain in the implementation until RFC-0026 migrates them. -- The measured shared-resource lifetime contention remains open in backlog - 000175. +- The measured shared-resource lifetime contention remained open when this RFC + was superseded and was subsequently resolved through backlog 000175. ## Open Questions @@ -362,8 +364,9 @@ deadlock and mutation policy remains separate follow-up work under backlog - Implement the RFC-0026 mandatory runtime and migrate DDL, maintenance, and transaction cleanup through its phases. [D7] -- Remove unnecessary hot-path shared-resource lifetime traffic and reassess - long-lived resource ownership under backlog 000175. [B3] +- Completed after this RFC closed: remove unnecessary hot-path shared-resource + lifetime traffic and document long-lived resource ownership under backlog + 000175. [B3] - Revisit exact-family lock-system policy independently under backlog 000171. [B4] @@ -376,5 +379,5 @@ deadlock and mutation policy remains separate follow-up work under backlog - `docs/tasks/000247-statement-public-transaction-cancellation-ownership.md` - `docs/backlogs/closed/000170-session-coordinated-cancellation-cleanup.md` - `docs/backlogs/closed/000124-statement-execution-cancellation-safety.md` -- `docs/backlogs/000175-scalable-shared-resource-lifetime-management.md` +- `docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md` - `docs/backlogs/000171-exact-family-lock-system-redesign.md` diff --git a/docs/tasks/000102-separate-readonly-pool-interface-and-simplify-ownership.md b/docs/tasks/000102-separate-readonly-pool-interface-and-simplify-ownership.md index 05683310..ab43a211 100644 --- a/docs/tasks/000102-separate-readonly-pool-interface-and-simplify-ownership.md +++ b/docs/tasks/000102-separate-readonly-pool-interface-and-simplify-ownership.md @@ -147,7 +147,7 @@ Reference: - `read_validated_block(&self, guard: &PoolGuard, block_id: PageID, validator: ReadonlyPageValidator) -> Result` - Keep `persisted_file_kind()`, `invalidate_block_id()`, and `invalidate_block_id_strict()` as readonly-specific metadata/invalidation - operations, and keep `pool_guard()` available for outer callers that need + operations, and keep `create_base_guard()` available for outer callers that need an explicit readonly guard source. - Implement the new methods on top of the existing internal load/dedup/validation machinery so task scope stays narrow. @@ -164,7 +164,7 @@ Reference: - readonly page tests in `doradb-storage/src/file/table_file.rs` to use `read_validated_block()` and the returned immutable guard rather than `PageSharedGuard`. - - Thread `PoolGuards::disk_guard()` or `ReadonlyBufferPool::pool_guard()` + - Thread `PoolGuards::disk_guard()` or `ReadonlyBufferPool::create_base_guard()` from outer call sites instead of reacquiring readonly guards inside the read helpers. - Remove readonly-only `BufferPool` imports from modules that no longer need @@ -224,7 +224,7 @@ Reference: - removed `impl BufferPool for ReadonlyBufferPool`; - added `ReadonlyBlockGuard` and readonly-specific `read_block(...)` / `read_validated_block(...)` entrypoints; - - kept `ReadonlyBufferPool::pool_guard()` public, and the final shipped API + - kept `ReadonlyBufferPool::create_base_guard()` public, and the final shipped API requires callers to pass `&PoolGuard` explicitly for readonly reads. 2. Simplified readonly ownership and runtime structure: - `GlobalReadonlyBufferPool` now owns `mappings`, `inflight_loads`, and @@ -241,7 +241,7 @@ Reference: explicit disk-pool guard threading. 4. Implementation review adjusted the final public shape: - readonly reads keep caller-owned guard provenance instead of reacquiring - `global.pool_guard()` internally; + `global.create_base_guard()` internally; - raw `read_block()` is now documented as a narrow COW root/meta-page helper and future expansion is explicitly cautioned; - the resident-hit validation-failure invalidation path keeps synchronous diff --git a/docs/tasks/000247-statement-public-transaction-cancellation-ownership.md b/docs/tasks/000247-statement-public-transaction-cancellation-ownership.md index 42831598..510ee62a 100644 --- a/docs/tasks/000247-statement-public-transaction-cancellation-ownership.md +++ b/docs/tasks/000247-statement-public-transaction-cancellation-ownership.md @@ -762,7 +762,8 @@ statement-boundary and stream-operation regressions. The amendment does not claim that the original successful-path budget passed: it records the fixed boundary cost as performance debt and defers eliminating unnecessary hot-path lifetime traffic, plus the broader shared-resource lifetime design, to -`docs/backlogs/000175-scalable-shared-resource-lifetime-management.md`. +`docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md`, +which subsequently implemented and measured that correction. Phase 3's cancellation ownership prerequisite is unchanged. ### Validation @@ -961,13 +962,12 @@ There are no unresolved implementation choices for this task. The following are explicit follow-ups rather than Phase 2 decisions: -1. Resolution measurements show contended statement-boundary and stream +1. Resolution measurements showed contended statement-boundary and stream operation regressions after local boxed-core work removed allocation and - copy costs. `docs/backlogs/000175-scalable-shared-resource-lifetime-management.md` - owns both the narrow removal of unnecessary session-coordinated hot-path - lifetime-counter traffic and the broader engine/buffer-pool/transaction- - system lifetime design. Do not move the transaction core into the public - handle as an incidental Phase 2 optimization. + copy costs. `docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md` + subsequently resolved the narrow hot-path lifetime traffic and documented + the broader engine/buffer-pool/transaction-system lifetime policy. The + transaction core remains outside the public handle. 2. Phase 3 replaces the private must-complete fallback with reliable whole-DDL/maintenance future ownership and background continuation. 3. A later task may add `DiscardOnly` cleanup only after proving absence of diff --git a/docs/tasks/000248-mandatory-operation-driver-and-concurrent-cleanup-executor.md b/docs/tasks/000248-mandatory-operation-driver-and-concurrent-cleanup-executor.md index c894e214..68493f62 100644 --- a/docs/tasks/000248-mandatory-operation-driver-and-concurrent-cleanup-executor.md +++ b/docs/tasks/000248-mandatory-operation-driver-and-concurrent-cleanup-executor.md @@ -982,8 +982,9 @@ because the first pair was noisy; the repeat still showed a slower candidate median, but the baseline was bimodal and its 19.020 ns IQR overlaps the candidate distribution. No mandatory-runtime access occurs on the public statement/transaction hot path, so this does not establish a deterministic -Phase 1 regression. Existing backlog 000175 continues to own the broader -contended shared-resource lifetime performance work. +Phase 1 regression. Existing backlog 000175 continued to own the broader +contended shared-resource lifetime performance work and subsequently resolved +it through session-local pool roots. ### Validation @@ -1174,8 +1175,8 @@ None within Phase 1. The noisy contended `stmt-noop` comparison does not establish a mandatory-runtime regression, but it also does not close the broader -shared-lifetime contention question. That work remains linked to -`docs/backlogs/000175-scalable-shared-resource-lifetime-management.md`. +shared-lifetime contention question. That work was subsequently resolved in +`docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md`. Future DDL and maintenance tasks must implement `PreparedExecution` with their own operation-specific preparation guards and accepted compensation/panic diff --git a/docs/tasks/000254-remove-engine-runtime-reference-accounting.md b/docs/tasks/000254-remove-engine-runtime-reference-accounting.md index 35f8fac5..1abc5d08 100644 --- a/docs/tasks/000254-remove-engine-runtime-reference-accounting.md +++ b/docs/tasks/000254-remove-engine-runtime-reference-accounting.md @@ -31,7 +31,7 @@ teardown. `- codex` `Source Backlogs:` -`- docs/backlogs/000175-scalable-shared-resource-lifetime-management.md` +`- docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md` `Benchmark Base:` `- 7151941aa9d9b5468adb864a3fdeb068ebcb020a` @@ -49,10 +49,11 @@ joined long-lived workers. Standalone observers were the remaining uncovered class because they intentionally coexist with an active effectful operation and therefore do not consume its slot. -Backlog 000175 recorded the contention evidence and still owns the wider -resource-lifetime investigation. This task delivered only the engine-accounting -slice; it did not redesign ordinary `Arc`, quiescent guards, pools, catalog or -file ownership, or transaction-system shared-resource guards. +Backlog 000175 recorded the contention evidence and, at this task's resolution, +still owned the wider resource-lifetime investigation. This task delivered only +the engine-accounting slice; it did not redesign ordinary `Arc`, quiescent +guards, pools, catalog or file ownership, or transaction-system shared-resource +guards. ## Goals @@ -82,7 +83,8 @@ file ownership, or transaction-system shared-resource guards. order remain unchanged. 6. Persisted formats, `doradb-bench` source, and CI timing policy remain unchanged. -7. Source backlog 000175 remains open for broader lifetime-management work. +7. Source backlog 000175 remained open at task resolution for broader + lifetime-management work. ## Plan @@ -242,9 +244,9 @@ No required row showed a repeatable regression outside baseline dispersion. - Focused line coverage: `engine.rs` 96.74%, `session.rs` 95.61%, combined 95.94% -No parent RFC is linked. Source backlog 000175 remains open intentionally -because its wider shared-resource lifetime investigation is not completed by -this task. +No parent RFC is linked. Source backlog 000175 remained open intentionally at +task resolution because its wider shared-resource lifetime investigation was +not completed by this task. ## Impacts @@ -291,10 +293,10 @@ this task. ## Open Questions -Backlog -[000175](../backlogs/000175-scalable-shared-resource-lifetime-management.md) -remains open for wider resource-lifetime work: ordinary `Arc` upgrades, +At task resolution, backlog +[000175](../backlogs/closed/000175-scalable-shared-resource-lifetime-management.md) +retained wider resource-lifetime work: ordinary `Arc` upgrades, engine-admission traffic, quiescent and component counters, and frequently -cloned pool, catalog, file, and transaction-system guards. Any architecture -change across those subsystems requires separate measured planning and may -require an RFC. +cloned pool, catalog, file, and transaction-system guards. Task 000255 and the +later pool-root audit subsequently completed that measured follow-up and closed +the backlog with a hybrid centralized, sharded, and retained-counting policy. diff --git a/docs/tasks/000255-session-local-runtime-reachability.md b/docs/tasks/000255-session-local-runtime-reachability.md index dc1b0476..3e0b4348 100644 --- a/docs/tasks/000255-session-local-runtime-reachability.md +++ b/docs/tasks/000255-session-local-runtime-reachability.md @@ -19,10 +19,12 @@ operation directly on the pinned state. Introduced one engine-owned `EngineCore` for shared runtime capabilities and a strong `SessionRuntime` wrapper around the upgraded state. Operation pins, observers, transaction attachments, mandatory work, and cleanup handoffs carry -that runtime instead of `EngineRef`. The final implementation removes -`EngineRef` and `WeakEngineRef` from storage source, removes normal -session-registry lookup from statement checkout, and borrows one canonical -pool-guard bundle. +that runtime instead of `EngineRef`. The task implementation removed +`EngineRef` and `WeakEngineRef` from storage source, removed normal +session-registry lookup from statement checkout, and at that revision borrowed +one canonical pool-guard bundle. The later backlog 000175 follow-up replaced +that bundle with one independent root bundle per session after proving the +canonical roots caused cross-session refcount contention. Lifecycle admission, shutdown authority, exact identity validation, public APIs, persisted formats, and component teardown order remain unchanged. @@ -35,7 +37,7 @@ APIs, persisted formats, and component teardown order remain unchanged. `- codex` `Source Backlogs:` -`- docs/backlogs/000175-scalable-shared-resource-lifetime-management.md` +`- docs/backlogs/closed/000175-scalable-shared-resource-lifetime-management.md` `Related Tasks:` `- docs/tasks/000247-statement-public-transaction-cancellation-ownership.md` @@ -57,9 +59,9 @@ redundant once the strong session runtime could reach immutable engine capabilities through the state. This was a bounded ownership-path refactor with no parent RFC. Backlog 000175 -remains open because it covers broader lifecycle admission, component guard, -buffer-page ownership, and shared-counter questions that this task did not -resolve. +remained open at task resolution because it covered broader lifecycle +admission, component guard, buffer-page ownership, and shared-counter questions +that this task did not resolve. ## Goals @@ -91,8 +93,8 @@ resolve. redo, undo, and persisted formats were not changed. 5. Public APIs, storage semantics, benchmark workloads, and CI timing policy were not changed. -6. The contended `index-stream` regression was investigated but not fixed; - backlog 000175 remains open for that and wider lifetime work. +6. The contended `index-stream` regression was investigated but not fixed in + this task; backlog 000175 remained open for that and wider lifetime work. ## Plan @@ -238,9 +240,9 @@ backlog 000175 rather than broadening this task without a root cause. 95.84% (9,757/10,180): 97.21%, 95.96%, and 95.14% respectively. - `rtk git diff --check`: passed. -No parent RFC synchronization is required. Source backlog 000175 remains open -intentionally because only its session-coordinated reachability slice was -implemented. +No parent RFC synchronization was required. Source backlog 000175 remained open +intentionally at task resolution because only its session-coordinated +reachability slice was implemented. ## Impacts @@ -250,14 +252,15 @@ implemented. retains owner orchestration. - Session-coordinated runtime owners carry `SessionRuntime`, while single-capability helpers use narrow guards. -- Pool guards are built once per engine and borrowed by session-coordinated - work. +- At this task revision, pool guards were built once per engine and borrowed by + session-coordinated work. The backlog follow-up later moved base roots back to + one bundle per session while retaining the same pool identities. - Shutdown, terminal cleanup, poison handling, and exact identity error classifications remain compatible. - Public APIs, dependencies, configuration, persisted formats, recovery protocols, benchmark CLI, and CI policy are unchanged. -- The contended index-stream finding is a documented performance risk owned by - backlog 000175. +- At task resolution, the contended index-stream finding remained a documented + performance risk owned by backlog 000175. ## Test Cases @@ -291,10 +294,9 @@ implemented. No task-scoped design question remains. -Backlog -[000175](../backlogs/000175-scalable-shared-resource-lifetime-management.md) -retains the unresolved engine-global admission cache line, broader -quiescent/component guard ownership, buffer-page reference-count contention, -and the measured choice between centralized, sharded, or retained counting. -Its future work should first reproduce the index-stream profile independently -and establish causality before changing index or buffer ownership. +At task resolution, backlog +[000175](../backlogs/closed/000175-scalable-shared-resource-lifetime-management.md) +retained the unresolved broader guard-ownership and buffer-page refcount work. +The follow-up reproduced the profile, isolated canonical `PoolGuard` roots as +the cause, restored per-session roots, audited base-root construction, and +closed with measured 1/1 neutrality and recovered 4/16 stream performance. diff --git a/docs/transaction-system.md b/docs/transaction-system.md index c3e9b92e..52f6e30e 100644 --- a/docs/transaction-system.md +++ b/docs/transaction-system.md @@ -246,11 +246,12 @@ instead owns one checkout continuously from direct construction through terminal conversion or synchronous panic parking. The checkout owns a `TrxAttachment` containing the exact `SessionRuntime` and exposes a copyable `TrxRuntime` value that pairs immutable `TrxContext` with -borrowed access to `EngineCore`, its canonical pool guards, and the -session-local user-table cache. `TrxContext` never -stores the attachment. Normal statement checkout/check-in does not reacquire -the session lifecycle mutex, allocate, touch the operation change notifier, or -send cleanup work. +borrowed access to `EngineCore`, the exact session's pool-guard roots, and the +session-local user-table cache. Those roots are created once with +`SessionState`; checkout and attachment accessors only borrow them. +`TrxContext` never stores the attachment. Normal statement checkout/check-in +does not reacquire the session lifecycle mutex, allocate, touch the operation +change notifier, or send cleanup work. Each session user-table cache entry contains one weak `Table` runtime hint and an optional `VersionedPageID`. The weak runtime is never authoritative for diff --git a/doradb-storage/src/buffer/arena.rs b/doradb-storage/src/buffer/arena.rs index 103c7c6c..0b60488d 100644 --- a/doradb-storage/src/buffer/arena.rs +++ b/doradb-storage/src/buffer/arena.rs @@ -184,9 +184,14 @@ impl QuiescentArena { self.keepalive.guard() } - /// Returns a cloneable pool guard for accesses into this arena. + /// Creates one independent clone root for accesses into this arena. + /// + /// Each call acquires the arena's pool-global quiescent counter once and + /// wraps that direct guard in a new `Arc`. Callers must keep the resulting + /// base guard at a natural ownership boundary and clone it for individual + /// page or task lifetimes. #[inline] - pub(crate) fn guard(&self) -> PoolGuard { + pub(crate) fn create_base_guard(&self) -> PoolGuard { PoolGuard::new(self.identity, self.quiescent_guard().into_sync()) } @@ -256,16 +261,48 @@ impl QuiescentArena { } } +#[cfg(test)] +pub(crate) use self::tests::outstanding_base_guard_count; + #[cfg(test)] mod tests { use super::*; + use crate::buffer::test_pool_guards_share_keepalive_root; + + /// Returns the number of independently created base guards. + #[inline] + pub(crate) fn outstanding_base_guard_count(arena: &QuiescentArena) -> usize { + arena.keepalive.outstanding_guard_count() + } #[test] #[should_panic(expected = "pool guard identity mismatch")] fn test_arena_guard_panics_on_foreign_guard() { let arena1 = Box::leak(Box::new(QuiescentArena::new(1).unwrap())); let arena2 = Box::leak(Box::new(QuiescentArena::new(1).unwrap())); - let foreign_guard = arena2.guard(); + let foreign_guard = arena2.create_base_guard(); let _ = arena1.arena_guard(foreign_guard); } + + #[test] + fn test_base_guard_creation_and_clone_lifecycle() { + let arena = QuiescentArena::new(1).unwrap(); + assert_eq!(outstanding_base_guard_count(&arena), 0); + + let first = arena.create_base_guard(); + let first_clone = first.clone(); + assert_eq!(outstanding_base_guard_count(&arena), 1); + assert!(test_pool_guards_share_keepalive_root(&first, &first_clone)); + + let second = arena.create_base_guard(); + assert_eq!(outstanding_base_guard_count(&arena), 2); + assert!(!test_pool_guards_share_keepalive_root(&first, &second)); + assert_eq!(first.identity(), second.identity()); + + drop(first); + drop(first_clone); + assert_eq!(outstanding_base_guard_count(&arena), 1); + drop(second); + assert_eq!(outstanding_base_guard_count(&arena), 0); + } } diff --git a/doradb-storage/src/buffer/evict.rs b/doradb-storage/src/buffer/evict.rs index fb4b3a30..c948fd89 100644 --- a/doradb-storage/src/buffer/evict.rs +++ b/doradb-storage/src/buffer/evict.rs @@ -379,8 +379,11 @@ impl EvictableBufferPool { ) -> (EvictableRuntime, PressureDeltaClockPolicy) { let policy = PressureDeltaClockPolicy::new(pool.in_mem.eviction_arbiter, MIN_IN_MEM_PAGES / 2); + // The evictor is detached engine-owned work with no session root to + // borrow. Create exactly one base root for the worker runtime and let + // every eviction page guard clone that root. let runtime = EvictableRuntime { - arena: pool.arena.arena_guard(pool.pool_guard()), + arena: pool.arena.arena_guard(pool.create_base_guard()), pool, }; (runtime, policy) @@ -459,8 +462,8 @@ impl BufferPool for EvictableBufferPool { } #[inline] - fn pool_guard(&self) -> PoolGuard { - self.arena.guard() + fn create_base_guard(&self) -> PoolGuard { + self.arena.create_base_guard() } #[inline] @@ -1921,7 +1924,7 @@ pub(crate) mod tests { pool: QuiescentGuard, page_count: usize, ) -> event_listener::EventListener { - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let mut page_guards = Vec::with_capacity(page_count); for idx in 0..page_count { let mut page_guard = pool @@ -1934,7 +1937,7 @@ pub(crate) mod tests { page_guards.push(page_guard); } let runtime = EvictableRuntime { - arena: pool.arena.arena_guard(pool.pool_guard()), + arena: pool.arena.arena_guard(pool.create_base_guard()), pool: pool.into_sync(), }; runtime.dispatch_io_writes(page_guards) @@ -1944,7 +1947,7 @@ pub(crate) mod tests { pool: QuiescentGuard, payload: &[u8], ) -> PageID { - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let mut page_guard = pool .allocate_page::(&pool_guard) .await @@ -1954,7 +1957,7 @@ pub(crate) mod tests { page_guard.bf_mut().set_dirty(true); page_guard.bf_mut().set_kind(FrameKind::Evicting); let runtime = EvictableRuntime { - arena: pool.arena.arena_guard(pool.pool_guard()), + arena: pool.arena.arena_guard(pool.create_base_guard()), pool: pool.clone().into_sync(), }; runtime.dispatch_io_writes(vec![page_guard]).await; @@ -2259,7 +2262,7 @@ pub(crate) mod tests { .max_file_size(128u64 * 1024 * 1024), ) .unwrap(); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); pool.signal_shutdown(); let err = match pool.allocate_page::(&pool_guard).await { @@ -2313,7 +2316,7 @@ pub(crate) mod tests { let temp_dir = TempDir::new().unwrap(); let (_fs_owner, owner, mut state_machine) = build_state_machine_for_test(temp_dir.path().join("data.swp")); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let err = backend_failure_for_test(); let read_page_id = make_evicted_reload_target_for_test(&owner, &pool_guard).await; @@ -2342,7 +2345,7 @@ pub(crate) mod tests { let temp_dir = TempDir::new().unwrap(); let (_fs_owner, owner, mut state_machine) = build_state_machine_for_test(temp_dir.path().join("data.swp")); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let err = backend_failure_for_test(); let read_page_id = make_evicted_reload_target_for_test(&owner, &pool_guard).await; @@ -2373,7 +2376,7 @@ pub(crate) mod tests { let temp_dir = TempDir::new().unwrap(); let (_fs_owner, owner, mut state_machine) = build_state_machine_for_test(temp_dir.path().join("data.swp")); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let err = backend_failure_for_test(); let read_page_id = make_evicted_reload_target_for_test(&owner, &pool_guard).await; @@ -2418,7 +2421,7 @@ pub(crate) mod tests { .max_mem_size(1024u64 * 1024 * 128) .max_file_size(1024u64 * 1024 * 256), ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); { let g = pool .allocate_page::(&pool_guard) @@ -2562,7 +2565,7 @@ pub(crate) mod tests { .max_mem_size(1024u64 * 1024 * 128) .max_file_size(1024u64 * 1024 * 256), ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let page_guard = pool .allocate_page::(&pool_guard) .await @@ -2592,7 +2595,7 @@ pub(crate) mod tests { .max_mem_size(1024u64 * 1024 * 128) .max_file_size(1024u64 * 1024 * 256), ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let g = pool .allocate_page::(&pool_guard) .await @@ -2669,7 +2672,7 @@ pub(crate) mod tests { .max_file_size(128u64 * 1024 * 1024), ) .unwrap(); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let mut page_guard = pool .allocate_page::(&pool_guard) @@ -2711,7 +2714,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let mut state_machine = EvictablePoolStateMachine { pool: owner.guard().into_sync(), file: storage, @@ -2766,7 +2769,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let mut page_guard = owner .allocate_page::(&pool_guard) .await @@ -2825,7 +2828,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let mut page_guard = owner .allocate_page::(&pool_guard) .await @@ -2836,7 +2839,7 @@ pub(crate) mod tests { page_guard.bf_mut().set_kind(FrameKind::Evicting); let baseline = owner.stats(); let runtime = EvictableRuntime { - arena: owner.arena.arena_guard(owner.pool_guard()), + arena: owner.arena.arena_guard(owner.create_base_guard()), pool: owner.guard().into_sync(), }; @@ -2861,7 +2864,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let sync_pool = owner.guard().into_sync(); let mut state_machine = EvictablePoolStateMachine { pool: sync_pool, @@ -2943,7 +2946,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let mut state_machine = EvictablePoolStateMachine { pool: owner.guard().into_sync(), file: storage, @@ -2993,7 +2996,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let mut state_machine = EvictablePoolStateMachine { pool: owner.guard().into_sync(), file: storage, @@ -3052,7 +3055,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let mut page_guard = owner .allocate_page::(&pool_guard) @@ -3063,7 +3066,7 @@ pub(crate) mod tests { page_guard.bf_mut().set_kind(FrameKind::Evicting); let runtime = EvictableRuntime { - arena: owner.arena.arena_guard(owner.pool_guard()), + arena: owner.arena.arena_guard(owner.create_base_guard()), pool: owner.guard().into_sync(), }; @@ -3106,7 +3109,7 @@ pub(crate) mod tests { ) .unwrap(); let owner = QuiescentBox::new(pool); - let pool_guard = owner.pool_guard(); + let pool_guard = owner.create_base_guard(); let mut page_guard = owner .allocate_page::(&pool_guard) .await @@ -3193,7 +3196,7 @@ pub(crate) mod tests { .max_file_size(128u64 * 1024 * 1024), ) .unwrap(); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let mut page_guard = pool .allocate_page::(&pool_guard) .await @@ -3238,7 +3241,7 @@ pub(crate) mod tests { .max_file_size(128u64 * 1024 * 130), ); let pool_ref = pool.owner_guard(); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let (tx, rx) = flume::unbounded(); let handle1 = { @@ -3290,7 +3293,7 @@ pub(crate) mod tests { .max_mem_size(1024u64 * 1024 * 64) .max_file_size(1024u64 * 1024 * 128), ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); println!( "max_nbr={}, max_nbr_in_mem={}", @@ -3321,7 +3324,7 @@ pub(crate) mod tests { .max_mem_size(64u64 * 1024 * 130) .max_file_size(128u64 * 1024 * 130), ); - let pool_guard = EvictableBufferPool::pool_guard(&pool); + let pool_guard = EvictableBufferPool::create_base_guard(&pool); let total_pages = pool.in_mem.max_count + 64; for i in 0..total_pages { @@ -3364,7 +3367,7 @@ pub(crate) mod tests { .unwrap(); let pool = QuiescentBox::new(pool); let guard = { - let pool_guard = EvictableBufferPool::pool_guard(&pool); + let pool_guard = EvictableBufferPool::create_base_guard(&pool); pool.allocate_page::(&pool_guard) .await .expect("test page allocation should succeed") @@ -3399,7 +3402,7 @@ pub(crate) mod tests { .max_mem_size(64u64 * 1024 * 1024) .max_file_size(64u64 * 1024 * 2048), ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); println!( "max_nbr={}, max_nbr_in_mem={}", @@ -3487,7 +3490,7 @@ pub(crate) mod tests { .dynamic_batch_bounds(3, 3), ), ); - let _pool_guard = pool.pool_guard(); + let _pool_guard = pool.create_base_guard(); let arbiter = pool.in_mem.eviction_arbiter; assert_eq!(arbiter.target_free(), 2); @@ -3531,8 +3534,8 @@ pub(crate) mod tests { .max_mem_size(1024u64 * 1024 * 32) .max_file_size(1024u64 * 1024 * 64), ); - let pool1_guard = pool1.pool_guard(); - let pool2_guard = pool2.pool_guard(); + let pool1_guard = pool1.create_base_guard(); + let pool2_guard = pool2.create_base_guard(); let page = pool1 .allocate_page::(&pool1_guard) diff --git a/doradb-storage/src/buffer/evictor.rs b/doradb-storage/src/buffer/evictor.rs index 71328294..743262d4 100644 --- a/doradb-storage/src/buffer/evictor.rs +++ b/doradb-storage/src/buffer/evictor.rs @@ -1235,8 +1235,12 @@ mod tests { ) { let mut buf = DirectBuf::zeroed(COW_FILE_PAGE_SIZE); buf.as_bytes_mut()[..payload.len()].copy_from_slice(payload); - let mutable = - MutableTableFile::fork(table_file, fs.background_writes(), readonly_pool.clone()); + let mutable = MutableTableFile::fork( + table_file, + fs.background_writes(), + readonly_pool.clone(), + readonly_pool.create_base_guard(), + ); mutable.write_block(block_id, buf).await.unwrap(); drop(mutable); } @@ -1333,7 +1337,7 @@ mod tests { } async fn allocate_with_pressure(pool: &EvictableBufferPool, total_pages: usize) { - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); for _ in 0..total_pages { let page = pool .allocate_page::(&pool_guard) @@ -1363,7 +1367,11 @@ mod tests { write_payload(fs, &table_file, &pool, block_id, payload.as_bytes()).await; } - let file = fs.open_table_file(table_id, pool.clone()).await.unwrap(); + let disk_guard = pool.create_base_guard(); + let file = fs + .open_table_file(table_id, pool.clone(), &disk_guard) + .await + .unwrap(); ReadonlyPressureFixture { file, pool, @@ -1374,7 +1382,7 @@ mod tests { async fn drive_read_pressure(fixture: &ReadonlyPressureFixture) { let pool = &fixture.pool; - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); for i in 0..fixture.block_count { let block_id = fixture.base_block_id + i as u64; let g = pool diff --git a/doradb-storage/src/buffer/fixed.rs b/doradb-storage/src/buffer/fixed.rs index c140cfe5..46158574 100644 --- a/doradb-storage/src/buffer/fixed.rs +++ b/doradb-storage/src/buffer/fixed.rs @@ -155,8 +155,8 @@ impl BufferPool for FixedBufferPool { } #[inline] - fn pool_guard(&self) -> PoolGuard { - self.arena.guard() + fn create_base_guard(&self) -> PoolGuard { + self.arena.create_base_guard() } // allocate a new page with exclusive lock. @@ -314,7 +314,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Meta, pool_bytes).unwrap(), ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let page = pool .allocate_page::(&pool_guard) .await @@ -344,7 +344,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Meta, pool_bytes).unwrap(), ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let page_id = test_page_id(0); let page = pool .allocate_page_at::(&pool_guard, page_id) @@ -415,7 +415,7 @@ mod tests { fn test_fixed_buffer_pool() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); { let g = pool .allocate_page::(&pool_guard) @@ -564,7 +564,7 @@ mod tests { fn test_fixed_buffer_pool_stats_track_resident_hits_only() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let page = pool .allocate_page::(&pool_guard) .await @@ -592,7 +592,7 @@ mod tests { fn test_facade_page_guard_lock_shared_and_try_into_shared() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let g = pool .allocate_page::(&pool_guard) .await @@ -665,7 +665,7 @@ mod tests { fn test_facade_page_guard_lock_exclusive_and_try_into_exclusive() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let g = pool .allocate_page::(&pool_guard) .await @@ -742,7 +742,7 @@ mod tests { fn test_facade_page_guard_try_upgrades_reject_generation_mismatch() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let (mut g, _, _) = stale_facade_guard(&pool, &pool_guard).await; assert!(g.try_shared().is_invalid()); @@ -762,7 +762,7 @@ mod tests { fn test_facade_page_guard_verify_upgrades_reject_generation_mismatch() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let (g, _, _) = stale_facade_guard(&pool, &pool_guard).await; assert!(g.verify_shared_async::().await.is_invalid()); @@ -777,7 +777,7 @@ mod tests { fn test_page_optimistic_guard_checked_upgrades_reject_generation_mismatch() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let page_id = allocate_test_row_page(&pool, &pool_guard).await; let g = pool @@ -817,7 +817,7 @@ mod tests { fn test_facade_page_guard_lock_exclusive_async_panics_on_shared_state() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let g = pool .allocate_page::(&pool_guard) .await @@ -843,7 +843,7 @@ mod tests { fn test_facade_page_guard_lock_shared_async_panics_on_exclusive_state() { smol::block_on(async { let pool = test_pool(); - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); let g = pool .allocate_page::(&pool_guard) .await @@ -871,7 +871,7 @@ mod tests { FixedBufferPool::with_capacity(PoolRole::Meta, 8 * 1024 * 1024).unwrap(), ); let guard = { - let pool_guard = FixedBufferPool::pool_guard(&pool); + let pool_guard = FixedBufferPool::create_base_guard(&pool); pool.allocate_page::(&pool_guard) .await .expect("test page allocation should succeed") @@ -900,8 +900,8 @@ mod tests { smol::block_on(async { let pool1 = test_pool(); let pool2 = test_pool(); - let pool1_guard = FixedBufferPool::pool_guard(&pool1); - let pool2_guard = FixedBufferPool::pool_guard(&pool2); + let pool1_guard = FixedBufferPool::create_base_guard(&pool1); + let pool2_guard = FixedBufferPool::create_base_guard(&pool2); let page = pool1 .allocate_page::(&pool1_guard) diff --git a/doradb-storage/src/buffer/guard.rs b/doradb-storage/src/buffer/guard.rs index 79307a23..356e308a 100644 --- a/doradb-storage/src/buffer/guard.rs +++ b/doradb-storage/src/buffer/guard.rs @@ -136,6 +136,9 @@ pub(crate) struct PageLatchGuard { // also bind `keepalive` before `raw` so reverse local-drop order preserves // the same guarantee across async suspension or future refactors. raw: RawHybridGuard, + // Callers clone one long-lived `PoolGuard` root into this field. That clone + // updates the root's outer `Arc`, so high-frequency session paths must use + // the session-local roots rather than `EnginePools`' canonical bundle. keepalive: PoolGuard, } diff --git a/doradb-storage/src/buffer/mod.rs b/doradb-storage/src/buffer/mod.rs index c2c41986..85bd62d7 100644 --- a/doradb-storage/src/buffer/mod.rs +++ b/doradb-storage/src/buffer/mod.rs @@ -11,6 +11,8 @@ mod pool_guard; mod readonly; mod util; +#[cfg(test)] +pub(crate) use self::arena::outstanding_base_guard_count as test_outstanding_base_guard_count; #[cfg(test)] pub(crate) use self::evict::tests::{ dispatch_dirty_pages_for_test as test_dispatch_dirty_pages, frame_kind as test_frame_kind, @@ -18,6 +20,8 @@ pub(crate) use self::evict::tests::{ persist_and_evict_page_for_test as test_persist_and_evict_page, }; #[cfg(test)] +pub(crate) use self::pool_guard::shares_keepalive_root as test_pool_guards_share_keepalive_root; +#[cfg(test)] pub(crate) use self::readonly::tests::{global_readonly_pool_scope, table_readonly_pool}; #[cfg(test)] pub(crate) use self::tests::test_page_id; @@ -198,8 +202,14 @@ pub(crate) trait BufferPool: Send + Sync { /// Returns the number of allocated pages. fn allocated(&self) -> usize; - /// Returns a cloneable keepalive guard for this pool. - fn pool_guard(&self) -> PoolGuard; + /// Creates a new clone root for keeping this pool alive. + /// + /// This is a lifecycle-boundary operation, not a cheap accessor. Every + /// call acquires one pool-global quiescent keepalive and allocates a fresh + /// outer `Arc` root. Ordinary storage work should receive an existing + /// owner- or session-scoped guard and clone that root instead of calling + /// this method per request, retry, or page access. + fn create_base_guard(&self) -> PoolGuard; /// Allocate a new page. /// diff --git a/doradb-storage/src/buffer/pool_guard.rs b/doradb-storage/src/buffer/pool_guard.rs index 35e39654..d79483b7 100644 --- a/doradb-storage/src/buffer/pool_guard.rs +++ b/doradb-storage/src/buffer/pool_guard.rs @@ -2,6 +2,11 @@ use crate::buffer::identity::{PoolIdentity, PoolRole, RowPoolRole}; use crate::quiescent::SyncQuiescentGuard; /// Cloneable keepalive guard branded with one exact buffer-pool identity. +/// +/// Page latch guards clone this value for every retained page access. Clones +/// from one root update the same outer `Arc` strong count, while separately +/// constructed guards retain the same pool through independent roots. Root +/// construction scope is therefore part of the buffer hot-path design. #[derive(Clone)] pub struct PoolGuard { identity: PoolIdentity, @@ -190,12 +195,21 @@ fn pool_guard_identity_mismatch( panic!("pool guard identity mismatch in {context}: expected {expected:?}, got {actual:?}"); } +#[cfg(test)] +pub(crate) use self::tests::shares_keepalive_root; + #[cfg(test)] mod tests { use super::*; use crate::quiescent::QuiescentBox; use std::panic::catch_unwind; + /// Returns whether two guards update the same keepalive `Arc` root. + #[inline] + pub(crate) fn shares_keepalive_root(first: &PoolGuard, second: &PoolGuard) -> bool { + crate::quiescent::test_sync_guards_share_root(&first._keepalive, &second._keepalive) + } + fn test_guard() -> PoolGuard { let owner = Box::leak(Box::new(QuiescentBox::new(()))); PoolGuard::new(owner.owner_identity(), owner.guard().into_sync()) diff --git a/doradb-storage/src/buffer/readonly.rs b/doradb-storage/src/buffer/readonly.rs index ccc9cdb6..14eb5f95 100644 --- a/doradb-storage/src/buffer/readonly.rs +++ b/doradb-storage/src/buffer/readonly.rs @@ -188,11 +188,15 @@ impl ReadonlyBufferPool { self.stats.snapshot() } - /// Returns a cloneable pool guard for this readonly pool. + /// Creates a new clone root for keeping this readonly pool alive. + /// + /// This has the same lifecycle-boundary semantics as + /// [`BufferPool::create_base_guard`]: every call acquires a pool-global + /// quiescent keepalive and allocates a fresh outer `Arc` root. #[inline] - pub(crate) fn pool_guard(&self) -> PoolGuard { + pub(crate) fn create_base_guard(&self) -> PoolGuard { debug_assert!(!matches!(self.role, PoolRole::Invalid)); - self.arena.guard() + self.arena.create_base_guard() } /// Returns the runtime identity of this pool instance. @@ -295,19 +299,25 @@ impl ReadonlyBufferPool { /// Invalidates a specific cache key and returns its old frame id. #[inline] - fn invalidate_key(&self, key: &BlockKey) -> Option { + fn invalidate_key(&self, guard: &PoolGuard, key: &BlockKey) -> Option { + self.validate_guard(guard); let frame_id = match self.mappings.remove(key) { Some((_, frame_id)) => frame_id, None => return None, }; - self.invalidate_frame_retry(frame_id, Some(*key)); + self.invalidate_frame_retry(guard, frame_id, Some(*key)); Some(frame_id) } /// Invalidates one physical block from one file. #[inline] - pub(crate) fn invalidate_block(&self, file_id: FileID, block_id: BlockID) -> Option { - self.invalidate_key(&BlockKey::new(file_id, block_id)) + pub(crate) fn invalidate_block( + &self, + guard: &PoolGuard, + file_id: FileID, + block_id: BlockID, + ) -> Option { + self.invalidate_key(guard, &BlockKey::new(file_id, block_id)) } #[inline] @@ -315,8 +325,11 @@ impl ReadonlyBufferPool { pool: SyncQuiescentGuard, ) -> (ReadonlyRuntime, PressureDeltaClockPolicy) { let policy = PressureDeltaClockPolicy::new(pool.eviction_arbiter, 1); + // The evictor is detached engine-owned work with no session root to + // borrow. Create exactly one base root for the worker runtime and let + // every eviction page guard clone that root. let runtime = ReadonlyRuntime { - arena: pool.arena.arena_guard(pool.pool_guard()), + arena: pool.arena.arena_guard(pool.create_base_guard()), pool, }; (runtime, policy) @@ -381,10 +394,15 @@ impl ReadonlyBufferPool { } #[inline] - fn invalidate_frame_retry(&self, frame_id: PageID, expected_key: Option) { + fn invalidate_frame_retry( + &self, + guard: &PoolGuard, + frame_id: PageID, + expected_key: Option, + ) { + self.validate_guard(guard); loop { - let guard = self.pool_guard(); - if let Some(page_guard) = self.try_lock_page_exclusive(&guard, frame_id) { + if let Some(page_guard) = self.try_lock_page_exclusive(guard, frame_id) { self.invalidate_frame_with_guard(page_guard, expected_key); let _ = self.residency.move_resident_to_free(frame_id); return; @@ -514,6 +532,7 @@ impl QuiescentGuard { async fn join_or_start_inflight_load( &self, file: &Arc, + guard: &PoolGuard, key: BlockKey, validation: Option, ) -> InternalResult> { @@ -543,7 +562,11 @@ impl QuiescentGuard { }; // The entry guard above is dropped before we reserve a frame or inspect // resident mappings, avoiding nested locks across the two DashMaps. - let task_arena = self.arena.arena_guard(self.pool_guard()); + // The reservation and submitted IO may outlive this caller, so clone + // the caller's root into the task-owned arena guard. Creating a new + // base root here would touch the pool-global quiescent counter on every + // cache miss and would discard the caller's session sharding domain. + let task_arena = self.arena.arena_guard(guard.clone()); match ReadonlyPageReservation::reserve_page(self, task_arena).await { Ok((frame_id, page_guard)) => { let reservation = ReadonlyPageReservation::from_reserved_page( @@ -592,13 +615,14 @@ impl QuiescentGuard { async fn get_or_load_frame_id( &self, file: &Arc, + guard: &PoolGuard, key: BlockKey, ) -> RuntimeResult<(PageID, bool)> { if let Some(frame_id) = self.try_get_frame_id(&key) { return Ok((frame_id, true)); } let inflight = self - .join_or_start_inflight_load(file, key, None) + .join_or_start_inflight_load(file, guard, key, None) .await .change_context(RuntimeError::BufferPageAccess)?; if let Some(frame_id) = self.try_get_frame_id(&key) { @@ -620,6 +644,7 @@ impl QuiescentGuard { &self, file_kind: FileKind, file: &Arc, + guard: &PoolGuard, key: BlockKey, validator: ReadonlyBlockValidator, ) -> RuntimeResult<(PageID, bool)> { @@ -629,6 +654,7 @@ impl QuiescentGuard { let inflight = self .join_or_start_inflight_load( file, + guard, key, Some(InflightLoadValidation { file_kind, @@ -667,19 +693,19 @@ impl QuiescentGuard { loop { let (frame_id, resident_hit) = match validation { Some(validator) => { - self.get_or_load_frame_id_validated(file_kind, file, key, validator) + self.get_or_load_frame_id_validated(file_kind, file, guard, key, validator) .await? } - None => self.get_or_load_frame_id(file, key).await?, + None => self.get_or_load_frame_id(file, guard, key).await?, }; - let guard = self + let page_guard = self .get_page_internal::(guard, frame_id, LatchFallbackMode::Shared) .await; - if !self.validate_guarded_frame_key(&guard, key) { + if !self.validate_guarded_frame_key(&page_guard, key) { self.invalidate_stale_mapping_if_same_frame(key, frame_id); continue; } - if let Some(shared) = guard.lock_shared_async().await { + if let Some(shared) = page_guard.lock_shared_async().await { let block = ReadonlyBlockGuard::new(block_id, shared); if let Some(validator) = validation && let Err(err) = validator(block.page(), file_kind, block_id) @@ -693,7 +719,7 @@ impl QuiescentGuard { // shared guard before invalidation so the retry loop does // not contend with our own latch; revisit this synchronous // path if raw readonly usage expands in the future. - let _ = self.invalidate_block(file.file_id(), block_id); + let _ = self.invalidate_block(guard, file.file_id(), block_id); return Err(err.change_context(RuntimeError::BufferPageAccess)); } if resident_hit { @@ -1256,12 +1282,17 @@ impl ReadonlyBlockGuard { /// It installs a per-key write-blocked state before invalidating resident /// mappings, so new misses cannot load old disk bytes while the physical block /// write is queued or in flight. +/// +/// The caller supplies its existing guard root. This path can run once per CoW +/// block write and must not create an independent base root for invalidation. #[inline] pub(crate) fn begin_write_barrier( pool: QuiescentGuard, + guard: &PoolGuard, file_id: FileID, block_id: BlockID, ) -> InternalResult { + pool.validate_guard(guard); let key = BlockKey::new(file_id, block_id); { // Keep the inflight entry guard scoped to this block; invalidation below @@ -1285,7 +1316,7 @@ pub(crate) fn begin_write_barrier( }, } } - let _ = pool.invalidate_key(&key); + let _ = pool.invalidate_key(guard, &key); Ok(ReadonlyWriteLease::new(pool, key)) } @@ -1293,7 +1324,7 @@ pub(crate) fn begin_write_barrier( pub(crate) mod tests { use super::*; use crate::buffer::page::Page; - use crate::buffer::test_page_id; + use crate::buffer::{test_outstanding_base_guard_count, test_page_id}; use crate::catalog::{ColumnAttributes, ColumnSpec, TableMetadata, USER_TABLE_ID_START}; use crate::conf::{EngineConfig, EvictableBufferPoolConfig, FileSystemConfig, TrxSysConfig}; use crate::engine::Engine; @@ -1449,7 +1480,7 @@ pub(crate) mod tests { async fn publish_test_frame(global: &GlobalReadOnlyPoolScope, key: BlockKey) -> PageID { let pool = global.guard(); - let task_arena = pool.arena.arena_guard(pool.pool_guard()); + let task_arena = pool.arena.arena_guard(pool.create_base_guard()); let (frame_id, page_guard) = ReadonlyPageReservation::reserve_page(&pool, task_arena) .await .unwrap(); @@ -1485,8 +1516,8 @@ pub(crate) mod tests { } #[inline] - pub(crate) fn pool_guard(&self) -> PoolGuard { - self.global.pool_guard() + pub(crate) fn create_base_guard(&self) -> PoolGuard { + self.global.create_base_guard() } #[inline] @@ -1588,8 +1619,12 @@ pub(crate) mod tests { let mut buf = DirectBuf::zeroed(COW_FILE_PAGE_SIZE); let bytes = buf.as_bytes_mut(); bytes[..payload.len()].copy_from_slice(payload); - let mutable = - MutableTableFile::fork(table_file, fs.background_writes(), readonly_pool.clone()); + let mutable = MutableTableFile::fork( + table_file, + fs.background_writes(), + readonly_pool.clone(), + readonly_pool.create_base_guard(), + ); mutable.write_block(page_id, buf).await.unwrap(); drop(mutable); } @@ -1608,6 +1643,7 @@ pub(crate) mod tests { table_file, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); mutable.write_block(page_id, buf).await.unwrap(); drop(mutable); @@ -1625,7 +1661,7 @@ pub(crate) mod tests { let scope = global_readonly_pool_scope(frame_page_bytes(4)); let pool = table_readonly_pool(&scope, test_user_table_id(120), &table_file); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let cold_start = pool.global_stats(); let cold_guard = pool @@ -1680,7 +1716,7 @@ pub(crate) mod tests { let scope = global_readonly_pool_scope(frame_page_bytes(4)); let pool_a = table_readonly_pool(&scope, test_user_table_id(121), &table_file_a); let pool_b = table_readonly_pool(&scope, test_user_table_id(122), &table_file_b); - let pool_a_guard = pool_a.pool_guard(); + let pool_a_guard = pool_a.create_base_guard(); let start_a = pool_a.global_stats(); let start_b = pool_b.global_stats(); @@ -1942,6 +1978,7 @@ pub(crate) mod tests { fn test_global_readonly_mapping_and_invalidation() { smol::block_on(async { let global = owned_global_pool(64 * 1024 * 1024); + let pool_guard = global.create_base_guard(); let key = BlockKey::new(test_file_id(7), test_block_id(11)); assert_eq!(global.allocated(), 0); @@ -1951,13 +1988,49 @@ pub(crate) mod tests { assert_eq!(global.try_get_block_key(frame_id), Some(key)); assert_eq!( - global.invalidate_block(key.file_id, key.block_id), + global.invalidate_block(&pool_guard, key.file_id, key.block_id), Some(frame_id) ); assert_eq!(global.allocated(), 0); }); } + #[test] + fn test_readonly_miss_and_invalidation_reuse_caller_base_guard() { + smol::block_on(async { + let (_temp_dir, fs) = build_test_fs(); + let table_id = test_user_table_id(121); + let table_file = fs + .create_table_file(table_id, make_metadata(), false) + .unwrap(); + let table_file = commit_table_file(&fs, table_file).await; + let block_id = test_block_id(19); + write_payload(&fs, &table_file, block_id, b"guard-root").await; + + let global = owned_global_pool(frame_page_bytes(2)); + let pool = owned_readonly_pool( + FileID::from(table_id), + FileKind::TableFile, + Arc::clone(table_file.sparse_file()), + &global, + ); + let guard = pool.create_base_guard(); + assert_eq!(test_outstanding_base_guard_count(&global.arena), 1); + + let block = pool.read_block(&guard, block_id).await.unwrap(); + assert_eq!(&block.page()[..10], b"guard-root"); + assert_eq!(test_outstanding_base_guard_count(&global.arena), 1); + drop(block); + + assert!( + global + .invalidate_block(&guard, FileID::from(table_id), block_id) + .is_some() + ); + assert_eq!(test_outstanding_base_guard_count(&global.arena), 1); + }); + } + #[test] fn test_readonly_reservation_retries_transient_free_frame_latch_contention() { smol::block_on(async { @@ -1970,14 +2043,14 @@ pub(crate) mod tests { .lock() .last() .expect("readonly test pool must contain one free frame"); - let pool_guard = global.pool_guard(); + let pool_guard = global.create_base_guard(); let shared = global .get_page_internal::(&pool_guard, frame_id, LatchFallbackMode::Shared) .await .lock_shared_async() .await .unwrap(); - let task_arena = global.arena.arena_guard(global.pool_guard()); + let task_arena = global.arena.arena_guard(global.create_base_guard()); let mut reserve = Box::pin(ReadonlyPageReservation::reserve_page(&global, task_arena)); assert!(futures::poll!(reserve.as_mut()).is_pending()); @@ -2008,7 +2081,7 @@ pub(crate) mod tests { let existing_frame_id = publish_test_frame(&global, key).await; let free_before = global.residency.free.lock().len(); - let task_arena = global.arena.arena_guard(global.pool_guard()); + let task_arena = global.arena.arena_guard(global.create_base_guard()); let (reserved_frame_id, page_guard) = ReadonlyPageReservation::reserve_page(&global, task_arena) .await @@ -2051,11 +2124,13 @@ pub(crate) mod tests { fn test_readonly_write_barrier_invalidates_resident_mapping() { smol::block_on(async { let global = owned_global_pool(64 * 1024 * 1024); + let pool_guard = global.create_base_guard(); let key = BlockKey::new(test_file_id(70), test_block_id(11)); let frame_id = publish_test_frame(&global, key).await; assert_eq!(global.try_get_frame_id(&key), Some(frame_id)); - let lease = begin_write_barrier(global.guard(), key.file_id, key.block_id).unwrap(); + let lease = begin_write_barrier(global.guard(), &pool_guard, key.file_id, key.block_id) + .unwrap(); assert_eq!(global.try_get_frame_id(&key), None); assert_eq!(global.try_get_block_key(frame_id), None); assert_eq!(global.allocated(), 0); @@ -2063,7 +2138,8 @@ pub(crate) mod tests { drop(lease); assert!(!global.inflights.contains_key(&key)); - let miss = begin_write_barrier(global.guard(), key.file_id, key.block_id).unwrap(); + let miss = begin_write_barrier(global.guard(), &pool_guard, key.file_id, key.block_id) + .unwrap(); assert!(key_state_is_write_blocked(&global, &key)); drop(miss); }); @@ -2072,13 +2148,15 @@ pub(crate) mod tests { #[test] fn test_readonly_write_barrier_rejects_same_key_inflight_load() { let global = owned_global_pool(64 * 1024 * 1024); + let pool_guard = global.create_base_guard(); let key = BlockKey::new(test_file_id(71), test_block_id(12)); global.inflights.insert( key, InflightBlockState::Loading(Arc::new(PageIOCompletion::new())), ); - let err = begin_write_barrier(global.guard(), key.file_id, key.block_id).unwrap_err(); + let err = begin_write_barrier(global.guard(), &pool_guard, key.file_id, key.block_id) + .unwrap_err(); assert_eq!( err.downcast_ref::().copied(), Some(InternalError::ReadonlyWriteInflight) @@ -2090,6 +2168,7 @@ pub(crate) mod tests { #[test] fn test_readonly_write_barrier_replaces_completed_same_key_load() { let global = owned_global_pool(64 * 1024 * 1024); + let pool_guard = global.create_base_guard(); let key = BlockKey::new(test_file_id(72), test_block_id(12)); let inflight = Arc::new(PageIOCompletion::new()); inflight.complete(Ok(test_page_id(2))); @@ -2097,7 +2176,8 @@ pub(crate) mod tests { .inflights .insert(key, InflightBlockState::Loading(inflight)); - let lease = begin_write_barrier(global.guard(), key.file_id, key.block_id).unwrap(); + let lease = + begin_write_barrier(global.guard(), &pool_guard, key.file_id, key.block_id).unwrap(); assert!(key_state_is_write_blocked(&global, &key)); drop(lease); assert!(!global.inflights.contains_key(&key)); @@ -2121,7 +2201,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(119)), block_id); let old = pool.read_block(&pool_guard, block_id).await.unwrap(); @@ -2129,7 +2209,8 @@ pub(crate) mod tests { drop(old); assert!(global.try_get_frame_id(&key).is_some()); - let lease = begin_write_barrier(global.guard(), key.file_id, key.block_id).unwrap(); + let lease = begin_write_barrier(global.guard(), &pool_guard, key.file_id, key.block_id) + .unwrap(); assert_eq!(global.try_get_frame_id(&key), None); assert!(key_state_is_write_blocked(&global, &key)); write_payload(&fs, &table_file, block_id, b"new-reuse-bytes").await; @@ -2160,9 +2241,10 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(120)), block_id); - let lease = begin_write_barrier(global.guard(), key.file_id, key.block_id).unwrap(); + let lease = begin_write_barrier(global.guard(), &pool_guard, key.file_id, key.block_id) + .unwrap(); assert!(key_state_is_write_blocked(&global, &key)); let err = match pool.read_block(&pool_guard, block_id).await { @@ -2197,7 +2279,7 @@ pub(crate) mod tests { let global = global_readonly_pool_scope(frame_page_bytes(2)); let disk_pool = table_readonly_pool(&global, test_user_table_id(121), &table_file); - let pool_guard = disk_pool.pool_guard(); + let pool_guard = disk_pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(121)), block_id); let write_hook = Arc::new(ControlledWriteHook::for_page( table_file.sparse_file().as_raw_fd(), @@ -2211,6 +2293,7 @@ pub(crate) mod tests { &table_file, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); let writer = smol::spawn(async move { mutable.write_block(block_id, buf).await }); write_hook.wait_started(1).await; @@ -2257,7 +2340,7 @@ pub(crate) mod tests { ); let key = BlockKey::new(test_user_file_id(TableID::new(118)), test_block_id(14)); let inflight = Arc::new(PageIOCompletion::new()); - let task_arena = global.arena.arena_guard(global.pool_guard()); + let task_arena = global.arena.arena_guard(global.create_base_guard()); let (frame_id, page_guard) = ReadonlyPageReservation::reserve_page(&global, task_arena) .await .unwrap(); @@ -2342,14 +2425,14 @@ pub(crate) mod tests { } else { test_page_id(0) }; - let global_guard = global.pool_guard(); + let global_guard = global.create_base_guard(); let mut page_guard = global .try_lock_page_exclusive(&global_guard, frame_id) .unwrap(); global.mappings.insert(key, mapped_frame_id); page_guard.bf_mut().set_kind(FrameKind::Evicting); let runtime = ReadonlyRuntime { - arena: global.arena.arena_guard(global.pool_guard()), + arena: global.arena.arena_guard(global.create_base_guard()), pool: global.guard().into_sync(), }; @@ -2364,14 +2447,14 @@ pub(crate) mod tests { let global = owned_global_pool(64 * 1024 * 1024); let key = BlockKey::new(test_file_id(302), test_block_id(12)); let frame_id = publish_test_frame(&global, key).await; - let global_guard = global.pool_guard(); + let global_guard = global.create_base_guard(); let mut page_guard = global .try_lock_page_exclusive(&global_guard, frame_id) .unwrap(); assert!(global.residency.move_resident_to_free(frame_id)); page_guard.bf_mut().set_kind(FrameKind::Evicting); let runtime = ReadonlyRuntime { - arena: global.arena.arena_guard(global.pool_guard()), + arena: global.arena.arena_guard(global.create_base_guard()), pool: global.guard().into_sync(), }; @@ -2385,7 +2468,7 @@ pub(crate) mod tests { smol::block_on(async { let global1 = owned_global_pool(64 * 1024 * 1024); let global2 = owned_global_pool(64 * 1024 * 1024); - let foreign_guard = (*global2).pool_guard(); + let foreign_guard = (*global2).create_base_guard(); let _ = global1 .get_page_internal::( &foreign_guard, @@ -2407,7 +2490,7 @@ pub(crate) mod tests { write_payload(&fs, &table_file, test_block_id(9), b"reload").await; let global = owned_global_pool(frame_page_bytes(2)); - let global_guard = (*global).pool_guard(); + let global_guard = (*global).create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(111)), test_block_id(9)); let stale_frame_id = publish_test_frame(&global, key).await; let mut stale_frame = global @@ -2426,7 +2509,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let reload_start = pool.global_stats(); let page = pool .read_block(&pool_guard, test_block_id(9)) @@ -2464,7 +2547,7 @@ pub(crate) mod tests { write_page_bytes(&fs, &table_file, test_block_id(12), &persisted_page).await; let global = owned_global_pool(frame_page_bytes(2)); - let global_guard = (*global).pool_guard(); + let global_guard = (*global).create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(123)), test_block_id(12)); let stale_frame_id = publish_test_frame(&global, key).await; let mut stale_frame = global @@ -2483,7 +2566,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let reload_start = pool.global_stats(); let page = pool @@ -2536,7 +2619,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let page = pool .read_block(&pool_guard, test_block_id(3)) .await @@ -2574,7 +2657,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(table_file.sparse_file().file_id(), block_id); let page = pool @@ -2592,7 +2675,7 @@ pub(crate) mod tests { let global_guard = global.guard(); let inflight = global_guard - .join_or_start_inflight_load(table_file.sparse_file(), key, None) + .join_or_start_inflight_load(table_file.sparse_file(), &pool_guard, key, None) .await .expect("duplicate load abort should not fail"); @@ -2636,7 +2719,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let mut tasks = vec![]; for _ in 0..16 { @@ -2681,7 +2764,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(112)), test_block_id(5)); let pool_for_loader = (*pool).clone(); @@ -2742,7 +2825,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(113)), test_block_id(9)); let pool_for_loader = (*pool).clone(); @@ -2799,7 +2882,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let observe = global.guard(); let pool_for_loader = (*pool).clone(); @@ -2857,7 +2940,9 @@ pub(crate) mod tests { .inflights .insert(key, InflightBlockState::Loading(Arc::clone(&inflight))); let global_guard = global.guard(); - let task_arena = global_guard.arena.arena_guard(global_guard.pool_guard()); + let task_arena = global_guard + .arena + .arena_guard(global_guard.create_base_guard()); let inflight_for_waiter = Arc::clone(&inflight); let reserve_waiter = { listener!(global.residency.evict_ev => evict_listener); @@ -2929,7 +3014,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(114)), test_block_id(7)); let pool_1 = (*pool).clone(); @@ -3001,7 +3086,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(115)), test_block_id(8)); let stats_start = pool.global_stats(); @@ -3075,7 +3160,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(107)), test_block_id(9)); let err = match pool @@ -3119,7 +3204,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(116)), test_block_id(10)); let err = match pool @@ -3156,7 +3241,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(108)), test_block_id(10)); let err = match pool @@ -3197,7 +3282,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let key = BlockKey::new(test_user_file_id(TableID::new(109)), test_block_id(11)); let err = match pool @@ -3266,10 +3351,14 @@ pub(crate) mod tests { let table_file = engine .inner() .table_fs - .open_table_file(test_user_table_id(103), engine.inner().pools.disk.clone()) + .open_table_file( + test_user_table_id(103), + engine.inner().pools.disk.clone(), + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap(); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); for i in 0..=capacity { let block_id = BlockID::from(base_page_id + i as u64); @@ -3335,7 +3424,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let g = pool .read_block(&pool_guard, test_block_id(9)) @@ -3364,7 +3453,7 @@ pub(crate) mod tests { Arc::clone(table_file.sparse_file()), &global, ); - let pool_guard = pool.pool_guard(); + let pool_guard = pool.create_base_guard(); let guard: ReadonlyBlockGuard = pool .read_block(&pool_guard, test_block_id(4)) .await diff --git a/doradb-storage/src/catalog/checkpoint.rs b/doradb-storage/src/catalog/checkpoint.rs index ba7a0a56..5cb8bed5 100644 --- a/doradb-storage/src/catalog/checkpoint.rs +++ b/doradb-storage/src/catalog/checkpoint.rs @@ -1,3 +1,4 @@ +use crate::buffer::PoolGuard; use crate::catalog::storage::tables::TABLE_ID_TABLES; use crate::catalog::{ Catalog, IndexDdlKind, IndexDdlRootProof, classify_index_ddl_root, is_catalog_table, @@ -382,7 +383,7 @@ impl MaintenanceExecution for CatalogCheckpointExecution { let engine = runtime.core(); engine .catalog() - .checkpoint_prepared(&engine.trx_sys) + .checkpoint_prepared(&engine.trx_sys, runtime.pool_guards().disk_guard()) .await .map_err(CompletionErrorBridge::capture_runtime_or_fatal) } @@ -417,9 +418,10 @@ impl Catalog { async fn checkpoint_prepared( &self, trx_sys: &TransactionSystem, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult { obs::info!("event=checkpoint_publish component=catalog action=start result=ok"); - self.checkpoint_prepared_inner(trx_sys) + self.checkpoint_prepared_inner(trx_sys, disk_guard) .await .inspect(|outcome| match outcome { CatalogCheckpointOutcome::Published { @@ -447,13 +449,14 @@ impl Catalog { async fn checkpoint_prepared_inner( &self, trx_sys: &TransactionSystem, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult { let scan_cfg = trx_sys.catalog_checkpoint_scan_config()?; let batch = self .scan_checkpoint_batch(trx_sys.persisted_watermark_cts(), scan_cfg) .await?; let publishable_progress = batch.redo_retention_progress(); - match self.apply_checkpoint_batch(batch).await { + match self.apply_checkpoint_batch(batch, disk_guard).await { Ok(CatalogCheckpointOutcome::Published { catalog_replay_start_ts, }) => { diff --git a/doradb-storage/src/catalog/index.rs b/doradb-storage/src/catalog/index.rs index 1ad4c16f..f4899126 100644 --- a/doradb-storage/src/catalog/index.rs +++ b/doradb-storage/src/catalog/index.rs @@ -458,13 +458,14 @@ impl<'a> CreateIndexRuntimeBuilder<'a> { #[inline] fn new( engine: &'a EngineCore, + guards: &'a PoolGuards, metadata: &'a TableMetadata, index_spec: &'a IndexSpec, build_ts: TrxID, ) -> Self { Self { index_pool: engine.pools.index.clone(), - index_guard: engine.pool_guards().index_guard(), + index_guard: guards.index_guard(), metadata, index_spec, build_ts, @@ -660,6 +661,7 @@ impl CreateIndexProgress { async fn execute_catalog_update( &mut self, engine: &EngineCore, + guards: &PoolGuards, new_metadata: &TableMetadata, ) -> RuntimeOrFatalResult<()> { debug_assert_eq!(self.phase, CreateIndexBuildPhase::LayoutStaged); @@ -677,10 +679,7 @@ impl CreateIndexProgress { match res { Ok(()) => Ok(()), Err(err) => { - if let Err(cleanup) = self - .rollback_before_catalog_commit(engine.pool_guards()) - .await - { + if let Err(cleanup) = self.rollback_before_catalog_commit(guards).await { return Err(err.merge_cleanup(cleanup.attach_with(|| { format!( "operation=create_index, phase=rollback_before_catalog_commit, table_id={}, index_no={}", @@ -750,10 +749,11 @@ impl CreateIndexProgress { async fn cleanup_after_catalog_commit_failure( &mut self, engine: &EngineCore, + guards: &PoolGuards, operation: &'static str, source: RuntimeOrFatalError, ) -> RuntimeOrFatalError { - self.cleanup_staged_runtime(engine.pool_guards()).await; + self.cleanup_staged_runtime(guards).await; self.phase = CreateIndexBuildPhase::Aborted; poison_index_after_catalog_commit_with_source( &engine.poisoner, @@ -1040,7 +1040,7 @@ impl AcceptedCreateIndex { }); let runtime = self.scope.engine().clone(); let engine = runtime.core(); - let guards = engine.pool_guards(); + let guards = runtime.pool_guards(); let table_id = plan.table_id; let index_no = plan.index_no; let index_no_usize = usize::from(index_no); @@ -1085,6 +1085,7 @@ impl AcceptedCreateIndex { plan.table.file(), engine.table_fs.background_writes(), plan.table.disk_pool().clone(), + guards.disk_guard().clone(), ); let disk_runtime = match SecondaryDiskTreeRuntime::new( index_no_usize, @@ -1154,6 +1155,7 @@ impl AcceptedCreateIndex { let runtime_builder = CreateIndexRuntimeBuilder::new( engine, + guards, plan.new_metadata.as_ref(), &plan.new_index_spec, build_ts, @@ -1220,7 +1222,7 @@ impl AcceptedCreateIndex { progress.stage_layout(new_layout); if let Err(err) = progress - .execute_catalog_update(engine, plan.new_metadata.as_ref()) + .execute_catalog_update(engine, guards, plan.new_metadata.as_ref()) .await { return Err(CompletionErrorBridge::capture_runtime_or_fatal(err)); @@ -1249,6 +1251,7 @@ impl AcceptedCreateIndex { progress .cleanup_after_catalog_commit_failure( engine, + guards, "table_root_publish", RuntimeOrFatalError::from(err), ) @@ -1411,7 +1414,7 @@ impl AcceptedDropIndex { }); let runtime = self.scope.engine().clone(); let engine = runtime.core(); - let guards = engine.pool_guards(); + let guards = runtime.pool_guards(); let table_id = plan.table_id; let index_no = plan.index_no; let index_no_usize = usize::from(index_no); @@ -1446,6 +1449,7 @@ impl AcceptedDropIndex { plan.table.file(), engine.table_fs.background_writes(), plan.table.disk_pool().clone(), + guards.disk_guard().clone(), ); mutable_file.replace_metadata_and_secondary_index_roots( Arc::clone(&plan.new_metadata), diff --git a/doradb-storage/src/catalog/mod.rs b/doradb-storage/src/catalog/mod.rs index 2c0c8b01..89f19cd0 100644 --- a/doradb-storage/src/catalog/mod.rs +++ b/doradb-storage/src/catalog/mod.rs @@ -135,16 +135,13 @@ impl Catalog { storage: CatalogStorage, poisoner: QuiescentGuard, config: CatalogConfig, + bootstrap_guards: &PoolGuards, ) -> RuntimeResult { - let pool_guards = PoolGuards::builder() - .push(PoolRole::Meta, storage.meta_pool.pool_guard()) - .push(PoolRole::Disk, storage.disk_pool.pool_guard()) - .build(); let snapshot = storage.checkpoint_snapshot(); storage .bootstrap_from_checkpoint( &snapshot, - &pool_guards, + bootstrap_guards, config.recovery_disable_dml_validation, ) .await?; @@ -194,9 +191,10 @@ impl Catalog { pub(crate) async fn apply_checkpoint_batch( &self, batch: CatalogCheckpointBatch, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult { self.storage - .apply_checkpoint_batch(batch, self.curr_next_table_id()) + .apply_checkpoint_batch(batch, self.curr_next_table_id(), disk_guard) .await } @@ -205,9 +203,10 @@ impl Catalog { pub(crate) async fn prepare_checkpoint_batch( &self, batch: CatalogCheckpointBatch, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult { self.storage - .prepare_checkpoint_batch(batch, self.curr_next_table_id()) + .prepare_checkpoint_batch(batch, self.curr_next_table_id(), disk_guard) .await } @@ -229,26 +228,22 @@ impl Catalog { index_pool: QuiescentGuard, table_fs: &FileSystem, disk_pool: QuiescentGuard, + guards: &PoolGuards, table_id: TableID, ) -> RuntimeResult { assert!( !self.user_tables.contains_key(&table_id), "catalog reload invariant violated: table runtime already exists, table_id={table_id}" ); - let guards = PoolGuards::builder() - .push(PoolRole::Meta, self.storage.meta_pool.pool_guard()) - .push(PoolRole::Index, index_pool.pool_guard()) - .push(PoolRole::Disk, disk_pool.pool_guard()) - .build(); let (table, metadata_in_catalog) = self - .user_table_metadata_from_catalog(&guards, table_id) + .user_table_metadata_from_catalog(guards, table_id) .await?; // Phase 2 allocator semantics: only table ids consume the global allocator. self.try_update_next_table_id(table.table_id.saturating_add(1)); let table_file = table_fs - .open_table_file(table.table_id, disk_pool.clone()) + .open_table_file(table.table_id, disk_pool.clone(), guards.disk_guard()) .await .change_context(RuntimeError::CatalogAccess) .attach_with(|| { @@ -877,13 +872,22 @@ impl Component for Catalog { let table_fs = registry.dependency::(); let disk_pool = registry.dependency::(); let poisoner = registry.dependency::(); + // Catalog bootstrap runs before sessions exist. Create one explicit + // component-build root per required pool and thread that bundle through + // file loading and catalog-table initialization. + let bootstrap_guards = PoolGuards::builder() + .push(PoolRole::Meta, meta_pool.create_base_guard()) + .push(PoolRole::Disk, disk_pool.create_base_guard()) + .build(); let storage = CatalogStorage::new( meta_pool.clone_inner(), table_fs.clone(), disk_pool.clone_inner(), + &bootstrap_guards, ) .await?; - registry.register::(Catalog::new(storage, poisoner, config).await?); + registry + .register::(Catalog::new(storage, poisoner, config, &bootstrap_guards).await?); Ok(()) } @@ -1823,7 +1827,10 @@ pub(crate) mod tests { .indexes() .list_uncommitted_by_table_id( &PoolGuards::builder() - .push(PoolRole::Meta, engine.inner().pools.meta.pool_guard()) + .push( + PoolRole::Meta, + engine.inner().pools.meta.create_base_guard(), + ) .build(), table_id, ) @@ -1988,7 +1995,13 @@ pub(crate) mod tests { .expect("catalog checkpoint should publish at least one root"); let root_block_id = BlockID::from(root.root_block_id.unwrap().get()); let block_id = { - let disk_pool_guard = engine.inner().core.catalog().storage.disk_pool.pool_guard(); + let disk_pool_guard = engine + .inner() + .core + .catalog() + .storage + .disk_pool + .create_base_guard(); let index = ColumnBlockIndex::new( root_block_id, root.pivot_row_id, @@ -2050,7 +2063,13 @@ pub(crate) mod tests { .expect("catalog checkpoint should publish at least one root"); let root_block_id = BlockID::from(root.root_block_id.unwrap().get()); let entry = { - let disk_pool_guard = engine.inner().core.catalog().storage.disk_pool.pool_guard(); + let disk_pool_guard = engine + .inner() + .core + .catalog() + .storage + .disk_pool + .create_base_guard(); let index = ColumnBlockIndex::new( root_block_id, root.pivot_row_id, @@ -2162,7 +2181,10 @@ pub(crate) mod tests { .inner() .core .catalog() - .apply_checkpoint_batch(batch1) + .apply_checkpoint_batch( + batch1, + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap(); let snap1 = engine.inner().core.catalog().storage.checkpoint_snapshot(); @@ -2184,7 +2206,10 @@ pub(crate) mod tests { .inner() .core .catalog() - .apply_checkpoint_batch(batch2) + .apply_checkpoint_batch( + batch2, + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap(); let snap2 = engine.inner().core.catalog().storage.checkpoint_snapshot(); diff --git a/doradb-storage/src/catalog/storage/mod.rs b/doradb-storage/src/catalog/storage/mod.rs index 0336b0b1..6d4dfc8c 100644 --- a/doradb-storage/src/catalog/storage/mod.rs +++ b/doradb-storage/src/catalog/storage/mod.rs @@ -6,7 +6,7 @@ mod object; mod table_replay_silent_watermarks; pub(crate) mod tables; -use crate::buffer::{BufferPool, FixedBufferPool, PoolGuard, PoolGuards, ReadonlyBufferPool}; +use crate::buffer::{FixedBufferPool, PoolGuard, PoolGuards, ReadonlyBufferPool}; use crate::catalog::storage::columns::*; use crate::catalog::storage::indexes::*; use crate::catalog::storage::merge::{CatalogFoldedRows, CatalogMergeKeyBuilder}; @@ -74,10 +74,10 @@ impl CatalogStorage { meta_pool: QuiescentGuard, table_fs: QuiescentGuard, disk_pool: QuiescentGuard, + bootstrap_guards: &PoolGuards, ) -> RuntimeResult { - let meta_pool_guard = meta_pool.pool_guard(); let mtb = table_fs - .open_or_create_multi_table_file(disk_pool.clone()) + .open_or_create_multi_table_file(disk_pool.clone(), bootstrap_guards.disk_guard()) .await .change_context(RuntimeError::CatalogAccess) .attach("operation=open_catalog_storage")?; @@ -94,7 +94,7 @@ impl CatalogStorage { // Make sure catalog table ids match their dense root slots. assert_eq!(cat.len(), must_catalog_table_slot(*table_id)); let metadata = Arc::new(metadata.clone()); - let blk_idx = BlockIndex::new_catalog(meta_pool.clone(), &meta_pool_guard) + let blk_idx = BlockIndex::new_catalog(meta_pool.clone(), bootstrap_guards.meta_guard()) .await .change_context(RuntimeError::CatalogAccess) .attach_with(|| { @@ -103,7 +103,7 @@ impl CatalogStorage { let table = Arc::new( CatalogTable::new( meta_pool.clone(), - &meta_pool_guard, + bootstrap_guards.meta_guard(), *table_id, blk_idx, metadata, @@ -309,6 +309,7 @@ impl CatalogStorage { &self, batch: CatalogCheckpointBatch, next_table_id: TableID, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult { let CatalogCheckpointBatch { replay_start_ts, @@ -385,6 +386,7 @@ impl CatalogStorage { current_root, &ops_by_table[idx], safe_cts, + disk_guard, ) .await?; new_roots[idx] = new_root; @@ -400,7 +402,8 @@ impl CatalogStorage { // Rewriting catalog table roots can make arbitrary old catalog // blocks unreachable, so rebuild the allocation map from the new // root graph before publishing. - self.rebuild_catalog_alloc_map(&mut mutable).await?; + self.rebuild_catalog_alloc_map(&mut mutable, disk_guard) + .await?; } else { // Metadata-only checkpoints do not change catalog table root // reachability. Reclaim the displaced metadata block directly and @@ -410,7 +413,6 @@ impl CatalogStorage { .change_context(RuntimeError::CatalogAccess) .attach("operation=prepare_catalog_checkpoint, phase=reserve_meta_block")?; } - let disk_pool_guard = self.disk_pool.pool_guard(); // Load the silent replay watermark overlay from `new_roots`, not from // the currently durable cache. The prepared checkpoint has already // materialized catalog-table changes into blocks, but its metadata root @@ -420,7 +422,7 @@ impl CatalogStorage { // root is committed. let checkpointed_silent_watermarks = self .load_checkpointed_table_replay_silent_watermark_map( - &disk_pool_guard, + disk_guard, new_roots[must_catalog_table_slot(TABLE_ID_TABLE_REPLAY_SILENT_WATERMARKS)], ) .await?; @@ -438,8 +440,11 @@ impl CatalogStorage { &self, batch: CatalogCheckpointBatch, next_table_id: TableID, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult { - let prepared = self.prepare_checkpoint_batch(batch, next_table_id).await?; + let prepared = self + .prepare_checkpoint_batch(batch, next_table_id, disk_guard) + .await?; Ok(prepared.commit(self).await?) } @@ -483,13 +488,14 @@ impl CatalogStorage { async fn rebuild_catalog_alloc_map( &self, mutable: &mut MutableMultiTableFile, + disk_guard: &PoolGuard, ) -> RuntimeResult { mutable .reserve_publish_meta_block() .change_context(RuntimeError::CatalogAccess) .attach("operation=rebuild_catalog_alloc_map, phase=reserve_meta_block")?; let reachable = self - .collect_catalog_reachable_blocks(mutable.root()) + .collect_catalog_reachable_blocks(mutable.root(), disk_guard) .await?; Ok(mutable.rebuild_alloc_map_from_reachable(&reachable)) } @@ -497,12 +503,12 @@ impl CatalogStorage { async fn collect_catalog_reachable_blocks( &self, root: &MultiTableActiveRoot, + disk_guard: &PoolGuard, ) -> RuntimeResult> { let mut reachable = BTreeSet::new(); reachable.insert(SUPER_BLOCK_ID); reachable.insert(root.meta_block_id); - let disk_pool_guard = self.disk_pool.pool_guard(); for (idx, table_root) in root.table_roots.iter().enumerate() { if catalog_table_slot(table_root.table_id) != Some(idx) { return Err( @@ -547,7 +553,7 @@ impl CatalogStorage { self.mtb.file_kind(), self.mtb.sparse_file(), &self.disk_pool, - &disk_pool_guard, + disk_guard, ); column_index .collect_reachable_blocks(&mut reachable) @@ -573,6 +579,10 @@ impl CatalogStorage { Ok(reachable) } + #[expect( + clippy::too_many_arguments, + reason = "catalog checkpoint folding keeps the root, mutation batch, and caller guard explicit" + )] async fn apply_table_ops( &self, mutable: &mut MutableMultiTableFile, @@ -581,11 +591,9 @@ impl CatalogStorage { root: CatalogTableRootDesc, table_ops: &[RowRedoKind], checkpoint_cts: TrxID, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult<(CatalogTableRootDesc, bool)> { - let disk_pool_guard = self.disk_pool.pool_guard(); - let base_rows = self - .load_rows_from_root(metadata, &disk_pool_guard, root) - .await?; + let base_rows = self.load_rows_from_root(metadata, disk_guard, root).await?; let mut folded = CatalogFoldedRows::from_base_rows(metadata, base_rows) .change_context(RuntimeError::CatalogAccess) .attach_with(|| format!("operation=apply_catalog_table_ops, table_id={table_id}"))?; @@ -691,7 +699,7 @@ impl CatalogStorage { self.mtb.file_kind(), self.mtb.sparse_file(), &self.disk_pool, - &disk_pool_guard, + disk_guard, ); let root_block_id = column_index .batch_insert(mutable, &new_entries, pivot_row_id, checkpoint_cts) @@ -1268,7 +1276,7 @@ fn validate_catalog_row( #[cfg(test)] pub(crate) mod tests { use super::*; - use crate::buffer::{PoolGuards, PoolRole}; + use crate::buffer::{BufferPool, PoolGuards, PoolRole}; use crate::catalog::USER_TABLE_ID_START; use crate::catalog::tests::{open_catalog_test_engine, table1, table2}; use crate::catalog::{ @@ -1386,8 +1394,13 @@ pub(crate) mod tests { next_table_id: TableID, ) -> Result<()> { let replay_start_ts = storage.checkpoint_snapshot().catalog_replay_start_ts; + let disk_guard = storage.disk_pool.create_base_guard(); storage - .apply_checkpoint_batch(metadata_only_batch(replay_start_ts), next_table_id) + .apply_checkpoint_batch( + metadata_only_batch(replay_start_ts), + next_table_id, + &disk_guard, + ) .await .map(|_| ()) .disclose() @@ -1461,7 +1474,7 @@ pub(crate) mod tests { let root = storage.checkpoint_snapshot().meta.table_roots[must_catalog_table_slot(table_id)]; let table = storage.get_catalog_table(table_id).unwrap(); - let disk_pool_guard = storage.disk_pool.pool_guard(); + let disk_pool_guard = storage.disk_pool.create_base_guard(); storage .load_rows_from_root(table.metadata(), &disk_pool_guard, root) .await @@ -1485,7 +1498,7 @@ pub(crate) mod tests { } assert_eq!(root.pivot_row_id, RowID::new(rows.len() as u64)); let root_block_id = BlockID::from(root.root_block_id.unwrap().get()); - let disk_pool_guard = storage.disk_pool.pool_guard(); + let disk_pool_guard = storage.disk_pool.create_base_guard(); let entries = storage .collect_index_entries(&disk_pool_guard, root_block_id) .await @@ -1526,7 +1539,7 @@ pub(crate) mod tests { mutable.write_block(block_id, page.buf).await.unwrap(); entries.push(page.shape.with_block_id(block_id)); } - let disk_pool_guard = storage.disk_pool.pool_guard(); + let disk_pool_guard = storage.disk_pool.create_base_guard(); let pivot_row_id = RowID::new(rows.len() as u64); let column_index = ColumnBlockIndex::new( SUPER_BLOCK_ID, @@ -1594,7 +1607,11 @@ pub(crate) mod tests { let err = expect_runtime_report( storage - .apply_checkpoint_batch(batch, engine.inner().core.catalog().curr_next_table_id()) + .apply_checkpoint_batch( + batch, + engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap_err(), ); @@ -1626,8 +1643,8 @@ pub(crate) mod tests { root.table_id = TABLE_ID_COLUMNS; let guards = PoolGuards::builder() - .push(PoolRole::Meta, storage.meta_pool.pool_guard()) - .push(PoolRole::Disk, storage.disk_pool.pool_guard()) + .push(PoolRole::Meta, storage.meta_pool.create_base_guard()) + .push(PoolRole::Disk, storage.disk_pool.create_base_guard()) .build(); let err = storage .bootstrap_from_checkpoint(&snapshot, &guards, false) @@ -1684,6 +1701,7 @@ pub(crate) mod tests { .apply_checkpoint_batch( batch, engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), ) .await .unwrap_err(), @@ -1857,7 +1875,7 @@ pub(crate) mod tests { ) .await; - let disk_pool_guard = storage.disk_pool.pool_guard(); + let disk_pool_guard = storage.disk_pool.create_base_guard(); storage .load_rows_from_root(table.metadata(), &disk_pool_guard, root) .await @@ -1895,7 +1913,7 @@ pub(crate) mod tests { ) .await; - let disk_pool_guard = storage.disk_pool.pool_guard(); + let disk_pool_guard = storage.disk_pool.create_base_guard(); let err = storage .load_rows_from_root(table.metadata(), &disk_pool_guard, root) .await @@ -2015,7 +2033,7 @@ pub(crate) mod tests { let storage = &engine.inner().core.catalog().storage; let snap = storage.checkpoint_snapshot(); - let disk_pool_guard = storage.disk_pool.pool_guard(); + let disk_pool_guard = storage.disk_pool.create_base_guard(); let mut catalog_index_blocks = BTreeSet::new(); for root in snap.meta.table_roots { let Some(root_block_id) = root.checkpoint_root_block_id() else { @@ -2032,11 +2050,11 @@ pub(crate) mod tests { } assert!(!catalog_index_blocks.is_empty()); for block_id in &catalog_index_blocks { - let _ = engine - .inner() - .pools - .disk - .invalidate_block(CATALOG_MTB_FILE_ID, *block_id); + let _ = engine.inner().pools.disk.invalidate_block( + &disk_pool_guard, + CATALOG_MTB_FILE_ID, + *block_id, + ); let key = BlockKey::new(CATALOG_MTB_FILE_ID, *block_id); assert!(engine.inner().pools.disk.try_get_frame_id(&key).is_none()); } @@ -2097,7 +2115,11 @@ pub(crate) mod tests { }; storage - .apply_checkpoint_batch(batch, engine.inner().core.catalog().curr_next_table_id()) + .apply_checkpoint_batch( + batch, + engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap(); @@ -2149,7 +2171,11 @@ pub(crate) mod tests { ); storage - .apply_checkpoint_batch(batch, engine.inner().core.catalog().curr_next_table_id()) + .apply_checkpoint_batch( + batch, + engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap(); @@ -2201,6 +2227,7 @@ pub(crate) mod tests { .apply_checkpoint_batch( batch, engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), ) .await .unwrap_err(), @@ -2251,7 +2278,11 @@ pub(crate) mod tests { ); storage - .apply_checkpoint_batch(batch, engine.inner().core.catalog().curr_next_table_id()) + .apply_checkpoint_batch( + batch, + engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap(); @@ -2296,7 +2327,10 @@ pub(crate) mod tests { roots, ); let err = storage - .rebuild_catalog_alloc_map(&mut mutable) + .rebuild_catalog_alloc_map( + &mut mutable, + engine.inner().core.pools.pool_guards().disk_guard(), + ) .await .unwrap_err(); @@ -2343,6 +2377,7 @@ pub(crate) mod tests { root, &table_ops, TrxID::new(7), + engine.inner().core.pools.pool_guards().disk_guard(), ) .await .unwrap(); @@ -2393,6 +2428,7 @@ pub(crate) mod tests { root, &table_ops, TrxID::new(8), + engine.inner().core.pools.pool_guards().disk_guard(), ) .await .unwrap(); @@ -2422,7 +2458,13 @@ pub(crate) mod tests { let snap = engine.inner().core.catalog().storage.checkpoint_snapshot(); let tables_root = snap.meta.table_roots[0]; let root_block_id = BlockID::from(tables_root.root_block_id.unwrap().get()); - let disk_pool_guard = engine.inner().core.catalog().storage.disk_pool.pool_guard(); + let disk_pool_guard = engine + .inner() + .core + .catalog() + .storage + .disk_pool + .create_base_guard(); let cached_before_first = engine.inner().pools.disk.allocated(); @@ -2563,11 +2605,12 @@ pub(crate) mod tests { vec![catalog_column_insert(table_id, 0, 30_000)], ), engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), ) .await .unwrap(); - let disk_pool_guard = storage.disk_pool.pool_guard(); + let disk_pool_guard = storage.disk_pool.create_base_guard(); let snap1 = storage.checkpoint_snapshot(); let columns_root1 = snap1.meta.table_roots[1]; assert_eq!(columns_root1.pivot_row_id, RowID::new(1)); @@ -2587,6 +2630,7 @@ pub(crate) mod tests { .apply_checkpoint_batch( checkpoint_batch_with_ops(storage, second_batch), engine.inner().core.catalog().curr_next_table_id(), + engine.inner().core.pools.pool_guards().disk_guard(), ) .await .unwrap(); diff --git a/doradb-storage/src/catalog/storage/tables.rs b/doradb-storage/src/catalog/storage/tables.rs index b570a074..93fa44d3 100644 --- a/doradb-storage/src/catalog/storage/tables.rs +++ b/doradb-storage/src/catalog/storage/tables.rs @@ -272,7 +272,10 @@ mod tests { let table_id = table1(&engine).await; { let guards = PoolGuards::builder() - .push(PoolRole::Meta, engine.inner().pools.meta.pool_guard()) + .push( + PoolRole::Meta, + engine.inner().pools.meta.create_base_guard(), + ) .build(); assert!( engine diff --git a/doradb-storage/src/catalog/table.rs b/doradb-storage/src/catalog/table.rs index 472d92e8..89caf586 100644 --- a/doradb-storage/src/catalog/table.rs +++ b/doradb-storage/src/catalog/table.rs @@ -125,7 +125,7 @@ enum CreateTablePhase { } enum CreateTableFile { - Mutable(MutableTableFile), + Mutable(Box), Published(Arc), } @@ -160,7 +160,7 @@ impl CreateTableProgress { #[inline] fn set_provisional_file(&mut self, mutable_file: MutableTableFile) { assert_eq!(self.phase, CreateTablePhase::Prepared); - self.file = Some(CreateTableFile::Mutable(mutable_file)); + self.file = Some(CreateTableFile::Mutable(Box::new(mutable_file))); self.phase = CreateTablePhase::FileCreated; } @@ -201,7 +201,7 @@ impl CreateTableProgress { panic!("create-table file is mutable before publish"); }; let table_file = trx_sys - .publish_table_file_root(mutable_file, root_ts, true) + .publish_table_file_root(*mutable_file, root_ts, true) .await .change_context(RuntimeError::CatalogAccess) .attach_with(|| { @@ -216,7 +216,11 @@ impl CreateTableProgress { } #[inline] - async fn build_runtime(&mut self, pools: &EnginePools) -> RuntimeResult<()> { + async fn build_runtime( + &mut self, + pools: &EnginePools, + guards: &PoolGuards, + ) -> RuntimeResult<()> { debug_assert_eq!(self.phase, CreateTablePhase::FilePublished); let Some(CreateTableFile::Published(table_file)) = self.file.as_ref() else { panic!("published table file is present before runtime build"); @@ -225,7 +229,7 @@ impl CreateTableProgress { let active_root = table_file.active_root_unchecked(); let blk_idx = BlockIndex::new( pools.meta.clone(), - pools.pool_guards().meta_guard(), + guards.meta_guard(), active_root.pivot_row_id, active_root.column_block_index_root, ) @@ -241,7 +245,7 @@ impl CreateTableProgress { Table::new( pools.mem.clone(), pools.index.clone(), - pools.pool_guards().index_guard(), + guards.index_guard(), self.table_id, blk_idx, table_file, @@ -296,7 +300,7 @@ impl CreateTableProgress { fn delete_provisional_file(&mut self, table_fs: &FileSystem) -> IoResult<()> { match self.file.take() { Some(CreateTableFile::Mutable(mutable_file)) => { - let _ = mutable_file.try_delete(); + let _ = (*mutable_file).try_delete(); } Some(CreateTableFile::Published(table_file)) => drop(table_file), None => {} @@ -322,13 +326,14 @@ impl CreateTableProgress { async fn abort_before_catalog_commit( &mut self, engine: &EngineCore, + guards: &PoolGuards, operation: &'static str, source: impl Into, ) -> RuntimeOrFatalError { let source = source.into(); let source_debug = format!("{source:?}"); let mut error = source; - if let Err(err) = self.destroy_staged_runtime(engine.pool_guards()).await { + if let Err(err) = self.destroy_staged_runtime(guards).await { let cleanup = poison_error_source( &engine.poisoner, RuntimeOrFatalError::from(err), @@ -371,11 +376,12 @@ impl CreateTableProgress { async fn abort_after_root_publish_commit_error( &mut self, engine: &EngineCore, + guards: &PoolGuards, operation: &'static str, source: RuntimeOrFatalError, ) -> RuntimeOrFatalError { let source_debug = format!("{source:?}"); - if let Err(err) = self.destroy_staged_runtime(engine.pool_guards()).await { + if let Err(err) = self.destroy_staged_runtime(guards).await { self.phase = CreateTablePhase::Aborted; return poison_error_source( &engine.poisoner, @@ -1285,6 +1291,7 @@ impl AcceptedCreateTable { .as_mut() .unwrap_or_else(|| panic!("accepted CREATE progress exists during execution")); let engine = scope.engine().clone(); + let guards = engine.pool_guards(); let table_id = progress.table_id; #[cfg(test)] @@ -1344,7 +1351,7 @@ impl AcceptedCreateTable { if let Err(err) = exec_res { return Err(CompletionErrorBridge::capture_runtime_or_fatal( progress - .abort_before_catalog_commit(&engine, "catalog_staging", err) + .abort_before_catalog_commit(&engine, guards, "catalog_staging", err) .await, )); } @@ -1363,7 +1370,7 @@ impl AcceptedCreateTable { { return Err(CompletionErrorBridge::capture_runtime_or_fatal( progress - .abort_before_catalog_commit(&engine, "test_after_catalog_staging", err) + .abort_before_catalog_commit(&engine, guards, "test_after_catalog_staging", err) .await, )); } @@ -1371,7 +1378,7 @@ impl AcceptedCreateTable { if let Err(err) = progress.publish_file(&engine.trx_sys).await { return Err(CompletionErrorBridge::capture_runtime_or_fatal( progress - .abort_before_catalog_commit(&engine, "file_publish", err) + .abort_before_catalog_commit(&engine, guards, "file_publish", err) .await, )); } @@ -1389,15 +1396,15 @@ impl AcceptedCreateTable { { return Err(CompletionErrorBridge::capture_runtime_or_fatal( progress - .abort_before_catalog_commit(&engine, "test_after_file_publish", err) + .abort_before_catalog_commit(&engine, guards, "test_after_file_publish", err) .await, )); } - if let Err(err) = progress.build_runtime(&engine.pools).await { + if let Err(err) = progress.build_runtime(&engine.pools, guards).await { return Err(CompletionErrorBridge::capture_runtime_or_fatal( progress - .abort_before_catalog_commit(&engine, "runtime_build", err) + .abort_before_catalog_commit(&engine, guards, "runtime_build", err) .await, )); } @@ -1415,7 +1422,7 @@ impl AcceptedCreateTable { { return Err(CompletionErrorBridge::capture_runtime_or_fatal( progress - .abort_before_catalog_commit(&engine, "test_after_runtime_build", err) + .abort_before_catalog_commit(&engine, guards, "test_after_runtime_build", err) .await, )); } @@ -1430,7 +1437,12 @@ impl AcceptedCreateTable { Err(err) => { return Err(CompletionErrorBridge::capture_runtime_or_fatal( progress - .abort_after_root_publish_commit_error(&engine, "catalog_commit", err) + .abort_after_root_publish_commit_error( + &engine, + guards, + "catalog_commit", + err, + ) .await, )); } @@ -1771,6 +1783,7 @@ pub(crate) fn reject_non_user_table_id( #[inline] pub(crate) async fn ensure_user_table_catalog_row( engine: &EngineCore, + guards: &PoolGuards, table_id: TableID, operation: &'static str, ) -> OperationOrRuntimeResult<()> { @@ -1778,7 +1791,7 @@ pub(crate) async fn ensure_user_table_catalog_row( .catalog() .storage .tables() - .find_uncommitted_by_id(engine.pool_guards(), table_id) + .find_uncommitted_by_id(guards, table_id) .await? .is_some() { @@ -1792,6 +1805,7 @@ pub(crate) async fn ensure_user_table_catalog_row( /// Return the validated runtime table for an index-DDL target. pub(crate) async fn validated_index_ddl_target( engine: &EngineCore, + guards: &PoolGuards, table_id: TableID, operation: &'static str, ) -> OperationOrRuntimeResult> { @@ -1801,7 +1815,7 @@ pub(crate) async fn validated_index_ddl_target( .validate_user_table_live(table_id) .await .attach_with(|| format!("operation={operation}"))?; - ensure_user_table_catalog_row(engine, table_id, operation).await?; + ensure_user_table_catalog_row(engine, guards, table_id, operation).await?; Ok(table) } diff --git a/doradb-storage/src/component.rs b/doradb-storage/src/component.rs index f4fb6986..cb128525 100644 --- a/doradb-storage/src/component.rs +++ b/doradb-storage/src/component.rs @@ -698,7 +698,7 @@ pool_access_newtype!(IndexPool, EvictableBufferPool); pool_access_newtype!(MemPool, EvictableBufferPool); pool_access_newtype!(DiskPool, ReadonlyBufferPool); -/// Canonical engine buffer-pool capability shared by session runtime work. +/// Engine buffer-pool capabilities and owner-scoped canonical guards. pub(crate) struct EnginePools { /// Metadata pool used for catalog and block-index pages. pub(crate) meta: QuiescentGuard, @@ -708,22 +708,14 @@ pub(crate) struct EnginePools { pub(crate) mem: QuiescentGuard, /// Readonly persisted-page pool. pub(crate) disk: QuiescentGuard, - /// Prebuilt guards for the exact four pool identities above. + /// Prebuilt guards for engine-owned work outside session hot paths. + /// + /// Session state must use [`Self::create_session_pool_guards`] instead. Page latch + /// guards clone these roots on every access, so sharing this bundle across + /// sessions would make each pool's outer `Arc` counter engine-global. guards: PoolGuards, } -impl Clone for EnginePools { - #[inline] - fn clone(&self) -> Self { - Self::new( - self.meta.clone(), - self.index.clone(), - self.mem.clone(), - self.disk.clone(), - ) - } -} - impl EnginePools { /// Create a bundle from explicit inner pool handles. #[inline] @@ -734,10 +726,10 @@ impl EnginePools { disk: QuiescentGuard, ) -> Self { let guards = PoolGuards::builder() - .push(PoolRole::Meta, meta.pool_guard()) - .push(PoolRole::Index, index.pool_guard()) - .push(PoolRole::Mem, mem.pool_guard()) - .push(PoolRole::Disk, disk.pool_guard()) + .push(PoolRole::Meta, meta.create_base_guard()) + .push(PoolRole::Index, index.create_base_guard()) + .push(PoolRole::Mem, mem.create_base_guard()) + .push(PoolRole::Disk, disk.create_base_guard()) .build(); Self { meta, @@ -753,6 +745,23 @@ impl EnginePools { pub(crate) fn pool_guards(&self) -> &PoolGuards { &self.guards } + + /// Build an independent guard-root bundle for one session lifetime. + /// + /// Each `create_base_guard` call acquires one long-lived arena keepalive + /// and wraps it in a fresh `Arc`. Page guards cloned by that session then + /// update only its root instead of contending on the canonical engine + /// roots. Keep this construction at session creation rather than moving it + /// into per-page or per-statement work. + #[inline] + pub(crate) fn create_session_pool_guards(&self) -> PoolGuards { + PoolGuards::builder() + .push(PoolRole::Meta, self.meta.create_base_guard()) + .push(PoolRole::Index, self.index.create_base_guard()) + .push(PoolRole::Mem, self.mem.create_base_guard()) + .push(PoolRole::Disk, self.disk.create_base_guard()) + .build() + } } /// Configuration for the metadata buffer pool. diff --git a/doradb-storage/src/engine.rs b/doradb-storage/src/engine.rs index 1ffc8712..b8357b90 100644 --- a/doradb-storage/src/engine.rs +++ b/doradb-storage/src/engine.rs @@ -4,8 +4,8 @@ //! including start, stop, recover, and execute commands. See //! `docs/engine-component-lifetime.md` for the runtime-versus-owner lifetime //! model that this module enforces with the component registry. +use crate::buffer::PoolRole; use crate::buffer::SharedPoolEvictorWorkers; -use crate::buffer::{PoolGuards, PoolRole}; #[cfg(test)] use crate::catalog::index::tests::IndexDdlTestController; #[cfg(test)] @@ -702,7 +702,7 @@ pub(crate) struct EngineCore { pub(crate) catalog: QuiescentGuard, /// Shared transaction-system handle. pub(crate) trx_sys: QuiescentGuard, - /// Canonical typed pool handles and matching guard bundle. + /// Typed pool handles, owner-scoped guards, and session-root factory. pub(crate) pools: EnginePools, /// Table-file subsystem that runs persistent page IO. pub(crate) table_fs: QuiescentGuard, @@ -728,12 +728,6 @@ impl EngineCore { &self.catalog } - /// Borrow the canonical pool guard bundle. - #[inline] - pub(crate) fn pool_guards(&self) -> &PoolGuards { - self.pools.pool_guards() - } - /// Return the shared logical lock manager. #[inline] pub(crate) fn lock_manager(&self) -> &QuiescentGuard { @@ -2543,7 +2537,12 @@ mod tests { config, engine.inner().poisoner.clone(), engine.inner().mandatory_runtime.clone(), - engine.inner().core.pools.clone(), + EnginePools::new( + engine.inner().core.pools.meta.clone(), + engine.inner().core.pools.index.clone(), + engine.inner().core.pools.mem.clone(), + engine.inner().core.pools.disk.clone(), + ), engine.inner().table_fs.clone(), engine.inner().catalog.clone(), ) diff --git a/doradb-storage/src/file/cow_file.rs b/doradb-storage/src/file/cow_file.rs index 1f5a1be5..a263b780 100644 --- a/doradb-storage/src/file/cow_file.rs +++ b/doradb-storage/src/file/cow_file.rs @@ -1,6 +1,6 @@ use crate::bitmap::AllocMap; use crate::buffer::page::PAGE_SIZE; -use crate::buffer::{ReadonlyBufferPool, ReadonlyWriteLease, begin_write_barrier}; +use crate::buffer::{PoolGuard, ReadonlyBufferPool, ReadonlyWriteLease, begin_write_barrier}; use crate::error::{ CompletionResult, DataIntegrityError, DataIntegrityResult, InternalResult, IoError, IoResult, ResourceError, ResourceResult, RuntimeError, RuntimeResult, @@ -66,7 +66,10 @@ pub(crate) trait MutableWriterFile { #[derive(Clone, Copy)] pub(crate) enum CowWriteBarrier<'a> { /// Block same-key readonly misses while the physical block is written. - ReadonlyPool(&'a QuiescentGuard), + ReadonlyPool { + pool: &'a QuiescentGuard, + guard: &'a PoolGuard, + }, /// Bypass readonly-cache write blocking when the caller has no relevant readonly path. Disabled, } @@ -74,8 +77,11 @@ pub(crate) enum CowWriteBarrier<'a> { impl<'a> CowWriteBarrier<'a> { /// Builds a user-table write barrier backed by the shared readonly pool. #[inline] - pub(crate) fn readonly_pool(pool: &'a QuiescentGuard) -> Self { - Self::ReadonlyPool(pool) + pub(crate) fn readonly_pool( + pool: &'a QuiescentGuard, + guard: &'a PoolGuard, + ) -> Self { + Self::ReadonlyPool { pool, guard } } /// Start the readonly-cache write barrier for one physical block. @@ -86,8 +92,8 @@ impl<'a> CowWriteBarrier<'a> { block_id: BlockID, ) -> InternalResult> { match self { - CowWriteBarrier::ReadonlyPool(pool) => { - begin_write_barrier(pool.clone(), file_id, block_id).map(Some) + CowWriteBarrier::ReadonlyPool { pool, guard } => { + begin_write_barrier(pool.clone(), guard, file_id, block_id).map(Some) } CowWriteBarrier::Disabled => Ok(None), } @@ -600,12 +606,12 @@ impl CowFile { &self, file_kind: FileKind, disk_pool: &QuiescentGuard, + disk_guard: &PoolGuard, ) -> RuntimeResult> { let file_id = self.file.file_id(); - let _ = disk_pool.invalidate_block(file_id, SUPER_BLOCK_ID); - let pool_guard = disk_pool.pool_guard(); + let _ = disk_pool.invalidate_block(disk_guard, file_id, SUPER_BLOCK_ID); let super_block_guard = disk_pool - .read_block(file_kind, &self.file, &pool_guard, SUPER_BLOCK_ID) + .read_block(file_kind, &self.file, disk_guard, SUPER_BLOCK_ID) .await .change_context(RuntimeError::FileRootAccess) .attach_with(|| { @@ -626,9 +632,9 @@ impl CowFile { drop(super_block_guard); let meta_block_id = super_block.body.meta_block_id; - let _ = disk_pool.invalidate_block(file_id, meta_block_id); + let _ = disk_pool.invalidate_block(disk_guard, file_id, meta_block_id); let meta_block_guard = disk_pool - .read_block(file_kind, &self.file, &pool_guard, meta_block_id) + .read_block(file_kind, &self.file, disk_guard, meta_block_id) .await .change_context(RuntimeError::FileRootAccess) .attach_with(|| { diff --git a/doradb-storage/src/file/fs.rs b/doradb-storage/src/file/fs.rs index d15605b3..8342b202 100644 --- a/doradb-storage/src/file/fs.rs +++ b/doradb-storage/src/file/fs.rs @@ -2,7 +2,7 @@ use crate::buffer::guard::PageExclusiveGuard; use crate::buffer::page::Page; use crate::buffer::{ EvictReadSubmission, EvictSubmission, EvictableBufferPool, EvictablePoolStateMachine, - PoolRequest, PoolRole, ReadSubmission, ReadonlyBufferPool, + PoolGuard, PoolRequest, PoolRole, ReadSubmission, ReadonlyBufferPool, }; use crate::catalog::is_user_table; use crate::catalog::table::TableMetadata; @@ -1854,6 +1854,7 @@ impl FileSystem { &self, table_id: TableID, disk_pool: QuiescentGuard, + disk_guard: &PoolGuard, ) -> RuntimeResult> { let file_path = self.user_table_file_path(table_id); let table_file = Arc::new( @@ -1863,7 +1864,9 @@ impl FileSystem { format!("operation=open_table_file, table_id={table_id}, file_path={file_path}") })?, ); - let active_root = table_file.load_active_root_from_pool(&disk_pool).await?; + let active_root = table_file + .load_active_root_from_pool(&disk_pool, disk_guard) + .await?; let old_root = table_file.install_loaded_root(active_root); debug_assert!(old_root.is_none()); Ok(table_file) @@ -1991,6 +1994,7 @@ impl FileSystem { pub(crate) async fn open_or_create_multi_table_file( &self, disk_pool: QuiescentGuard, + disk_guard: &PoolGuard, ) -> RuntimeResult> { let file_path = self.catalog_mtb_file_path(); match MultiTableFile::open_or_create(&file_path) @@ -2000,7 +2004,9 @@ impl FileSystem { format!("operation=open_or_create_catalog_file, file_path={file_path}") })? { MultiTableFileOpenOutcome::Opened(mtb) => { - let active_root = mtb.load_active_root_from_pool(&disk_pool).await?; + let active_root = mtb + .load_active_root_from_pool(&disk_pool, disk_guard) + .await?; let old_root = mtb.install_loaded_root(active_root); debug_assert!(old_root.is_none()); Ok(mtb) @@ -2371,6 +2377,7 @@ pub(crate) mod tests { table_file, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); mutable.write_block(block_id, buf).await.unwrap(); drop(mutable); @@ -2968,7 +2975,10 @@ pub(crate) mod tests { let _hook = install_storage_backend_test_hook(Arc::new( FailingFirstWriteHook::new(path.clone()), )); - let err = match fs.open_or_create_multi_table_file(global.guard()).await { + let err = match fs + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) + .await + { Ok(_) => panic!("expected initial catalog.mtb publish failure"), Err(err) => err, }; @@ -2993,7 +3003,7 @@ pub(crate) mod tests { ); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let snapshot = mtb.load_snapshot(); @@ -3055,6 +3065,7 @@ pub(crate) mod tests { &table_file, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); let err = match mutable.commit(TrxID::new(2), false).await { Ok(_) => panic!("expected table commit fsync failure"), @@ -3131,11 +3142,15 @@ pub(crate) mod tests { write_payload(&fs, &table_file, BlockID::from(7usize), b"table-read").await; let reopened = fs - .open_table_file(table_id, fs.disk_pool().clone_inner()) + .open_table_file( + table_id, + fs.disk_pool().clone_inner(), + &fs.disk_pool().create_base_guard(), + ) .await .unwrap(); let readonly_pool = fs.disk_pool().clone_inner(); - let readonly_guard = readonly_pool.pool_guard(); + let readonly_guard = readonly_pool.create_base_guard(); let index_pool = fs.index_pool(); let background_file = StorageBackendFileIdentity::from_path(temp_dir.path().join("index.swp")).unwrap(); @@ -3227,7 +3242,7 @@ pub(crate) mod tests { let read_stats_start = mem_pool.stats(); let mem_pool_probe = mem_pool.clone(); - let pool_guard = mem_pool.pool_guard(); + let pool_guard = mem_pool.create_base_guard(); let reload_task = smol::spawn(async move { let g = mem_pool .get_page::(&pool_guard, reload_page_id, LatchFallbackMode::Shared) diff --git a/doradb-storage/src/file/multi_table_file.rs b/doradb-storage/src/file/multi_table_file.rs index 6105e794..7235790b 100644 --- a/doradb-storage/src/file/multi_table_file.rs +++ b/doradb-storage/src/file/multi_table_file.rs @@ -1,5 +1,5 @@ use crate::bitmap::AllocMap; -use crate::buffer::ReadonlyBufferPool; +use crate::buffer::{PoolGuard, ReadonlyBufferPool}; use crate::catalog::{ USER_TABLE_ID_LIMIT, USER_TABLE_ID_START, catalog_table_id_from_slot, catalog_table_slot, }; @@ -209,9 +209,10 @@ impl MultiTableFile { pub(crate) async fn load_active_root_from_pool( &self, disk_pool: &QuiescentGuard, + disk_guard: &PoolGuard, ) -> RuntimeResult { self.file - .load_active_root_from_pool(FileKind::CatalogMultiTableFile, disk_pool) + .load_active_root_from_pool(FileKind::CatalogMultiTableFile, disk_pool, disk_guard) .await } @@ -710,7 +711,7 @@ mod tests { let (_dir, fs) = build_test_fs(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let displaced_meta_block_id = (1..mtb.active_root_unchecked().alloc_map.len()) @@ -760,7 +761,7 @@ mod tests { let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let s0 = mtb.load_snapshot(); @@ -789,7 +790,7 @@ mod tests { drop(mtb); let mtb2 = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let s1 = mtb2.load_snapshot(); @@ -811,7 +812,7 @@ mod tests { let background_writes = fs.background_writes(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); @@ -841,7 +842,7 @@ mod tests { let fs = build_test_fs_in(dir.path()); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let reloaded = mtb.load_snapshot(); @@ -857,7 +858,7 @@ mod tests { let background_writes = fs.background_writes(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); @@ -898,7 +899,7 @@ mod tests { let path = fs.catalog_mtb_file_path(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); drop(mtb); @@ -921,7 +922,9 @@ mod tests { file.sync_all().unwrap(); let fs = build_test_fs_in(dir.path()); - let res = fs.open_or_create_multi_table_file(global.guard()).await; + let res = fs + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) + .await; assert!(res.is_err()); }); } @@ -933,7 +936,7 @@ mod tests { let path = fs.catalog_mtb_file_path(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let active_meta_block_id = mtb.active_root_unchecked().meta_block_id; @@ -956,7 +959,10 @@ mod tests { file.sync_all().unwrap(); let fs = build_test_fs_in(dir.path()); - let err = match fs.open_or_create_multi_table_file(global.guard()).await { + let err = match fs + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) + .await + { Ok(_) => panic!("expected multi-table meta version corruption"), Err(err) => err, }; @@ -975,7 +981,7 @@ mod tests { let path = fs.catalog_mtb_file_path(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let active_meta_block_id = mtb.active_root_unchecked().meta_block_id; @@ -987,7 +993,10 @@ mod tests { overwrite_file_bytes(&path, checksum_offset, &[0xff]); let fs = build_test_fs_in(dir.path()); - let err = match fs.open_or_create_multi_table_file(global.guard()).await { + let err = match fs + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) + .await + { Ok(_) => panic!("expected multi-table meta checksum corruption"), Err(err) => err, }; @@ -1007,7 +1016,7 @@ mod tests { let path = fs.catalog_mtb_file_path(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); @@ -1048,7 +1057,7 @@ mod tests { let fs = build_test_fs_in(dir.path()); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); let snapshot = mtb.load_snapshot(); @@ -1068,7 +1077,7 @@ mod tests { let path = fs.catalog_mtb_file_path(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); @@ -1130,7 +1139,10 @@ mod tests { ); let fs = build_test_fs_in(dir.path()); - let err = match fs.open_or_create_multi_table_file(global.guard()).await { + let err = match fs + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) + .await + { Ok(_) => panic!("expected newest multi-table root invariant failure"), Err(err) => err, }; @@ -1149,7 +1161,7 @@ mod tests { let (_dir, fs) = build_test_fs(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); @@ -1164,7 +1176,7 @@ mod tests { let (_dir, fs) = build_test_fs(); let global = global_readonly_pool_scope(64 * 1024 * 1024); let mtb = fs - .open_or_create_multi_table_file(global.guard()) + .open_or_create_multi_table_file(global.guard(), &global.create_base_guard()) .await .unwrap(); diff --git a/doradb-storage/src/file/table_file.rs b/doradb-storage/src/file/table_file.rs index 941d8c62..23bbccb8 100644 --- a/doradb-storage/src/file/table_file.rs +++ b/doradb-storage/src/file/table_file.rs @@ -1,5 +1,5 @@ use crate::bitmap::AllocMap; -use crate::buffer::ReadonlyBufferPool; +use crate::buffer::{PoolGuard, ReadonlyBufferPool}; use crate::catalog::table::TableMetadata; use crate::error::{ CompletionErrorBridge, CompletionResult, DataIntegrityResult, IoResult, MultiDomainResultExt, @@ -166,9 +166,10 @@ impl TableFile { pub(crate) async fn load_active_root_from_pool( &self, disk_pool: &QuiescentGuard, + disk_guard: &PoolGuard, ) -> RuntimeResult { self.file - .load_active_root_from_pool(FileKind::TableFile, disk_pool) + .load_active_root_from_pool(FileKind::TableFile, disk_pool, disk_guard) .await } @@ -262,18 +263,27 @@ impl MutableTableFile { } /// Fork the whole table file with readonly-cache write barriers enabled. + /// + /// `disk_guard` must come from the operation/session that owns this fork. + /// The mutable file retains that same root for every block invalidation; + /// creating a base guard here would turn a routine fork into a hidden + /// pool-global lifecycle acquisition. #[inline] pub(crate) fn fork( table_file: &Arc, background_writes: &IOClient, disk_pool: QuiescentGuard, + disk_guard: PoolGuard, ) -> Self { let writer_claim = MutableWriterClaim::new(table_file); MutableTableFile { file: Arc::clone(table_file), new_root: MutableCowRoot::fork(table_file.active_root_unchecked()), background_writes: background_writes.clone(), - write_barrier: MutableTableWriteBarrier::ReadonlyPool(disk_pool), + write_barrier: MutableTableWriteBarrier::ReadonlyPool { + pool: disk_pool, + guard: disk_guard, + }, writer_claim, } } @@ -427,10 +437,10 @@ impl MutableTableFile { heap_redo_start_ts: TrxID, ts: TrxID, disk_pool: &QuiescentGuard, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult<()> { let table_file = Arc::clone(&self.file); let background_writes = self.background_writes.clone(); - let disk_pool_guard = disk_pool.pool_guard(); let mut max_row_id = self.root().pivot_row_id; let mut writes = Vec::with_capacity(lwc_blocks.len()); let mut new_entries = Vec::with_capacity(lwc_blocks.len()); @@ -490,7 +500,7 @@ impl MutableTableFile { table_file.file_kind(), table_file.sparse_file(), disk_pool, - &disk_pool_guard, + disk_guard, ); let new_root = column_index .batch_insert(self, &new_entries, max_row_id, ts) @@ -561,7 +571,10 @@ impl MutableCowFile for MutableTableFile { } enum MutableTableWriteBarrier { - ReadonlyPool(QuiescentGuard), + ReadonlyPool { + pool: QuiescentGuard, + guard: PoolGuard, + }, Disabled, } @@ -569,7 +582,9 @@ impl MutableTableWriteBarrier { #[inline] fn as_cow_write_barrier(&self) -> CowWriteBarrier<'_> { match self { - MutableTableWriteBarrier::ReadonlyPool(pool) => CowWriteBarrier::readonly_pool(pool), + MutableTableWriteBarrier::ReadonlyPool { pool, guard } => { + CowWriteBarrier::readonly_pool(pool, guard) + } MutableTableWriteBarrier::Disabled => CowWriteBarrier::Disabled, } } @@ -737,7 +752,7 @@ mod tests { ) -> Result { let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(0), table_file); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let page = disk_pool .read_validated_block(&disk_pool_guard, page_id, accept_any_page) .await @@ -754,8 +769,9 @@ mod tests { ts: TrxID, disk_pool: &QuiescentGuard, ) -> Result<(Arc, Option)> { + let disk_guard = disk_pool.create_base_guard(); mutable_file - .apply_lwc_blocks(lwc_blocks, heap_redo_start_ts, ts, disk_pool) + .apply_lwc_blocks(lwc_blocks, heap_redo_start_ts, ts, disk_pool, &disk_guard) .await .disclose()?; mutable_file.commit(ts, false).await.disclose() @@ -799,6 +815,7 @@ mod tests { &table_file, background_writes, test_disk_pool.global_pool().clone(), + test_disk_pool.create_base_guard(), ); let res = mutable.write_block(test_block_id(3), buf).await; assert!(res.is_ok()); @@ -811,12 +828,20 @@ mod tests { drop(table_file); - let table_file2 = fs.open_table_file(table_id, global.guard()).await.unwrap(); + let disk_guard = global.create_base_guard(); + let table_file2 = fs + .open_table_file(table_id, global.guard(), &disk_guard) + .await + .unwrap(); let disk_pool = global.guard(); assert_eq!(table_file2.active_root_unchecked().root_ts, TrxID::new(1)); - let mut mutable = - MutableTableFile::fork(&table_file2, background_writes, disk_pool.clone()); + let mut mutable = MutableTableFile::fork( + &table_file2, + background_writes, + disk_pool.clone(), + disk_guard.clone(), + ); let secondary_root = mutable.allocate_block().unwrap(); mutable.set_secondary_index_root(0, secondary_root); assert_eq!(mutable.secondary_index_root(0), secondary_root); @@ -831,7 +856,7 @@ mod tests { let (table_file3, old_root) = mutable.commit(TrxID::new(2), false).await.unwrap(); drop(old_root); let active_root = table_file3 - .load_active_root_from_pool(&disk_pool) + .load_active_root_from_pool(&disk_pool, &disk_guard) .await .unwrap(); assert_eq!(active_root.slot_no, 1); @@ -857,10 +882,12 @@ mod tests { let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, table_id, &table_file); + let disk_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table_file, background_writes, disk_pool.global_pool().clone(), + disk_guard.clone(), ); let inherited_root = mutable.allocate_block().unwrap(); mutable.set_secondary_index_root(0, inherited_root); @@ -872,6 +899,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); let allocated_before = mutable.root().alloc_map.allocated(); let panic = catch_unwind(AssertUnwindSafe(|| { @@ -921,7 +949,7 @@ mod tests { let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, table_id, &table_file); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let block_id = first_unallocated_blocks(table_file.active_root_unchecked(), 1) .pop() .unwrap(); @@ -937,6 +965,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); let allocated = mutable.allocate_block().unwrap(); assert_eq!(allocated, block_id); @@ -966,7 +995,7 @@ mod tests { let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, table_id, &table_file); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let meta_block_id = first_unallocated_blocks(table_file.active_root_unchecked(), 1) .pop() .unwrap(); @@ -982,6 +1011,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); let (table_file, old_root) = mutable.commit(TrxID::new(2), false).await.unwrap(); drop(old_root); @@ -1009,7 +1039,7 @@ mod tests { let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, table_id, &table_file); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let before_root = table_file.active_root_unchecked().clone(); let cached_blocks = first_unallocated_blocks(&before_root, 8); for block_id in &cached_blocks { @@ -1049,6 +1079,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); mutable .apply_lwc_blocks( @@ -1056,6 +1087,7 @@ mod tests { TrxID::new(7), TrxID::new(2), disk_pool.global_pool(), + &disk_pool_guard, ) .await .unwrap(); @@ -1095,7 +1127,7 @@ mod tests { let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, table_id, &table_file); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let out_of_range_page_id = test_block_id(1_000_000); let res = disk_pool .read_validated_block(&disk_pool_guard, out_of_range_page_id, accept_any_page) @@ -1133,7 +1165,7 @@ mod tests { .unwrap(), ); let disk_pool = global.guard(); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); // We first shut down the file system, then send the IO request, // it should fail. @@ -1233,7 +1265,10 @@ mod tests { let fs = build_test_fs_in(temp_dir.path()); let global = global_readonly_pool_scope(64 * 1024 * 1024); - let err = match fs.open_table_file(table_id, global.guard()).await { + let err = match fs + .open_table_file(table_id, global.guard(), &global.create_base_guard()) + .await + { Ok(_) => panic!("expected table meta checksum corruption"), Err(err) => err, }; @@ -1270,7 +1305,10 @@ mod tests { let fs = build_test_fs_in(temp_dir.path()); let global = global_readonly_pool_scope(64 * 1024 * 1024); - let err = match fs.open_table_file(table_id, global.guard()).await { + let err = match fs + .open_table_file(table_id, global.guard(), &global.create_base_guard()) + .await + { Ok(_) => panic!("expected table meta version corruption"), Err(err) => err, }; @@ -1321,6 +1359,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ), lwc_blocks, TrxID::new(7), @@ -1338,7 +1377,7 @@ mod tests { assert_eq!(active_root.deletion_cutoff_ts, TrxID::new(1)); assert_ne!(active_root.column_block_index_root, SUPER_BLOCK_ID); let disk_pool = table_readonly_pool(&global, table_id, &table_file); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let column_index = ColumnBlockIndex::new( active_root.column_block_index_root, @@ -1414,6 +1453,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ), lwc_blocks, TrxID::new(7), @@ -1443,11 +1483,13 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); let _second = MutableTableFile::fork( &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); }); } @@ -1469,6 +1511,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); drop(first); @@ -1476,6 +1519,7 @@ mod tests { &table_file, background_writes, disk_pool.global_pool().clone(), + disk_pool.create_base_guard(), ); }); } diff --git a/doradb-storage/src/index/block_index.rs b/doradb-storage/src/index/block_index.rs index 735d4413..c430a2d7 100644 --- a/doradb-storage/src/index/block_index.rs +++ b/doradb-storage/src/index/block_index.rs @@ -367,8 +367,8 @@ mod tests { } #[inline] - fn pool_guard(&self) -> PoolGuard { - self.inner.pool_guard() + fn create_base_guard(&self) -> PoolGuard { + self.inner.create_base_guard() } #[inline] @@ -456,7 +456,7 @@ mod tests { fn test_block_index_root_accessors_and_update() { smol::block_on(async { let meta_pool = owned_index_pool(64 * 1024 * 1024); - let meta_guard = (*meta_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); let blk_idx = BlockIndex::new( meta_pool.guard(), &meta_guard, @@ -501,7 +501,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 64 * 1024 * 1024).unwrap(), ); - let meta_guard = (*pool).pool_guard(); + let meta_guard = (*pool).create_base_guard(); let blk_idx = smol::block_on(BlockIndex::new( pool.guard(), &meta_guard, @@ -520,8 +520,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(64 * 1024 * 1024); let mem_pool = owned_mem_pool(64 * 1024 * 1024); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let blk_idx = BlockIndex::new( meta_pool.guard(), diff --git a/doradb-storage/src/index/btree/cursor.rs b/doradb-storage/src/index/btree/cursor.rs index f3d249b5..2dbed870 100644 --- a/doradb-storage/src/index/btree/cursor.rs +++ b/doradb-storage/src/index/btree/cursor.rs @@ -302,7 +302,7 @@ mod tests { fn test_btree_cursor_resumes_with_raw_upper_fence_before_strict_successor() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let fixture = build_exact_boundary_resume_fixture(pool.guard(), &pool_guard).await; @@ -366,7 +366,7 @@ mod tests { fn test_btree_compactor_parent_done_buffers_raw_upper_fence() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let fixture = build_exact_boundary_resume_fixture(pool.guard(), &pool_guard).await; let mut compactor = diff --git a/doradb-storage/src/index/btree/mod.rs b/doradb-storage/src/index/btree/mod.rs index 35e2913c..74e3fca7 100644 --- a/doradb-storage/src/index/btree/mod.rs +++ b/doradb-storage/src/index/btree/mod.rs @@ -1994,7 +1994,7 @@ mod tests { async fn run_lookup_against_map(hints_enabled: bool) { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, hints_enabled, TrxID::new(200)) .await .expect("test btree construction should succeed"); @@ -2033,7 +2033,7 @@ mod tests { fn test_btree_merge_partial_branch_suffix_drops_lower_fence_child() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await .expect("test btree construction should succeed"); @@ -2101,7 +2101,7 @@ mod tests { fn test_btree_merge_full_deletes_parent_separator_with_branch_value_width() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await .expect("test btree construction should succeed"); @@ -2168,7 +2168,7 @@ mod tests { fn test_btree_delete_exact_checks_value_and_delete_state() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(100)) .await .expect("test btree construction should succeed"); @@ -2235,7 +2235,7 @@ mod tests { ) .unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await .expect("test btree construction should succeed"); @@ -2294,7 +2294,7 @@ mod tests { fn test_btree_replace_or_insert_semantics() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await .expect("test btree construction should succeed"); @@ -2371,7 +2371,7 @@ mod tests { fn test_btree_replace_or_insert_splits_for_absent_keys() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await .expect("test btree construction should succeed"); @@ -2409,7 +2409,7 @@ mod tests { fn test_btree_single_node() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await @@ -2564,7 +2564,7 @@ mod tests { fn test_btree_scale() { smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await @@ -2583,7 +2583,7 @@ mod tests { fn test_btree_delete() { smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await @@ -2604,7 +2604,7 @@ mod tests { fn test_btree_compact() { smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await @@ -2651,7 +2651,7 @@ mod tests { const ROWS: u64 = 10_000; const MAX_VALUE: u64 = 100_000; let pool = owned_index_pool(20 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(1)) .await @@ -2726,7 +2726,7 @@ mod tests { fn test_btree_split() { smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = Arc::new( BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) @@ -2805,7 +2805,7 @@ mod tests { fn test_btree_concurrent_split() { smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = Arc::new( BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) @@ -2861,7 +2861,7 @@ mod tests { fn test_btree_low_count_delete_insert_churn_reclaims_without_split() { smol::block_on(async { let pool = owned_index_pool(16 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let tree = BTree::new(pool.guard(), &pool_guard, true, TrxID::new(300)) .await .expect("test btree construction should succeed"); @@ -2934,7 +2934,7 @@ mod tests { fn test_btree_merge_partial() { smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); { let tree = BTree::new(pool.guard(), &pool_guard, false, TrxID::new(200)) .await @@ -2967,7 +2967,7 @@ mod tests { const H2_ROWS: u64 = WIDE_HEIGHT2_ROWS; smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); assert!(pool.allocated() == 0); // height=0 { diff --git a/doradb-storage/src/index/btree/node.rs b/doradb-storage/src/index/btree/node.rs index 18bf38f0..f4b7725e 100644 --- a/doradb-storage/src/index/btree/node.rs +++ b/doradb-storage/src/index/btree/node.rs @@ -2452,7 +2452,7 @@ mod tests { fn test_btree_node_insert() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { let mut page_guard = buf_pool @@ -2483,7 +2483,7 @@ mod tests { fn test_btree_node_delete() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { let mut page_guard = buf_pool @@ -2533,7 +2533,7 @@ mod tests { fn test_btree_node_delete_exact_checks_delete_state() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); let mut page_guard = buf_pool .allocate_page::(&buf_pool_guard) @@ -2819,7 +2819,7 @@ mod tests { fn test_btree_node_update() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { let mut page_guard = buf_pool @@ -2884,7 +2884,7 @@ mod tests { fn test_btree_node_compact_non_empty() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { // Create source leaf node with data @@ -2935,7 +2935,7 @@ mod tests { fn test_btree_node_compact_empty() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { // Create empty source node @@ -2984,7 +2984,7 @@ mod tests { fn test_btree_node_space_estimation() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { let mut page1_guard = buf_pool @@ -3056,7 +3056,7 @@ mod tests { fn test_btree_node_update_key() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { let mut page_guard = buf_pool @@ -3145,7 +3145,7 @@ mod tests { fn test_btree_node_enable_hints_seq() { smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { let mut page_guard = buf_pool .allocate_page::(&buf_pool_guard) @@ -3179,7 +3179,7 @@ mod tests { const COUNT: usize = 100; smol::block_on(async { let buf_pool = test_buf_pool(); - let buf_pool_guard = FixedBufferPool::pool_guard(&buf_pool); + let buf_pool_guard = FixedBufferPool::create_base_guard(&buf_pool); { let mut rng = ChaCha8Rng::seed_from_u64(0u64); let uniform = Uniform::new(0u64, 1u64 << 63).unwrap(); diff --git a/doradb-storage/src/index/column_block_index.rs b/doradb-storage/src/index/column_block_index.rs index c03e9a94..fde1caca 100644 --- a/doradb-storage/src/index/column_block_index.rs +++ b/doradb-storage/src/index/column_block_index.rs @@ -3702,9 +3702,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let disk_pool_guard = disk_pool.create_base_guard(); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let index = ColumnBlockIndex::new( SUPER_BLOCK_ID, RowID::new(0), @@ -3776,9 +3780,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let disk_pool_guard = disk_pool.create_base_guard(); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let root_block_id = ColumnBlockIndex::new( SUPER_BLOCK_ID, RowID::new(0), @@ -3883,9 +3891,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let disk_pool_guard = disk_pool.create_base_guard(); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let root_block_id = ColumnBlockIndex::new( SUPER_BLOCK_ID, RowID::new(0), @@ -3963,9 +3975,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let disk_pool_guard = disk_pool.create_base_guard(); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let entries = vec![ dense_entry(RowID::new(0), RowID::new(4), test_block_id(1001)), sparse_entry( @@ -4058,9 +4074,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let disk_pool_guard = disk_pool.create_base_guard(); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let root_v1 = ColumnBlockIndex::new( SUPER_BLOCK_ID, RowID::new(0), @@ -4083,8 +4103,12 @@ mod tests { .unwrap(); let (_table, _old_root) = mutable.commit(TrxID::new(2), false).await.unwrap(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let root_v2 = ColumnBlockIndex::new( root_v1, RowID::new(8), @@ -4141,15 +4165,19 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let row_ids = test_row_id_range(0, 96); let delete_deltas: Vec = (0..96).collect(); let entry = ColumnBlockEntryShape::new(RowID::new(0), RowID::new(96), row_ids, delete_deltas) .with_block_id(test_block_id(1001)); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let root = ColumnBlockIndex::new( SUPER_BLOCK_ID, RowID::new(0), @@ -4199,7 +4227,7 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let seed = dense_entry_with_delete_domain( RowID::new(0), @@ -4208,8 +4236,12 @@ mod tests { ColumnDeleteDomain::Ordinal, test_block_id(1001), ); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let root_v1 = ColumnBlockIndex::new( SUPER_BLOCK_ID, RowID::new(0), @@ -4223,8 +4255,12 @@ mod tests { .unwrap(); let (_table, _old_root) = mutable.commit(TrxID::new(2), false).await.unwrap(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let root_v2 = ColumnBlockIndex::new( root_v1, RowID::new(8), @@ -4280,9 +4316,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); - let mut mutable = - MutableTableFile::fork(&table, background_writes, disk_pool.global_pool().clone()); + let disk_pool_guard = disk_pool.create_base_guard(); + let mut mutable = MutableTableFile::fork( + &table, + background_writes, + disk_pool.global_pool().clone(), + disk_pool_guard.clone(), + ); let mut entries = Vec::new(); for idx in 0..(COLUMN_BLOCK_MAX_ENTRIES + 32) as u64 { entries.push(dense_entry( diff --git a/doradb-storage/src/index/column_deletion_blob.rs b/doradb-storage/src/index/column_deletion_blob.rs index 7b284eca..2577d58b 100644 --- a/doradb-storage/src/index/column_deletion_blob.rs +++ b/doradb-storage/src/index/column_deletion_blob.rs @@ -722,12 +722,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool_guard.clone(), ); let blob = vec![9u8; 513]; let blob_ref = { @@ -761,12 +762,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool_guard.clone(), ); let blob = vec![9u8; 513]; let blob_ref = { @@ -809,12 +811,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool_guard.clone(), ); let blob = vec![7u8; COLUMN_DELETION_BLOB_PAGE_BODY_SIZE * 2 + 113]; let blob_ref = { @@ -848,12 +851,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool_guard.clone(), ); let blob = vec![7u8; COLUMN_DELETION_BLOB_PAGE_BODY_SIZE * 2 + 113]; let blob_ref = { @@ -893,12 +897,13 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(1), &table); - let disk_pool_guard = disk_pool.pool_guard(); + let disk_pool_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_pool_guard.clone(), ); let first_blob = vec![3u8; COLUMN_DELETION_BLOB_PAGE_BODY_SIZE - COLUMN_AUX_BLOB_HEADER_SIZE]; diff --git a/doradb-storage/src/index/disk_tree.rs b/doradb-storage/src/index/disk_tree.rs index 3a320761..068dd630 100644 --- a/doradb-storage/src/index/disk_tree.rs +++ b/doradb-storage/src/index/disk_tree.rs @@ -551,12 +551,6 @@ impl DiskTreeRuntime { DiskTree::from_root_snapshot(root_block_id, self, disk_pool_guard) } - /// Returns a readonly buffer-pool guard for opening DiskTree snapshots. - #[inline] - pub(crate) fn disk_pool_guard(&self) -> PoolGuard { - self.disk_pool.pool_guard() - } - /// Returns the shared key encoder for this DiskTree shape. #[inline] pub(crate) fn encoder(&self) -> Arc { @@ -2721,7 +2715,7 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(301), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); assert_eq!(tree.lookup(&[Val::from(1u32)]).await.unwrap(), None); @@ -2743,7 +2737,7 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(302), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let runtime = non_unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); assert!( @@ -2775,11 +2769,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(303), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -2848,11 +2843,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(310), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -2932,11 +2928,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(311), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = non_unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -2978,11 +2975,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(312), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -3044,11 +3042,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(304), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = non_unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -3129,11 +3128,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(309), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = non_unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -3187,11 +3187,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(310), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let inner = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let allocated_before = inner.root().alloc_map.allocated(); let mut mutable = FailingDiskTreeWriteFile::new(inner, Some(1), None); @@ -3232,11 +3233,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(311), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let inner = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let allocated_before = inner.root().alloc_map.allocated(); let mut mutable = FailingDiskTreeWriteFile::new(inner, None, Some(0)); @@ -3280,11 +3282,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(305), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let unique_runtime = unique_runtime!(metadata, disk_pool); @@ -3443,11 +3446,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(306), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -3529,11 +3533,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(307), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = non_unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -3643,11 +3648,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(308), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -3730,11 +3736,12 @@ mod tests { let write_global = global_readonly_pool_scope(64 * 1024 * 1024); let write_disk_pool = table_readonly_pool(&write_global, test_user_table_id(309), &table); - let write_guard = write_disk_pool.pool_guard(); + let write_guard = write_disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), write_disk_pool.global_pool().clone(), + write_guard.clone(), ); let write_runtime = unique_runtime!(metadata, write_disk_pool); let tree = write_runtime.open(SUPER_BLOCK_ID, &write_guard); @@ -3758,7 +3765,7 @@ mod tests { let inspect_global = global_readonly_pool_scope(64 * 1024 * 1024); let inspect_disk_pool = table_readonly_pool(&inspect_global, test_user_table_id(309), &table); - let inspect_guard = inspect_disk_pool.pool_guard(); + let inspect_guard = inspect_disk_pool.create_base_guard(); let inspect_runtime = unique_runtime!(metadata, inspect_disk_pool); let inspect_tree = inspect_runtime.open(root, &inspect_guard); let root_guard = inspect_tree.read_node(root).await.unwrap(); @@ -3772,7 +3779,7 @@ mod tests { let collect_global = global_readonly_pool_scope(64 * 1024 * 1024); let collect_disk_pool = table_readonly_pool(&collect_global, test_user_table_id(309), &table); - let collect_guard = collect_disk_pool.pool_guard(); + let collect_guard = collect_disk_pool.create_base_guard(); let collect_runtime = unique_runtime!(metadata, collect_disk_pool); let collect_tree = collect_runtime.open(root, &collect_guard); let start_stats = collect_disk_pool.global_stats(); @@ -3806,11 +3813,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(313), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -3920,11 +3928,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(314), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); @@ -4014,11 +4023,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(305), &table); - let guard = disk_pool.pool_guard(); + let guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + guard.clone(), ); let runtime = unique_runtime!(metadata, disk_pool); let tree = runtime.open(SUPER_BLOCK_ID, &guard); diff --git a/doradb-storage/src/index/non_unique_index.rs b/doradb-storage/src/index/non_unique_index.rs index 3eb4a225..55a77887 100644 --- a/doradb-storage/src/index/non_unique_index.rs +++ b/doradb-storage/src/index/non_unique_index.rs @@ -388,7 +388,7 @@ mod tests { FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); { - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_non_unique_mem_index( &pool, &pool_guard, @@ -406,7 +406,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_non_unique_mem_index( &pool, &pool_guard, @@ -460,7 +460,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 64 * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_non_unique_mem_index( &pool, &pool_guard, @@ -508,7 +508,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_non_unique_mem_index( &pool, &pool_guard, @@ -569,7 +569,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_non_unique_mem_index( &pool, &pool_guard, @@ -604,7 +604,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_non_unique_mem_index( &pool, &pool_guard, @@ -696,7 +696,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_non_unique_mem_index( &pool, &pool_guard, diff --git a/doradb-storage/src/index/row_page_index.rs b/doradb-storage/src/index/row_page_index.rs index 2ffbfea7..2eae7c58 100644 --- a/doradb-storage/src/index/row_page_index.rs +++ b/doradb-storage/src/index/row_page_index.rs @@ -1958,8 +1958,8 @@ mod tests { } #[inline] - fn pool_guard(&self) -> PoolGuard { - self.inner.pool_guard() + fn create_base_guard(&self) -> PoolGuard { + self.inner.create_base_guard() } #[inline] @@ -2055,7 +2055,7 @@ mod tests { .unwrap(); { let metadata = make_test_metadata(); - let meta_guard = engine.inner().pools.meta.pool_guard(); + let meta_guard = engine.inner().pools.meta.create_base_guard(); let blk_idx = RowPageIndex::new( engine.inner().pools.meta.clone(), &meta_guard, @@ -2063,7 +2063,7 @@ mod tests { ) .await .expect("test row-page-index construction should succeed"); - let mem_guard = engine.inner().pools.mem.pool_guard(); + let mem_guard = engine.inner().pools.mem.create_base_guard(); let p1 = blk_idx .get_insert_page( &meta_guard, @@ -2115,7 +2115,7 @@ mod tests { .unwrap(); { let metadata = make_test_metadata(); - let meta_guard = engine.inner().pools.meta.pool_guard(); + let meta_guard = engine.inner().pools.meta.create_base_guard(); let blk_idx = RowPageIndex::new( engine.inner().pools.meta.clone(), &meta_guard, @@ -2123,7 +2123,7 @@ mod tests { ) .await .expect("test row-page-index construction should succeed"); - let mem_guard = engine.inner().pools.mem.pool_guard(); + let mem_guard = engine.inner().pools.mem.create_base_guard(); let p1 = blk_idx .get_insert_page_exclusive( &meta_guard, @@ -2159,8 +2159,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(fixed_pool_bytes(1)); let mem_pool = owned_mem_pool(fixed_pool_bytes(1)); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let blk_idx = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await .expect("test row-page-index construction should succeed"); @@ -2231,8 +2231,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(64 * 1024 * 1024); let mem_pool = owned_mem_pool(64 * 1024 * 1024); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let blk_idx = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await @@ -2268,8 +2268,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(64 * 1024 * 1024); let mem_pool = owned_mem_pool(64 * 1024 * 1024); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let index = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await @@ -2308,8 +2308,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(64 * 1024 * 1024); let mem_pool = owned_mem_pool(64 * 1024 * 1024); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let blk_idx = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await @@ -2340,8 +2340,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(64 * 1024 * 1024); let mem_pool = owned_mem_pool(64 * 1024 * 1024); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let blk_idx = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await @@ -2379,8 +2379,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(fixed_pool_bytes(1)); let mem_pool = owned_mem_pool(fixed_pool_bytes(1)); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let blk_idx = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await @@ -2408,8 +2408,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(fixed_pool_bytes(1)); let mem_pool = owned_mem_pool(fixed_pool_bytes(1)); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let blk_idx = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await @@ -2437,8 +2437,8 @@ mod tests { smol::block_on(async { let meta_pool = owned_index_pool(fixed_pool_bytes(1)); let mem_pool = owned_mem_pool(fixed_pool_bytes(1)); - let meta_guard = (*meta_pool).pool_guard(); - let mem_guard = (*mem_pool).pool_guard(); + let meta_guard = (*meta_pool).create_base_guard(); + let mem_guard = (*mem_pool).create_base_guard(); let metadata = make_test_metadata(); let blk_idx = RowPageIndex::new(meta_pool.guard(), &meta_guard, RowID::new(0)) .await @@ -2497,7 +2497,7 @@ mod tests { .unwrap(); { let metadata = make_test_metadata(); - let meta_guard = engine.inner().pools.meta.pool_guard(); + let meta_guard = engine.inner().pools.meta.create_base_guard(); let blk_idx = RowPageIndex::new( engine.inner().pools.meta.clone(), &meta_guard, @@ -2505,7 +2505,7 @@ mod tests { ) .await .expect("test row-page-index construction should succeed"); - let mem_guard = engine.inner().pools.mem.pool_guard(); + let mem_guard = engine.inner().pools.mem.create_base_guard(); for _ in 0..row_pages { let _ = blk_idx .get_insert_page( @@ -2537,7 +2537,7 @@ mod tests { fn test_row_page_index_cursor_two_level_tree() { smol::block_on(async { let pool = owned_index_pool(1024usize * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let blk_idx = RowPageIndex::new(pool.guard(), &pool_guard, RowID::new(0)) .await .expect("test row-page-index construction should succeed"); @@ -2607,7 +2607,7 @@ mod tests { fn test_prune_checkpoint_prefix_in_root_leaf_and_reset_empty_root() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = RowPageIndex::new(pool.guard(), &pool_guard, RowID::new(0)) .await .unwrap(); @@ -2672,7 +2672,7 @@ mod tests { fn test_prune_checkpoint_prefix_reclaims_leaf_and_collapses_root() { smol::block_on(async { let pool = owned_index_pool(128 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = RowPageIndex::new(pool.guard(), &pool_guard, RowID::new(0)) .await .unwrap(); @@ -2714,7 +2714,7 @@ mod tests { fn test_prune_checkpoint_prefix_across_multiple_branch_levels() { smol::block_on(async { let pool = owned_index_pool(64 * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = RowPageIndex::new(pool.guard(), &pool_guard, RowID::new(0)) .await .unwrap(); @@ -2817,7 +2817,7 @@ mod tests { fn test_row_page_index_search() { smol::block_on(async { let pool = owned_index_pool(512usize * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let blk_idx = RowPageIndex::new(pool.guard(), &pool_guard, RowID::new(0)) .await .expect("test row-page-index construction should succeed"); @@ -2859,7 +2859,7 @@ mod tests { fn test_row_page_index_split() { smol::block_on(async { let pool = owned_index_pool(1024usize * 1024 * 1024); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let blk_idx = RowPageIndex::new(pool.guard(), &pool_guard, RowID::new(0)) .await .expect("test row-page-index construction should succeed"); @@ -2939,7 +2939,7 @@ mod tests { .unwrap(); { let metadata = make_test_metadata(); - let meta_guard = engine.inner().pools.meta.pool_guard(); + let meta_guard = engine.inner().pools.meta.create_base_guard(); let blk_idx = RowPageIndex::new( engine.inner().pools.meta.clone(), &meta_guard, @@ -2947,7 +2947,7 @@ mod tests { ) .await .expect("test row-page-index construction should succeed"); - let mem_guard = engine.inner().pools.mem.pool_guard(); + let mem_guard = engine.inner().pools.mem.create_base_guard(); let redo_ctx = RowPageCreateRedoCtx::new(&engine.inner().trx_sys, TableID::new(104)); let page_guard = blk_idx @@ -3022,7 +3022,7 @@ mod tests { .await .unwrap(); let metadata = make_test_metadata(); - let meta_guard = engine.inner().pools.meta.pool_guard(); + let meta_guard = engine.inner().pools.meta.create_base_guard(); let blk_idx = RowPageIndex::new( engine.inner().pools.meta.clone(), &meta_guard, @@ -3030,7 +3030,7 @@ mod tests { ) .await .expect("test row-page-index construction should succeed"); - let mem_guard = engine.inner().pools.mem.pool_guard(); + let mem_guard = engine.inner().pools.mem.create_base_guard(); let redo_ctx = RowPageCreateRedoCtx::new(&engine.inner().trx_sys, TableID::new(206)); let _ = engine .inner() @@ -3097,7 +3097,7 @@ mod tests { .unwrap(); { let meta_pool = &engine.inner().pools.meta; - let meta_guard = meta_pool.pool_guard(); + let meta_guard = meta_pool.create_base_guard(); let blk_idx = RowPageIndex::new(meta_pool.clone(), &meta_guard, RowID::new(0)) .await .expect("test row-page-index construction should succeed"); @@ -3109,7 +3109,7 @@ mod tests { let blk_idx = &blk_idx; let meta_pool_ref = meta_pool; async move { - let pool_guard = meta_pool_ref.pool_guard(); + let pool_guard = meta_pool_ref.create_base_guard(); let start = worker * pages_per_worker; let end = (start + pages_per_worker).min(total_pages); for page_no in start..end { diff --git a/doradb-storage/src/index/secondary_index.rs b/doradb-storage/src/index/secondary_index.rs index ab2ce8fb..fdb2d54f 100644 --- a/doradb-storage/src/index/secondary_index.rs +++ b/doradb-storage/src/index/secondary_index.rs @@ -187,15 +187,6 @@ impl SecondaryDiskTreeRuntime { self.index_no } - /// Borrow a guard for opening one or more DiskTree readers on this runtime. - #[inline] - pub(crate) fn disk_pool_guard(&self) -> PoolGuard { - match &self.kind { - SecondaryDiskTreeRuntimeKind::Unique(runtime) => runtime.disk_pool_guard(), - SecondaryDiskTreeRuntimeKind::NonUnique(runtime) => runtime.disk_pool_guard(), - } - } - /// Returns the shared key encoder for this secondary index. #[inline] pub(crate) fn key_encoder(&self) -> Arc { @@ -1179,11 +1170,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(611), &table); - let disk_guard = disk_pool.pool_guard(); + let disk_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_guard.clone(), ); let disk_runtime = unique_runtime!(metadata, disk_pool); let disk = disk_runtime.open(SUPER_BLOCK_ID, &disk_guard); @@ -1219,7 +1211,7 @@ mod tests { let index_pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 64 * 1024 * 1024).unwrap(), ); - let index_guard = (*index_pool).pool_guard(); + let index_guard = (*index_pool).create_base_guard(); let mem = unique_mem_index(&index_pool, &index_guard).await; assert!( mem.bind(&index_guard) @@ -1237,8 +1229,8 @@ mod tests { .unwrap(); let index = SecondaryIndex::Unique { mem, disk: runtime }; let pool_guards = PoolGuards::builder() - .push(PoolRole::Index, (*index_pool).pool_guard()) - .push(PoolRole::Disk, disk_pool.pool_guard()) + .push(PoolRole::Index, (*index_pool).create_base_guard()) + .push(PoolRole::Disk, disk_pool.create_base_guard()) .build(); let bound = index.bind_unique_unchecked(&pool_guards, root).unwrap(); @@ -1509,7 +1501,7 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(614), &table); - let disk_guard = disk_pool.pool_guard(); + let disk_guard = disk_pool.create_base_guard(); let key1 = [Val::from(1u32)]; let key2 = [Val::from(2u32)]; @@ -1517,6 +1509,7 @@ mod tests { &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_guard.clone(), ); let disk_runtime = unique_runtime!(metadata, disk_pool); let disk = disk_runtime.open(SUPER_BLOCK_ID, &disk_guard); @@ -1538,7 +1531,7 @@ mod tests { disk_pool.global_pool().clone(), ) .unwrap(); - let opened_a_guard = runtime.disk_pool_guard(); + let opened_a_guard = disk_pool.create_base_guard(); let opened_a = runtime.open_unique_at(root_a, &opened_a_guard).unwrap(); assert_eq!(opened_a.lookup(&key1).await.unwrap(), Some(RowID::new(10))); assert_eq!(opened_a.lookup(&key2).await.unwrap(), None); @@ -1547,6 +1540,7 @@ mod tests { &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_guard.clone(), ); let disk = disk_runtime.open(root_a, &disk_guard); let root_b = { @@ -1569,7 +1563,7 @@ mod tests { assert_eq!(opened_a.lookup(&key1).await.unwrap(), Some(RowID::new(10))); assert_eq!(opened_a.lookup(&key2).await.unwrap(), None); - let opened_b_guard = runtime.disk_pool_guard(); + let opened_b_guard = disk_pool.create_base_guard(); let opened_b = runtime.open_unique_at(root_b, &opened_b_guard).unwrap(); assert_eq!(opened_b.lookup(&key1).await.unwrap(), Some(RowID::new(10))); assert_eq!(opened_b.lookup(&key2).await.unwrap(), Some(RowID::new(20))); @@ -1588,11 +1582,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(612), &table); - let disk_guard = disk_pool.pool_guard(); + let disk_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_guard.clone(), ); let disk_runtime = non_unique_runtime!(metadata, disk_pool); let disk = disk_runtime.open(SUPER_BLOCK_ID, &disk_guard); @@ -1628,7 +1623,7 @@ mod tests { let index_pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 64 * 1024 * 1024).unwrap(), ); - let index_guard = (*index_pool).pool_guard(); + let index_guard = (*index_pool).create_base_guard(); let mem = non_unique_mem_index(&index_pool, &index_guard).await; assert!( mem.bind(&index_guard) @@ -1646,8 +1641,8 @@ mod tests { .unwrap(); let index = SecondaryIndex::NonUnique { mem, disk: runtime }; let pool_guards = PoolGuards::builder() - .push(PoolRole::Index, (*index_pool).pool_guard()) - .push(PoolRole::Disk, disk_pool.pool_guard()) + .push(PoolRole::Index, (*index_pool).create_base_guard()) + .push(PoolRole::Disk, disk_pool.create_base_guard()) .build(); let bound = index.bind_non_unique_unchecked(&pool_guards, root).unwrap(); @@ -1817,11 +1812,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(615), &table); - let disk_guard = disk_pool.pool_guard(); + let disk_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_guard.clone(), ); let disk_runtime = unique_runtime!(metadata, disk_pool); let disk = disk_runtime.open(SUPER_BLOCK_ID, &disk_guard); @@ -1857,7 +1853,7 @@ mod tests { let index_pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 64 * 1024 * 1024).unwrap(), ); - let index_guard = (*index_pool).pool_guard(); + let index_guard = (*index_pool).create_base_guard(); let mem = unique_mem_index(&index_pool, &index_guard).await; assert!( mem.bind(&index_guard) @@ -1875,8 +1871,8 @@ mod tests { .unwrap(); let index = SecondaryIndex::Unique { mem, disk: runtime }; let pool_guards = PoolGuards::builder() - .push(PoolRole::Index, (*index_pool).pool_guard()) - .push(PoolRole::Disk, disk_pool.pool_guard()) + .push(PoolRole::Index, (*index_pool).create_base_guard()) + .push(PoolRole::Disk, disk_pool.create_base_guard()) .build(); let bound = index.bind_unique_unchecked(&pool_guards, root).unwrap(); @@ -1924,11 +1920,12 @@ mod tests { drop(old_root); let global = global_readonly_pool_scope(64 * 1024 * 1024); let disk_pool = table_readonly_pool(&global, test_user_table_id(616), &table); - let disk_guard = disk_pool.pool_guard(); + let disk_guard = disk_pool.create_base_guard(); let mut mutable = MutableTableFile::fork( &table, fs.background_writes(), disk_pool.global_pool().clone(), + disk_guard.clone(), ); let disk_runtime = non_unique_runtime!(metadata, disk_pool); let disk = disk_runtime.open(SUPER_BLOCK_ID, &disk_guard); @@ -1959,7 +1956,7 @@ mod tests { let index_pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 64 * 1024 * 1024).unwrap(), ); - let index_guard = (*index_pool).pool_guard(); + let index_guard = (*index_pool).create_base_guard(); let mem = non_unique_mem_index(&index_pool, &index_guard).await; assert!( mem.bind(&index_guard) @@ -1977,8 +1974,8 @@ mod tests { .unwrap(); let index = SecondaryIndex::NonUnique { mem, disk: runtime }; let pool_guards = PoolGuards::builder() - .push(PoolRole::Index, (*index_pool).pool_guard()) - .push(PoolRole::Disk, disk_pool.pool_guard()) + .push(PoolRole::Index, (*index_pool).create_base_guard()) + .push(PoolRole::Disk, disk_pool.create_base_guard()) .build(); let bound = index.bind_non_unique_unchecked(&pool_guards, root).unwrap(); @@ -2029,7 +2026,7 @@ mod tests { let index_pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 64 * 1024 * 1024).unwrap(), ); - let index_guard = (*index_pool).pool_guard(); + let index_guard = (*index_pool).create_base_guard(); let mem = unique_mem_index(&index_pool, &index_guard).await; let shadow_key = [Val::from(9u32)]; let guarded = mem.bind(&index_guard); diff --git a/doradb-storage/src/index/unique_index.rs b/doradb-storage/src/index/unique_index.rs index c91f5873..8ac9119d 100644 --- a/doradb-storage/src/index/unique_index.rs +++ b/doradb-storage/src/index/unique_index.rs @@ -383,7 +383,7 @@ mod tests { FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); { - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_unique_mem_index( &pool, &pool_guard, @@ -405,7 +405,7 @@ mod tests { FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); { - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_unique_mem_index( &pool, &pool_guard, @@ -432,7 +432,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_unique_mem_index( &pool, &pool_guard, @@ -474,7 +474,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_unique_mem_index( &pool, &pool_guard, @@ -533,7 +533,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_unique_mem_index( &pool, &pool_guard, @@ -619,7 +619,7 @@ mod tests { let pool = QuiescentBox::new( FixedBufferPool::with_capacity(PoolRole::Index, 1024usize * 1024 * 1024).unwrap(), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let index = test_unique_mem_index( &pool, &pool_guard, diff --git a/doradb-storage/src/quiescent.rs b/doradb-storage/src/quiescent.rs index 9b5dfa9d..a150f896 100644 --- a/doradb-storage/src/quiescent.rs +++ b/doradb-storage/src/quiescent.rs @@ -195,6 +195,10 @@ impl QuiescentGuard { /// /// The wrapped direct guard still holds exactly one quiescent keepalive. /// Further clones only clone the outer `Arc` and do not touch guard_count. + /// The caller therefore chooses the contention domain: every call creates + /// a distinct `Arc` root, while every clone of its result updates that + /// root's strong count. High-frequency users should create roots at their + /// natural ownership boundary instead of sharing one process-wide root. #[inline] pub(crate) fn into_sync(self) -> SyncQuiescentGuard { SyncQuiescentGuard { @@ -296,6 +300,9 @@ fn guard_count_underflow() -> ! { panic!("quiescent guard count underflow"); } +#[cfg(test)] +pub(crate) use self::tests::shares_root as test_sync_guards_share_root; + #[cfg(test)] mod tests { use super::*; @@ -311,6 +318,15 @@ mod tests { dropped: Arc, } + /// Returns whether two wrappers update the same outer `Arc` strong count. + #[inline] + pub(crate) fn shares_root( + first: &SyncQuiescentGuard, + second: &SyncQuiescentGuard, + ) -> bool { + Arc::ptr_eq(&first.guard, &second.guard) + } + impl Drop for DropSpy { fn drop(&mut self) { self.dropped.store(true, Ordering::Release); @@ -502,4 +518,24 @@ mod tests { drop(guard); drop(owner); } + + #[test] + fn test_sync_quiescent_guard_fresh_roots_shard_outer_arc() { + let owner = QuiescentBox::new(()); + let first = owner.guard().into_sync(); + let second = owner.guard().into_sync(); + assert_eq!(owner.outstanding_guard_count(), 2); + assert!(!shares_root(&first, &second)); + + let first_clone = first.clone(); + assert!(shares_root(&first, &first_clone)); + assert_eq!(owner.outstanding_guard_count(), 2); + + drop(first); + assert_eq!(owner.outstanding_guard_count(), 2); + drop(first_clone); + assert_eq!(owner.outstanding_guard_count(), 1); + drop(second); + assert_eq!(owner.outstanding_guard_count(), 0); + } } diff --git a/doradb-storage/src/recovery/mod.rs b/doradb-storage/src/recovery/mod.rs index d94582a4..47a49ee9 100644 --- a/doradb-storage/src/recovery/mod.rs +++ b/doradb-storage/src/recovery/mod.rs @@ -348,6 +348,7 @@ impl<'a> RecoveryCoordinator<'a> { self.resources.pools.index.clone(), &self.resources.table_fs, self.resources.pools.disk.clone(), + &self.resources.pool_guards, table.table_id, ) .await?; @@ -705,6 +706,7 @@ impl<'a> RecoveryCoordinator<'a> { self.resources.pools.index.clone(), &self.resources.table_fs, self.resources.pools.disk.clone(), + &self.resources.pool_guards, table_id, ) .await?; @@ -1244,6 +1246,7 @@ mod tests { IndexKey, IndexObject, IndexOrder, IndexSpec, TableMetadata, TableObject, TableSpec, USER_TABLE_ID_START, }; + use crate::component::EnginePools; use crate::conf::{EngineConfig, EvictableBufferPoolConfig, FileSystemConfig, TrxSysConfig}; use crate::engine::Engine; use crate::error::{ @@ -1629,7 +1632,12 @@ mod tests { catalog_replay_start_ts: TrxID, ) -> RecoveryCoordinator<'a> { let resources = RecoveryResources::new( - engine.inner().core.pools.clone(), + EnginePools::new( + engine.inner().core.pools.meta.clone(), + engine.inner().core.pools.index.clone(), + engine.inner().core.pools.mem.clone(), + engine.inner().core.pools.disk.clone(), + ), engine.inner().table_fs.clone(), engine.inner().core.catalog(), ); @@ -1949,6 +1957,7 @@ mod tests { &table_file, engine.inner().table_fs.background_writes(), table.disk_pool().clone(), + engine.inner().core.pools.pool_guards().disk_guard().clone(), ); mutable.replace_metadata_and_secondary_index_roots(metadata, roots); engine @@ -4332,7 +4341,7 @@ mod tests { let active_root = table.file().active_root_unchecked(); let block_id = { - let disk_pool_guard = table.disk_pool().pool_guard(); + let disk_pool_guard = table.disk_pool().create_base_guard(); let index = ColumnBlockIndex::new( active_root.column_block_index_root, active_root.pivot_row_id, @@ -4494,7 +4503,7 @@ mod tests { let active_root = table.file().active_root_unchecked(); let blob_ref = { - let disk_pool_guard = table.disk_pool().pool_guard(); + let disk_pool_guard = table.disk_pool().create_base_guard(); let index = ColumnBlockIndex::new( active_root.column_block_index_root, active_root.pivot_row_id, diff --git a/doradb-storage/src/session.rs b/doradb-storage/src/session.rs index 0e68e9ad..2907e1f1 100644 --- a/doradb-storage/src/session.rs +++ b/doradb-storage/src/session.rs @@ -636,7 +636,7 @@ pub(crate) trait SessionRuntimeAccess { fn engine(&self) -> &EngineCore { self.runtime().core() } - /// Borrows the canonical pool-guard bundle. + /// Borrows the exact session's pool-guard roots. fn pool_guards(&self) -> &PoolGuards { self.runtime().pool_guards() } @@ -675,10 +675,10 @@ impl SessionRuntime { WeakSessionRef::new(&self.0) } - /// Borrow the canonical engine pool guard bundle. + /// Borrow the exact session's pool-guard roots. #[inline] pub(crate) fn pool_guards(&self) -> &PoolGuards { - self.core().pool_guards() + &self.0.pool_guards } /// Returns whether owner-side shutdown has closed operation admission. @@ -1034,9 +1034,10 @@ impl Session { .attach_with(|| format!("prepare CREATE INDEX locks: table_id={table_id}")) .disclose()?; let engine = scope.engine(); - let table = validated_index_ddl_target(engine, table_id, "create_index") - .await - .disclose()?; + let table = + validated_index_ddl_target(engine, engine.pool_guards(), table_id, "create_index") + .await + .disclose()?; engine.poisoner.ensure_healthy().disclose()?; let gates = IndexDdlGateScope::acquire(Arc::clone(&table), engine.catalog_guard()) .await @@ -1080,9 +1081,10 @@ impl Session { .attach_with(|| format!("prepare DROP INDEX locks: table_id={table_id}")) .disclose()?; let engine = scope.engine(); - let table = validated_index_ddl_target(engine, table_id, "drop_index") - .await - .disclose()?; + let table = + validated_index_ddl_target(engine, engine.pool_guards(), table_id, "drop_index") + .await + .disclose()?; engine.poisoner.ensure_healthy().disclose()?; let gates = IndexDdlGateScope::acquire(Arc::clone(&table), engine.catalog_guard()) .await @@ -1839,7 +1841,7 @@ impl SessionOperationPin { .reject_table_ddl_explicit_session_lock(table_id, self.operation_lock_owner()) } - /// Returns a cloned guard bundle for this foreground operation. + /// Borrows the exact session's pool-guard roots. #[inline] pub(crate) fn pool_guards(&self) -> &PoolGuards { self.runtime.pool_guards() @@ -2368,6 +2370,13 @@ impl SessionTableCacheEntry { /// Shared mutable state referenced by transactions started from one [`Session`]. pub(crate) struct SessionState { id: SessionID, + /// Per-session roots for page-guard `Arc` clone/drop traffic. + /// + /// Keep this field before `core`: Rust drops fields in declaration order, + /// so the session roots release their arena keepalives before the shared + /// engine capabilities. Do not replace them with `EngineCore`'s canonical + /// bundle; page lookup clones would again contend across all sessions. + pool_guards: PoolGuards, core: Arc, admission: Arc, lifecycle: Mutex, @@ -2383,8 +2392,12 @@ impl SessionState { admission: Arc, id: SessionID, ) -> Self { + // Four allocations and arena acquisitions are intentionally paid once + // at session creation to shard millions of page-guard Arc operations. + let pool_guards = core.pools.create_session_pool_guards(); SessionState { id, + pool_guards, core, admission, lifecycle: Mutex::new(SessionLifecycle { @@ -3178,7 +3191,7 @@ impl TrxAttachment { self.trx_id } - /// Borrows the canonical engine pool guards. + /// Borrows the exact session's pool-guard roots. #[inline] pub(crate) fn pool_guards(&self) -> &PoolGuards { self.runtime.pool_guards() @@ -3365,6 +3378,7 @@ async fn wait_for_maintenance_boundary( pub(crate) mod tests { use super::*; use crate::buffer::guard::PageGuard; + use crate::buffer::{PoolRole, test_pool_guards_share_keepalive_root}; use crate::catalog::storage::tables::TABLE_ID_TABLES; use crate::catalog::tests::{table1, table2, wait_for_dropped_table_floor}; use crate::catalog::{ @@ -4572,6 +4586,54 @@ pub(crate) mod tests { }); } + #[test] + fn test_sessions_use_independent_pool_guard_arc_roots() { + smol::block_on(async { + let root = TempDir::new().unwrap(); + let engine = Engine::bootstrap(EngineConfig::default().storage_root(root.path())) + .await + .unwrap(); + let session1 = engine.new_session().unwrap(); + let session2 = engine.new_session().unwrap(); + let runtime1 = test_session_runtime(&session1).unwrap(); + let runtime2 = test_session_runtime(&session2).unwrap(); + let guards1 = runtime1.pool_guards().clone(); + let guards1_clone = runtime1.pool_guards().clone(); + let guards2 = runtime2.pool_guards().clone(); + let canonical = engine.inner().core.pools.pool_guards().clone(); + + for role in [ + PoolRole::Meta, + PoolRole::Index, + PoolRole::Mem, + PoolRole::Disk, + ] { + let guard1 = guards1.guard(role); + let guard1_clone = guards1_clone.guard(role); + let guard2 = guards2.guard(role); + let canonical_guard = canonical.guard(role); + assert_eq!(guard1.identity(), guard2.identity(), "role={role:?}"); + assert_eq!( + guard1.identity(), + canonical_guard.identity(), + "role={role:?}" + ); + assert!( + test_pool_guards_share_keepalive_root(guard1, guard1_clone), + "same-session clones must share one root: role={role:?}" + ); + assert!( + !test_pool_guards_share_keepalive_root(guard1, guard2), + "different sessions must shard roots: role={role:?}" + ); + assert!( + !test_pool_guards_share_keepalive_root(guard1, canonical_guard), + "session and canonical engine work must shard roots: role={role:?}" + ); + } + }); + } + #[test] fn test_shutdown_inspection_collects_exact_claimable_transaction() { smol::block_on(async { diff --git a/doradb-storage/src/table/access.rs b/doradb-storage/src/table/access.rs index da44075b..6abd92f0 100644 --- a/doradb-storage/src/table/access.rs +++ b/doradb-storage/src/table/access.rs @@ -5603,9 +5603,11 @@ mod tests { let block_id = entry.block_id(); let table_file_path = engine.inner().table_fs.user_table_file_path(table_id); corrupt_page_checksum(table_file_path, block_id); - let _ = table - .disk_pool() - .invalidate_block(table.file().sparse_file().file_id(), block_id); + let _ = table.disk_pool().invalidate_block( + session.pool_guards().disk_guard(), + table.file().sparse_file().file_id(), + block_id, + ); let mut trx = session.begin_trx().unwrap(); let err = trx @@ -5651,9 +5653,11 @@ mod tests { let table_file_path = engine.inner().table_fs.user_table_file_path(table_id); corrupt_leaf_row_codec(table_file_path, entry.leaf_block_id, 0); - let _ = table - .disk_pool() - .invalidate_block(table.file().sparse_file().file_id(), entry.leaf_block_id); + let _ = table.disk_pool().invalidate_block( + session.pool_guards().disk_guard(), + table.file().sparse_file().file_id(), + entry.leaf_block_id, + ); let mut trx = session.begin_trx().unwrap(); let res = trx_select_row_mvcc_by_id(&mut trx, table_id, &key, &[0, 1]).await; @@ -5696,9 +5700,11 @@ mod tests { let table_file_path = engine.inner().table_fs.user_table_file_path(table_id); corrupt_leaf_block_id(table_file_path, entry.leaf_block_id, 0); - let _ = table - .disk_pool() - .invalidate_block(table.file().sparse_file().file_id(), entry.leaf_block_id); + let _ = table.disk_pool().invalidate_block( + session.pool_guards().disk_guard(), + table.file().sparse_file().file_id(), + entry.leaf_block_id, + ); let mut trx = session.begin_trx().unwrap(); let res = trx_select_row_mvcc_by_id(&mut trx, table_id, &key, &[0, 1]).await; @@ -5741,9 +5747,11 @@ mod tests { let table_file_path = engine.inner().table_fs.user_table_file_path(table_id); corrupt_lwc_row_shape_fingerprint(table_file_path, entry.block_id()); - let _ = table - .disk_pool() - .invalidate_block(table.file().sparse_file().file_id(), entry.block_id()); + let _ = table.disk_pool().invalidate_block( + session.pool_guards().disk_guard(), + table.file().sparse_file().file_id(), + entry.block_id(), + ); let mut trx = session.begin_trx().unwrap(); let res = trx_select_row_mvcc_by_id(&mut trx, table_id, &key, &[0, 1]).await; diff --git a/doradb-storage/src/table/gc.rs b/doradb-storage/src/table/gc.rs index 48c9d6bd..b97581b0 100644 --- a/doradb-storage/src/table/gc.rs +++ b/doradb-storage/src/table/gc.rs @@ -1705,9 +1705,11 @@ mod tests { let table_file_path = engine.inner().table_fs.user_table_file_path(table_id); corrupt_lwc_row_shape_fingerprint(table_file_path, block_id); - let _ = table - .disk_pool() - .invalidate_block(table.file().sparse_file().file_id(), block_id); + let _ = table.disk_pool().invalidate_block( + session.pool_guards().disk_guard(), + table.file().sparse_file().file_id(), + block_id, + ); let err = session .cleanup_secondary_mem_indexes(table_id, true) diff --git a/doradb-storage/src/table/layout.rs b/doradb-storage/src/table/layout.rs index ae265d7d..5a3ec364 100644 --- a/doradb-storage/src/table/layout.rs +++ b/doradb-storage/src/table/layout.rs @@ -378,7 +378,10 @@ mod tests { ); let guards = PoolGuards::builder() - .push(PoolRole::Index, engine.inner().pools.index.pool_guard()) + .push( + PoolRole::Index, + engine.inner().pools.index.create_base_guard(), + ) .build(); assert_eq!( table_for_internal_assertion(&engine, table_id) diff --git a/doradb-storage/src/table/mem_table.rs b/doradb-storage/src/table/mem_table.rs index 285e64bd..5c3c4356 100644 --- a/doradb-storage/src/table/mem_table.rs +++ b/doradb-storage/src/table/mem_table.rs @@ -3388,8 +3388,8 @@ mod tests { mem_table_id: TableID, metadata: Arc, ) -> TestMemTable { - let meta_guard = engine.inner().pools.meta.pool_guard(); - let index_guard = engine.inner().pools.index.pool_guard(); + let meta_guard = engine.inner().pools.meta.create_base_guard(); + let index_guard = engine.inner().pools.index.create_base_guard(); let mem_pool = engine.inner().pools.mem.clone(); let blk_idx = BlockIndex::new( engine.inner().pools.meta.clone(), @@ -4926,7 +4926,7 @@ mod tests { FixedBufferPool::with_capacity(PoolRole::Index, pool_bytes) .expect("one-page fixed index pool should be constructible"), ); - let pool_guard = (*pool).pool_guard(); + let pool_guard = (*pool).create_base_guard(); let metadata = TableMetadata::try_new( vec![ColumnSpec::new( "id", diff --git a/doradb-storage/src/table/persistence.rs b/doradb-storage/src/table/persistence.rs index c892d448..96d6a4e6 100644 --- a/doradb-storage/src/table/persistence.rs +++ b/doradb-storage/src/table/persistence.rs @@ -3,8 +3,8 @@ use super::checkpoint_workflow::{ PreparedFreezeAttempt, PreparedTransitionPage, }; use super::lifecycle::{CheckpointPublishLease, TableCheckpointRootMutationScope, TableTerminal}; -use crate::buffer::PoolGuards; use crate::buffer::guard::PageGuard; +use crate::buffer::{PoolGuard, PoolGuards}; use crate::catalog::{IndexSpec, SilentWatermarkObject, TableColumnLayout, TableMetadata}; use crate::error::{ CompletionErrorBridge, CompletionResult, DataIntegrityError, DataIntegrityResult, FatalError, @@ -226,7 +226,13 @@ where // Step 1: claim one mutable root snapshot and initialize checkpoint // boundaries. This is checkpoint-internal current-root access after the // post-lease liveness check above. - let mut mutable_file = MutableTableFile::fork(table_file, table_writes, disk_pool.clone()); + let disk_guard = pool_guards.disk_guard(); + let mut mutable_file = MutableTableFile::fork( + table_file, + table_writes, + disk_pool.clone(), + disk_guard.clone(), + ); let pivot_row_id = mutable_file.root().pivot_row_id; let mut secondary_sidecar = SecondaryCheckpointSidecar::new(metadata); @@ -377,7 +383,13 @@ where // Step 5: apply checkpoint changes to the already-checked mutable root. if !lwc_blocks.is_empty() { mutable_file - .apply_lwc_blocks(lwc_blocks, heap_redo_start_ts, checkpoint_ts, disk_pool) + .apply_lwc_blocks( + lwc_blocks, + heap_redo_start_ts, + checkpoint_ts, + disk_pool, + disk_guard, + ) .await .change_runtime_context(RuntimeError::CheckpointExecution) .attach_with(|| { @@ -399,6 +411,7 @@ where &mut secondary_sidecar, cutoff_ts, checkpoint_ts, + disk_guard, ) .await .change_runtime_context(RuntimeError::CheckpointExecution) @@ -417,6 +430,7 @@ where &layout, &mut secondary_sidecar, checkpoint_ts, + disk_guard, #[cfg(test)] &session.engine().maintenance_test, ) @@ -432,7 +446,7 @@ where // mutable root, rebuild its allocation map from the current active root // and the mutable root that will be published. table - .rebuild_reachable_alloc_map(&mut mutable_file, &layout) + .rebuild_reachable_alloc_map(&mut mutable_file, &layout, disk_guard) .await .change_context(RuntimeError::CheckpointExecution) .attach_with(|| { @@ -1034,6 +1048,7 @@ impl Table { root: &ActiveRoot, layout: &TableRuntimeLayout, reachable: &mut BTreeSet, + disk_guard: &PoolGuard, ) -> RuntimeResult<()> { if root.secondary_index_roots.len() != layout.index_slot_count() { return Err(Report::new(DataIntegrityError::InvalidRootInvariant) @@ -1056,14 +1071,13 @@ impl Table { if root.column_block_index_root != SUPER_BLOCK_ID { let disk_pool = self.disk_pool(); - let disk_pool_guard = disk_pool.pool_guard(); let column_index = ColumnBlockIndex::new( root.column_block_index_root, root.pivot_row_id, self.file().file_kind(), self.file().sparse_file(), disk_pool, - &disk_pool_guard, + disk_guard, ); column_index .collect_reachable_blocks(&mut root_reachable) @@ -1097,9 +1111,8 @@ impl Table { continue; } let runtime = index.disk_runtime(); - let disk_pool_guard = runtime.disk_pool_guard(); runtime - .collect_reachable_blocks(root_block_id, &disk_pool_guard, &mut root_reachable) + .collect_reachable_blocks(root_block_id, disk_guard, &mut root_reachable) .await .change_context(RuntimeError::CheckpointExecution) .attach_with(|| { @@ -1128,15 +1141,17 @@ impl Table { &self, mutable_file: &mut MutableTableFile, layout: &TableRuntimeLayout, + disk_guard: &PoolGuard, ) -> RuntimeResult { let mut reachable = BTreeSet::new(); self.collect_root_reachable_blocks( self.file().active_root_unchecked(), layout, &mut reachable, + disk_guard, ) .await?; - self.collect_root_reachable_blocks(mutable_file.root(), layout, &mut reachable) + self.collect_root_reachable_blocks(mutable_file.root(), layout, &mut reachable, disk_guard) .await?; Ok(mutable_file.rebuild_alloc_map_from_reachable(&reachable)) } @@ -1148,9 +1163,9 @@ impl Table { secondary_sidecar: &mut SecondaryCheckpointSidecar, cutoff_ts: TrxID, checkpoint_ts: TrxID, + disk_guard: &PoolGuard, ) -> RuntimeOrFatalResult<()> { let disk_pool = self.disk_pool(); - let disk_pool_guard = disk_pool.pool_guard(); let (column_block_index_root, pivot_row_id, deletion_cutoff_ts) = { let root = mutable_file.root(); ( @@ -1217,7 +1232,7 @@ impl Table { self.file().file_kind(), self.file().sparse_file(), disk_pool, - &disk_pool_guard, + disk_guard, ); let mut groups: Vec = Vec::new(); @@ -1326,6 +1341,7 @@ impl Table { &new_deltas, metadata, secondary_sidecar, + disk_guard, ) .await?; @@ -1362,6 +1378,7 @@ impl Table { layout: &TableRuntimeLayout, sidecar: &mut SecondaryCheckpointSidecar, checkpoint_ts: TrxID, + disk_guard: &PoolGuard, #[cfg(test)] maintenance_test: &MaintenanceTestController, ) -> RuntimeOrFatalResult<()> { let metadata = layout.metadata(); @@ -1390,8 +1407,6 @@ impl Table { .into()); } - let disk_pool = self.disk_pool(); - let disk_pool_guard = disk_pool.pool_guard(); for active in &mut sidecar.indexes { let index_no = active.index_no; // The sidecar is built directly from this immutable metadata @@ -1412,7 +1427,7 @@ impl Table { SecondaryIndexSidecar::Unique { puts, deletes, .. } => { // Use one writer per affected index so same-run puts and // conditional deletes produce a single new DiskTree root. - let tree = runtime.open_unique_at(old_root, &disk_pool_guard)?; + let tree = runtime.open_unique_at(old_root, disk_guard)?; let mut writer = tree.batch_writer(mutable_file, checkpoint_ts); let put_entries = puts .iter() @@ -1440,7 +1455,7 @@ impl Table { } => { // Non-unique roots are exact-entry sets. Inserts and // deletes are independent facts keyed by (key, row_id). - let tree = runtime.open_non_unique_at(old_root, &disk_pool_guard)?; + let tree = runtime.open_non_unique_at(old_root, disk_guard)?; let mut writer = tree.batch_writer(mutable_file, checkpoint_ts); let insert_entries = inserts .iter() @@ -1471,6 +1486,7 @@ impl Table { delete_deltas: &[u32], metadata: &TableMetadata, secondary_sidecar: &mut SecondaryCheckpointSidecar, + disk_guard: &PoolGuard, ) -> RuntimeResult<()> { if secondary_sidecar.indexes.is_empty() || delete_deltas.is_empty() { return Ok(()); @@ -1501,16 +1517,11 @@ impl Table { .enumerate() .all(|(idx, row_id)| *row_id == entry.start_row_id + idx as u64); - let disk_pool = self.disk_pool(); - let disk_pool_guard = disk_pool.pool_guard(); // Decode the persisted LWC block once for this block group, then derive // all secondary delete keys from the selected row indexes. let file_kind = self.file().file_kind(); let block_id = entry.block_id(); - let persisted = self - .storage - .load_lwc_block(&disk_pool_guard, block_id) - .await?; + let persisted = self.storage.load_lwc_block(disk_guard, block_id).await?; let block = persisted.block(); if block.row_count() != row_ids.len() || block.row_shape_fingerprint() != entry.row_shape_fingerprint() @@ -3664,9 +3675,11 @@ mod tests { let table_file_path = engine.inner().table_fs.user_table_file_path(table_id); corrupt_leaf_delete_codec(table_file_path, entry.leaf_block_id, 0); - let _ = table - .disk_pool() - .invalidate_block(table.file().sparse_file().file_id(), entry.leaf_block_id); + let _ = table.disk_pool().invalidate_block( + session.pool_guards().disk_guard(), + table.file().sparse_file().file_id(), + entry.leaf_block_id, + ); let err = session.checkpoint_table(table_id).await.unwrap_err(); assert_table_data_integrity( @@ -3734,9 +3747,11 @@ mod tests { let table_file_path = engine.inner().table_fs.user_table_file_path(table_id); corrupt_leaf_short_delete_section_header(table_file_path, entry.leaf_block_id, 0); - let _ = table - .disk_pool() - .invalidate_block(table.file().sparse_file().file_id(), entry.leaf_block_id); + let _ = table.disk_pool().invalidate_block( + session.pool_guards().disk_guard(), + table.file().sparse_file().file_id(), + entry.leaf_block_id, + ); let err = session.checkpoint_table(table_id).await.unwrap_err(); assert_table_data_integrity( @@ -6064,7 +6079,11 @@ mod tests { let table_file = engine .inner() .table_fs - .open_table_file(table_id, engine.inner().pools.disk.clone()) + .open_table_file( + table_id, + engine.inner().pools.disk.clone(), + session.pool_guards().disk_guard(), + ) .await .unwrap(); let root_after = table_file.active_root_unchecked(); diff --git a/doradb-storage/src/trx/mod.rs b/doradb-storage/src/trx/mod.rs index cacd1cbd..7823fa5f 100644 --- a/doradb-storage/src/trx/mod.rs +++ b/doradb-storage/src/trx/mod.rs @@ -6614,6 +6614,7 @@ pub(crate) mod tests { &table_file, engine.inner().table_fs.background_writes(), engine.inner().pools.disk.clone(), + session.pool_guards().disk_guard().clone(), ); let table_file = engine .inner() diff --git a/doradb-storage/src/trx/purge.rs b/doradb-storage/src/trx/purge.rs index dd8a93ed..4ae311a5 100644 --- a/doradb-storage/src/trx/purge.rs +++ b/doradb-storage/src/trx/purge.rs @@ -1475,10 +1475,19 @@ mod tests { #[inline] fn full_pool_guards(engine: &Engine) -> PoolGuards { PoolGuards::builder() - .push(PoolRole::Meta, engine.inner().pools.meta.pool_guard()) - .push(PoolRole::Index, engine.inner().pools.index.pool_guard()) - .push(PoolRole::Mem, engine.inner().pools.mem.pool_guard()) - .push(PoolRole::Disk, engine.inner().pools.disk.pool_guard()) + .push( + PoolRole::Meta, + engine.inner().pools.meta.create_base_guard(), + ) + .push( + PoolRole::Index, + engine.inner().pools.index.create_base_guard(), + ) + .push(PoolRole::Mem, engine.inner().pools.mem.create_base_guard()) + .push( + PoolRole::Disk, + engine.inner().pools.disk.create_base_guard(), + ) .build() } @@ -2703,7 +2712,7 @@ mod tests { .mem .mem_pool() .get_page::( - &table.mem.mem_pool().pool_guard(), + &table.mem.mem_pool().create_base_guard(), page_id, LatchFallbackMode::Shared, ) @@ -2821,7 +2830,7 @@ mod tests { .mem .mem_pool() .get_page::( - &table.mem.mem_pool().pool_guard(), + &table.mem.mem_pool().create_base_guard(), page_id, LatchFallbackMode::Shared, ) diff --git a/doradb-storage/src/trx/retention.rs b/doradb-storage/src/trx/retention.rs index aa6a30c8..3ad933e6 100644 --- a/doradb-storage/src/trx/retention.rs +++ b/doradb-storage/src/trx/retention.rs @@ -1,3 +1,4 @@ +use crate::buffer::PoolGuard; use crate::catalog::{CatalogCheckpointOutcome, CatalogCheckpointScope}; use crate::error::{ CompletionErrorBridge, CompletionResult, DataIntegrityError, DataIntegrityResult, FatalError, @@ -161,6 +162,7 @@ impl MaintenanceExecution for CatalogRedoMaintenanceExecution { .trx_sys .checkpoint_catalog_and_truncate_redo_log_prepared( || self.catalog_scope.release(), + runtime.pool_guards().disk_guard(), #[cfg(test)] &engine.maintenance_test, ) @@ -359,6 +361,7 @@ impl TransactionSystem { async fn checkpoint_catalog_and_truncate_redo_log_prepared( &self, release_catalog: F, + disk_guard: &PoolGuard, #[cfg(test)] maintenance_test: &MaintenanceTestController, ) -> RuntimeOrFatalResult where @@ -383,7 +386,10 @@ impl TransactionSystem { .await .map_err(RuntimeOrFatalError::from)?; let checkpoint_progress = batch.redo_retention_progress(); - let mut prepared = self.catalog.prepare_checkpoint_batch(batch).await?; + let mut prepared = self + .catalog + .prepare_checkpoint_batch(batch, disk_guard) + .await?; let checkpoint_will_publish = prepared.will_publish(); // 3. Build truncation inputs from the projected post-checkpoint state.