Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,14 @@ data coordination is described in [Lock System](./lock-system.md).
## Mandatory Background Runtime

The engine owns one fixed-thread asynchronous executor for obligations that
must reach a supervised terminal outcome after acceptance. Caller preparation
and operation-lock waiting remain outside runtime capacity and are
cancellable. A synchronous consuming acceptance edge transfers all prepared
resources into mandatory ownership before the task is detached.
must reach a supervised terminal outcome after acceptance. Effectful session
maintenance uses this runtime beside table and index DDL: table freeze and
checkpoint, catalog checkpoint, redo truncation, combined catalog/redo
maintenance, and secondary `MemIndex` cleanup all prepare their complete
authority before mandatory admission. Caller preparation and operation-lock
waiting remain outside runtime capacity and are cancellable. A synchronous
consuming acceptance edge transfers all prepared resources into mandatory
ownership before the task is detached.

The same executor replaces the former sequential transaction-cleanup thread.
Abandoned transactions, explicit terminal rollback, and failed-precommit
Expand Down
19 changes: 14 additions & 5 deletions docs/checkpoint.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,14 @@ silently changing the batch or its snapshot boundary.

One checkpoint attempt follows these conceptual phases:

1. From an idle session, acquire scoped `TableMetadata(S)` and `TableData(IS)`
locks, revalidate the live table, and claim its checkpoint workflow.
2. Hold root-mutation exclusion and verify that the currently active root is no
longer visible to an active snapshot before forking a mutable CoW root.
1. From an idle session, acquire owned `TableMetadata(S)` and `TableData(IS)`
locks, revalidate the live table, claim its checkpoint workflow, and acquire
lifetime-free root-mutation exclusion while the caller future remains
cancellable.
2. After the complete table, workflow, and root authority is prepared, transfer
it synchronously to the mandatory runtime. Accepted execution acquires none
of those resources and verifies that the currently active root is no longer
visible to an active snapshot before forking a mutable CoW root.
3. Use the purge-published GC horizon as the exclusive cutoff for both frozen
row images and cold-row delete selection, then allocate `checkpoint_ts`.
4. Convert a ready frozen-page prefix into LWC blocks and collect matching
Expand Down Expand Up @@ -134,7 +138,12 @@ checkpoint attempt and returns listener-only wait state, so the subsequent
sleep owns no table runtime, layout, frozen page, page guard, checkpoint
attempt, or logical table lock. Completion only means a retry may be useful.
`checkpoint_table_with_wait` retries delayed outcomes and returns published,
cancelled, or error outcomes unchanged.
cancelled, or error outcomes unchanged. Each delayed checkpoint attempt first
reaches its own terminal operation state and releases its mandatory permit,
table runtime, workflow/root authority, and logical locks. The standalone
observer wait then owns only listener state. A useful wake starts a new
checkpoint operation with a fresh operation key and freshly prepared
authority; retries never reuse the completed attempt's operation owner.

### Root Liveness and Reclamation

Expand Down
13 changes: 13 additions & 0 deletions docs/engine-component-lifetime.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,19 @@ publishes the outer operation terminal. Unexpected execution unwind instead
retains `FailedRetained`, publishes mandatory-runtime poison, and releases the
caller permit after the accepted owner is dropped.

Effectful maintenance uses the same contract. Table freeze/checkpoint transfer
the exact live table, owned maintenance locks, workflow attempt, and checkpoint
root-mutation scope. Catalog checkpoint and redo maintenance transfer their
catalog-checkpoint and redo-retention scopes; combined maintenance preserves
both through catalog publication, releases catalog admission before unlink,
and retains redo exclusion until unlink accounting finishes. Secondary
`MemIndex` cleanup transfers its table scope and stores each private
transaction in accepted progress before scanning or awaiting. On normal
completion domain workflow/gate and private-transaction resources release
first, then prepared maintenance locks, and only then the outer session entry
publishes `Terminal`. A dropped result observer owns none of these resources
and cannot cancel accepted work.

The supervisor catches both synchronous future construction and polling
unwinds while the accepted operation or cleanup job remains in an outer owner.
Its domain policy first releases or moves residual unsafe ownership into fatal
Expand Down
18 changes: 11 additions & 7 deletions docs/lock-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ The current implementation uses the following table-level mapping:
| Full-table MVCC mutation | transaction `S` | transaction `X` | transaction |
| Explicit shared table lock | session `S` | session `S` | explicit session |
| Explicit exclusive table lock | session `S` | session `X` | explicit session |
| Freeze/checkpoint | scoped `S` | scoped `IS` | maintenance operation |
| Freeze/checkpoint | prepared owned `S` | prepared owned `IS` | prepared maintenance operation, then mandatory owner |
| CREATE TABLE on a new id | target `X`; catalog slots 0-3 `S` | catalog slots 0-3 `IX` | prepared DDL operation, then mandatory owner |
| DROP TABLE | target `X`; catalog slots 0-4 `S` | target `X`; catalog slots 0-4 `IX` | prepared DDL operation, then mandatory owner |
| CREATE INDEX | target `X`; catalog slots 0,2,3 `S` | target `X`; catalog slots 0,2,3 `IX` | prepared DDL operation, then mandatory owner |
Expand All @@ -145,7 +145,10 @@ synchronously transfers the same `OwnerLockState` and operation owner to
accepted execution; there is no release/reacquire window. Catalog statements
receive a typed prepared-write authority that proves metadata S plus data IX
for each catalog table and bypasses ordinary transaction lock acquisition.
Maintenance continues through its scoped lock-manager path.
Effectful maintenance likewise prepares an owned lock scope before mandatory
admission and transfers the exact `OwnerLockState` without a release/reacquire
window. The read-only `total_row_pages` observation remains caller-owned and
uses the borrowed scoped lock-manager path.

Recovery, purge, and no-transaction replay do not acquire logical locks. They
run at lifecycle boundaries where foreground lock owners do not exist. Logical
Expand Down Expand Up @@ -327,11 +330,12 @@ global `release_owner()` scan during session cleanup.

Transactions, DDL, maintenance, and explicit-lock mutations reserve ids from
one plain session-local sequence. One public DDL call retains one
`Operation` owner through its typed `SessionDdlContext`; one public maintenance
workflow retains one `Operation` owner across every
`ScopedTableRuntimeAccess`, bounded recheck, and internal retry. Operation
cleanup still uses fresh-lock and scoped guards rather than an authoritative
`LockScopeState`.
`Operation` owner through its typed `SessionDdlContext`; one effectful public
maintenance call retains one `Operation` owner through its prepared and
accepted owned scope. A delayed checkpoint completes that operation before its
observer-only wait and allocates a fresh operation owner for the next attempt.
The caller-owned `total_row_pages` observation continues to use
`ScopedTableRuntimeAccess`.

Maintenance always records its own exact claims even when a covering
`SessionExplicit` claim admits it. Releasing maintenance therefore cannot
Expand Down
14 changes: 6 additions & 8 deletions docs/public-error-audit.csv
Original file line number Diff line number Diff line change
Expand Up @@ -11,28 +11,26 @@ doradb-storage/src/error.rs,SharedFatalError::disclose,1
doradb-storage/src/log/mod.rs,LogSync::from_str,1
doradb-storage/src/session.rs,Session::begin_trx,4
doradb-storage/src/session.rs,Session::buffer_pool_stats,1
doradb-storage/src/session.rs,Session::checkpoint_catalog,2
doradb-storage/src/session.rs,Session::checkpoint_catalog_and_truncate_redo_log,2
doradb-storage/src/session.rs,Session::checkpoint_table,3
doradb-storage/src/session.rs,Session::checkpoint_table_with_wait,3
doradb-storage/src/session.rs,Session::cleanup_secondary_mem_indexes,3
doradb-storage/src/session.rs,Session::checkpoint_catalog,3
doradb-storage/src/session.rs,Session::checkpoint_catalog_and_truncate_redo_log,3
doradb-storage/src/session.rs,Session::checkpoint_table,5
doradb-storage/src/session.rs,Session::cleanup_secondary_mem_indexes,5
doradb-storage/src/session.rs,Session::close,3
doradb-storage/src/session.rs,Session::create_index,9
doradb-storage/src/session.rs,Session::create_table,4
doradb-storage/src/session.rs,Session::drop_index,8
doradb-storage/src/session.rs,Session::drop_table,4
doradb-storage/src/session.rs,Session::freeze_table,3
doradb-storage/src/session.rs,Session::freeze_table,5
doradb-storage/src/session.rs,Session::list_table_ids,1
doradb-storage/src/session.rs,Session::lock_table,2
doradb-storage/src/session.rs,Session::storage_io_stats,1
doradb-storage/src/session.rs,Session::total_row_pages,3
doradb-storage/src/session.rs,Session::transaction_system_stats,1
doradb-storage/src/session.rs,Session::truncate_redo_log,2
doradb-storage/src/session.rs,Session::truncate_redo_log,3
doradb-storage/src/session.rs,Session::unlock_table,2
doradb-storage/src/session.rs,Session::wait_for_checkpoint_retry,2
doradb-storage/src/session.rs,Session::wait_for_gc_horizon_after,1
doradb-storage/src/session.rs,Session::wait_for_purge_completion_after,1
doradb-storage/src/session.rs,wait_for_checkpoint_retry_in_operation,2
doradb-storage/src/session.rs,wait_for_maintenance_boundary,4
doradb-storage/src/table/access.rs,LazyRow::val,2
doradb-storage/src/table/access.rs,UserTableAccessor::delete_known_cold_row,3
Expand Down
8 changes: 4 additions & 4 deletions docs/rfcs/0026-engine-owned-mandatory-background-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -1238,10 +1238,10 @@ focused validation.
completion.
- Non-goals: Do not migrate standalone wait/diagnostic APIs, change
checkpoint/redo policy or formats, or parallelize recovery/checkpoint.
- Task Doc: `docs/tasks/TBD.md`
- Task Issue: `#0`
- Phase Status: `pending`
- Implementation Summary: `pending`
- Task Doc: `docs/tasks/000251-runtime-owned-mandatory-maintenance.md`
- Task Issue: `#928`
- Phase Status: done
- Implementation Summary: Implemented all six effectful maintenance roots as caller-prepared, mandatory-runtime-owned operations while preserving observer-only waits and finite read-only observations. [Task Resolve Sync: docs/tasks/000251-runtime-owned-mandatory-maintenance.md @ 2026-08-03]

- **Phase 5: Lifecycle, Fairness, And Evolution Readiness**
- Scope: Remove superseded foreground-handoff transitions and queue paths;
Expand Down
Loading
Loading