diff --git a/CHANGELOG.md b/CHANGELOG.md index 6072c932f..c3b4487c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,9 +17,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 updating a referenced element propagates the new hash along every chain, and deleting/overwriting it cascades the chains away (each affected reference must opt in via `cascade_on_update`). Maintenance is automatic - on V4, with `BackwardReferencesPolicy::Skip` as an explicit opt-out; batch - maintenance is gated by the new `apply_batch.backward_references_maintenance` - version slot. The referrer list is stored on the element itself under a + on V4, and every operation declares what it displaces (`DisplacedValue`, + see Changed); batch maintenance is gated by the new + `apply_batch.backward_references_maintenance` version slot. The referrer list is stored on the element itself under a two-layer hash (`combine(inner, backrefs)`), so registering a referrer never re-hashes what existing referrers committed to; public reads return the stripped element, and proofs authenticate these elements through the @@ -43,16 +43,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 capacity, the ceiling for writes that cannot see the element they displace, ≤10-hop chains, 1 referrer per reference) while pre-V4 estimation stays byte-stable for replay. See - `adr/bidirectional_references.md`. `clear_subtree` now exposes the same policy: - default Maintain refuses participant-containing subtrees before mutation. - `drop_flat_subtree` adds a required policy argument, and both it and batch - `DropFlat` require explicit Skip to preserve O(1) cost. Recursive deletions - under Maintain include participant-scan costs in the V4 cost pins. Ordinary - batches that touch no participants retain their original executor semantics. - Estimation charges the displaced-participant fan-out only in layers that - declare it (`EstimatedLayerInformation::may_contain_backward_references`, - or the `*WithBackwardReferences` worst-case variants); undeclared layers - estimate plain writes exactly as `Skip` does. + `adr/bidirectional_references.md`. `clear_subtree` exposes the same + declaration: `MayBeParticipant` refuses participant-containing subtrees + before mutation, `NotParticipant` is trusted for a raw clear. + `drop_flat_subtree` takes the declaration as a required argument, and both + it and batch `DropFlat` require `NotParticipant` to preserve O(1) cost. + Ordinary batches that touch no participants retain their original executor + semantics. Estimation charges the displaced-participant fan-out only for + ops declared `MayBeParticipant`. - **BREAKING**: Added `add_parent_tree_on_subquery` feature to PathQuery (#379) - New field in `Query` struct: `add_parent_tree_on_subquery: bool` - When set to `true`, parent tree elements (like CountTree or SumTree) are included in query results when performing subqueries @@ -61,21 +59,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Updated proof verification logic to handle parent tree inclusion ### Changed -- **BREAKING**: `EstimatedLayerInformation` gains - `may_contain_backward_references: bool` (declare `false` for layers that - never hold backward-reference participants), and `WorstCaseLayerInformation` - gains `MaxElementsNumberWithBackwardReferences` and - `NumberOfLevelsWithBackwardReferences`. Under the default `Maintain` policy - the estimators charge the displaced-participant fan-out and delete probe - only in declared layers, so ordinary V4 estimates no longer inflate for - every write. -- **BREAKING**: Replace `propagate_backward_references` in insert, delete, - and batch options with `backward_references_policy` (`Maintain` by default, - or explicit `Skip`). V4 observes old values through retained Merk nodes so - ordinary mutations need no separate old-value fetch for classification. - Partial batches reject displaced participants; subtree removal/replacement - refuses unsupported descendant maintenance before commit. Earlier protocol - versions retain their historical behavior. +- **BREAKING**: Replace `propagate_backward_references` with a per-operation + declaration of the stored value an operation displaces, + `DisplacedValue::{MayBeParticipant, NotParticipant}`, on `InsertOptions`, + `DeleteOptions`, `ClearOptions` and every `QualifiedGroveDbOp` + (`with_displaced_value`; `MayBeParticipant` is the default everywhere). + `BatchApplyOptions` carries no backward-references policy. V4 has one write + path: the displaced value is read for the write anyway, so + `MayBeParticipant` maintains a participant it finds and `NotParticipant` + refuses the operation before anything commits; where nothing reads the + contents (a flat drop, a raw `clear_subtree`, replacing a populated + subtree, a live recursive delete) `NotParticipant` is trusted. A batch + `DeleteTree` is never pre-scanned: `DontCheckWithNoCleanup` declares that + the batch's own deletes emptied the subtree, `Error` and `Skip` verify + that at apply time, and `DeleteChildren` checks the declaration on the + cleanup walk it makes anyway, refusing a participant the batch does not + explicitly delete. A delete-up-tree chain and a batch recursive removal + therefore cost the plain removal on V4. Partial batches still refuse + participant mutations, and earlier protocol versions retain their + historical behavior. +- **BREAKING**: `EstimatedLayerInformation::may_contain_backward_references` + and the `*WithBackwardReferences` variants of `WorstCaseLayerInformation` + are removed. The estimators charge the displaced-participant fan-out and + delete probe per op declared `MayBeParticipant` instead of per layer; ops + that write a participant themselves are charged from the op regardless. - Bumped the GroveDB workspace crates and their internal dependency requirements to **6.0.0** for the public API changes since 5.0.1. This package version is independent of the existing `GroveVersion` runtime compatibility versions. diff --git a/adr/bidirectional_references.md b/adr/bidirectional_references.md index c132f2d59..0b5db09fc 100644 --- a/adr/bidirectional_references.md +++ b/adr/bidirectional_references.md @@ -51,17 +51,18 @@ chain origin. When such behavior is required, a different type of element should Moreover, these types are incompatible, which will be discussed in the "Rules" section. On `GROVE_V4`, ordinary inserts, replacements, deletes, and full batches -maintain backward references automatically. Callers do not need to predict -whether a plain operation will displace a participant. `InsertOptions`, -`DeleteOptions`, `ClearOptions`, and `BatchApplyOptions` expose `backward_references_policy`, -whose default is `BackwardReferencesPolicy::Maintain`. - -`BackwardReferencesPolicy::Skip` deliberately disables maintenance for an -operation. It permits dangling references and stale hashes; it is not a hint -that a key is known to have no references. Inserting a bidirectional reference -through the live API still registers its edge even with `Skip`. Full batches -with `Skip` reject family payloads, while allowing ordinary mutations that -intentionally bypass maintenance. +maintain backward references automatically, and every operation declares +what it displaces: `InsertOptions`, `DeleteOptions`, `ClearOptions` and each +`QualifiedGroveDbOp` carry a `DisplacedValue`, `MayBeParticipant` by default. +GroveDB reads the displaced value for the write anyway, so for a keyed +operation the declaration only decides what happens when that value takes +part in backward references: `MayBeParticipant` maintains the references, +`NotParticipant` refuses the operation before anything commits. Where nothing +reads the contents — a flat drop, a raw `clear_subtree`, the replacement of a +populated subtree, a live recursive delete — `NotParticipant` is trusted and +leaves any participant's registrations stale, exactly like the storage it +strands. There is no policy that skips maintenance on a value known to +participate. Inserting a bidirectional reference always registers its edge. ## Versioning and scope @@ -90,16 +91,26 @@ Current limitations: specialized/indexed descendants. Remove those descendants first. - Live participant maintenance below an indexed primary requires a full batch; the reference cache refuses that propagation before commit. -- Recursive delete and subtree replacement inspect descendants under `Maintain`. - Those scans add reads and are charged in the V4 default cost tests. -- Flat drop retains its O(1) contract. Standalone `drop_flat_subtree` requires - an explicit policy argument; it and batch `DropFlat` reject `Maintain` - before scanning. Use `Skip` to acknowledge stale or dangling registrations, - or use recursive delete when maintenance is required. -- `clear_subtree` defaults to `Maintain`: it scans and refuses a subtree +- `Delete` of a populated tree and subtree replacement inspect descendants + when declared `MayBeParticipant`; those scans add reads and are charged in + the V4 default cost tests. A batch `DeleteTree` is never pre-scanned: + `DontCheckWithNoCleanup` declares that the batch's own deletes emptied the + subtree, `Error` and `Skip` verify that at apply time, and `DeleteChildren` + (with the defensive `Error`/`Skip` sweeps) checks the declaration on the + cleanup walk it makes anyway, refusing a participant the batch does not + explicitly delete. A delete-up-tree chain and a batch recursive removal + therefore cost the plain removal under either declaration. +- Flat drop retains its O(1) contract. Standalone `drop_flat_subtree` takes + the declaration as a required argument; it and batch `DropFlat` refuse + `MayBeParticipant` before reading anything and trust `NotParticipant`. Use + recursive delete when maintenance is required. +- `clear_subtree` under `MayBeParticipant` scans and refuses a subtree containing participants before making any mutation, including with a caller - transaction. Delete the participants through the normal API first. Explicit - `ClearOptions::backward_references_policy = Skip` permits a raw clear. + transaction; delete the participants through the normal API first. + `NotParticipant` is trusted for a raw clear. A live recursive `delete` + declared `NotParticipant` likewise trusts its contents: a clear runs one + such delete per nested subtree inside the caller's transaction, where a + refusal part-way through could not be undone. Ordinary full batches keep their original operation set when neither stored nor incoming values participate in references. Preparation retains the Merks @@ -116,13 +127,17 @@ transaction's original subtree contents. Cross-segment conflict checks prevent those committed-state inspections from overlooking changes staged by the first segment. A refusal discards the storage batch and preserves the caller's transaction. These scans have real costs, pinned alongside the full-batch costs; -this API does not promise a no-scan recursive removal. +this API does not promise a no-scan recursive removal, only that `DeleteTree` +removals add no participant-scan reads of their own (`Error` and `Skip` still +read the tree to check emptiness): their contents are checked on the cleanup +walk. ## Rules Next, we’ll go over the rules and limitations for using bidirectional references. -These rules apply by default; `BackwardReferencesPolicy::Skip` explicitly opts out. +These rules always apply; `DisplacedValue::NotParticipant` is a checked claim, not an +opt-out. An 'Element with backward references' refers to `ItemWithBackwardsReferences`, `SumItemWithBackwardsReferences`, `ItemWithSumItemWithBackwardsReferences`, and @@ -149,7 +164,8 @@ registered element) and the node growth registrations can inflict on a target. insertion.__ Public reads enforce the declared budget deterministically, so an edge whose chain is already longer than its declaration would never resolve; the write path rejects such dead edges instead of persisting them. (An edge can still fall out of budget later — -e.g. its target is overwritten into a plain reference with `BackwardReferencesPolicy::Skip` — and +e.g. its target is overwritten into a plain reference behind a raw clear declared +`NotParticipant` — and reads then return `ReferenceLimit`.) - __Both ends of a bidirectional edge must sit at most 32 subtree levels deep__ (`MAX_BACKWARD_REFERENCES_GROVE_DEPTH`, enforced at registration). Every later derived @@ -226,15 +242,13 @@ estimation is preserved byte-for-byte for replay of historical admission decisions. The estimator cannot see stored state, so the bound for a write that may -displace a participant is a per-layer declaration: -`EstimatedLayerInformation::may_contain_backward_references` (average case) -and the `*WithBackwardReferences` variants of `WorstCaseLayerInformation`. -Undeclared layers charge no displaced-state fan-out and no delete probe — -their `Maintain` estimates equal `Skip` estimates byte-for-byte — while ops -that themselves write a participant (family items, bidirectional references) -are charged from the op regardless. Declaring the layers that hold -participants is the caller's responsibility; an undeclared layer that does -hold them under-estimates cascades. +displace a participant follows the op's own declaration: an op declared +`MayBeParticipant` charges the displaced-state fan-out and the delete probe, +an op declared `NotParticipant` charges the plain write alone, and ops that +themselves write a participant (family items, bidirectional references) are +charged from the op regardless. Declaring correctly is the caller's +responsibility; the apply path refuses a false `NotParticipant` claim rather +than running an unpriced cascade. Live writes use the same preparation observer. Ordinary values retain the existing parent and indexed-tree propagation; participating values reuse the @@ -310,7 +324,7 @@ Previous read: [Merk cache](./merk_cache.md). Deletion or an update of an element with backward references triggers a cascade hash update or a deletion, both of which alter the state of affected subtrees, leading to regular hash propagation to ancestor subtrees up to the GroveDB root. In short, operations -under `BackwardReferencesPolicy::Maintain` (the V4 default) can trigger updates across +on V4 can trigger updates across several subtrees simultaneously. Thus, there are two ongoing propagations: diff --git a/docs/book/src/batch-operations.md b/docs/book/src/batch-operations.md index b43143a8b..4901b4e8e 100644 --- a/docs/book/src/batch-operations.md +++ b/docs/book/src/batch-operations.md @@ -255,27 +255,35 @@ replacement or delete displaces a registered element. Preparation observes old values and retains their Merk nodes for execution and storage accounting. The existing consent and batch-conflict rules still apply. -Set `BatchApplyOptions { backward_references_policy: -BackwardReferencesPolicy::Skip, ..Default::default() }` only when deliberately -bypassing maintenance. This can leave stale hashes or dangling references. -Partial batches refuse participant mutations; use a full batch for reference -maintenance. Recursive removal of a subtree containing participants is -supported by live `delete`; see the bidirectional references ADR for scope. - -Under `Maintain`, ordinary batches retain their executor semantics when no old -or new value participates in backward references. Reference planner conflict -rules apply only to batches that touch participants. Recursive subtree deletion -and replacement inspect descendants and incur additional read costs. - -Cost estimation follows the same split. Layers whose -`EstimatedLayerInformation` sets `may_contain_backward_references` (or use -the `WithBackwardReferences` worst-case variants) charge the -displaced-participant fan-out for plain writes and deletes; undeclared layers -estimate them exactly as `Skip` would. Ops that write a participant -themselves are always charged. - -`SubelementsDeletionBehavior::DropFlat` requires explicit -`BatchApplyOptions::backward_references_policy = BackwardReferencesPolicy::Skip`. -`Maintain` refuses flat drop before scanning, preserving its O(1) contract. -Partial batches refuse participant mutations in either segment; their subtree -inspection occurs before commit and can also incur recursive read costs. +Every `QualifiedGroveDbOp` declares what it displaces: `MayBeParticipant` +(the default) maintains a participant the write lands on, `NotParticipant` +refuses the write if the stored value turns out to participate. The +declaration is checked from the value the batch reads for the write anyway, +so it never costs a read; declare `NotParticipant` through +`with_displaced_value` on ops whose positions are known to hold no +participants. Partial batches refuse participant mutations; use a full batch +for reference maintenance. Recursive removal of a subtree containing +participants is supported by live `delete`; see the bidirectional references +ADR for scope. + +Ordinary batches retain their executor semantics when no old or new value +participates in backward references. Reference planner conflict rules apply +only to batches that touch participants. A `Delete` of a populated tree or a +tree replacement declared `MayBeParticipant` inspects the descendants and +incurs additional read costs. A `DeleteTree` is never pre-scanned: +`DontCheckWithNoCleanup` declares that the batch's own deletes emptied the +subtree, `Error` and `Skip` verify that at apply time, and `DeleteChildren` +checks the declaration on the cleanup walk it already makes, refusing a +participant the batch does not explicitly delete. A delete-up-tree chain and +a recursive removal therefore cost the plain removal. + +Cost estimation follows the same split: an op declared `MayBeParticipant` +charges the displaced-participant fan-out for its plain write or delete, an +op declared `NotParticipant` estimates the plain write alone, and ops that +write a participant themselves are always charged. + +`SubelementsDeletionBehavior::DropFlat` requires the op to declare +`DisplacedValue::NotParticipant`; `MayBeParticipant` is refused before +reading anything, preserving the O(1) contract. Partial batches refuse +participant mutations in either segment; their inspection of tree +replacements occurs before commit and can also incur recursive read costs. diff --git a/docs/crates/grovedb.md b/docs/crates/grovedb.md index 4883ac01a..b4dacabf7 100644 --- a/docs/crates/grovedb.md +++ b/docs/crates/grovedb.md @@ -127,7 +127,7 @@ pub fn insert>( #### Delete Operations - **delete**: Standard deletion - **delete_up_tree**: Remove empty parents recursively -- **clear_subtree**: Bulk deletion; V4 Maintain scans and refuses backward-reference participants. Explicit Skip permits raw clearing. +- **clear_subtree**: Bulk deletion; on V4, `MayBeParticipant` scans and refuses backward-reference participants, `NotParticipant` is trusted for a raw clear. #### Query System @@ -434,8 +434,10 @@ GroveDB is designed with several core principles: - Performance optimizations -On Grove V4, insert, delete, and full-batch options default to -`BackwardReferencesPolicy::Maintain`. Old-value preparation retains fetched -Merk nodes for the mutation. `BackwardReferencesPolicy::Skip` is an explicit -opt-out that permits stale or dangling references. See +On Grove V4, every insert, delete and batch operation declares what it +displaces (`DisplacedValue`, `MayBeParticipant` by default). Old-value +preparation retains fetched Merk nodes for the mutation, so the declaration +is checked for free: `MayBeParticipant` maintains a participant it finds, +`NotParticipant` refuses the operation. Only routes that read nothing (flat +drop, raw clear) trust the claim. See [the bidirectional references design](../../adr/bidirectional_references.md). diff --git a/grovedb-version/src/version/grovedb_versions.rs b/grovedb-version/src/version/grovedb_versions.rs index 1ae67a90e..cfb1ba7ed 100644 --- a/grovedb-version/src/version/grovedb_versions.rs +++ b/grovedb-version/src/version/grovedb_versions.rs @@ -239,11 +239,11 @@ pub struct GroveDBApplyBatchVersions { /// - `0` (V1..V3): no reference planning. Backward-reference family /// payloads are rejected, ordinary ops run the released executor /// unchanged, and `DropFlat` needs no policy. - /// - `1` (V4+): under `BackwardReferencesPolicy::Maintain` (the default) - /// the batch is prepared through the backward-references planner, whose - /// old-value reads are retained for execution; partial batches refuse + /// - `1` (V4+): the batch is prepared through the backward-references + /// planner, whose old-value reads are retained for execution and check + /// each op's `DisplacedValue` declaration; partial batches refuse /// participant mutations through the old-value gate; and `DropFlat` - /// requires an explicit `Skip`. `Skip` keeps the released executor. + /// requires `NotParticipant`. /// /// This slot owns the batch surface. The live insert and delete routers /// are versioned by their own `insert_on_transaction` and @@ -558,8 +558,8 @@ pub struct GroveDBOperationsAverageCaseVersions { /// refused. Matches those versions' apply path, which rejects the /// family in batches, so historical admission decisions replay /// byte-identically. - /// - `1` (V4+): family-carrying ops and (under - /// `BatchApplyOptions::backward_references_policy`) deletes charge + /// - `1` (V4+): family-carrying ops and (when declared + /// `DisplacedValue::MayBeParticipant`) plain writes and deletes charge /// the derived registration / propagation / cascade fan-out, bounded /// by the apply path's budgets (≤32 referrers per item, ≤10-hop /// chains, 1 referrer per reference), and the derived op itself gets diff --git a/grovedb-version/src/version/v4.rs b/grovedb-version/src/version/v4.rs index 53be7cdec..46d1ee417 100644 --- a/grovedb-version/src/version/v4.rs +++ b/grovedb-version/src/version/v4.rs @@ -148,9 +148,13 @@ //! keyless ops before the batch structure is built). //! //! - `apply_batch.backward_references_maintenance: 1` — batches maintain -//! backward references by default: planner preparation with retained -//! old-value reads, the partial-batch participant gate, and explicit -//! `Skip` for `DropFlat`. +//! backward references: planner preparation with retained old-value +//! reads, the partial-batch participant gate, each op's `DisplacedValue` +//! declaration checked from the value the write reads anyway, and +//! `NotParticipant` required for `DropFlat`. A `Delete` of a populated +//! tree or a tree replacement declared `MayBeParticipant` is scanned; a +//! `DeleteTree` never is, `DeleteChildren` checking the declaration on +//! its cleanup walk instead. //! - `apply_batch.non_merk_parent_keyed_ops_rejection: 1` — batch execution //! refuses ordinary keyed ops at a level whose parent is a non-Merk data //! tree (`CommitmentTree`, `MmrTree`, `BulkAppendTree`, `DenseTree`, @@ -357,9 +361,10 @@ pub const GROVE_V4: GroveVersion = GroveVersion { keyless_op_cost_dispatch: 1, add_on_op_collision: 1, non_merk_parent_keyed_ops_rejection: 1, - // v1: batches maintain backward references by default — planner + // v1: batches maintain backward references — planner // preparation with retained old-value reads, the partial-batch - // participant gate, and explicit Skip for DropFlat. + // participant gate, per-op DisplacedValue declarations, and + // NotParticipant required for DropFlat. backward_references_maintenance: 1, }, element: GroveDBElementMethodVersions { @@ -485,7 +490,8 @@ pub const GROVE_V4: GroveVersion = GroveVersion { // keeps the legacy reopen byte-for-byte for replay // compatibility. // v2: automatic backward-reference maintenance with cached - // old-value observation. Skip retains the v1 route. + // old-value observation, one route for both declarations; + // `NotParticipant` refuses a participant it finds. delete_internal_on_transaction: 2, average_case_delete_operation_for_delete: 0, worst_case_delete_operation_for_delete: 0, diff --git a/grovedb/src/batch/backward_references.rs b/grovedb/src/batch/backward_references.rs index eff365bd6..11080ad36 100644 --- a/grovedb/src/batch/backward_references.rs +++ b/grovedb/src/batch/backward_references.rs @@ -1,6 +1,6 @@ //! The backward-references batch preprocessor (batching milestones M2–M4). //! -//! With the default [`crate::BackwardReferencesPolicy::Maintain`], +//! With the default [`crate::DisplacedValue::MayBeParticipant`], //! user operations touching the backward-references family — the three ITEM //! variants and `BidirectionalReference` itself — expand into the derived //! operations the live flow would perform. The decisions come from @@ -85,7 +85,7 @@ use crate::{ }, operations::get::MAX_REFERENCE_HOPS, reference_path::{path_from_reference_path_type, ReferencePathType}, - Element, Error, GroveDb, Transaction, + DisplacedValue, Element, Error, GroveDb, Transaction, }; /// [`ChainStore`] over the batch's prospective state: an overlay of staged @@ -558,6 +558,7 @@ impl<'db, 'g> Expansion<'db, 'g> { node_value_hash, end_hash, }, + displaced_value: DisplacedValue::MayBeParticipant, }, ); self.store.stage(position, Some(element)); @@ -576,6 +577,7 @@ impl<'db, 'g> Expansion<'db, 'g> { node_value_hash, end_hash, }, + displaced_value: DisplacedValue::MayBeParticipant, }, ); self.store.stage(position, Some(element)); @@ -698,21 +700,41 @@ pub(super) fn expand_backward_references_ops<'db>( _ => continue, }; let previous = cost_return_on_error!(&mut cost, expansion.store.element_at(&path, &key)); - needs_reference_planning |= new_element.is_some_and(Element::supports_backward_references) - || previous - .as_ref() - .is_some_and(Element::supports_backward_references); - let removes_subtree = !matches!( - op.op, - GroveOp::InsertIfNotExists { .. } - | GroveOp::RefreshReference { .. } - | GroveOp::DeleteTree( - _, - super::SubelementsDeletionBehavior::Skip - | super::SubelementsDeletionBehavior::Error - ) - ); - if removes_subtree + // The displaced value is in hand, so a `NotParticipant` claim costs + // nothing to check and fails closed. + let previous_participates = previous + .as_ref() + .is_some_and(Element::supports_backward_references); + // A conditional insert over an existing key writes nothing (or fails + // for existing), so it displaces nothing to check. + if previous_participates + && !op.displaced_value.may_be_participant() + && !matches!(op.op, GroveOp::InsertIfNotExists { .. }) + { + return Err(Error::NotSupported( + "operation declared DisplacedValue::NotParticipant but the stored value takes \ + part in backward references" + .to_owned(), + )) + .wrap_with_cost(cost); + } + needs_reference_planning |= + new_element.is_some_and(Element::supports_backward_references) || previous_participates; + // A removal whose contents nothing in this batch reads is scanned for + // participants when the op declares there may be some: `Delete` on a + // populated tree and a tree replacement. Every `DeleteTree` is + // exempt: `DeleteChildren`, `Error` and `Skip` are checked on the + // cleanup walk that already decodes the contents, and + // `DontCheckWithNoCleanup` declares that the batch's own deletes, + // each planned here, emptied the subtree. + let removes_unread_subtree = op.displaced_value.may_be_participant() + && !matches!( + op.op, + GroveOp::InsertIfNotExists { .. } + | GroveOp::RefreshReference { .. } + | GroveOp::DeleteTree(..) + ); + if removes_unread_subtree && previous.as_ref().is_some_and(|old| { old.is_any_tree() && !old.uses_non_merk_data_storage() @@ -727,7 +749,7 @@ pub(super) fn expand_backward_references_ops<'db>( &mut cost, db.backward_reference_participants(&qualified, tx, grove_version) ); - if matches!(op.op, GroveOp::Delete | GroveOp::DeleteTree(..)) { + if matches!(op.op, GroveOp::Delete) { if participants.iter().any(|(path, key, _)| { !user_deleted_positions.contains(&(path.clone(), key.clone())) }) { diff --git a/grovedb/src/batch/batch_structure.rs b/grovedb/src/batch/batch_structure.rs index 9ea066a61..915005f99 100644 --- a/grovedb/src/batch/batch_structure.rs +++ b/grovedb/src/batch/batch_structure.rs @@ -187,6 +187,7 @@ where path: op_path, key: op_key, op: grove_op, + displaced_value: _, } = op; // Keyless ops (append-only tree ops: CommitmentTreeInsert, diff --git a/grovedb/src/batch/estimated_costs/average_case_costs.rs b/grovedb/src/batch/estimated_costs/average_case_costs.rs index 80820c96f..d081ebdc7 100644 --- a/grovedb/src/batch/estimated_costs/average_case_costs.rs +++ b/grovedb/src/batch/estimated_costs/average_case_costs.rs @@ -33,7 +33,6 @@ use integer_encoding::VarInt; #[cfg(feature = "minimal")] use itertools::Itertools; -use crate::Element; #[cfg(feature = "minimal")] use crate::{ batch::{ @@ -42,6 +41,7 @@ use crate::{ }, Error, GroveDb, }; +use crate::{DisplacedValue, Element}; #[cfg(feature = "minimal")] impl GroveOp { @@ -63,12 +63,11 @@ impl GroveOp { // and compaction terms by `2^chunk_power`, which the op itself does // not carry. Ignored by every other op type. append_tree_chunk_power: Option, - // Whether the batch maintains backward references - // (`BatchApplyOptions::backward_references_policy`): participant - // writes then charge the derived fan-out on GROVE_V4+, and plain - // writes/deletes charge the displaced-state bound only in layers - // declaring that they may contain participants. - backward_references_enabled: bool, + // The op's own declaration about the value it displaces: participant + // payloads charge the derived fan-out on GROVE_V4+ regardless, and a + // plain write or delete charges the displaced-state bound only when + // it declares `MayBeParticipant`. + displaced_value: DisplacedValue, propagate: bool, grove_version: &GroveVersion, ) -> CostResult<(), Error> { @@ -89,7 +88,7 @@ impl GroveOp { // documented shape (see the model in `super`). `None` when the op // triggers no bookkeeping or the flag/version leaves it inactive. let backward_references_fan_out = |element: Option<&Element>| { - if !backward_references_enabled || fan_out_version == 0 { + if fan_out_version == 0 { return None; } match element { @@ -121,16 +120,12 @@ impl GroveOp { ) => Some(super::BackwardReferencesFanOut::average_item_with_capacity( element.max_incoming_references().unwrap_or(0), )), - // The estimator cannot see the STORED element the op - // displaces (or deletes): any other write can land on a - // registered family element needing propagation/cascade - // work, so every write charges the typical item shape. // A plain write or delete can only owe maintenance for the // participant it displaces, and the estimator cannot see - // stored state: charge that bound only where the caller - // declared the layer may hold participants. - Some(_) | None => layer_element_estimates - .may_contain_backward_references + // stored state: charge that bound only when the op declares + // the displaced value may be a participant. + Some(_) | None => displaced_value + .may_be_participant() .then(super::BackwardReferencesFanOut::average_item), } }; @@ -154,10 +149,7 @@ impl GroveOp { // deletion — charged whenever the fan-out is active. let flagged_delete_probe = || { let mut probe = OperationCost::default(); - if backward_references_enabled - && fan_out_version != 0 - && layer_element_estimates.may_contain_backward_references - { + if fan_out_version != 0 && displaced_value.may_be_participant() { let key_width = GroveDb::average_case_layer_key_size( &layer_element_estimates.estimated_layer_sizes, ); @@ -902,6 +894,8 @@ pub(in crate::batch) struct AverageCaseTreeCacheKnownPaths { /// The axes each pending path maintains, so the emitted op carries one /// entry per axis exactly as a real apply would. pending_cidx_axes: HashMap>, Vec>, + /// Each op's declaration about the value it displaces, by position. + displaced_values: HashMap<(KeyInfoPath, KeyInfo), DisplacedValue>, } #[cfg(feature = "minimal")] @@ -909,12 +903,14 @@ impl AverageCaseTreeCacheKnownPaths { /// Updates the cache to the default setting with the given subtree paths pub(in crate::batch) fn new_with_estimated_layer_information( paths: HashMap, + displaced_values: HashMap<(KeyInfoPath, KeyInfo), DisplacedValue>, ) -> Self { AverageCaseTreeCacheKnownPaths { paths, cached_merks: HashMap::default(), pending_cidx_secondary: HashSet::default(), pending_cidx_axes: HashMap::default(), + displaced_values, } } } @@ -980,7 +976,7 @@ impl TreeCache for AverageCaseTreeCacheKnownPaths { path: &KeyInfoPath, ops_at_path_by_key: BTreeMap, _ops_by_qualified_paths: &BTreeMap>, GroveOp>, - batch_apply_options: &BatchApplyOptions, + _batch_apply_options: &BatchApplyOptions, _flags_update: &mut G, _split_removal_bytes: &mut SR, grove_version: &GroveVersion, @@ -1146,7 +1142,10 @@ impl TreeCache for AverageCaseTreeCacheKnownPaths { &key, layer_element_estimates, append_tree_chunk_power, - batch_apply_options.backward_references_policy.maintains(), + self.displaced_values + .get(&(path.clone(), key.clone())) + .copied() + .unwrap_or_default(), false, grove_version ) @@ -1277,8 +1276,15 @@ impl TreeCache for AverageCaseTreeCacheKnownPaths { #[cfg(feature = "minimal")] #[cfg(test)] mod tests { - use crate::batch::BatchApplyOptions; - use crate::BackwardReferencesPolicy; + /// The estimator charges the displaced-participant fan-out only for ops + /// that declare `MayBeParticipant`; these plain-write pins declare none. + fn not_participant(ops: Vec) -> Vec { + ops.into_iter() + .map(|op| op.with_displaced_value(DisplacedValue::NotParticipant)) + .collect() + } + + use crate::DisplacedValue; use std::collections::HashMap; use grovedb_costs::{ @@ -1314,17 +1320,16 @@ mod tests { let db = make_empty_grovedb(); let tx = db.start_transaction(); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(0), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -1332,10 +1337,7 @@ mod tests { let average_case_cost = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1388,17 +1390,16 @@ mod tests { let db = make_empty_grovedb(); let tx = db.start_transaction(); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree_with_flags(Some(b"cat".to_vec())), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(0, true), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, Some(3)), }, @@ -1407,7 +1408,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"key1".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(0, true), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -1415,10 +1415,7 @@ mod tests { let average_case_cost = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1461,17 +1458,16 @@ mod tests { let db = make_empty_grovedb(); let tx = db.start_transaction(); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::new_item(b"cat".to_vec()), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(0, true), estimated_layer_sizes: AllItems(4, 3, None), }, @@ -1479,10 +1475,7 @@ mod tests { let average_case_cost = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1539,17 +1532,16 @@ mod tests { .unwrap() .expect("successful root tree leaf insert"); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -1558,10 +1550,7 @@ mod tests { let average_case_cost = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1630,17 +1619,16 @@ mod tests { .unwrap() .expect("successful root tree leaf insert"); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![b"0".to_vec()], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(0, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -1650,7 +1638,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"0".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(0, true), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -1659,10 +1646,7 @@ mod tests { let average_case_cost = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1715,18 +1699,17 @@ mod tests { #[test] fn test_batch_root_one_sum_item_replace_op_average_case_costs() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::replace_op( vec![vec![7]], hex::decode("46447a3b4c8939fd4cf8b610ba7da3d3f6b52b39ab2549bf91503b9b07814055") .unwrap(), Element::new_sum_item(500), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees( 1, @@ -1749,7 +1732,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![vec![7]]), EstimatedLayerInformation { tree_type: TreeType::SumTree, - may_contain_backward_references: false, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems(32, 8, None), }, @@ -1757,10 +1739,7 @@ mod tests { let average_case_cost = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(paths), ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1807,17 +1786,16 @@ mod tests { .unwrap() .expect("successful root tree leaf insert"); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -1827,7 +1805,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"0".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(0, true), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -1859,17 +1836,16 @@ mod tests { let db = make_empty_grovedb(); let tx = db.start_transaction(); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_dense_tree(3), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(0), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -1877,10 +1853,7 @@ mod tests { let average_case_cost = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1906,7 +1879,7 @@ mod tests { #[test] fn test_refresh_reference_average_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::refresh_reference_op( + let ops = not_participant(vec![QualifiedGroveDbOp::refresh_reference_op( vec![vec![7]], b"ref_key".to_vec(), ReferencePathType::AbsolutePathReference(vec![b"target".to_vec()]), @@ -1914,13 +1887,12 @@ mod tests { None, /* non_counted = */ false, true, - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -1929,7 +1901,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![vec![7]]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems(32, 64, None), }, @@ -1964,22 +1935,23 @@ mod tests { #[test] fn test_refresh_reference_with_sum_item_average_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::refresh_reference_with_sum_item_op( - vec![vec![7]], - b"ref_key".to_vec(), - ReferencePathType::AbsolutePathReference(vec![b"target".to_vec()]), - Some(5), - 42, // sum_value - None, // flags - false, // non_counted - true, // trust_refresh_reference - )]; + let ops = not_participant(vec![ + QualifiedGroveDbOp::refresh_reference_with_sum_item_op( + vec![vec![7]], + b"ref_key".to_vec(), + ReferencePathType::AbsolutePathReference(vec![b"target".to_vec()]), + Some(5), + 42, // sum_value + None, // flags + false, // non_counted + true, // trust_refresh_reference + ), + ]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -1988,7 +1960,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![vec![7]]), EstimatedLayerInformation { tree_type: TreeType::SumTree, - may_contain_backward_references: false, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems(32, 64, None), }, @@ -2052,7 +2023,6 @@ mod tests { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -2061,7 +2031,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![vec![7]]), EstimatedLayerInformation { tree_type: TreeType::CountSumTree, - may_contain_backward_references: false, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems(32, 64, None), }, @@ -2112,18 +2081,17 @@ mod tests { #[test] fn test_patch_average_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::patch_op( + let ops = not_participant(vec![QualifiedGroveDbOp::patch_op( vec![vec![7]], b"patch_key".to_vec(), Element::new_item(b"patched_value".to_vec()), 5, // change_in_bytes - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -2132,7 +2100,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![vec![7]]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems(32, 64, None), }, @@ -2166,16 +2133,15 @@ mod tests { #[test] fn test_delete_average_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::delete_op( + let ops = not_participant(vec![QualifiedGroveDbOp::delete_op( vec![vec![7]], b"del_key".to_vec(), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -2184,7 +2150,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![vec![7]]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllItems(32, 64, None), }, @@ -2218,18 +2183,17 @@ mod tests { #[test] fn test_delete_tree_average_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::delete_tree_op( + let ops = not_participant(vec![QualifiedGroveDbOp::delete_tree_op( vec![vec![7]], b"tree_key".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::Error, - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(1, NoSumTrees, None), }, @@ -2238,7 +2202,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![vec![7]]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: PotentiallyAtMaxElements, estimated_layer_sizes: AllSubtrees(32, NoSumTrees, None), }, @@ -2285,7 +2248,6 @@ mod tests { let key = KeyInfo::KnownKey(b"tree_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(10), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2297,7 +2259,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version ) @@ -2309,7 +2271,7 @@ mod tests { &key, &layer_info, Some(10), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2355,7 +2317,6 @@ mod tests { let key = KeyInfo::KnownKey(b"mmr_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(5), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2365,7 +2326,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2405,7 +2366,6 @@ mod tests { let key = KeyInfo::KnownKey(b"bulk_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(5), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2415,7 +2375,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2450,7 +2410,6 @@ mod tests { let key = KeyInfo::KnownKey(b"pds_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(5), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2460,7 +2419,7 @@ mod tests { &key, &layer_info, Some(4), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2496,7 +2455,7 @@ mod tests { &key, &layer_info, Some(4), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2511,7 +2470,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version ) @@ -2530,7 +2489,7 @@ mod tests { &key, &layer_info, Some(2), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2542,7 +2501,7 @@ mod tests { &key, &layer_info, Some(10), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2588,7 +2547,6 @@ mod tests { let key = KeyInfo::KnownKey(b"dense_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(5), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2598,7 +2556,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2641,7 +2599,6 @@ mod tests { let key = KeyInfo::KnownKey(b"nmerk_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(5), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2651,7 +2608,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2685,7 +2642,6 @@ mod tests { let key = KeyInfo::KnownKey(b"inmerk_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(0), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2695,7 +2651,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2734,7 +2690,6 @@ mod tests { let key = KeyInfo::KnownKey(b"merk_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(0), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2753,7 +2708,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2798,7 +2753,6 @@ mod tests { let key = KeyInfo::KnownKey(b"inmerk_key".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(0), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2816,7 +2770,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2844,7 +2798,6 @@ mod tests { let key = KeyInfo::KnownKey(b"agg_idx".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(8), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -2864,7 +2817,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2884,7 +2837,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, true, grove_version, ) @@ -2926,18 +2879,17 @@ mod tests { .unwrap() .expect("create pcit"); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![b"cidx".to_vec()], b"k1".to_vec(), Element::new_item(b"v1".to_vec()), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -2948,7 +2900,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"cidx".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::ProvableCountIndexedTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllItems(2, 2, None), }, @@ -3036,18 +2987,17 @@ mod tests { .expect("seed entry"); // Same key, same count contribution, different bytes. - let ops = vec![QualifiedGroveDbOp::replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::replace_op( vec![b"cidx".to_vec()], b"k1".to_vec(), Element::new_item(b"v2".to_vec()), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -3056,7 +3006,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"cidx".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::ProvableCountIndexedTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllItems(2, 2, None), }, @@ -3151,18 +3100,17 @@ mod tests { .unwrap() .expect("seed entry"); - let ops = vec![QualifiedGroveDbOp::replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::replace_op( vec![b"idx".to_vec()], b"k1".to_vec(), Element::new_item_with_sum_item(b"v2".to_vec(), 777), - )]; + )]); let mut paths = HashMap::new(); paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllSubtrees(3, NoSumTrees, None), }, @@ -3171,7 +3119,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"idx".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::ProvableCountProvableSumIndexedTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: grovedb_merk::estimated_costs::average_case_costs::EstimatedLayerSizes::AllItemsWithSumItem(2, 2, None), @@ -3270,7 +3217,6 @@ mod tests { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -3279,7 +3225,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"cidx".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::ProvableCountIndexedTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(n as u32), estimated_layer_sizes: AllItems(2, 2, None), }, @@ -3371,7 +3316,6 @@ mod tests { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllSubtrees(3, NoSumTrees, None), }, @@ -3380,7 +3324,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"idx".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::ProvableCountProvableSumIndexedTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(n as u32), estimated_layer_sizes: grovedb_merk::estimated_costs::average_case_costs::EstimatedLayerSizes::AllItemsWithSumItem(2, 2, None), @@ -3468,7 +3411,6 @@ mod tests { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(1), estimated_layer_sizes: AllSubtrees(3, NoSumTrees, None), }, @@ -3477,7 +3419,6 @@ mod tests { KeyInfoPath(vec![KeyInfo::KnownKey(b"idx".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::ProvableCountProvableSumIndexedTree, - may_contain_backward_references: false, estimated_layer_count: ApproximateElements(4), estimated_layer_sizes: grovedb_merk::estimated_costs::average_case_costs::EstimatedLayerSizes::AllItemsWithSumItem(2, 2, None), @@ -3549,7 +3490,6 @@ mod tests { let key = KeyInfo::KnownKey(b"pool".to_vec()); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }; @@ -3560,7 +3500,7 @@ mod tests { &key, &layer_info, None, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -3574,7 +3514,7 @@ mod tests { &key, &layer_info, Some(4), - false, + DisplacedValue::NotParticipant, false, grove_version, ) diff --git a/grovedb/src/batch/estimated_costs/mod.rs b/grovedb/src/batch/estimated_costs/mod.rs index 42f91c564..9dab41140 100644 --- a/grovedb/src/batch/estimated_costs/mod.rs +++ b/grovedb/src/batch/estimated_costs/mod.rs @@ -43,7 +43,7 @@ pub(in crate::batch) fn wrapper_overhead_for( // ── Backward-references fan-out estimation model ──────────────────────── // -// Under `BatchApplyOptions::backward_references_policy` (GROVE_V4+), a +// Under `BatchApplyOptions::displaced_value` (GROVE_V4+), a // single op can expand into derived operations in OTHER subtrees: // registering on a target, rewriting every referrer chain with a new end // hash, cascading deletions through referrer chains. The estimator cannot diff --git a/grovedb/src/batch/estimated_costs/worst_case_costs.rs b/grovedb/src/batch/estimated_costs/worst_case_costs.rs index 2c5c23785..6cd2ecbd5 100644 --- a/grovedb/src/batch/estimated_costs/worst_case_costs.rs +++ b/grovedb/src/batch/estimated_costs/worst_case_costs.rs @@ -31,7 +31,6 @@ use grovedb_version::{error::GroveVersionError, version::GroveVersion}; #[cfg(feature = "minimal")] use itertools::Itertools; -use crate::Element; #[cfg(feature = "minimal")] use crate::{ batch::{ @@ -40,6 +39,7 @@ use crate::{ }, Error, GroveDb, }; +use crate::{DisplacedValue, Element}; #[cfg(feature = "minimal")] impl GroveOp { @@ -52,12 +52,11 @@ impl GroveOp { key: &KeyInfo, in_parent_tree_type: TreeType, worst_case_layer_element_estimates: &WorstCaseLayerInformation, - // Whether the batch maintains backward references - // (`BatchApplyOptions::backward_references_policy`): participant - // writes then charge the derived fan-out on GROVE_V4+, and plain - // writes/deletes charge the displaced-state bound only in layers - // declaring that they may contain participants. - backward_references_enabled: bool, + // The op's own declaration about the value it displaces: participant + // payloads charge the derived fan-out on GROVE_V4+ regardless, and a + // plain write or delete charges the displaced-state bound only when + // it declares `MayBeParticipant`. + displaced_value: DisplacedValue, propagate: bool, grove_version: &GroveVersion, ) -> CostResult<(), Error> { @@ -76,7 +75,7 @@ impl GroveOp { // The derived fan-out charged on top of an op's own model, per the // documented worst-case bounds (see the model in `super`). let backward_references_fan_out = |element: Option<&Element>| { - if !backward_references_enabled || fan_out_version == 0 { + if fan_out_version == 0 { return None; } match element { @@ -115,10 +114,10 @@ impl GroveOp { // is the full item bound at the protocol ceiling. // A plain write or delete can only owe maintenance for the // participant it displaces, and the estimator cannot see - // stored state: charge that bound only where the caller - // declared the layer may hold participants. - Some(_) | None => worst_case_layer_element_estimates - .may_contain_backward_references() + // stored state: charge that bound only when the op declares + // the displaced value may be a participant. + Some(_) | None => displaced_value + .may_be_participant() .then(super::BackwardReferencesFanOut::worst_item), } }; @@ -142,10 +141,7 @@ impl GroveOp { // deletion — charged whenever the fan-out is active. let flagged_delete_probe = || { let mut probe = OperationCost::default(); - if backward_references_enabled - && fan_out_version != 0 - && worst_case_layer_element_estimates.may_contain_backward_references() - { + if fan_out_version != 0 && displaced_value.may_be_participant() { for _ in 0..2 { let _ = add_worst_case_get_merk_node( &mut probe, @@ -799,6 +795,8 @@ fn add_worst_case_backward_references_fan_out( pub(in crate::batch) struct WorstCaseTreeCacheKnownPaths { paths: HashMap, cached_merks: HashSet, + /// Each op's declaration about the value it displaces, by position. + displaced_values: HashMap<(KeyInfoPath, KeyInfo), DisplacedValue>, } #[cfg(feature = "minimal")] @@ -806,10 +804,12 @@ impl WorstCaseTreeCacheKnownPaths { /// Updates the cache with the default settings and the given paths pub(in crate::batch) fn new_with_worst_case_layer_information( paths: HashMap, + displaced_values: HashMap<(KeyInfoPath, KeyInfo), DisplacedValue>, ) -> Self { WorstCaseTreeCacheKnownPaths { paths, cached_merks: HashSet::default(), + displaced_values, } } } @@ -849,7 +849,7 @@ impl TreeCache for WorstCaseTreeCacheKnownPaths { path: &KeyInfoPath, ops_at_path_by_key: BTreeMap, _ops_by_qualified_paths: &BTreeMap>, GroveOp>, - batch_apply_options: &BatchApplyOptions, + _batch_apply_options: &BatchApplyOptions, _flags_update: &mut G, _split_removal_bytes: &mut SR, grove_version: &GroveVersion, @@ -900,7 +900,10 @@ impl TreeCache for WorstCaseTreeCacheKnownPaths { &key, TreeType::NormalTree, worst_case_layer_element_estimates, - batch_apply_options.backward_references_policy.maintains(), + self.displaced_values + .get(&(path.clone(), key.clone())) + .copied() + .unwrap_or_default(), false, grove_version ) @@ -944,8 +947,15 @@ impl TreeCache for WorstCaseTreeCacheKnownPaths { #[cfg(feature = "minimal")] #[cfg(test)] mod tests { - use crate::batch::BatchApplyOptions; - use crate::BackwardReferencesPolicy; + /// The estimator charges the displaced-participant fan-out only for ops + /// that declare `MayBeParticipant`; these plain-write pins declare none. + fn not_participant(ops: Vec) -> Vec { + ops.into_iter() + .map(|op| op.with_displaced_value(DisplacedValue::NotParticipant)) + .collect() + } + + use crate::DisplacedValue; use std::collections::HashMap; use grovedb_costs::{ @@ -973,20 +983,17 @@ mod tests { let db = make_empty_grovedb(); let tx = db.start_transaction(); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); let worst_case_cost = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1032,20 +1039,17 @@ mod tests { let db = make_empty_grovedb(); let tx = db.start_transaction(); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree_with_flags(Some(b"cat".to_vec())), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(0)); let worst_case_cost = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1091,20 +1095,17 @@ mod tests { let db = make_empty_grovedb(); let tx = db.start_transaction(); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::new_item(b"cat".to_vec()), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(0)); let worst_case_cost = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1161,20 +1162,17 @@ mod tests { .unwrap() .expect("successful root tree leaf insert"); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(u32::MAX)); let worst_case_cost = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1231,11 +1229,11 @@ mod tests { .unwrap() .expect("successful root tree leaf insert"); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![b"0".to_vec()], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); paths.insert( @@ -1245,10 +1243,7 @@ mod tests { let worst_case_cost = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -1299,11 +1294,11 @@ mod tests { .unwrap() .expect("successful root tree leaf insert"); - let ops = vec![QualifiedGroveDbOp::insert_or_replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::insert_or_replace_op( vec![], b"key1".to_vec(), Element::empty_tree(), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(u32::MAX)); let worst_case_cost_result = GroveDb::estimated_case_operations_for_batch( @@ -1334,7 +1329,7 @@ mod tests { #[test] fn test_refresh_reference_worst_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::refresh_reference_op( + let ops = not_participant(vec![QualifiedGroveDbOp::refresh_reference_op( vec![vec![7]], b"ref_key".to_vec(), ReferencePathType::AbsolutePathReference(vec![b"target".to_vec()]), @@ -1342,7 +1337,7 @@ mod tests { None, /* non_counted = */ false, true, - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); paths.insert( @@ -1368,16 +1363,18 @@ mod tests { #[test] fn test_refresh_reference_with_sum_item_worst_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::refresh_reference_with_sum_item_op( - vec![vec![7]], - b"ref_key".to_vec(), - ReferencePathType::AbsolutePathReference(vec![b"target".to_vec()]), - Some(5), - 42, // sum_value - None, // flags - false, // non_counted - true, // trust_refresh_reference - )]; + let ops = not_participant(vec![ + QualifiedGroveDbOp::refresh_reference_with_sum_item_op( + vec![vec![7]], + b"ref_key".to_vec(), + ReferencePathType::AbsolutePathReference(vec![b"target".to_vec()]), + Some(5), + 42, // sum_value + None, // flags + false, // non_counted + true, // trust_refresh_reference + ), + ]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); paths.insert( @@ -1476,12 +1473,12 @@ mod tests { #[test] fn test_patch_worst_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::patch_op( + let ops = not_participant(vec![QualifiedGroveDbOp::patch_op( vec![vec![7]], b"patch_key".to_vec(), Element::new_item(b"patched_value".to_vec()), 5, - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); paths.insert( @@ -1507,10 +1504,10 @@ mod tests { #[test] fn test_delete_worst_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::delete_op( + let ops = not_participant(vec![QualifiedGroveDbOp::delete_op( vec![vec![7]], b"del_key".to_vec(), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); paths.insert( @@ -1535,12 +1532,12 @@ mod tests { #[test] fn test_delete_tree_worst_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::delete_tree_op( + let ops = not_participant(vec![QualifiedGroveDbOp::delete_tree_op( vec![vec![7]], b"tree_key".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::Error, - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); paths.insert( @@ -1580,7 +1577,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1606,7 +1603,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, true, grove_version, ) @@ -1631,7 +1628,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1656,7 +1653,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1680,7 +1677,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1716,7 +1713,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1744,7 +1741,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, true, grove_version, ) @@ -1768,7 +1765,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(50), - false, + DisplacedValue::NotParticipant, true, grove_version, ) @@ -1800,7 +1797,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1832,7 +1829,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, true, grove_version, ) @@ -1868,7 +1865,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1924,7 +1921,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -1944,11 +1941,11 @@ mod tests { #[test] fn test_replace_worst_case_cost() { let grove_version = GroveVersion::latest(); - let ops = vec![QualifiedGroveDbOp::replace_op( + let ops = not_participant(vec![QualifiedGroveDbOp::replace_op( vec![vec![7]], b"key1".to_vec(), Element::new_item(b"val".to_vec()), - )]; + )]); let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(1)); paths.insert( @@ -1995,7 +1992,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2022,7 +2019,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, true, grove_version, ) @@ -2072,7 +2069,7 @@ mod tests { &key, TreeType::NormalTree, &layer_info, - false, + DisplacedValue::NotParticipant, false, grove_version, ) @@ -2132,7 +2129,7 @@ mod tests { &key, TreeType::NormalTree, &MaxElementsNumber(100), - false, + DisplacedValue::NotParticipant, false, grove_version, ) diff --git a/grovedb/src/batch/mod.rs b/grovedb/src/batch/mod.rs index b057a3708..9c97fd9c8 100644 --- a/grovedb/src/batch/mod.rs +++ b/grovedb/src/batch/mod.rs @@ -39,7 +39,7 @@ mod single_sum_item_deletion_cost_tests; #[cfg(test)] mod single_sum_item_insert_cost_tests; -use crate::BackwardReferencesPolicy; +use crate::{bidirectional_references::batch_maintains_backward_references, DisplacedValue}; use core::fmt; use std::{ cmp::Ordering, @@ -153,6 +153,14 @@ pub enum SubelementsDeletionBehavior { /// O(1), storage reclaimed) or [`Self::DeleteChildren`] (recursive /// cleanup, O(contents)). /// + /// Under `DisplacedValue::MayBeParticipant` (`GROVE_V4`+) the + /// declaration also stands in for the removed subtree's + /// backward-reference participant scan: every element the batch removed + /// beneath the tree passed through the old-value observer (or the full + /// batch's preparation), so nothing unobserved can remain and nothing is + /// read. A false declaration therefore also strands the registrations of + /// any participant it leaks, exactly like the storage. + /// /// One exception to "touches no child storage": an indexed primary /// still gets its per-axis secondary namespaces swept, because those /// live outside the primary's prefix and can hold stale rows even when @@ -170,7 +178,9 @@ pub enum SubelementsDeletionBehavior { /// still perform post-apply storage cleanup to remove the child /// subtree's storage (and any nested subtrees), walking the structure /// via `find_subtrees` — O(contents). Use this when the subtree may - /// contain children that should be recursively cleaned up. + /// contain children that should be recursively cleaned up. Under + /// `DisplacedValue::MayBeParticipant` the contents are also scanned + /// for backward-reference participants before commit. DeleteChildren, /// Check emptiness at apply time. If the subtree is non-empty, /// silently skip this `DeleteTree` operation (no error, no deletion). @@ -185,8 +195,8 @@ pub enum SubelementsDeletionBehavior { /// GroveDB owns the transaction, at the caller's next /// `flush_pending_prefix_drops` otherwise. /// - /// Requires explicit `BatchApplyOptions::backward_references_policy = Skip`; - /// Maintain refuses this operation before scanning. + /// Requires the op to declare `DisplacedValue::NotParticipant`; + /// `MayBeParticipant` is refused before reading anything. /// The caller declares the subtree contains **no child subtrees**; a /// false declaration leaks the children's storage (unreachable, /// invisible to hashes/proofs/sync) but never corrupts state. The @@ -964,6 +974,39 @@ impl KeyInfoPath { } } +/// Per-position facts a batch consults after apply, keyed by qualified path +/// (path segments plus key) for ops with known keys. +#[derive(Default)] +struct OpDeclarations { + /// Each op's declaration about the value it displaces. + displaced_values: HashMap>, DisplacedValue>, + /// Positions the batch deletes explicitly (`Delete` / `DeleteTree`), + /// which account for participants found under a removed subtree. + deleted_positions: HashSet>>, +} + +impl OpDeclarations { + fn from_ops(ops: &[QualifiedGroveDbOp]) -> Self { + let mut declarations = Self::default(); + declarations.extend_from(ops); + declarations + } + + fn extend_from(&mut self, ops: &[QualifiedGroveDbOp]) { + for op in ops { + let Some(key) = op.key.as_ref() else { + continue; + }; + let mut qualified = op.path.to_path(); + qualified.push(key.get_key_clone()); + if matches!(op.op, GroveOp::Delete | GroveOp::DeleteTree(..)) { + self.deleted_positions.insert(qualified.clone()); + } + self.displaced_values.insert(qualified, op.displaced_value); + } + } +} + /// Batch operation #[derive(Clone, PartialEq, Eq, Hash)] pub struct QualifiedGroveDbOp { @@ -976,6 +1019,10 @@ pub struct QualifiedGroveDbOp { pub key: Option, /// Operation to perform on the key pub op: GroveOp, + /// What this operation declares about the stored value it displaces. + /// Every constructor defaults to `MayBeParticipant`; see + /// [`Self::with_displaced_value`]. + pub displaced_value: DisplacedValue, } impl fmt::Debug for QualifiedGroveDbOp { @@ -1092,6 +1139,29 @@ impl fmt::Debug for QualifiedGroveDbOp { } impl QualifiedGroveDbOp { + /// Declare what this operation displaces. Every constructor starts at + /// [`DisplacedValue::MayBeParticipant`]; a caller that knows the stored + /// value takes no part in backward references declares + /// [`DisplacedValue::NotParticipant`] here. + pub fn with_displaced_value(mut self, displaced_value: DisplacedValue) -> Self { + self.displaced_value = displaced_value; + self + } + + /// Each op's declaration by `(path, key)`, for the estimators, which see + /// ops keyed by position rather than as `QualifiedGroveDbOp`s. + pub(in crate::batch) fn displaced_values_by_position( + ops: &[Self], + ) -> HashMap<(KeyInfoPath, KeyInfo), DisplacedValue> { + ops.iter() + .filter_map(|op| { + op.key + .clone() + .map(|key| ((op.path.clone(), key), op.displaced_value)) + }) + .collect() + } + /// An insert op using a known owned path and known key. /// The caller asserts the key is new — no existence check is performed. /// This is a performance optimization hint. @@ -1105,6 +1175,7 @@ impl QualifiedGroveDbOp { path, key: Some(KnownKey(key)), op: GroveOp::InsertWithKnownToNotAlreadyExist { element }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1127,6 +1198,7 @@ impl QualifiedGroveDbOp { element, error_if_exists: true, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1145,6 +1217,7 @@ impl QualifiedGroveDbOp { element, error_if_exists: false, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1155,6 +1228,7 @@ impl QualifiedGroveDbOp { path, key: Some(KnownKey(key)), op: GroveOp::InsertOrReplace { element }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1164,6 +1238,7 @@ impl QualifiedGroveDbOp { path, key: Some(key), op: GroveOp::InsertOrReplace { element }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1174,6 +1249,7 @@ impl QualifiedGroveDbOp { path, key: Some(KnownKey(key)), op: GroveOp::Replace { element }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1183,6 +1259,7 @@ impl QualifiedGroveDbOp { path, key: Some(key), op: GroveOp::Replace { element }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1201,6 +1278,7 @@ impl QualifiedGroveDbOp { element, change_in_bytes, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1218,6 +1296,7 @@ impl QualifiedGroveDbOp { element, change_in_bytes, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1265,6 +1344,7 @@ impl QualifiedGroveDbOp { flags, non_counted, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1308,6 +1388,7 @@ impl QualifiedGroveDbOp { flags, non_counted, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1347,6 +1428,7 @@ impl QualifiedGroveDbOp { flags, non_counted, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1357,6 +1439,7 @@ impl QualifiedGroveDbOp { path, key: Some(KnownKey(key)), op: GroveOp::Delete, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1372,6 +1455,7 @@ impl QualifiedGroveDbOp { path, key: Some(KnownKey(key)), op: GroveOp::DeleteTree(tree_type, subelements_deletion_behavior), + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1381,6 +1465,7 @@ impl QualifiedGroveDbOp { path, key: Some(key), op: GroveOp::Delete, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1395,6 +1480,7 @@ impl QualifiedGroveDbOp { path, key: Some(key), op: GroveOp::DeleteTree(tree_type, subelements_deletion_behavior), + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1418,6 +1504,7 @@ impl QualifiedGroveDbOp { cv_net, payload, }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1442,6 +1529,7 @@ impl QualifiedGroveDbOp { path, key: None, op: GroveOp::MmrTreeAppend { value }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1452,6 +1540,7 @@ impl QualifiedGroveDbOp { path, key: None, op: GroveOp::BulkAppend { value }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1463,6 +1552,7 @@ impl QualifiedGroveDbOp { path, key: None, op: GroveOp::DenseTreeInsert { value }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1475,6 +1565,7 @@ impl QualifiedGroveDbOp { path, key: None, op: GroveOp::PrivateDocumentStoreInsert { entry }, + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -1649,7 +1740,12 @@ impl GroveDbOpConsistencyResults { /// Cache for Merk trees by their paths. struct TreeCacheMerkByPath { backward_references_prepared: bool, - unprepared_subtree_removals: Vec>>, + /// Each op's declaration by qualified path, for the ops this cache is + /// applying: the initial segment's, then the continuation's added on top. + declarations: HashMap>, DisplacedValue>, + /// Populated Merk subtrees the observer saw removed or replaced, each + /// with the declaration of the op that displaced it. + unprepared_subtree_removals: Vec<(Vec>, DisplacedValue)>, merks: HashMap>, Merk>, /// Empty Merks reserved while scanning tree insertions, with no path /// operations applied yet. A skipped insertion must not carry its @@ -1727,7 +1823,7 @@ impl fmt::Debug for TreeCacheMerkByPath { /// empty on V1..V3. #[derive(Default)] struct BatchApplyCaptures { - unprepared_subtree_removals: Vec>>, + unprepared_subtree_removals: Vec<(Vec>, DisplacedValue)>, /// Cidx primary paths displaced by a safe-subset overwrite; their old /// primary subtree storage + per-axis secondary namespaces get cleared. cidx_overwrite_cleanup_paths: Vec>>, @@ -1950,7 +2046,7 @@ trait TreeCache { /// (primary subtree + secondary namespace) must be cleaned up /// because a safe-subset overwrite replaced them with a non-cidx /// element or an empty cidx. Default impl returns an empty Vec. - fn take_unprepared_subtree_removals(&mut self) -> Vec>> { + fn take_unprepared_subtree_removals(&mut self) -> Vec<(Vec>, DisplacedValue)> { Vec::new() } @@ -2925,7 +3021,7 @@ where .insert(qualified_path, element.clone()); } - fn take_unprepared_subtree_removals(&mut self) -> Vec>> { + fn take_unprepared_subtree_removals(&mut self) -> Vec<(Vec>, DisplacedValue)> { std::mem::take(&mut self.unprepared_subtree_removals) } @@ -4513,9 +4609,7 @@ where return; } if !self.backward_references_prepared - && batch_apply_options - .backward_references_policy - .maintains_in_batch(grove_version) + && batch_maintains_backward_references(grove_version) { match Element::deserialize(old_value, grove_version) { Ok(element) => { @@ -4527,12 +4621,24 @@ where } if element.is_any_tree() && !element.uses_non_merk_data_storage() + && element + .root_key_and_tree_type() + .is_some_and(|(root, _)| root.is_some()) && (matches!(disposition, OldValueDisposition::Deleted) || pending_overwrite_inspections.contains_key(key)) { let mut qualified = path.clone(); qualified.push(key.to_vec()); - self.unprepared_subtree_removals.push(qualified); + // The declaration of the op displacing this + // tree, recorded here so a later op at the + // same path (another segment) cannot speak + // for it. + let declared = self + .declarations + .get(&qualified) + .copied() + .unwrap_or_default(); + self.unprepared_subtree_removals.push((qualified, declared)); } } Err(error) => { @@ -5370,6 +5476,7 @@ impl GroveDb { ) -> CostResult)>, Error>, { + let declarations = OpDeclarations::from_ops(&ops).displaced_values; check_grovedb_v0_with_cost!( "apply_body", grove_version.grovedb_versions.apply_batch.apply_body @@ -5383,6 +5490,7 @@ impl GroveDb { split_removed_bytes_function, TreeCacheMerkByPath { backward_references_prepared, + declarations, unprepared_subtree_removals: Vec::new(), merks: Default::default(), unused_new_merks: Default::default(), @@ -5454,6 +5562,13 @@ impl GroveDb { .apply_batch .continue_partial_apply_body ); + // The continuation applies the initial segment's leftover ops (already + // declared) plus the add-on ops; positions cannot collide across + // segments except validated ancestor merges, where the add-on op is + // the one executing. + merk_tree_cache + .declarations + .extend(OpDeclarations::from_ops(&additional_ops).displaced_values); // The first segment's conditional insertions may have been skipped. // Keep their actual stored values authoritative for references and // discard only unused placeholders reserved for the attempted trees. @@ -5551,7 +5666,13 @@ impl GroveDb { path_slices.as_slice(), key.as_slice(), element.to_owned(), - options.clone().map(|o| o.as_insert_options()), + Some( + options + .as_ref() + .map(BatchApplyOptions::as_insert_options) + .unwrap_or_default() + .with_displaced_value(op.displaced_value), + ), transaction, grove_version, ) @@ -5572,7 +5693,13 @@ impl GroveDb { path_slices.as_slice(), key.as_slice(), element.to_owned(), - options.clone().map(|o| o.as_insert_options()), + Some( + options + .as_ref() + .map(BatchApplyOptions::as_insert_options) + .unwrap_or_default() + .with_displaced_value(op.displaced_value), + ), transaction, grove_version, ) @@ -5592,9 +5719,10 @@ impl GroveDb { ); if error_if_exists { let mut insert_options = options - .clone() - .map(|o| o.as_insert_options()) - .unwrap_or_default(); + .as_ref() + .map(BatchApplyOptions::as_insert_options) + .unwrap_or_default() + .with_displaced_value(op.displaced_value); insert_options.validate_insertion_does_not_override = true; cost_return_on_error!( &mut cost, @@ -5634,7 +5762,13 @@ impl GroveDb { self.delete( path_slices.as_slice(), key.as_slice(), - options.clone().map(|o| o.as_delete_options()), + Some( + options + .as_ref() + .map(BatchApplyOptions::as_delete_options) + .unwrap_or_default() + .with_displaced_value(op.displaced_value), + ), transaction, grove_version ) @@ -5661,10 +5795,7 @@ impl GroveDb { self.drop_flat_subtree( path_slices.as_slice(), key.as_slice(), - options - .as_ref() - .map(|o| o.backward_references_policy) - .unwrap_or_default(), + op.displaced_value, transaction, grove_version ) @@ -5694,10 +5825,7 @@ impl GroveDb { validate_tree_at_path_exists: false, // Same decision as `as_delete_options`: the batch's // opt-in extends to its deletes. - backward_references_policy: options - .as_ref() - .map(|o| o.backward_references_policy) - .unwrap_or_default(), + displaced_value: op.displaced_value, }; cost_return_on_error!( &mut cost, @@ -6442,30 +6570,31 @@ impl GroveDb { Ok(scan).wrap_with_cost(cost) } - /// Flat drop must never acquire a descendant scan from the default policy. - fn reject_flat_drop_with_maintenance( + /// Flat drop never reads the subtree it drops, so it cannot check a + /// participant claim: the op must declare `DisplacedValue::NotParticipant` + /// on versions that maintain backward references. + fn reject_flat_drop_declaring_participants( ops: &[QualifiedGroveDbOp], - policy: BackwardReferencesPolicy, grove_version: &GroveVersion, ) -> Result<(), Error> { - if policy.maintains_in_batch(grove_version) + if batch_maintains_backward_references(grove_version) && ops.iter().any(|op| { matches!( op.op, GroveOp::DeleteTree(_, SubelementsDeletionBehavior::DropFlat) - ) + ) && op.displaced_value.may_be_participant() }) { return Err(Error::NotSupported( - "flat drop requires explicit BackwardReferencesPolicy::Skip; use recursive delete for maintenance".to_owned(), + "flat drop requires the DeleteTree op to declare DisplacedValue::NotParticipant; use recursive delete for maintenance".to_owned(), )); } Ok(()) } - /// Family payloads require full-batch planning on V4 (the default policy). - /// Explicit Skip and partial batches reject them because those paths - /// cannot register their edges or plan cross-subtree reference mutations. + /// Family payloads require full-batch planning on V4. Earlier versions + /// and partial batches reject them because those paths cannot register + /// their edges or plan cross-subtree reference mutations. fn reject_backward_references_elements_in_batch( ops: &[QualifiedGroveDbOp], allow_family: bool, @@ -6500,8 +6629,8 @@ impl GroveDb { } if !allow_family { return Err(Error::NotSupported( - "backward-references family elements require \ - BatchApplyOptions::backward_references_policy (GROVE_V4+)" + "backward-references family elements require a full batch on \ + GROVE_V4+" .to_owned(), )); } @@ -6590,27 +6719,16 @@ impl GroveDb { } } - // V4 maintains backward references by default. The prepared Merks - // carry observations into execution without fetching the nodes twice. - let backward_references_enabled = batch_apply_options - .as_ref() - .map(|options| options.backward_references_policy) - .unwrap_or_default() - .maintains_in_batch(grove_version); + // V4 maintains backward references. The prepared Merks carry + // observations into execution without fetching the nodes twice. + let backward_references_enabled = batch_maintains_backward_references(grove_version); cost_return_on_error_no_add!( cost, Self::reject_backward_references_elements_in_batch(&ops, backward_references_enabled) ); cost_return_on_error_no_add!( cost, - Self::reject_flat_drop_with_maintenance( - &ops, - batch_apply_options - .as_ref() - .map(|options| options.backward_references_policy) - .unwrap_or_default(), - grove_version - ) + Self::reject_flat_drop_declaring_participants(&ops, grove_version) ); let storage_batch = StorageBatch::new(); let (ops, prepared_merks) = if backward_references_enabled { @@ -6709,7 +6827,7 @@ impl GroveDb { mut merk_delete_paths, mut cidx_primary_delete_paths, skipped_delete_paths, - delete_tree_behaviors, + mut delete_tree_behaviors, } = cost_return_on_error!( &mut cost, self.scan_delete_tree_ops(&ops, &storage_batch, tx.as_ref(), grove_version) @@ -6733,6 +6851,12 @@ impl GroveDb { } else { ops }; + // A skipped `DeleteTree` executed nothing: its behavior must not exempt + // a later replacement of the same populated tree from the participant + // scan, so only executed ops keep their entry. + for path in &skipped_delete_paths { + delete_tree_behaviors.remove(path); + } // With the only one difference (if there is a transaction) do the following: // 2. If nothing left to do and we were on a non-leaf subtree or we're done with @@ -6744,6 +6868,9 @@ impl GroveDb { // 5. Remove operation from the tree, repeat until there are operations to do; // 6. Add root leaves save operation to the batch // 7. Apply storage_cost batch + // The final ops, derived cascades included, account for the + // participants a recursive removal's cleanup walk may reach. + let op_declarations = OpDeclarations::from_ops(&ops); let (_leftover, batch_apply_captures, _, _) = cost_return_on_error!( &mut cost, self.apply_body( @@ -6871,6 +6998,7 @@ impl GroveDb { .apply_batch .delete_tree_recursive_secondary_cleanup >= 1, + Some(&op_declarations.deleted_positions), "batch delete", grove_version, ) @@ -6946,6 +7074,7 @@ impl GroveDb { tx.as_ref(), &storage_batch, true, + Some(&op_declarations.deleted_positions), "batch overwrite", grove_version, ) @@ -7072,14 +7201,7 @@ impl GroveDb { ); cost_return_on_error_no_add!( cost, - Self::reject_flat_drop_with_maintenance( - &ops, - batch_apply_options - .as_ref() - .map(|options| options.backward_references_policy) - .unwrap_or_default(), - grove_version - ) + Self::reject_flat_drop_declaring_participants(&ops, grove_version) ); cost_return_on_error!( @@ -7140,6 +7262,7 @@ impl GroveDb { // here from the DECLARED tree types; on V4+ they stay empty and are // filled after apply_body from the ACTUAL stored types captured by // the merk old-value observer. See `scan_delete_tree_ops`. + let mut op_declarations = OpDeclarations::from_ops(&ops); let DeleteTreePreScan { mut non_merk_delete_paths, mut merk_delete_paths, @@ -7169,6 +7292,12 @@ impl GroveDb { } else { ops }; + // A skipped `DeleteTree` executed nothing: its behavior must not exempt + // a later replacement of the same populated tree from the participant + // scan, so only executed ops keep their entry. + for path in &skipped_delete_paths { + delete_tree_behaviors.remove(path); + } if batch_apply_options.batch_pause_height.is_none() { // we default to pausing at the root tree, which is the most common case batch_apply_options.batch_pause_height = Some(1); @@ -7402,11 +7531,7 @@ impl GroveDb { ); cost_return_on_error_no_add!( cost, - Self::reject_flat_drop_with_maintenance( - &new_operations, - batch_apply_options.backward_references_policy, - grove_version - ) + Self::reject_flat_drop_declaring_participants(&new_operations, grove_version) ); // Add-on typed appends (CommitmentTreeInsert, MmrTreeAppend, @@ -7496,7 +7621,7 @@ impl GroveDb { merk_delete_paths: add_on_merk_delete_paths, cidx_primary_delete_paths: add_on_cidx_primary_delete_paths, skipped_delete_paths: add_on_skipped_delete_paths, - delete_tree_behaviors: add_on_delete_tree_behaviors, + delete_tree_behaviors: mut add_on_delete_tree_behaviors, } = cost_return_on_error!( &mut cost, self.scan_delete_tree_ops( @@ -7509,6 +7634,13 @@ impl GroveDb { non_merk_delete_paths.extend(add_on_non_merk_delete_paths); merk_delete_paths.extend(add_on_merk_delete_paths); cidx_primary_delete_paths.extend(add_on_cidx_primary_delete_paths); + // A skipped add-on `DeleteTree` executed nothing: drop its entry + // before merging so it neither exempts a replacement nor erases the + // cleanup behavior of a deletion the initial segment executed at the + // same path. + for path in &add_on_skipped_delete_paths { + add_on_delete_tree_behaviors.remove(path); + } delete_tree_behaviors.extend(add_on_delete_tree_behaviors); for op in &new_operations { if is_merged_ancestor(op) @@ -7541,6 +7673,9 @@ impl GroveDb { } else { new_operations }; + // Only executed add-on deletes account for participants under a + // removed subtree. + op_declarations.extend_from(&new_operations); // we are trying to finalize batch_apply_options.batch_pause_height = None; @@ -7572,10 +7707,20 @@ impl GroveDb { indexed_mirror_rekey_churn_bytes: continue_rekey_churn_bytes, } = continue_captures; - for path in partial_subtree_removals + // A removed subtree whose contents nothing in this batch reads is + // scanned for participants before commit when its op declares there + // may be some: a tree replacement or a `Delete` of a populated tree. + // Every `DeleteTree` is exempt: `DeleteChildren`, `Error` and `Skip` + // are checked on the cleanup walk below, which already decodes the + // contents, and `DontCheckWithNoCleanup` declares that the batch's + // own deletes, each gated by the observer, emptied the subtree. + for (path, declared) in partial_subtree_removals .into_iter() .chain(continue_subtree_removals) { + if delete_tree_behaviors.contains_key(&path) || !declared.may_be_participant() { + continue; + } if !cost_return_on_error!( &mut cost, self.backward_reference_participants(&path, tx.as_ref(), grove_version) @@ -7658,6 +7803,7 @@ impl GroveDb { .apply_batch .delete_tree_recursive_secondary_cleanup >= 1, + Some(&op_declarations.deleted_positions), "batch delete", grove_version, ) @@ -7725,6 +7871,7 @@ impl GroveDb { tx.as_ref(), &storage_batch, true, + Some(&op_declarations.deleted_positions), "batch overwrite", grove_version, ) @@ -7803,6 +7950,7 @@ impl GroveDb { if ops.is_empty() { return Ok(()).wrap_with_cost(cost); } + let displaced_values = QualifiedGroveDbOp::displaced_values_by_position(&ops); match estimated_costs_type { EstimatedCostsType::AverageCaseCostsType(estimated_layer_information) => { @@ -7813,7 +7961,8 @@ impl GroveDb { update_element_flags_function, split_removal_bytes_function, AverageCaseTreeCacheKnownPaths::new_with_estimated_layer_information( - estimated_layer_information + estimated_layer_information, + displaced_values, ), grove_version ) @@ -7836,7 +7985,8 @@ impl GroveDb { update_element_flags_function, split_removal_bytes_function, WorstCaseTreeCacheKnownPaths::new_with_worst_case_layer_information( - worst_case_layer_information + worst_case_layer_information, + displaced_values, ), grove_version ) @@ -7858,7 +8008,7 @@ impl GroveDb { #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use grovedb_costs::storage_cost::removal::StorageRemovedBytes::NoStorageRemoval; use grovedb_merk::proofs::Query; @@ -8039,7 +8189,6 @@ mod tests { disable_operation_consistency_check: true, base_root_storage_is_free: true, batch_pause_height: None, - backward_references_policy: BackwardReferencesPolicy::Skip, }), None, grove_version @@ -8645,7 +8794,6 @@ mod tests { disable_operation_consistency_check: false, base_root_storage_is_free: true, batch_pause_height: None, - backward_references_policy: BackwardReferencesPolicy::Skip, }), None, grove_version @@ -8687,7 +8835,6 @@ mod tests { validate_insertion_does_not_override: true, base_root_storage_is_free: true, batch_pause_height: None, - backward_references_policy: BackwardReferencesPolicy::Skip, }), None, grove_version @@ -8721,7 +8868,6 @@ mod tests { disable_operation_consistency_check: false, base_root_storage_is_free: true, batch_pause_height: None, - backward_references_policy: BackwardReferencesPolicy::Skip, }), None, grove_version diff --git a/grovedb/src/batch/options.rs b/grovedb/src/batch/options.rs index 968be691f..91f957c8c 100644 --- a/grovedb/src/batch/options.rs +++ b/grovedb/src/batch/options.rs @@ -1,7 +1,7 @@ //! Options #[cfg(feature = "minimal")] -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use grovedb_merk::MerkOptions; #[cfg(feature = "minimal")] @@ -78,15 +78,6 @@ pub struct BatchApplyOptions { /// At what height do we want to pause applying batch operations /// Most of the time this should be not set pub batch_pause_height: Option, - /// Maintain registrations, propagate updates, and cascade deletions by - /// default on V4. Old values are observed in Merks retained for execution. - /// `Skip` deliberately permits stale/dangling references and rejects - /// family payloads. Partial batches reject participant mutations while - /// maintenance is enabled; use a full batch for reference planning. - /// Recursive removals inspect descendants under Maintain, including in - /// partial batches before commit. DropFlat requires explicit Skip so it - /// preserves its O(1) contract without a hidden descendant scan. - pub backward_references_policy: BackwardReferencesPolicy, } #[cfg(feature = "minimal")] @@ -98,7 +89,6 @@ impl Default for BatchApplyOptions { disable_operation_consistency_check: false, base_root_storage_is_free: true, batch_pause_height: None, - backward_references_policy: BackwardReferencesPolicy::Maintain, } } } @@ -112,7 +102,9 @@ impl BatchApplyOptions { validate_insertion_does_not_override_tree: self .validate_insertion_does_not_override_tree, base_root_storage_is_free: self.base_root_storage_is_free, - backward_references_policy: self.backward_references_policy, + // Per operation: the caller overrides this with the op's own + // declaration. + displaced_value: DisplacedValue::MayBeParticipant, } } @@ -123,8 +115,9 @@ impl BatchApplyOptions { deleting_non_empty_trees_returns_error: true, base_root_storage_is_free: self.base_root_storage_is_free, validate_tree_at_path_exists: false, - // Preserve the batch's maintenance policy for generated deletes. - backward_references_policy: self.backward_references_policy, + // Per operation: the caller overrides this with the op's own + // declaration. + displaced_value: DisplacedValue::MayBeParticipant, } } diff --git a/grovedb/src/batch/single_insert_cost_tests.rs b/grovedb/src/batch/single_insert_cost_tests.rs index 5a592e0fd..4abf7a39e 100644 --- a/grovedb/src/batch/single_insert_cost_tests.rs +++ b/grovedb/src/batch/single_insert_cost_tests.rs @@ -2,8 +2,7 @@ #[cfg(feature = "minimal")] mod tests { - use crate::batch::BatchApplyOptions; - use crate::BackwardReferencesPolicy; + use grovedb_costs::{ storage_cost::{ removal::{ @@ -1649,17 +1648,9 @@ mod tests { Element::empty_tree(), ), ]; - db.apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - Some(&tx), - grove_version, - ) - .value - .expect("expected to execute setup batch"); + db.apply_batch(ops, None, Some(&tx), grove_version) + .value + .expect("expected to execute setup batch"); // Every op shape the V4 gates watch: an item overwrite via // InsertOrReplace, a reference overwrite via Replace, and a @@ -1682,15 +1673,7 @@ mod tests { SubelementsDeletionBehavior::Error, ), ]; - db.apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - Some(&tx), - grove_version, - ) + db.apply_batch(ops, None, Some(&tx), grove_version) }; let v3 = run(&grovedb_version::version::v3::GROVE_V3); diff --git a/grovedb/src/bidirectional_references/mod.rs b/grovedb/src/bidirectional_references/mod.rs index 92c1a3ef1..8c681b600 100644 --- a/grovedb/src/bidirectional_references/mod.rs +++ b/grovedb/src/bidirectional_references/mod.rs @@ -15,40 +15,51 @@ use grovedb_version::version::GroveVersion; pub(crate) use handling::*; -/// Whether a mutation maintains backward references on GROVE_V4 and later. -/// Ordinary operations maintain references automatically. `Skip` is an -/// explicit choice to permit dangling references and stale reference hashes; -/// it is not an assertion that the stored element has no references. -#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] -pub enum BackwardReferencesPolicy { - /// Register references, propagate changed values, and cascade deletions - /// with each referrer's consent before committing the mutation. APIs that - /// cannot plan that maintenance refuse participant mutations: partial - /// batches and clear_subtree. Flat drop always requires explicit Skip. +/// What a write or removal declares about the stored value it displaces. +/// +/// GroveDB reads the displaced value anyway on `GROVE_V4`+ (the batch +/// old-value observer, the Merk retained for a live write), so for a keyed +/// operation the declaration decides only what happens when that value takes +/// part in backward references: `MayBeParticipant` maintains the references, +/// `NotParticipant` refuses the operation before anything commits. Where +/// nothing reads the contents — a flat drop, a raw `clear_subtree`, the +/// replacement of a populated subtree — `NotParticipant` is trusted and +/// leaves any participant's registrations stale, exactly like the storage it +/// strands. Recursive removals that already walk their contents check the +/// claim on the way at no extra cost. +/// +/// The estimator cannot see stored state, so the same declaration decides +/// whether a plain write or delete is charged the displaced-participant +/// fan-out. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash)] +pub enum DisplacedValue { + /// The displaced value, or for a subtree removal its contents, may take + /// part in backward references: register, propagate and cascade with + /// each referrer's consent before committing. Partial batches cannot + /// plan that maintenance and refuse participant mutations. #[default] - Maintain, - /// Skip maintenance of displaced values. Live bidirectional-reference - /// insertion still registers its edge; batches reject family payloads. - Skip, + MayBeParticipant, + /// The caller knows the displaced value takes no part in backward + /// references. Checked for free wherever the value is read; refused if + /// the claim is false. Required for a flat drop. + NotParticipant, } -impl BackwardReferencesPolicy { - pub(crate) fn maintains(self) -> bool { - matches!(self, Self::Maintain) +impl DisplacedValue { + pub(crate) fn may_be_participant(self) -> bool { + matches!(self, Self::MayBeParticipant) } +} - /// Whether a batch run under this policy maintains backward references on - /// `grove_version`: `Maintain` plus - /// `apply_batch.backward_references_maintenance` (V4+). Released versions - /// never plan references, so they answer `false` under either policy. - pub(crate) fn maintains_in_batch(self, grove_version: &GroveVersion) -> bool { - self.maintains() - && grove_version - .grovedb_versions - .apply_batch - .backward_references_maintenance - >= 1 - } +/// Whether batches maintain backward references on `grove_version` +/// (`apply_batch.backward_references_maintenance`, V4+). Released versions +/// never plan references and reject the family payloads outright. +pub(crate) fn batch_maintains_backward_references(grove_version: &GroveVersion) -> bool { + grove_version + .grovedb_versions + .apply_batch + .backward_references_maintenance + >= 1 } /// Maximum Grove path depth (number of subtree levels) of any position diff --git a/grovedb/src/debugger.rs b/grovedb/src/debugger.rs index 0ab254deb..4b67cdcac 100644 --- a/grovedb/src/debugger.rs +++ b/grovedb/src/debugger.rs @@ -1309,7 +1309,7 @@ fn node_to_update( #[cfg(test)] mod tests { use super::*; - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; #[test] fn element_to_grovedbg_converts_item_with_sum_item() { @@ -1763,7 +1763,7 @@ mod tests { Some(vec![7]), ), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, diff --git a/grovedb/src/estimated_costs/average_case_costs.rs b/grovedb/src/estimated_costs/average_case_costs.rs index 6520ccc01..b1f1c03cc 100644 --- a/grovedb/src/estimated_costs/average_case_costs.rs +++ b/grovedb/src/estimated_costs/average_case_costs.rs @@ -538,7 +538,6 @@ impl GroveDb { .min(u32::MAX as usize) as u32; let secondary_layer = EstimatedLayerInformation { tree_type: axis_secondary_tree_type(*axis), - may_contain_backward_references: false, // 1:1 with the primary. estimated_layer_count: primary_layer_information.estimated_layer_count, // The row shape's OWN variant, not `AllItems`: the two diff --git a/grovedb/src/lib.rs b/grovedb/src/lib.rs index b44e6ae9d..a5f97926e 100644 --- a/grovedb/src/lib.rs +++ b/grovedb/src/lib.rs @@ -175,7 +175,7 @@ use std::sync::Arc; use std::{collections::HashMap, option::Option::None, path::Path}; #[cfg(feature = "minimal")] -pub use bidirectional_references::{BackwardReferencesPolicy, BidirectionalReference}; +pub use bidirectional_references::{BidirectionalReference, DisplacedValue}; #[cfg(feature = "grovedbg")] use debugger::start_visualizer; #[cfg(any(feature = "minimal", feature = "verify"))] diff --git a/grovedb/src/operations/auxiliary.rs b/grovedb/src/operations/auxiliary.rs index 78c21fa08..79268f534 100644 --- a/grovedb/src/operations/auxiliary.rs +++ b/grovedb/src/operations/auxiliary.rs @@ -30,6 +30,8 @@ mod find_subtrees; +use std::collections::HashSet; + use grovedb_costs::{ cost_return_on_error, storage_cost::key_value_cost::KeyValueStorageCost, CostResult, CostsExt, OperationCost, @@ -158,6 +160,12 @@ impl GroveDb { /// and hashes for ordinary trees. Direct deletion keeps it enabled on /// every version because its legacy loop already included the sweep. /// + /// With `accounted_deletes`, the V4+ walk also checks the removal's + /// claim about its contents on the elements it decodes anyway: a + /// backward-reference participant not deleted explicitly by the caller + /// (a position in the set) refuses the removal. Earlier versions never + /// hold participants and keep their historical walk. + /// /// `context` names the calling operation in error messages. pub(crate) fn clear_subtree_storage_recursively<'db, B: AsRef<[u8]>>( &'db self, @@ -165,15 +173,36 @@ impl GroveDb { transaction: &'db Transaction, batch: &'db StorageBatch, sweep_secondary_namespaces: bool, + accounted_deletes: Option<&HashSet>>>, context: &str, grove_version: &GroveVersion, ) -> CostResult<(), Error> { let mut cost = OperationCost::default(); - let subtrees_paths = cost_return_on_error!( - &mut cost, - self.find_subtrees(path, Some(transaction), grove_version) - ); + let subtrees_paths = match accounted_deletes { + Some(accounted_deletes) + if grove_version + .grovedb_versions + .operations + .non_merk_tree + .subtree_discovery + >= 1 => + { + cost_return_on_error!( + &mut cost, + self.find_subtrees_refusing_participants_v1( + path, + Some(transaction), + grove_version, + accounted_deletes, + ) + ) + } + _ => cost_return_on_error!( + &mut cost, + self.find_subtrees(path, Some(transaction), grove_version) + ), + }; for subtree_path in subtrees_paths { let p: SubtreePath<_> = subtree_path.as_slice().into(); let mut storage = self diff --git a/grovedb/src/operations/auxiliary/find_subtrees/v1.rs b/grovedb/src/operations/auxiliary/find_subtrees/v1.rs index 4282dffd6..81211356c 100644 --- a/grovedb/src/operations/auxiliary/find_subtrees/v1.rs +++ b/grovedb/src/operations/auxiliary/find_subtrees/v1.rs @@ -3,11 +3,15 @@ //! Non-Merk descendants remain in the cleanup result but are never traversed: //! their data records are not Merk nodes and cannot contain child subtrees. -use grovedb_costs::{cost_return_on_error, CostResult, CostsExt, OperationCost}; +use grovedb_costs::{ + cost_return_on_error, cost_return_on_error_no_add, CostResult, CostsExt, OperationCost, +}; use grovedb_path::SubtreePath; use grovedb_storage::{Storage, StorageContext}; use grovedb_version::version::GroveVersion; +use std::collections::HashSet; + use crate::{ element::elements_iterator::ElementIteratorExtensions, util::TxRef, Element, Error, GroveDb, TransactionArg, @@ -19,6 +23,50 @@ impl GroveDb { path: &SubtreePath, transaction: TransactionArg, grove_version: &GroveVersion, + ) -> CostResult>>, Error> { + self.walk_subtrees_v1(path, transaction, grove_version, |_, _, _| Ok(())) + } + + /// `find_subtrees_v1` that also checks the claim a recursive removal + /// makes about its contents on the elements the walk decodes anyway: + /// a backward-reference participant not in `accounted_deletes` (the + /// positions the batch deletes explicitly) refuses the removal. + pub(crate) fn find_subtrees_refusing_participants_v1>( + &self, + path: &SubtreePath, + transaction: TransactionArg, + grove_version: &GroveVersion, + accounted_deletes: &HashSet>>, + ) -> CostResult>>, Error> { + self.walk_subtrees_v1( + path, + transaction, + grove_version, + |subtree_path, key, element| { + if !element.supports_backward_references() { + return Ok(()); + } + let mut qualified = subtree_path.to_vec(); + qualified.push(key.to_vec()); + if accounted_deletes.contains(&qualified) { + return Ok(()); + } + Err(Error::NotSupported( + "a recursive subtree removal reached a backward-reference participant it does \ + not explicitly delete; delete the participant first, or use live delete with \ + DisplacedValue::MayBeParticipant for recursive maintenance" + .to_owned(), + )) + }, + ) + } + + fn walk_subtrees_v1>( + &self, + path: &SubtreePath, + transaction: TransactionArg, + grove_version: &GroveVersion, + mut visit: impl FnMut(&[Vec], &[u8], &Element) -> Result<(), Error>, ) -> CostResult>>, Error> { let mut cost = OperationCost::default(); @@ -49,6 +97,7 @@ impl GroveDb { while let Some((key, value)) = cost_return_on_error!(&mut cost, raw_iter.next_element(grove_version)) { + cost_return_on_error_no_add!(cost, visit(&q, &key, &value)); if value.is_any_tree() { let mut sub_path = q.clone(); sub_path.push(key.to_vec()); diff --git a/grovedb/src/operations/bulk_append_tree.rs b/grovedb/src/operations/bulk_append_tree.rs index 876295f0b..c6191c263 100644 --- a/grovedb/src/operations/bulk_append_tree.rs +++ b/grovedb/src/operations/bulk_append_tree.rs @@ -665,6 +665,8 @@ impl GroveDb { chunk_power, }, }, + // The displaced value is the tree element itself, never a participant. + displaced_value: crate::DisplacedValue::NotParticipant, }; replacements.insert(tree_path.clone(), replacement); } diff --git a/grovedb/src/operations/commitment_tree.rs b/grovedb/src/operations/commitment_tree.rs index 8c93e657c..cb4b16b55 100644 --- a/grovedb/src/operations/commitment_tree.rs +++ b/grovedb/src/operations/commitment_tree.rs @@ -691,6 +691,8 @@ impl GroveDb { chunk_power, }, }, + // The displaced value is the tree element itself, never a participant. + displaced_value: crate::DisplacedValue::NotParticipant, }; replacements.insert(tree_path.clone(), replacement); } diff --git a/grovedb/src/operations/delete/clear_subtree/mod.rs b/grovedb/src/operations/delete/clear_subtree/mod.rs index d706f811a..80fcce1bb 100644 --- a/grovedb/src/operations/delete/clear_subtree/mod.rs +++ b/grovedb/src/operations/delete/clear_subtree/mod.rs @@ -31,8 +31,8 @@ //! through ancestors with the indexed-aware walk, refreshing the entry's //! canonical secondary row when the parent is an indexed primary. //! -//! v1 also refuses clears containing backward-reference participants under -//! the default Maintain policy. See [v0] / [v1]. +//! v1 also refuses clears containing backward-reference participants when +//! declared `MayBeParticipant`. See [v0] / [v1]. //! //! [v0]: self::v0 //! [v1]: self::v1 @@ -51,10 +51,11 @@ impl GroveDb { /// Delete all elements in a specified subtree. /// Returns if we successfully cleared the subtree. /// - /// On V4, the default `Maintain` policy scans for backward-reference - /// participants and refuses the clear before mutation if any are found. - /// Delete those participants through the normal delete API first, or - /// explicitly choose `ClearOptions::backward_references_policy = Skip`. + /// On V4, the default `MayBeParticipant` declaration scans for + /// backward-reference participants and refuses the clear before mutation + /// if any are found. Delete those participants through the normal delete + /// API first, or declare `ClearOptions::displaced_value = NotParticipant` + /// for a raw clear that trusts the claim. /// Ordinary `Reference` elements have no registrations and remain the /// caller's responsibility. V1–V3 retain their historical behavior. pub fn clear_subtree<'b, B, P>( diff --git a/grovedb/src/operations/delete/clear_subtree/v1.rs b/grovedb/src/operations/delete/clear_subtree/v1.rs index 5bdebb600..46e95be54 100644 --- a/grovedb/src/operations/delete/clear_subtree/v1.rs +++ b/grovedb/src/operations/delete/clear_subtree/v1.rs @@ -221,7 +221,7 @@ impl GroveDb { // Validate the entire removal before any child delete can modify the // caller's transaction. A raw clear cannot repair cross-subtree edges. - if options.backward_references_policy.maintains() + if options.displaced_value.may_be_participant() && !cost_return_on_error!( &mut cost, self.backward_reference_participants( @@ -233,7 +233,7 @@ impl GroveDb { .is_empty() { return Err(Error::NotSupported( - "clear_subtree cannot remove backward-reference participants under Maintain; delete the participants normally first or explicitly select Skip".to_owned(), + "clear_subtree cannot remove backward-reference participants; delete them normally first or declare DisplacedValue::NotParticipant for a raw clear".to_owned(), )).wrap_with_cost(cost); } @@ -267,7 +267,7 @@ impl GroveDb { Some(DeleteOptions { allow_deleting_non_empty_trees: true, deleting_non_empty_trees_returns_error: false, - backward_references_policy: options.backward_references_policy, + displaced_value: options.displaced_value, ..Default::default() }), Some(tx.as_ref()), diff --git a/grovedb/src/operations/delete/delete_internal_on_transaction/v0.rs b/grovedb/src/operations/delete/delete_internal_on_transaction/v0.rs index 81858de8d..f5c5326df 100644 --- a/grovedb/src/operations/delete/delete_internal_on_transaction/v0.rs +++ b/grovedb/src/operations/delete/delete_internal_on_transaction/v0.rs @@ -233,6 +233,7 @@ impl GroveDb { transaction, batch, true, + None, "delete", grove_version, ) diff --git a/grovedb/src/operations/delete/delete_internal_on_transaction/v1.rs b/grovedb/src/operations/delete/delete_internal_on_transaction/v1.rs index 92e09b1cd..a749c353c 100644 --- a/grovedb/src/operations/delete/delete_internal_on_transaction/v1.rs +++ b/grovedb/src/operations/delete/delete_internal_on_transaction/v1.rs @@ -265,6 +265,13 @@ impl GroveDb { transaction, batch, true, + // A live recursive removal trusts its declaration about the + // contents: `clear_subtree` runs one such delete per nested + // subtree inside the caller's transaction, where a refusal + // part-way through could not be undone. Batches, whose + // refusal discards the whole staged batch, check the claim + // on this same walk. + None, "delete", grove_version, ) diff --git a/grovedb/src/operations/delete/delete_internal_on_transaction/v2.rs b/grovedb/src/operations/delete/delete_internal_on_transaction/v2.rs index a8edeaf80..4272a0b2b 100644 --- a/grovedb/src/operations/delete/delete_internal_on_transaction/v2.rs +++ b/grovedb/src/operations/delete/delete_internal_on_transaction/v2.rs @@ -1,7 +1,7 @@ //! Automatic deletion with cached old-value observation on GROVE_V4. use crate::operations::indexed_tree::reject_generic_write_into_indexed_primary; -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use grovedb_costs::{ cost_return_on_error, cost_return_on_error_no_add, storage_cost::removal::StorageRemovedBytes, CostResult, CostsExt, @@ -45,27 +45,15 @@ impl GroveDb { batch: &StorageBatch, grove_version: &GroveVersion, ) -> CostResult { - if options.backward_references_policy.maintains() { - self.delete_with_backward_references_v2( - path, - key, - options, - transaction, - sectioned_removal, - batch, - grove_version, - ) - } else { - self.delete_internal_on_transaction_v1( - path, - key, - options, - transaction, - sectioned_removal, - batch, - grove_version, - ) - } + self.delete_with_backward_references_v2( + path, + key, + options, + transaction, + sectioned_removal, + batch, + grove_version, + ) } /// The `MerkCache`-based delete flow with backward-references cascade. @@ -119,7 +107,22 @@ impl GroveDb { let element = cost_return_on_error_no_add!(cost, observed.transpose().map_err(Error::from) .and_then(|value| value.ok_or_else(|| Error::PathKeyNotFound(hex::encode(key))))); - let descendants_need_maintenance = if element.is_any_tree() + // The displaced value is in hand, so a `NotParticipant` claim costs + // nothing to check and fails closed. + if !options.displaced_value.may_be_participant() && element.supports_backward_references() { + return Err(Error::NotSupported( + "delete declared DisplacedValue::NotParticipant but the stored value takes part in \ + backward references" + .to_owned(), + )) + .wrap_with_cost(cost); + } + // A populated subtree is scanned for participants to maintain only + // when the caller says there may be some; a `NotParticipant` removal + // takes the ordinary route, whose cleanup walk checks the claim on + // the elements it decodes anyway. + let descendants_need_maintenance = if options.displaced_value.may_be_participant() + && element.is_any_tree() && !element.uses_non_merk_data_storage() && element .root_key_and_tree_type() @@ -212,12 +215,7 @@ impl GroveDb { let visitor = GroveVisitor::new( &self.db, transaction, - DeletionVisitor::new( - &cache, - options.backward_references_policy, - true, - sectioned_removal, - ), + DeletionVisitor::new(&cache, options.displaced_value, true, sectioned_removal), true, grove_version, ); @@ -321,7 +319,7 @@ impl GroveDb { /// we're good as long as we do nothing outside of the cache, then finalize /// it, and only then merge with the final deletion batches. struct DeletionVisitor<'c, 'db, 'b, 's, B: AsRef<[u8]>> { - backward_references_policy: BackwardReferencesPolicy, + displaced_value: DisplacedValue, allow_deleting_subtrees: bool, cache: &'c MerkCache<'db, 'b, B>, /// The caller's removal-accounting policy, applied to every referrer a @@ -332,12 +330,12 @@ struct DeletionVisitor<'c, 'db, 'b, 's, B: AsRef<[u8]>> { impl<'c, 'db, 'b, 's, B: AsRef<[u8]>> DeletionVisitor<'c, 'db, 'b, 's, B> { fn new( cache: &'c MerkCache<'db, 'b, B>, - backward_references_policy: BackwardReferencesPolicy, + displaced_value: DisplacedValue, allow_deleting_subtrees: bool, sectioned_removal: bidirectional_references::SectionedRemovalFn<'s>, ) -> Self { Self { - backward_references_policy, + displaced_value, allow_deleting_subtrees, cache, sectioned_removal, @@ -393,7 +391,7 @@ impl<'b, B: AsRef<[u8]>> Visit<'b, B> for DeletionVisitor<'_, '_, 'b, '_, B> { // Step 2: perform backward references' deletion on top of cached // data: - if self.backward_references_policy.maintains() + if self.displaced_value.may_be_participant() && matches!( element, Element::ItemWithBackwardsReferences(..) diff --git a/grovedb/src/operations/delete/delete_up_tree.rs b/grovedb/src/operations/delete/delete_up_tree.rs index 4aa5b31cf..51bc9f845 100644 --- a/grovedb/src/operations/delete/delete_up_tree.rs +++ b/grovedb/src/operations/delete/delete_up_tree.rs @@ -1,6 +1,6 @@ //! Delete up tree -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use grovedb_costs::{ cost_return_on_error, cost_return_on_error_no_add, storage_cost::removal::{StorageRemovedBytes, StorageRemovedBytes::BasicStorageRemoval}, @@ -52,7 +52,7 @@ impl DeleteUpTreeOptions { deleting_non_empty_trees_returns_error: self.deleting_non_empty_trees_returns_error, base_root_storage_is_free: self.base_root_storage_is_free, validate_tree_at_path_exists: self.validate_tree_at_path_exists, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, } } } diff --git a/grovedb/src/operations/delete/flat_drop.rs b/grovedb/src/operations/delete/flat_drop.rs index ef68a2340..4928cafbb 100644 --- a/grovedb/src/operations/delete/flat_drop.rs +++ b/grovedb/src/operations/delete/flat_drop.rs @@ -11,8 +11,9 @@ //! its key, the parent Merk's shape, and the path. The subtree's contents //! are **never opened, checked, or metered**, which is what makes the cost //! O(1) in the subtree's size: dropping a tree of ten million entries -//! costs the same as dropping a tree of ten. Both entry points require explicit -//! `BackwardReferencesPolicy::Skip`; Maintain is refused before reading contents. +//! costs the same as dropping a tree of ten. Both entry points require +//! `DisplacedValue::NotParticipant`; `MayBeParticipant` is refused before +//! reading anything. //! //! Atomically with the delete, a durable redo record //! ([`PendingPrefixDropRecord`]) is committed into a reserved namespace of @@ -70,7 +71,7 @@ //! lifecycle is the caller's responsibility. use crate::operations::indexed_tree::reject_generic_write_into_indexed_primary; -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use std::collections::HashMap; use grovedb_costs::{ @@ -209,7 +210,7 @@ impl GroveDb { /// or sweeping its contents, and stage its storage prefixes for /// reclamation. See the [module documentation](self) for the full /// contract: the caller declares the subtree contains **no child - /// subtrees** and explicitly selects `BackwardReferencesPolicy::Skip`. + /// subtrees** and explicitly selects `DisplacedValue::NotParticipant`. /// Incoming references may dangle, and the dropped path must /// not be re-created before its record drains. /// @@ -221,7 +222,7 @@ impl GroveDb { &self, path: P, key: &[u8], - backward_references_policy: BackwardReferencesPolicy, + displaced_value: DisplacedValue, transaction: TransactionArg, grove_version: &GroveVersion, ) -> CostResult<(), Error> @@ -244,9 +245,9 @@ impl GroveDb { ) ); - if backward_references_policy.maintains() { + if displaced_value.may_be_participant() { return Err(Error::NotSupported( - "flat drop requires explicit BackwardReferencesPolicy::Skip; use recursive delete for maintenance".to_owned(), + "flat drop requires DisplacedValue::NotParticipant; use recursive delete for maintenance".to_owned(), )).wrap_with_cost(cost); } diff --git a/grovedb/src/operations/delete/mod.rs b/grovedb/src/operations/delete/mod.rs index f54d3b537..03c70abd4 100644 --- a/grovedb/src/operations/delete/mod.rs +++ b/grovedb/src/operations/delete/mod.rs @@ -15,7 +15,7 @@ //! //! The exception is the bidirectional-references machinery (`GROVE_V4`+): //! deleting with -//! the default [`crate::BackwardReferencesPolicy::Maintain`] cascades any +//! the default [`crate::DisplacedValue::MayBeParticipant`] cascades any //! [`BidirectionalReference`](crate::Element::BidirectionalReference) //! chains that point at the deleted element (each affected reference must //! allow `cascade_on_update`, otherwise the delete errors instead). See @@ -42,7 +42,7 @@ pub mod flat_drop; mod worst_case; #[cfg(feature = "minimal")] -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use std::collections::BTreeSet; #[cfg(feature = "minimal")] @@ -83,10 +83,11 @@ pub struct ClearOptions { /// If we check for subtrees, and we don't allow deleting and there are /// some, should we error? pub trying_to_clear_with_subtrees_returns_error: bool, - /// On V4, Maintain scans for participants and refuses the clear before - /// mutation if any are found. Delete those participants through the normal - /// delete API first. Skip explicitly permits dangling registrations. - pub backward_references_policy: BackwardReferencesPolicy, + /// On V4, `MayBeParticipant` scans for participants and refuses the clear + /// before mutation if any are found; delete those participants through the + /// normal delete API first. `NotParticipant` is trusted: a raw clear reads + /// nothing, so a false claim strands the participants' registrations. + pub displaced_value: DisplacedValue, } #[cfg(feature = "minimal")] @@ -96,7 +97,7 @@ impl Default for ClearOptions { check_for_subtrees: true, allow_deleting_subtrees: false, trying_to_clear_with_subtrees_returns_error: true, - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, } } } @@ -113,11 +114,24 @@ pub struct DeleteOptions { pub base_root_storage_is_free: bool, /// Validate tree at path exists pub validate_tree_at_path_exists: bool, - /// Maintain backward references by default on V4. Every cascaded reference - /// must consent through `cascade_on_update`, or the operation fails - /// atomically. The initial value is observed in the Merk used for deletion. - /// `Skip` deliberately allows references to the deleted position to dangle. - pub backward_references_policy: BackwardReferencesPolicy, + /// What the delete declares about the value it removes. The stored value + /// is observed in the Merk used for the deletion: under + /// `MayBeParticipant` a participant cascades with each referrer's + /// consent (or the delete fails atomically) and a populated subtree is + /// scanned so its participants are maintained; under `NotParticipant` a + /// participant is refused, and a recursive removal trusts the claim for + /// its contents (a batch removal checks it on the cleanup walk it makes + /// anyway). + pub displaced_value: DisplacedValue, +} + +#[cfg(feature = "minimal")] +impl DeleteOptions { + /// Replace the declaration about the displaced value. + pub fn with_displaced_value(mut self, displaced_value: DisplacedValue) -> Self { + self.displaced_value = displaced_value; + self + } } #[cfg(feature = "minimal")] @@ -128,7 +142,7 @@ impl Default for DeleteOptions { deleting_non_empty_trees_returns_error: true, base_root_storage_is_free: true, validate_tree_at_path_exists: false, - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, } } } @@ -148,7 +162,7 @@ impl GroveDb { /// /// # Dangling references /// - /// With [`crate::BackwardReferencesPolicy::Skip`], this + /// With [`crate::DisplacedValue::NotParticipant`], this /// operation does **not** check for incoming references. If other /// elements hold [`Reference`](crate::Element::Reference) paths that point /// to the deleted element, those references become dangling. Following a @@ -220,10 +234,11 @@ impl GroveDb { /// Ordinary [`Reference`](crate::Element::Reference) elements are not /// tracked: any that point to the deleted element become dangling, and /// callers must manage their lifecycle. Under the default - /// [`Maintain`](crate::BackwardReferencesPolicy::Maintain) policy - /// (`GROVE_V4`+), bidirectional references pointing at the deleted element - /// are cascaded or the delete is refused, as described on [`Self::delete`]; - /// explicit `Skip` leaves them dangling too. See the + /// [`MayBeParticipant`](crate::DisplacedValue::MayBeParticipant) + /// declaration (`GROVE_V4`+), bidirectional references pointing at the + /// deleted element are cascaded or the delete is refused, as described on + /// [`Self::delete`]; a `NotParticipant` delete of a participant is refused + /// outright. See the /// [module-level documentation](self) for details. pub fn delete_with_sectional_storage_function>( &self, @@ -310,10 +325,11 @@ impl GroveDb { /// Ordinary [`Reference`](crate::Element::Reference) elements are not /// tracked: any that point to the deleted tree become dangling, and /// callers must manage their lifecycle. Under the default - /// [`Maintain`](crate::BackwardReferencesPolicy::Maintain) policy - /// (`GROVE_V4`+), bidirectional references pointing at the deleted tree - /// are cascaded or the delete is refused, as described on [`Self::delete`]; - /// explicit `Skip` leaves them dangling too. See the + /// [`MayBeParticipant`](crate::DisplacedValue::MayBeParticipant) + /// declaration (`GROVE_V4`+), bidirectional references pointing at the + /// deleted tree are cascaded or the delete is refused, as described on + /// [`Self::delete`]; a `NotParticipant` delete of a participant is refused + /// outright. See the /// [module-level documentation](self) for details. pub fn delete_if_empty_tree<'b, B, P>( &self, @@ -438,9 +454,8 @@ impl GroveDb { /// This builds a batch operation; it performs no reference check itself. /// Ordinary [`Reference`](crate::Element::Reference) elements pointing at /// the deleted element become dangling when the batch applies. Whether - /// bidirectional references are cascaded is decided by the - /// [`BackwardReferencesPolicy`] of the - /// batch that applies the operation. See the + /// bidirectional references are cascaded is decided by the operation's + /// own [`DisplacedValue`] declaration. See the /// [module-level documentation](self) for details. pub fn delete_operation_for_delete_internal>( &self, @@ -600,7 +615,7 @@ impl GroveDb { #[cfg(feature = "minimal")] #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_costs::{ storage_cost::{removal::StorageRemovedBytes::BasicStorageRemoval, StorageCost}, OperationCost, @@ -1990,7 +2005,7 @@ mod tests { .clear_subtree( [TEST_LEAF, b"key1"].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: false, trying_to_clear_with_subtrees_returns_error: false, @@ -2005,7 +2020,7 @@ mod tests { .clear_subtree( [TEST_LEAF, b"key1"].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: true, trying_to_clear_with_subtrees_returns_error: false, @@ -2158,7 +2173,7 @@ mod tests { deleting_non_empty_trees_returns_error: true, base_root_storage_is_free: true, validate_tree_at_path_exists: true, - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, }), None, version, @@ -2197,7 +2212,7 @@ mod tests { deleting_non_empty_trees_returns_error: false, base_root_storage_is_free: true, validate_tree_at_path_exists: true, - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, }), Some(&transaction), version, diff --git a/grovedb/src/operations/dense_tree.rs b/grovedb/src/operations/dense_tree.rs index 3f76927bf..1807d6e4f 100644 --- a/grovedb/src/operations/dense_tree.rs +++ b/grovedb/src/operations/dense_tree.rs @@ -485,6 +485,8 @@ impl GroveDb { height, }, }, + // The displaced value is the tree element itself, never a participant. + displaced_value: crate::DisplacedValue::NotParticipant, }; replacements.insert(tree_path.clone(), replacement); } diff --git a/grovedb/src/operations/indexed_tree.rs b/grovedb/src/operations/indexed_tree.rs index f91ddc009..a01379ad9 100644 --- a/grovedb/src/operations/indexed_tree.rs +++ b/grovedb/src/operations/indexed_tree.rs @@ -514,6 +514,7 @@ impl GroveDb { transaction, batch, true, + None, "dedicated indexed-tree overwrite/delete", grove_version, ) diff --git a/grovedb/src/operations/insert/add_element_on_transaction/v1.rs b/grovedb/src/operations/insert/add_element_on_transaction/v1.rs index 12c1fa2c8..7f9010bec 100644 --- a/grovedb/src/operations/insert/add_element_on_transaction/v1.rs +++ b/grovedb/src/operations/insert/add_element_on_transaction/v1.rs @@ -277,7 +277,7 @@ impl GroveDb { // The backward-references item variants store exactly like // their plain counterparts; the backward-reference // bookkeeping only runs when the caller opts in via - // `backward_references_policy` (routed before this call). + // `displaced_value` (routed before this call). // // DELIBERATE TRADEOFF (see adr/bidirectional_references.md): // without the flag, overwriting a key that carries backward diff --git a/grovedb/src/operations/insert/add_element_on_transaction/v2.rs b/grovedb/src/operations/insert/add_element_on_transaction/v2.rs index 9513ba1c3..622eb7669 100644 --- a/grovedb/src/operations/insert/add_element_on_transaction/v2.rs +++ b/grovedb/src/operations/insert/add_element_on_transaction/v2.rs @@ -359,7 +359,7 @@ impl GroveDb { // The backward-references item variants store exactly like // their plain counterparts; the backward-reference // bookkeeping only runs when the caller opts in via - // `backward_references_policy` (routed before this call). + // `displaced_value` (routed before this call). // // DELIBERATE TRADEOFF (see adr/bidirectional_references.md): // without the flag, overwriting a key that carries backward diff --git a/grovedb/src/operations/insert/insert_on_transaction/v1.rs b/grovedb/src/operations/insert/insert_on_transaction/v1.rs index d20065f97..089ec9c9f 100644 --- a/grovedb/src/operations/insert/insert_on_transaction/v1.rs +++ b/grovedb/src/operations/insert/insert_on_transaction/v1.rs @@ -40,21 +40,6 @@ pub(super) fn insert_on_transaction<'db, 'b, B: AsRef<[u8]>>( )) .wrap_with_cost(Default::default()); } - // A new bidirectional reference must register its target even under Skip. - if !options.backward_references_policy.maintains() - && !matches!(element, Element::BidirectionalReference(..)) - { - return super::v0::insert_on_transaction_body( - db, - path, - key, - element, - options, - transaction, - batch, - grove_version, - ); - } let mut cost = Default::default(); let mut merk = cost_return_on_error!( &mut cost, @@ -101,21 +86,40 @@ pub(super) fn insert_on_transaction<'db, 'b, B: AsRef<[u8]>>( )) .wrap_with_cost(cost); } - if previous.as_ref().is_some_and(|old| { - old.is_any_tree() - && !old.uses_non_merk_data_storage() - && old - .root_key_and_tree_type() - .is_some_and(|(root, _)| root.is_some()) - }) && !cost_return_on_error!( - &mut cost, - db.backward_reference_participants( - &path.derive_owned_with_child(key).to_vec(), - transaction, - grove_version, + // The displaced value is in hand, so a `NotParticipant` claim costs + // nothing to check and fails closed. + if !options.displaced_value.may_be_participant() + && previous + .as_ref() + .is_some_and(Element::supports_backward_references) + { + return Err(Error::NotSupported( + "insert declared DisplacedValue::NotParticipant but the stored value takes part in \ + backward references" + .to_owned(), + )) + .wrap_with_cost(cost); + } + // Replacing a populated subtree reads none of its contents: scan it for + // participants when the caller says there may be some, trust the + // caller otherwise. + if options.displaced_value.may_be_participant() + && previous.as_ref().is_some_and(|old| { + old.is_any_tree() + && !old.uses_non_merk_data_storage() + && old + .root_key_and_tree_type() + .is_some_and(|(root, _)| root.is_some()) + }) + && !cost_return_on_error!( + &mut cost, + db.backward_reference_participants( + &path.derive_owned_with_child(key).to_vec(), + transaction, + grove_version, + ) ) - ) - .is_empty() + .is_empty() { return Err(Error::NotSupported( "delete a subtree containing backward-reference participants before replacing it" diff --git a/grovedb/src/operations/insert/mod.rs b/grovedb/src/operations/insert/mod.rs index 23a336796..562f7fcb4 100644 --- a/grovedb/src/operations/insert/mod.rs +++ b/grovedb/src/operations/insert/mod.rs @@ -1,6 +1,6 @@ //! Insert operations -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use std::option::Option::None; use grovedb_costs::{cost_return_on_error, CostResult, CostsExt, OperationCost}; @@ -25,11 +25,21 @@ pub struct InsertOptions { pub validate_insertion_does_not_override_tree: bool, /// Base root storage is free pub base_root_storage_is_free: bool, - /// Maintain backward references by default on V4. The stored value is - /// observed in the same Merk used by the write. `Skip` explicitly permits - /// stale reference hashes and dangling registrations. A newly inserted + /// What the write declares about the value it displaces. The stored + /// value is observed in the same Merk used for the write, so on V4 a + /// participant is maintained under `MayBeParticipant` and refused under + /// `NotParticipant`. Replacing a populated subtree is scanned for + /// participants only under `MayBeParticipant`. A newly inserted /// bidirectional reference always registers its edge. - pub backward_references_policy: BackwardReferencesPolicy, + pub displaced_value: DisplacedValue, +} + +impl InsertOptions { + /// Replace the declaration about the displaced value. + pub fn with_displaced_value(mut self, displaced_value: DisplacedValue) -> Self { + self.displaced_value = displaced_value; + self + } } impl Default for InsertOptions { @@ -38,7 +48,7 @@ impl Default for InsertOptions { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: true, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, } } } @@ -279,7 +289,7 @@ impl GroveDb { #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_costs::{ storage_cost::{removal::StorageRemovedBytes::NoStorageRemoval, StorageCost}, OperationCost, @@ -473,7 +483,7 @@ mod tests { validate_insertion_does_not_override: true, validate_insertion_does_not_override_tree: true, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), None, gv, @@ -551,7 +561,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, } } @@ -3312,7 +3322,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), Some(&tx), grove_version, @@ -3602,7 +3612,7 @@ mod tests { b"key5", Element::new_item_allowing_bidirectional_references(b"certainly new value".to_vec()), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -3632,7 +3642,7 @@ mod tests { b"key5", Element::new_item(b"hello".to_vec()), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, diff --git a/grovedb/src/operations/mmr_tree.rs b/grovedb/src/operations/mmr_tree.rs index 7c3b90c4e..98d17cd27 100644 --- a/grovedb/src/operations/mmr_tree.rs +++ b/grovedb/src/operations/mmr_tree.rs @@ -525,6 +525,8 @@ impl GroveDb { mmr_size: new_mmr_size, }, }, + // The displaced value is the tree element itself, never a participant. + displaced_value: crate::DisplacedValue::NotParticipant, }; replacements.insert(tree_path.clone(), replacement); } diff --git a/grovedb/src/operations/private_document_store.rs b/grovedb/src/operations/private_document_store.rs index 4b07a3b94..4f33bc512 100644 --- a/grovedb/src/operations/private_document_store.rs +++ b/grovedb/src/operations/private_document_store.rs @@ -578,6 +578,8 @@ impl GroveDb { chunk_power, }, }, + // The displaced value is the tree element itself, never a participant. + displaced_value: crate::DisplacedValue::NotParticipant, }; replacements.insert(tree_path.clone(), replacement); } diff --git a/grovedb/src/tests/append_family_cost_bound_tests.rs b/grovedb/src/tests/append_family_cost_bound_tests.rs index b17096470..41e7e4996 100644 --- a/grovedb/src/tests/append_family_cost_bound_tests.rs +++ b/grovedb/src/tests/append_family_cost_bound_tests.rs @@ -64,7 +64,6 @@ fn average_case_estimate( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -73,7 +72,6 @@ fn average_case_estimate( KeyInfoPath::from_known_owned_path(vec![key.to_vec()]), EstimatedLayerInformation { tree_type, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(16, false), estimated_layer_sizes: AllItems(8, value_size, None), }, diff --git a/grovedb/src/tests/append_storage_accounting_tests.rs b/grovedb/src/tests/append_storage_accounting_tests.rs index 5327125a5..0057bf2d0 100644 --- a/grovedb/src/tests/append_storage_accounting_tests.rs +++ b/grovedb/src/tests/append_storage_accounting_tests.rs @@ -604,7 +604,6 @@ fn bulk_average_case_estimate( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -614,7 +613,6 @@ fn bulk_average_case_estimate( KeyInfoPath::from_known_owned_path(vec![b"bulk".to_vec()]), EstimatedLayerInformation { tree_type: TreeType::BulkAppendTree(chunk_power), - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(16, false), estimated_layer_sizes: AllItems(8, value_size, None), }, diff --git a/grovedb/src/tests/automatic_backward_references_tests.rs b/grovedb/src/tests/automatic_backward_references_tests.rs index bb32e91a8..27f09dfc9 100644 --- a/grovedb/src/tests/automatic_backward_references_tests.rs +++ b/grovedb/src/tests/automatic_backward_references_tests.rs @@ -9,7 +9,7 @@ use crate::{ operations::{delete::DeleteOptions, insert::InsertOptions}, reference_path::ReferencePathType, tests::{make_test_grovedb, TempGroveDb, TEST_LEAF}, - BackwardReferencesPolicy, BidirectionalReference, Element, Error, + BidirectionalReference, DisplacedValue, Element, Error, }; fn reference(target: &[u8], cascade: bool) -> Element { @@ -24,6 +24,12 @@ fn reference(target: &[u8], cascade: bool) -> Element { ) } +fn not_participant(ops: Vec) -> Vec { + ops.into_iter() + .map(|op| op.with_displaced_value(DisplacedValue::NotParticipant)) + .collect() +} + fn chain(cascade: bool) -> TempGroveDb { let version = GroveVersion::latest(); let db = make_test_grovedb(version); @@ -133,60 +139,64 @@ fn default_live_and_batch_updates_preserve_registrations_and_refresh_hashes() { } #[test] -fn default_plain_overwrite_cascades_and_explicit_skip_keeps_dangling_references() { +fn default_plain_overwrite_cascades_and_a_false_not_participant_claim_is_refused() { let version = GroveVersion::latest(); for batch in [false, true] { - for skip in [false, true] { + for declared in [ + DisplacedValue::MayBeParticipant, + DisplacedValue::NotParticipant, + ] { let db = chain(true); - let policy = if skip { - BackwardReferencesPolicy::Skip - } else { - BackwardReferencesPolicy::Maintain - }; + let before = db.root_hash(None, version).unwrap().unwrap(); let item = Element::new_item(b"plain".to_vec()); - if batch { + let result = if batch { db.apply_batch( vec![QualifiedGroveDbOp::insert_or_replace_op( vec![TEST_LEAF.to_vec()], b"value".to_vec(), item.clone(), - )], - Some(BatchApplyOptions { - backward_references_policy: policy, - ..Default::default() - }), + ) + .with_displaced_value(declared)], + None, None, version, ) .unwrap() - .unwrap(); } else { db.insert( &[TEST_LEAF], b"value", item.clone(), Some(InsertOptions { - backward_references_policy: policy, + displaced_value: declared, ..Default::default() }), None, version, ) .unwrap() - .unwrap(); + }; + let refused = !declared.may_be_participant(); + if refused { + // The stored value is read for the write anyway, so the + // false claim is caught for free and nothing changes. + assert!(matches!(result, Err(Error::NotSupported(_)))); + assert_eq!(db.root_hash(None, version).unwrap().unwrap(), before); + } else { + result.unwrap(); + assert_eq!( + db.get(&[TEST_LEAF], b"value", None, version) + .unwrap() + .unwrap(), + item + ); } - assert_eq!( - db.get(&[TEST_LEAF], b"value", None, version) - .unwrap() - .unwrap(), - item - ); for key in [b"r1".as_slice(), b"r2"] { assert_eq!( db.get_raw(SubtreePath::from(&[TEST_LEAF]), key, None, version) .unwrap() .is_ok(), - skip + refused ); } } @@ -345,15 +355,7 @@ fn ordinary_mutations_reuse_preparation_reads() { .cost_as_result() .unwrap(); let skipped_cost = skipped - .apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - version, - ) + .apply_batch(not_participant(ops), None, None, version) .cost_as_result() .unwrap(); assert_eq!(observed_cost, skipped_cost); @@ -365,21 +367,22 @@ fn ordinary_mutations_reuse_preparation_reads() { } #[test] -fn skip_delete_is_an_explicit_opt_out() { +fn not_participant_delete_of_a_participant_is_refused() { let version = GroveVersion::latest(); let db = chain(true); - db.delete( + let before = db.root_hash(None, version).unwrap().unwrap(); + let result = db.delete( &[TEST_LEAF], b"value", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, ..Default::default() }), None, version, - ) - .unwrap() - .unwrap(); + ); + assert!(matches!(result.unwrap(), Err(Error::NotSupported(_)))); + assert_eq!(db.root_hash(None, version).unwrap().unwrap(), before); assert!(db .get_raw(SubtreePath::from(&[TEST_LEAF]), b"r1", None, version) .unwrap() @@ -389,6 +392,7 @@ fn skip_delete_is_an_explicit_opt_out() { #[test] fn recursive_removal_cannot_bypass_descendant_maintenance() { use crate::batch::SubelementsDeletionBehavior; + use grovedb_merk::TreeType; let version = GroveVersion::latest(); let db = make_test_grovedb(version); @@ -636,7 +640,7 @@ fn clear_subtree_refuses_incoming_and_outgoing_edges_before_mutating_the_transac &[TEST_LEAF], Some(ClearOptions { allow_deleting_subtrees: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, ..Default::default() }), Some(&tx), @@ -650,7 +654,7 @@ fn clear_subtree_refuses_incoming_and_outgoing_edges_before_mutating_the_transac .is_err()); } else { // Public get_raw strips the registered referrers; inspect the - // stored element to verify the explicit Skip contract. + // stored element to verify the trusted NotParticipant contract. let merk = db .open_transactional_merk_at_path( SubtreePath::from(root.as_slice()), @@ -666,7 +670,7 @@ fn clear_subtree_refuses_incoming_and_outgoing_edges_before_mutating_the_transac assert_eq!( target.backward_references().unwrap().len(), 1, - "Skip leaves the registration deliberately stale" + "a raw clear declared NotParticipant leaves the registration stale" ); } } @@ -674,7 +678,7 @@ fn clear_subtree_refuses_incoming_and_outgoing_edges_before_mutating_the_transac } #[test] -fn flat_drop_requires_explicit_skip_in_live_full_and_both_partial_segments() { +fn flat_drop_requires_not_participant_in_live_full_and_both_partial_segments() { use crate::batch::SubelementsDeletionBehavior; use grovedb_merk::TreeType; let version = GroveVersion::latest(); @@ -692,7 +696,7 @@ fn flat_drop_requires_explicit_skip_in_live_full_and_both_partial_segments() { 0 => db.drop_flat_subtree( &[] as &[&[u8]], TEST_LEAF, - BackwardReferencesPolicy::Maintain, + DisplacedValue::MayBeParticipant, Some(&tx), version, ), @@ -737,9 +741,9 @@ fn ordinary_batch_conditionals_and_duplicate_positions_keep_executor_semantics() let version = GroveVersion::latest(); for scenario in 0..4 { let mut roots = Vec::new(); - for policy in [ - BackwardReferencesPolicy::Maintain, - BackwardReferencesPolicy::Skip, + for declared in [ + DisplacedValue::MayBeParticipant, + DisplacedValue::NotParticipant, ] { let db = make_test_grovedb(version); db.insert( @@ -793,10 +797,11 @@ fn ordinary_batch_conditionals_and_duplicate_positions_keep_executor_semantics() ], }; db.apply_batch( - ops, + ops.into_iter() + .map(|op| op.with_displaced_value(declared)) + .collect(), Some(BatchApplyOptions { disable_operation_consistency_check: true, - backward_references_policy: policy, ..Default::default() }), None, @@ -858,3 +863,528 @@ fn partial_subtree_refusal_is_atomic_in_the_continuation_and_caller_transaction( .is_empty()); } } + +/// Delete-up-tree shape: the batch deletes a leaf, then removes the emptied +/// trees above it with `DontCheckWithNoCleanup`. Every element the batch +/// removes is read as an old value on the way, so `Maintain` has nothing +/// left to scan and adds no reads over `Skip`, in full and partial batches +/// alike. `DeleteChildren` keeps its scan: GroveDB cannot see what that +/// removal takes with it. +#[test] +fn declared_empty_subtree_removals_skip_the_participant_scan() { + use crate::batch::SubelementsDeletionBehavior; + use grovedb_merk::TreeType; + let version = GroveVersion::latest(); + let seed = |db: &TempGroveDb| { + db.insert( + &[TEST_LEAF], + b"a", + Element::empty_tree(), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"a"], + b"b", + Element::empty_tree(), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"a", b"b"], + b"c", + Element::new_item(vec![1]), + None, + None, + version, + ) + .unwrap() + .unwrap(); + }; + // Every declared-empty or apply-time-checked removal behaves the same: + // `DontCheckWithNoCleanup` trusts the batch's own deletes, `Error` and + // `Skip` verify emptiness against them at apply time. + for behavior in [ + SubelementsDeletionBehavior::DontCheckWithNoCleanup, + SubelementsDeletionBehavior::Error, + SubelementsDeletionBehavior::Skip, + ] { + let delete_up_tree = [ + QualifiedGroveDbOp::delete_op( + vec![TEST_LEAF.to_vec(), b"a".to_vec(), b"b".to_vec()], + b"c".to_vec(), + ), + QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec(), b"a".to_vec()], + b"b".to_vec(), + TreeType::NormalTree, + behavior, + ), + QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec()], + b"a".to_vec(), + TreeType::NormalTree, + behavior, + ), + ]; + for partial in [false, true] { + let mut runs = Vec::new(); + for declared in [ + DisplacedValue::MayBeParticipant, + DisplacedValue::NotParticipant, + ] { + let db = make_test_grovedb(version); + seed(&db); + let ops: Vec = delete_up_tree + .iter() + .cloned() + .map(|op| op.with_displaced_value(declared)) + .collect(); + let result = if partial { + db.apply_partial_batch(ops, None, |_, _| Ok(vec![]), None, version) + } else { + db.apply_batch(ops, None, None, version) + }; + result.value.expect("delete up the tree"); + // `Skip` checks emptiness before the same-batch deletes land + // and silently keeps the populated chain; the other two + // remove it. + assert_eq!( + db.get_raw(SubtreePath::from(&[TEST_LEAF]), b"a", None, version) + .unwrap() + .is_err(), + !matches!(behavior, SubelementsDeletionBehavior::Skip), + "{behavior:?} partial={partial}" + ); + runs.push((result.cost, db.root_hash(None, version).unwrap().unwrap())); + } + let (may_be, not) = (&runs[0], &runs[1]); + assert_eq!(may_be.1, not.1, "{behavior:?} partial={partial}: root hash"); + assert_eq!( + may_be.0, not.0, + "{behavior:?} partial={partial}: MayBeParticipant must cost exactly what \ + NotParticipant costs" + ); + } + } + + let recursive = [QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec()], + b"a".to_vec(), + TreeType::NormalTree, + SubelementsDeletionBehavior::DeleteChildren, + )]; + let mut costs = Vec::new(); + for declared in [ + DisplacedValue::MayBeParticipant, + DisplacedValue::NotParticipant, + ] { + let db = make_test_grovedb(version); + seed(&db); + let ops: Vec = recursive + .iter() + .cloned() + .map(|op| op.with_displaced_value(declared)) + .collect(); + let result = db.apply_batch(ops, None, None, version); + result.value.expect("recursive delete"); + costs.push(result.cost); + } + assert_eq!( + costs[0], costs[1], + "DeleteChildren checks its contents on the cleanup walk it makes anyway" + ); +} + +/// A declared-empty removal relies on the batch's own deletes for +/// maintenance: the explicitly deleted participant cascades its referrer in +/// a full batch, and a partial batch still refuses the participant mutation +/// before anything is committed. +#[test] +fn declared_empty_subtree_removal_still_maintains_the_deleted_participant() { + use crate::batch::SubelementsDeletionBehavior; + use grovedb_merk::TreeType; + let version = GroveVersion::latest(); + let seed = |db: &TempGroveDb| { + db.insert( + &[TEST_LEAF], + b"tree", + Element::empty_tree(), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"tree"], + b"value", + Element::new_item_allowing_bidirectional_references(vec![1]), + None, + None, + version, + ) + .unwrap() + .unwrap(); + let outside = Element::BidirectionalReference( + BidirectionalReference { + forward_reference_path: ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"tree".to_vec(), + b"value".to_vec(), + ]), + backward_references: Vec::new(), + cascade_on_update: true, + max_hop: None, + }, + None, + ); + db.insert(&[TEST_LEAF], b"outside", outside, None, None, version) + .unwrap() + .unwrap(); + }; + let ops = vec![ + QualifiedGroveDbOp::delete_op( + vec![TEST_LEAF.to_vec(), b"tree".to_vec()], + b"value".to_vec(), + ), + QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + TreeType::NormalTree, + SubelementsDeletionBehavior::DontCheckWithNoCleanup, + ), + ]; + + let db = make_test_grovedb(version); + seed(&db); + db.apply_batch(ops.clone(), None, None, version) + .unwrap() + .expect("the explicit participant delete cascades its referrer"); + assert!(db + .get_raw(SubtreePath::from(&[TEST_LEAF]), b"outside", None, version) + .unwrap() + .is_err()); + assert!(db + .get_raw(SubtreePath::from(&[TEST_LEAF]), b"tree", None, version) + .unwrap() + .is_err()); + assert!(db + .verify_grovedb(None, true, true, version) + .unwrap() + .is_empty()); + + let db = make_test_grovedb(version); + seed(&db); + let before = db.root_hash(None, version).unwrap().unwrap(); + let result = db.apply_partial_batch(ops, None, |_, _| Ok(vec![]), None, version); + assert!(matches!(result.unwrap(), Err(Error::NotSupported(_)))); + assert_eq!(db.root_hash(None, version).unwrap().unwrap(), before); +} + +/// A `DeleteTree(Skip)` over a populated tree executes nothing, so its +/// behavior must not exempt a callback replacement of that same tree from +/// the participant scan: the replacement is refused before commit and the +/// outside reference keeps resolving. +#[test] +fn skipped_delete_tree_does_not_exempt_a_callback_replacement() { + use crate::batch::SubelementsDeletionBehavior; + use grovedb_merk::TreeType; + let version = GroveVersion::latest(); + let db = make_test_grovedb(version); + db.insert( + &[TEST_LEAF], + b"tree", + Element::empty_tree(), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"tree"], + b"value", + Element::new_item_allowing_bidirectional_references(vec![1]), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF], + b"outside", + Element::BidirectionalReference( + BidirectionalReference { + forward_reference_path: ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"tree".to_vec(), + b"value".to_vec(), + ]), + backward_references: Vec::new(), + cascade_on_update: true, + max_hop: None, + }, + None, + ), + None, + None, + version, + ) + .unwrap() + .unwrap(); + let before = db.root_hash(None, version).unwrap().unwrap(); + let result = db.apply_partial_batch( + vec![QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + TreeType::NormalTree, + SubelementsDeletionBehavior::Skip, + )], + None, + |_, _| { + Ok(vec![QualifiedGroveDbOp::insert_or_replace_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + Element::empty_tree(), + )]) + }, + None, + version, + ); + assert!(matches!(result.unwrap(), Err(Error::NotSupported(_)))); + assert_eq!(db.root_hash(None, version).unwrap().unwrap(), before); + db.get(&[TEST_LEAF], b"outside", None, version) + .unwrap() + .expect("the outside reference still resolves"); + assert!(db + .verify_grovedb(None, true, true, version) + .unwrap() + .is_empty()); +} + +/// The non-batched adapter must carry each op's declaration whether or not +/// batch options are supplied: a false `NotParticipant` claim is refused +/// with `None` exactly as with `Some(BatchApplyOptions::default())`. +#[test] +fn non_batched_apply_keeps_the_op_declaration_without_options() { + let version = GroveVersion::latest(); + let overwrite = QualifiedGroveDbOp::insert_or_replace_op( + vec![TEST_LEAF.to_vec()], + b"value".to_vec(), + Element::new_item(vec![2]), + ) + .with_displaced_value(DisplacedValue::NotParticipant); + let delete = QualifiedGroveDbOp::delete_op(vec![TEST_LEAF.to_vec()], b"value".to_vec()) + .with_displaced_value(DisplacedValue::NotParticipant); + for op in [overwrite, delete] { + for options in [None, Some(BatchApplyOptions::default())] { + let db = chain(true); + let before = db.root_hash(None, version).unwrap().unwrap(); + let result = + db.apply_operations_without_batching(vec![op.clone()], options, None, version); + assert!( + matches!(result.unwrap(), Err(Error::NotSupported(_))), + "{op:?} must be refused" + ); + assert_eq!(db.root_hash(None, version).unwrap().unwrap(), before); + } + } +} + +/// A conditional insert over an existing key writes nothing, so it has no +/// displaced value to check: declaring `NotParticipant` on it is not a false +/// claim even when the existing value participates. +#[test] +fn skipped_conditional_insert_checks_no_displaced_value() { + let version = GroveVersion::latest(); + let db = chain(true); + let before = db.root_hash(None, version).unwrap().unwrap(); + db.apply_batch( + vec![QualifiedGroveDbOp::insert_if_not_exists_or_skip_op( + vec![TEST_LEAF.to_vec()], + b"value".to_vec(), + Element::new_item(vec![2]), + ) + .with_displaced_value(DisplacedValue::NotParticipant)], + None, + None, + version, + ) + .unwrap() + .expect("a skipped conditional insert displaces nothing"); + assert_eq!(db.root_hash(None, version).unwrap().unwrap(), before); +} + +/// A removal observed in the initial segment keeps the declaration of the +/// op that displaced it: a callback op at the same path declared +/// `NotParticipant` (a plain overwrite, or a `DeleteTree(Skip)` that ends up +/// skipped) cannot speak for the earlier tree replacement. +#[test] +fn initial_segment_removal_keeps_its_own_declaration() { + use crate::batch::SubelementsDeletionBehavior; + use grovedb_merk::TreeType; + let version = GroveVersion::latest(); + for callback_deletes in [false, true] { + let db = make_test_grovedb(version); + db.insert( + &[TEST_LEAF], + b"tree", + Element::empty_tree(), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"tree"], + b"value", + Element::new_item_allowing_bidirectional_references(vec![1]), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF], + b"outside", + Element::BidirectionalReference( + BidirectionalReference { + forward_reference_path: ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"tree".to_vec(), + b"value".to_vec(), + ]), + backward_references: Vec::new(), + cascade_on_update: true, + max_hop: None, + }, + None, + ), + None, + None, + version, + ) + .unwrap() + .unwrap(); + let before = db.root_hash(None, version).unwrap().unwrap(); + let result = db.apply_partial_batch( + vec![QualifiedGroveDbOp::insert_or_replace_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + Element::new_item(vec![1]), + )], + None, + |_, _| { + Ok(vec![if callback_deletes { + QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + TreeType::NormalTree, + SubelementsDeletionBehavior::Skip, + ) + } else { + QualifiedGroveDbOp::insert_or_replace_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + Element::new_item(vec![2]), + ) + } + .with_displaced_value(DisplacedValue::NotParticipant)]) + }, + None, + version, + ); + assert!( + matches!(result.unwrap(), Err(Error::NotSupported(_))), + "callback_deletes={callback_deletes}" + ); + assert_eq!(db.root_hash(None, version).unwrap().unwrap(), before); + db.get(&[TEST_LEAF], b"outside", None, version) + .unwrap() + .expect("the outside reference still resolves"); + assert!(db + .verify_grovedb(None, true, true, version) + .unwrap() + .is_empty()); + } +} + +/// A skipped callback `DeleteTree(Skip)` must not erase the cleanup behavior +/// of the `DeleteTree(DeleteChildren)` the initial segment executed at the +/// same path: the removed subtree's storage is still cleaned. +#[test] +fn skipped_callback_delete_keeps_the_executed_deletions_cleanup() { + use crate::batch::SubelementsDeletionBehavior; + use grovedb_merk::TreeType; + use grovedb_storage::{Storage, StorageContext}; + let version = GroveVersion::latest(); + let db = make_test_grovedb(version); + db.insert( + &[TEST_LEAF], + b"tree", + Element::empty_tree(), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"tree"], + b"child", + Element::new_item(vec![1]), + None, + None, + version, + ) + .unwrap() + .unwrap(); + db.apply_partial_batch( + vec![QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + TreeType::NormalTree, + SubelementsDeletionBehavior::DeleteChildren, + )], + None, + |_, _| { + Ok(vec![QualifiedGroveDbOp::delete_tree_op( + vec![TEST_LEAF.to_vec()], + b"tree".to_vec(), + TreeType::NormalTree, + SubelementsDeletionBehavior::Skip, + )]) + }, + None, + version, + ) + .unwrap() + .expect("the executed deletion applies; the skipped one is dropped"); + assert!(db + .get_raw(SubtreePath::from(&[TEST_LEAF]), b"tree", None, version) + .unwrap() + .is_err()); + let tx = db.start_transaction(); + let storage = db + .db + .get_transactional_storage_context(SubtreePath::from(&[TEST_LEAF, b"tree"]), None, &tx) + .unwrap(); + assert!( + storage.get(b"child").unwrap().unwrap().is_none(), + "the executed DeleteChildren must still clean the subtree's storage" + ); + assert!(db + .verify_grovedb(None, true, true, version) + .unwrap() + .is_empty()); +} diff --git a/grovedb/src/tests/batch_backward_references_cost_tests.rs b/grovedb/src/tests/batch_backward_references_cost_tests.rs index b8bf1f21d..2b9f6a1d8 100644 --- a/grovedb/src/tests/batch_backward_references_cost_tests.rs +++ b/grovedb/src/tests/batch_backward_references_cost_tests.rs @@ -1,13 +1,13 @@ //! Estimated-cost coverage for backward-references batch ops (batching -//! M5): under `BatchApplyOptions::backward_references_policy`, the -//! GROVE_V4 estimators charge the derived fan-out (registration, chain -//! propagation, cascade deletion) so `worst-case estimate >= actual` holds -//! for maintained family batches, while pre-V4 estimation stays byte-stable -//! for replay. Plain writes and deletes charge the displaced-state bound -//! only in layers declaring `may_contain_backward_references`; undeclared -//! layers estimate them exactly as `Skip` would. - -use crate::BackwardReferencesPolicy; +//! M5): the GROVE_V4 estimators charge the derived fan-out (registration, +//! chain propagation, cascade deletion) so `worst-case estimate >= actual` +//! holds for maintained family batches, while pre-V4 estimation stays +//! byte-stable for replay. A plain write or delete charges the +//! displaced-state bound only when the op declares +//! `DisplacedValue::MayBeParticipant`; an op declared `NotParticipant` +//! estimates the plain write alone. + +use crate::DisplacedValue; use std::collections::HashMap; use grovedb_merk::estimated_costs::{ @@ -17,9 +17,7 @@ use grovedb_merk::estimated_costs::{ EstimatedLayerSizes::{AllItems, AllSubtrees}, EstimatedSumTrees::NoSumTrees, }, - worst_case_costs::WorstCaseLayerInformation::{ - self, MaxElementsNumber, MaxElementsNumberWithBackwardReferences, - }, + worst_case_costs::WorstCaseLayerInformation::{self, MaxElementsNumber}, }; use grovedb_merk::tree_type::TreeType; use grovedb_version::version::GroveVersion; @@ -36,11 +34,10 @@ use crate::{ Element, Error, GroveDb, }; -fn batch_flag_on() -> Option { - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, - ..Default::default() - }) +fn not_participant(ops: Vec) -> Vec { + ops.into_iter() + .map(|op| op.with_displaced_value(DisplacedValue::NotParticipant)) + .collect() } fn sibling_bidi(key: &[u8]) -> Element { @@ -78,11 +75,9 @@ fn worst_case_layers( { let mut paths = HashMap::new(); paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(4)); - // `db_with_chain` stores its participants directly under TEST_LEAF, so - // the layer declares them; the root layer holds none. paths.insert( KeyInfoPath(vec![KeyInfo::KnownKey(TEST_LEAF.to_vec())]), - MaxElementsNumberWithBackwardReferences(16), + MaxElementsNumber(16), ); paths } @@ -93,7 +88,6 @@ fn average_case_layers() -> HashMap { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(32, NoSumTrees, None), }, @@ -102,7 +96,6 @@ fn average_case_layers() -> HashMap { KeyInfoPath(vec![KeyInfo::KnownKey(TEST_LEAF.to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: true, estimated_layer_count: EstimatedLevel(2, true), estimated_layer_sizes: AllItems(32, 128, None), }, @@ -182,9 +175,9 @@ fn worst_case_estimate_covers_flagged_family_overwrite() { b"value".to_vec(), Element::new_item_allowing_bidirectional_references(b"updated".to_vec()), )]; - let estimate = worst_case_estimate(ops.clone(), batch_flag_on(), grove_version); + let estimate = worst_case_estimate(ops.clone(), None, grove_version); let actual = db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("apply succeeds"); @@ -203,9 +196,9 @@ fn worst_case_estimate_covers_flagged_delete_cascade() { vec![TEST_LEAF.to_vec()], b"value".to_vec(), )]; - let estimate = worst_case_estimate(ops.clone(), batch_flag_on(), grove_version); + let estimate = worst_case_estimate(ops.clone(), None, grove_version); let actual = db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("apply succeeds"); @@ -232,9 +225,9 @@ fn worst_case_estimate_covers_bidi_insert_with_in_batch_target() { sibling_bidi(b"value"), ), ]; - let estimate = worst_case_estimate(ops.clone(), batch_flag_on(), grove_version); + let estimate = worst_case_estimate(ops.clone(), None, grove_version); let actual = db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("apply succeeds"); @@ -245,7 +238,7 @@ fn worst_case_estimate_covers_bidi_insert_with_in_batch_target() { } #[test] -fn fan_out_terms_are_default_and_skip_disables_them() { +fn fan_out_terms_follow_the_op_declaration() { let grove_version = GroveVersion::latest(); let family_op = || { @@ -256,57 +249,34 @@ fn fan_out_terms_are_default_and_skip_disables_them() { )] }; - // Automatic maintenance includes fan-out on GROVE_V4+. - let flagged = worst_case_estimate(family_op(), None, grove_version); + // A participant payload is charged from the op itself on GROVE_V4+, + // whatever it declares about the value it displaces. + let family = worst_case_estimate(family_op(), None, grove_version); assert_eq!( - flagged, - worst_case_estimate(family_op(), batch_flag_on(), grove_version) - ); - let unflagged = worst_case_estimate( - family_op(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - grove_version, + family, + worst_case_estimate(not_participant(family_op()), None, grove_version), + "a participant payload charges its fan-out regardless of the declaration" ); - assert!( - flagged.seek_count > unflagged.seek_count - && flagged.storage_cost.replaced_bytes > unflagged.storage_cost.replaced_bytes, - "the flag must activate the fan-out terms: {flagged:?} vs {unflagged:?}" - ); - let flagged_avg = average_case_estimate(family_op(), batch_flag_on(), grove_version); - let unflagged_avg = average_case_estimate( - family_op(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - grove_version, + assert_eq!( + average_case_estimate(family_op(), None, grove_version), + average_case_estimate(not_participant(family_op()), None, grove_version) ); - assert!(flagged_avg.seek_count > unflagged_avg.seek_count); - // …and a PLAIN-item op also charges the displaced-state fan-out under - // the flag: the estimator cannot see the stored element the write - // lands on, which may be a registered family element whose - // propagation/cascade the preprocessor must perform. + // A PLAIN-item op charges the displaced-state fan-out only when it + // declares the stored element it lands on may be a registered family + // element whose propagation/cascade the preprocessor must perform. let plain_op = vec![QualifiedGroveDbOp::insert_or_replace_op( vec![TEST_LEAF.to_vec()], b"value".to_vec(), Element::new_item(b"hello".to_vec()), )]; - let flagged_plain = worst_case_estimate(plain_op.clone(), batch_flag_on(), grove_version); - let unflagged_plain = worst_case_estimate( - plain_op, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - grove_version, - ); + let may_be_plain = worst_case_estimate(plain_op.clone(), None, grove_version); + let not_plain = worst_case_estimate(not_participant(plain_op), None, grove_version); assert!( - flagged_plain.seek_count > unflagged_plain.seek_count, - "plain writes must charge the displaced-state bound in a declared layer" + may_be_plain.seek_count > not_plain.seek_count + && may_be_plain.storage_cost.replaced_bytes > not_plain.storage_cost.replaced_bytes, + "the declaration must activate the displaced-state terms: {may_be_plain:?} vs \ + {not_plain:?}" ); } @@ -323,9 +293,9 @@ fn worst_case_estimate_covers_flagged_plain_overwrite_of_registered_target() { b"value".to_vec(), Element::new_item(b"plain".to_vec()), )]; - let estimate = worst_case_estimate(ops.clone(), batch_flag_on(), grove_version); + let estimate = worst_case_estimate(ops.clone(), None, grove_version); let actual = db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("apply succeeds — the chain cascades"); @@ -400,7 +370,7 @@ fn worst_case_estimate_covers_deep_origin_registration_growth() { let estimate = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(paths), ops.clone(), - batch_flag_on(), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok(( @@ -413,7 +383,7 @@ fn worst_case_estimate_covers_deep_origin_registration_growth() { .cost_as_result() .expect("expected worst case costs"); let actual = db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("apply succeeds"); @@ -438,11 +408,11 @@ fn pre_v4_estimation_is_byte_stable_for_replay() { )] }; assert_eq!( - worst_case_estimate(family_op(), batch_flag_on(), v3), + worst_case_estimate(family_op(), None, v3), worst_case_estimate(family_op(), None, v3), ); assert_eq!( - average_case_estimate(family_op(), batch_flag_on(), v3), + average_case_estimate(family_op(), None, v3), average_case_estimate(family_op(), None, v3), ); } @@ -464,6 +434,7 @@ fn derived_op_estimation_is_version_gated() { node_value_hash: [7; 32], end_hash: None, }, + displaced_value: DisplacedValue::MayBeParticipant, }] }; @@ -592,12 +563,12 @@ fn worst_case_estimate_covers_max_fan_out_deep_component() { paths.insert(KeyInfoPath(vec![]), MaxElementsNumber(8)); paths.insert( KeyInfoPath(vec![KeyInfo::KnownKey(TEST_LEAF.to_vec())]), - MaxElementsNumberWithBackwardReferences(128), + MaxElementsNumber(128), ); let estimate = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(paths), ops.clone(), - batch_flag_on(), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok(( @@ -610,7 +581,7 @@ fn worst_case_estimate_covers_max_fan_out_deep_component() { .cost_as_result() .expect("expected worst case costs"); let actual = db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("apply succeeds — full-width propagation"); @@ -643,9 +614,9 @@ fn worst_case_estimate_covers_flagged_empty_tree_deletion() { vec![TEST_LEAF.to_vec()], b"sub".to_vec(), )]; - let estimate = worst_case_estimate(ops.clone(), batch_flag_on(), grove_version); + let estimate = worst_case_estimate(ops.clone(), None, grove_version); let actual = db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("an empty subtree deletes under the flag"); @@ -679,13 +650,13 @@ fn declared_capacity_tightens_the_worst_case_estimate() { Element::new_item(b"updated".to_vec()), )]; - let tight = worst_case_estimate(family(1), batch_flag_on(), grove_version); + let tight = worst_case_estimate(family(1), None, grove_version); let default = worst_case_estimate( family(grovedb_element::DEFAULT_BACKWARD_REFERENCES_CAPACITY), - batch_flag_on(), + None, grove_version, ); - let ceiling = worst_case_estimate(plain, batch_flag_on(), grove_version); + let ceiling = worst_case_estimate(plain, None, grove_version); assert!( tight.seek_count < default.seek_count && tight.hash_node_calls < default.hash_node_calls, "a capacity of one must estimate below the default: {tight:?} vs {default:?}" @@ -702,7 +673,7 @@ fn declared_capacity_tightens_the_worst_case_estimate() { // propagation along the whole chain. let db = db_with_chain(grove_version); let actual = db - .apply_batch(family(1), batch_flag_on(), None, grove_version) + .apply_batch(family(1), None, None, grove_version) .cost_as_result() .expect("one registered referrer fits a capacity of one"); assert!( @@ -715,10 +686,10 @@ fn declared_capacity_tightens_the_worst_case_estimate() { .is_empty()); // The average model is capped by the declaration too. - let tight_average = average_case_estimate(family(0), batch_flag_on(), grove_version); + let tight_average = average_case_estimate(family(0), None, grove_version); let default_average = average_case_estimate( family(grovedb_element::DEFAULT_BACKWARD_REFERENCES_CAPACITY), - batch_flag_on(), + None, grove_version, ); assert!( @@ -728,49 +699,13 @@ fn declared_capacity_tightens_the_worst_case_estimate() { ); } -fn undeclared_worst_case_layers() -> HashMap { - worst_case_layers() - .into_iter() - .map(|(path, layer)| { - let undeclared = match layer { - MaxElementsNumber(n) | MaxElementsNumberWithBackwardReferences(n) => { - MaxElementsNumber(n) - } - WorstCaseLayerInformation::NumberOfLevels(n) - | WorstCaseLayerInformation::NumberOfLevelsWithBackwardReferences(n) => { - WorstCaseLayerInformation::NumberOfLevels(n) - } - }; - (path, undeclared) - }) - .collect() -} - -fn undeclared_average_case_layers() -> HashMap { - average_case_layers() - .into_iter() - .map(|(path, mut layer)| { - layer.may_contain_backward_references = false; - (path, layer) - }) - .collect() -} - -fn skip() -> Option { - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }) -} - -/// The estimator cannot see stored state. A layer that does not declare -/// `may_contain_backward_references` charges neither the displaced-state -/// fan-out nor the delete probe, so default (`Maintain`) estimates for plain -/// writes and deletes are byte-identical to explicit `Skip` estimates, in -/// both estimators. Declaring the layer reinstates the bound, and an op that -/// itself writes a participant is charged from the op regardless. +/// The estimator cannot see stored state. An op declared `NotParticipant` +/// charges neither the displaced-state fan-out nor the delete probe, so its +/// estimate is the plain write's alone in both estimators; the default +/// `MayBeParticipant` reinstates the bound, and an op that itself writes a +/// participant is charged from the op regardless. #[test] -fn undeclared_layers_estimate_plain_writes_exactly_like_skip() { +fn not_participant_ops_estimate_plain_writes_without_the_displaced_bound() { let grove_version = GroveVersion::latest(); let plain_ops = [ ( @@ -788,50 +723,19 @@ fn undeclared_layers_estimate_plain_writes_exactly_like_skip() { ]; for (name, op) in plain_ops { let ops = || vec![op.clone()]; - let default_average = average_case_estimate_with_layers( - undeclared_average_case_layers(), - ops(), - None, - grove_version, - ); - let skip_average = average_case_estimate_with_layers( - undeclared_average_case_layers(), - ops(), - skip(), - grove_version, - ); - assert_eq!( - default_average, skip_average, - "{name}: an undeclared layer must estimate the default policy exactly like Skip" - ); - let default_worst = worst_case_estimate_with_layers( - undeclared_worst_case_layers(), - ops(), - None, - grove_version, - ); - let skip_worst = worst_case_estimate_with_layers( - undeclared_worst_case_layers(), - ops(), - skip(), - grove_version, - ); - assert_eq!( - default_worst, skip_worst, - "{name}: an undeclared layer must estimate the default policy exactly like Skip" - ); - - let declared_average = average_case_estimate(ops(), None, grove_version); - let declared_worst = worst_case_estimate(ops(), None, grove_version); + let not_average = average_case_estimate(not_participant(ops()), None, grove_version); + let not_worst = worst_case_estimate(not_participant(ops()), None, grove_version); + let may_be_average = average_case_estimate(ops(), None, grove_version); + let may_be_worst = worst_case_estimate(ops(), None, grove_version); assert!( - declared_average.seek_count > default_average.seek_count, - "{name}: the declaration must reinstate the average-case bound: {declared_average:?} \ - vs {default_average:?}" + may_be_average.seek_count > not_average.seek_count, + "{name}: the declaration must reinstate the average-case bound: {may_be_average:?} \ + vs {not_average:?}" ); assert!( - declared_worst.seek_count > default_worst.seek_count, - "{name}: the declaration must reinstate the worst-case bound: {declared_worst:?} vs \ - {default_worst:?}" + may_be_worst.seek_count > not_worst.seek_count, + "{name}: the declaration must reinstate the worst-case bound: {may_be_worst:?} vs \ + {not_worst:?}" ); } @@ -842,43 +746,23 @@ fn undeclared_layers_estimate_plain_writes_exactly_like_skip() { Element::new_item_allowing_bidirectional_references(b"hello".to_vec()), )] }; - let family_average = average_case_estimate_with_layers( - undeclared_average_case_layers(), - family(), - None, - grove_version, - ); - let family_average_skip = average_case_estimate_with_layers( - undeclared_average_case_layers(), - family(), - skip(), - grove_version, - ); - assert!( - family_average.seek_count > family_average_skip.seek_count, - "a participant write is charged from the op even in an undeclared layer" - ); - let family_worst = worst_case_estimate_with_layers( - undeclared_worst_case_layers(), - family(), - None, - grove_version, + assert_eq!( + average_case_estimate(family(), None, grove_version), + average_case_estimate(not_participant(family()), None, grove_version), + "a participant write is charged from the op whatever it declares" ); - let family_worst_skip = worst_case_estimate_with_layers( - undeclared_worst_case_layers(), - family(), - skip(), - grove_version, + assert_eq!( + worst_case_estimate(family(), None, grove_version), + worst_case_estimate(not_participant(family()), None, grove_version) ); - assert!(family_worst.seek_count > family_worst_skip.seek_count); } /// Overwriting a registered target with a plain item cascades its chain. -/// Only the declared layer's worst-case estimate covers that work; the -/// undeclared estimate is the `Skip` estimate, which is the caller's -/// acknowledged trade-off for not declaring the layer. +/// Only the `MayBeParticipant` worst-case estimate covers that work; a +/// `NotParticipant` estimate is the plain write's, and the apply refuses the +/// false claim rather than running the cascade unpriced. #[test] -fn declared_layer_covers_the_displaced_cascade_an_undeclared_layer_cannot_see() { +fn may_be_participant_covers_the_displaced_cascade_not_participant_cannot_see() { let grove_version = GroveVersion::latest(); let db = db_with_chain(grove_version); let ops = || { @@ -890,17 +774,12 @@ fn declared_layer_covers_the_displaced_cascade_an_undeclared_layer_cannot_see() }; let declared = worst_case_estimate(ops(), None, grove_version); - let undeclared = - worst_case_estimate_with_layers(undeclared_worst_case_layers(), ops(), None, grove_version); - assert_eq!( - undeclared, - worst_case_estimate_with_layers( - undeclared_worst_case_layers(), - ops(), - skip(), - grove_version - ) - ); + let undeclared = worst_case_estimate(not_participant(ops()), None, grove_version); + assert!(matches!( + db.apply_batch(not_participant(ops()), None, None, grove_version) + .unwrap(), + Err(Error::NotSupported(_)) + )); let actual = db .apply_batch(ops(), None, None, grove_version) diff --git a/grovedb/src/tests/batch_backward_references_tests.rs b/grovedb/src/tests/batch_backward_references_tests.rs index 2a38fa95f..3f650032f 100644 --- a/grovedb/src/tests/batch_backward_references_tests.rs +++ b/grovedb/src/tests/batch_backward_references_tests.rs @@ -1,16 +1,16 @@ //! Batch support for the backward-references family (batching M2–M4): the //! master invariant is that a batch under -//! `BatchApplyOptions::backward_references_policy` produces the exact +//! `BatchApplyOptions::displaced_value` produces the exact //! root hash the live flagged flow produces for the same logical //! operations — including `BidirectionalReference` ops, in-batch targets //! and chains, retargets, identical-edge no-ops, and the M4 conflict //! rules. -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use grovedb_version::version::GroveVersion; use crate::{ - batch::{BatchApplyOptions, QualifiedGroveDbOp}, + batch::QualifiedGroveDbOp, bidirectional_references::BidirectionalReference, operations::{delete::DeleteOptions, insert::InsertOptions}, reference_path::ReferencePathType, @@ -20,14 +20,7 @@ use crate::{ fn flag_on() -> Option { Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, - ..Default::default() - }) -} - -fn batch_flag_on() -> Option { - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }) } @@ -109,7 +102,7 @@ fn batch_fresh_insert_matches_live() { b"value".to_vec(), Element::new_item_allowing_bidirectional_references(b"hello".to_vec()), )], - batch_flag_on(), + None, None, grove_version, ) @@ -143,7 +136,7 @@ fn batch_overwrite_propagates_along_the_chain_like_live() { b"value".to_vec(), updated.clone(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -212,7 +205,7 @@ fn batch_sum_twin_overwrite_matches_live() { b"twin".to_vec(), updated.clone(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -244,7 +237,7 @@ fn batch_delete_cascades_like_live() { vec![TEST_LEAF.to_vec()], b"value".to_vec(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -255,7 +248,7 @@ fn batch_delete_cascades_like_live() { &[TEST_LEAF], b"value", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -289,7 +282,7 @@ fn batch_overwrite_with_plain_item_cascades_like_live() { b"value".to_vec(), plain.clone(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -341,7 +334,7 @@ fn batch_cascade_requires_consent() { vec![TEST_LEAF.to_vec()], b"value".to_vec(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -367,7 +360,7 @@ fn batch_clears_caller_supplied_referrer_lists() { b"planted".to_vec(), Element::ItemWithBackwardsReferences(b"x".to_vec(), vec![forged].into(), None), )], - batch_flag_on(), + None, None, grove_version, ) @@ -396,37 +389,17 @@ fn batch_rejections_hold() { let grove_version = GroveVersion::latest(); let (db, _other) = twin_dbs_with_chain(grove_version); - // Family item ops with maintenance explicitly skipped: rejected. + // A family payload declared `NotParticipant` over a stored participant: + // the claim is false and is refused from the value already in hand. assert!(matches!( db.apply_batch( vec![QualifiedGroveDbOp::insert_or_replace_op( vec![TEST_LEAF.to_vec()], - b"fresh".to_vec(), + b"value".to_vec(), Element::new_item_allowing_bidirectional_references(b"x".to_vec()), - )], - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + ) + .with_displaced_value(DisplacedValue::NotParticipant)], None, - grove_version, - ) - .unwrap(), - Err(Error::NotSupported(_)) - )); - - // BidirectionalReference element ops with maintenance explicitly skipped: rejected. - assert!(matches!( - db.apply_batch( - vec![QualifiedGroveDbOp::insert_or_replace_op( - vec![TEST_LEAF.to_vec()], - b"newref".to_vec(), - sibling_bidi(b"value", true), - )], - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), None, grove_version, ) @@ -445,8 +418,9 @@ fn batch_rejections_hold() { node_value_hash: [7; 32], end_hash: None, }, + displaced_value: DisplacedValue::MayBeParticipant, }], - batch_flag_on(), + None, None, grove_version, ) @@ -467,7 +441,7 @@ fn batch_rejections_hold() { ), QualifiedGroveDbOp::delete_op(vec![TEST_LEAF.to_vec()], b"r1".to_vec()), ], - batch_flag_on(), + None, None, grove_version, ) @@ -484,7 +458,7 @@ fn batch_rejections_hold() { b"fresh".to_vec(), Element::new_item_allowing_bidirectional_references(b"x".to_vec()), )], - batch_flag_on(), + None, None, v3, ) @@ -521,7 +495,7 @@ fn batch_bidi_insert_with_existing_target_matches_live() { b"ref".to_vec(), sibling_bidi(b"value", true), )], - batch_flag_on(), + None, None, grove_version, ) @@ -578,7 +552,7 @@ fn batch_bidi_insert_with_in_batch_target_matches_live_in_any_op_order() { let batch_db = make_test_grovedb(grove_version); let live_db = make_test_grovedb(grove_version); batch_db - .apply_batch(vec![op_a, op_b], batch_flag_on(), None, grove_version) + .apply_batch(vec![op_a, op_b], None, None, grove_version) .unwrap() .unwrap(); // The live twin's only valid sequential order is target first. @@ -634,7 +608,7 @@ fn batch_whole_chain_created_in_one_batch_matches_live() { Element::new_item_allowing_bidirectional_references(b"hello".to_vec()), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -688,7 +662,7 @@ fn batch_retarget_matches_live() { b"ref".to_vec(), sibling_bidi(b"b", true), )], - batch_flag_on(), + None, None, grove_version, ) @@ -737,7 +711,7 @@ fn batch_retarget_with_upstream_referrer_matches_live() { b"r1".to_vec(), sibling_bidi(b"other", true), )], - batch_flag_on(), + None, None, grove_version, ) @@ -771,7 +745,7 @@ fn batch_identical_edge_reinsert_is_a_no_op() { b"r1".to_vec(), sibling_bidi(b"value", true), )], - batch_flag_on(), + None, None, grove_version, ) @@ -798,7 +772,7 @@ fn batch_bidi_delete_matches_live() { vec![TEST_LEAF.to_vec()], b"r1".to_vec(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -809,7 +783,7 @@ fn batch_bidi_delete_matches_live() { &[TEST_LEAF], b"r1", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -840,7 +814,7 @@ fn batch_overwrite_bidi_with_plain_item_matches_live() { b"r1".to_vec(), plain.clone(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -887,7 +861,7 @@ fn batch_two_refs_to_same_target_matches_live() { sibling_bidi(b"value", true), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -933,7 +907,7 @@ fn batch_ref_plus_target_overwrite_in_same_batch_matches_live() { sibling_bidi(b"value", true), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -985,7 +959,7 @@ fn batch_component_budget_enforced_against_prospective_state() { sibling_bidi(format!("t{}", i - 1).as_bytes(), true), )); } - db.apply_batch(ops.clone(), batch_flag_on(), None, grove_version) + db.apply_batch(ops.clone(), None, None, grove_version) .unwrap() .expect("a chain at the hop budget is valid"); @@ -1002,8 +976,7 @@ fn batch_component_budget_enforced_against_prospective_state() { sibling_bidi(format!("t{}", MAX_REFERENCE_HOPS - 1).as_bytes(), true), )); assert!(matches!( - db.apply_batch(ops, batch_flag_on(), None, grove_version) - .unwrap(), + db.apply_batch(ops, None, None, grove_version).unwrap(), Err(Error::BidirectionalReferenceRule(_)) )); } @@ -1041,8 +1014,7 @@ fn batch_ref_insert_with_target_deleted_in_same_batch_errors() { } assert!( matches!( - db.apply_batch(ops, batch_flag_on(), None, grove_version) - .unwrap(), + db.apply_batch(ops, None, None, grove_version).unwrap(), Err(Error::InvalidBatchOperation(_)) | Err(Error::CorruptedReferencePathKeyNotFound(_)) ), @@ -1067,7 +1039,7 @@ fn batch_cascade_hitting_a_user_write_errors() { Element::new_item(b"squatter".to_vec()), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -1091,9 +1063,10 @@ fn batch_refresh_reference_on_bidi_errors() { flags: None, non_counted: false, }, + displaced_value: DisplacedValue::MayBeParticipant, }; assert!(matches!( - db.apply_batch(vec![refresh], batch_flag_on(), None, grove_version) + db.apply_batch(vec![refresh], None, None, grove_version) .unwrap(), Err(Error::NotSupported(_)) )); @@ -1121,7 +1094,7 @@ fn batch_plain_reference_can_point_at_in_batch_family_target() { Element::new_reference(ReferencePathType::SiblingReference(b"value".to_vec())), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -1187,7 +1160,7 @@ fn batch_bidi_ops_keep_caller_authority_rules() { b"ref".to_vec(), reference, )], - batch_flag_on(), + None, None, grove_version, ) @@ -1256,7 +1229,7 @@ fn batch_no_op_insert_if_not_exists_does_not_swallow_registration() { sibling_bidi(b"value", true), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -1324,7 +1297,7 @@ fn batch_later_plain_overwrite_supersedes_propagation() { ops.reverse(); } batch_db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .unwrap() .unwrap(); @@ -1410,7 +1383,7 @@ fn batch_populates_a_subtree_created_in_the_same_batch() { let batch_db = make_test_grovedb(grove_version); batch_db - .apply_batch(ops(), batch_flag_on(), None, grove_version) + .apply_batch(ops(), None, None, grove_version) .unwrap() .expect("a batch may create and populate a subtree under the flag"); @@ -1421,7 +1394,7 @@ fn batch_populates_a_subtree_created_in_the_same_batch() { let mut reversed = ops(); reversed.reverse(); reversed_db - .apply_batch(reversed, batch_flag_on(), None, grove_version) + .apply_batch(reversed, None, None, grove_version) .unwrap() .unwrap(); assert_eq!( @@ -1474,7 +1447,7 @@ fn batch_populates_a_subtree_created_in_the_same_batch() { } /// A flagged overwrite of a family item carrying a DANGLING registration -/// (its referrer was removed through an unflagged batch) plans a stale- +/// (its referrer was removed through a raw clear) plans a stale- /// entry cleanup targeting the op's own position: that cleanup must fold /// into the op itself, not become a second op that fails consistency. #[test] @@ -1494,29 +1467,47 @@ fn batch_flagged_overwrite_folds_own_stale_cleanup() { .unwrap(); db.insert( &[TEST_LEAF], + b"refs", + Element::empty_tree(), + None, + None, + grove_version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"refs"], b"ref", - sibling_bidi(b"value", true), + Element::BidirectionalReference( + BidirectionalReference { + forward_reference_path: ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"value".to_vec(), + ]), + backward_references: Vec::new(), + cascade_on_update: true, + max_hop: None, + }, + None, + ), None, None, grove_version, ) .unwrap() .unwrap(); - // Remove the referrer through the supported explicit Skip batch path: - // the registration on `value` is left dangling. - db.apply_batch( - vec![QualifiedGroveDbOp::delete_op( - vec![TEST_LEAF.to_vec()], - b"ref".to_vec(), - )], - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, + // Remove the referrer through a trusted path that reads nothing: a + // raw `clear_subtree` declared `NotParticipant` leaves the + // registration on `value` dangling. + db.clear_subtree( + &[TEST_LEAF, b"refs"], + Some(crate::operations::delete::ClearOptions { + displaced_value: DisplacedValue::NotParticipant, ..Default::default() }), None, grove_version, ) - .unwrap() .unwrap(); db }; @@ -1530,7 +1521,7 @@ fn batch_flagged_overwrite_folds_own_stale_cleanup() { b"value".to_vec(), updated.clone(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -1573,7 +1564,7 @@ fn batch_patch_created_subtree_is_fresh() { Element::new_item(b"i".to_vec()), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -1661,7 +1652,7 @@ fn batch_registration_depth_is_bounded() { b"ref".to_vec(), ref_to_value(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -1690,7 +1681,7 @@ fn batch_registration_depth_is_bounded() { b"ref".to_vec(), ref_to_value(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -1792,7 +1783,7 @@ fn batch_flags_mutation_on_derived_rewrite_rehashes_final_bytes() { let db = build(); db.apply_batch_with_element_flags_update( ops(), - batch_flag_on(), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok(( @@ -1818,7 +1809,7 @@ fn batch_flags_mutation_on_derived_rewrite_rehashes_final_bytes() { let mut mutated = 0usize; db.apply_batch_with_element_flags_update( ops(), - batch_flag_on(), + None, |_cost, _old_flags, new_flags| { new_flags.push(7); mutated += 1; @@ -1887,7 +1878,7 @@ fn batch_refuses_wrapped_backward_references_elements() { Default::default(), None, ))); - for flag in [batch_flag_on(), None] { + for flag in [None, None] { let err = db .apply_batch( vec![QualifiedGroveDbOp::insert_or_replace_op( @@ -1952,7 +1943,7 @@ fn batch_flagged_non_empty_subtree_deletion_is_refused() { vec![TEST_LEAF.to_vec()], b"sub".to_vec(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -1978,7 +1969,7 @@ fn batch_flagged_non_empty_subtree_deletion_is_refused() { ), QualifiedGroveDbOp::delete_op(vec![TEST_LEAF.to_vec()], b"sub".to_vec()), ], - batch_flag_on(), + None, None, grove_version, ) @@ -2002,7 +1993,7 @@ fn batch_flagged_non_empty_subtree_deletion_is_refused() { vec![TEST_LEAF.to_vec()], b"empty_sub".to_vec(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -2079,7 +2070,7 @@ fn batch_paired_upstream_updates_validate_against_pending_edges() { ops.reverse(); } batch_db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .unwrap() .unwrap_or_else(|e| { panic!( @@ -2133,7 +2124,7 @@ fn batch_paired_upstream_updates_validate_against_pending_edges() { ops.reverse(); } batch_db - .apply_batch(ops, batch_flag_on(), None, grove_version) + .apply_batch(ops, None, None, grove_version) .unwrap() .unwrap_or_else(|e| panic!("A detaches from B in the same batch; B's retarget is free (flip: {flip}): {e:?}")); assert!(batch_db @@ -2152,7 +2143,7 @@ fn batch_paired_upstream_updates_validate_against_pending_edges() { b"b".to_vec(), sibling_bidi(b"d", true), )], - batch_flag_on(), + None, None, grove_version, ) @@ -2223,8 +2214,7 @@ fn batch_skipped_conditional_does_not_relax_upstream_budget() { } assert!( matches!( - db.apply_batch(ops, batch_flag_on(), None, grove_version) - .unwrap(), + db.apply_batch(ops, None, None, grove_version).unwrap(), Err(Error::BidirectionalReferenceRule(_)) ), "a skipped conditional must not relax the stored budget (flip: {flip})" @@ -2306,7 +2296,7 @@ fn batch_detached_ancestor_frees_full_downstream_budget() { if flip { ops.reverse(); } - db.apply_batch(ops, batch_flag_on(), None, grove_version) + db.apply_batch(ops, None, None, grove_version) .unwrap() .unwrap_or_else(|e| { panic!("the detached A must not count against B's component (flip: {flip}): {e:?}") @@ -2358,7 +2348,7 @@ fn batch_enforces_declared_capacity() { b"r2".to_vec(), referrer(), )], - batch_flag_on(), + None, None, grove_version, ) @@ -2376,7 +2366,7 @@ fn batch_enforces_declared_capacity() { b"target".to_vec(), Element::new_item_allowing_bidirectional_references_with_capacity(b"w".to_vec(), 0), )], - batch_flag_on(), + None, None, grove_version, ) @@ -2400,7 +2390,7 @@ fn batch_enforces_declared_capacity() { referrer(), ), ], - batch_flag_on(), + None, None, grove_version, ) @@ -2431,7 +2421,7 @@ fn batch_enforces_declared_capacity() { b"r3".to_vec(), referrer(), )], - batch_flag_on(), + None, None, grove_version, ) diff --git a/grovedb/src/tests/batch_coverage_tests.rs b/grovedb/src/tests/batch_coverage_tests.rs index 19997de17..39cbb9ce4 100644 --- a/grovedb/src/tests/batch_coverage_tests.rs +++ b/grovedb/src/tests/batch_coverage_tests.rs @@ -19,6 +19,8 @@ mod tests { Element, Error, }; + use crate::DisplacedValue; + // =================================================================== // 3. Batch with Patch operation // =================================================================== @@ -502,6 +504,7 @@ mod tests { root_key: None, aggregate_data: AggregateData::NoAggregateData, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let options = Some(BatchApplyOptions { @@ -548,6 +551,7 @@ mod tests { not_summed: false, not_counted_or_summed: false, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let options = Some(BatchApplyOptions { @@ -593,6 +597,7 @@ mod tests { non_counted: false, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let options = Some(BatchApplyOptions { @@ -2057,6 +2062,7 @@ mod tests { chunk_power: 4, }, }, + displaced_value: DisplacedValue::MayBeParticipant, }; // With consistency check enabled (default), this should fail diff --git a/grovedb/src/tests/batch_rejection_tests.rs b/grovedb/src/tests/batch_rejection_tests.rs index b6427592f..b47c97f4f 100644 --- a/grovedb/src/tests/batch_rejection_tests.rs +++ b/grovedb/src/tests/batch_rejection_tests.rs @@ -23,7 +23,7 @@ use crate::{ key_info::KeyInfo::KnownKey, GroveOp, KeyInfoPath, NonMerkTreeMeta, QualifiedGroveDbOp, }, tests::{common::EMPTY_PATH, make_empty_grovedb}, - Element, Error, + DisplacedValue, Element, Error, }; #[test] @@ -39,6 +39,7 @@ fn test_apply_batch_rejects_replace_tree_root_key() { root_key: None, aggregate_data: AggregateData::NoAggregateData, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let result = db.apply_batch(vec![op], None, None, grove_version).value; @@ -79,6 +80,7 @@ fn test_apply_batch_rejects_insert_tree_with_root_hash() { not_summed: false, not_counted_or_summed: false, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let result = db.apply_batch(vec![op], None, None, grove_version).value; @@ -118,6 +120,7 @@ fn test_apply_batch_rejects_insert_non_merk_tree() { non_counted: false, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let result = db.apply_batch(vec![op], None, None, grove_version).value; @@ -204,6 +207,7 @@ fn test_apply_batch_replace_non_merk_tree_root_wrong_meta() { height: 3, }, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let result = db.apply_batch(vec![op], None, None, grove_version).value; diff --git a/grovedb/src/tests/batch_unit_tests.rs b/grovedb/src/tests/batch_unit_tests.rs index 3fdb8aa27..6d00161d8 100644 --- a/grovedb/src/tests/batch_unit_tests.rs +++ b/grovedb/src/tests/batch_unit_tests.rs @@ -12,11 +12,13 @@ mod tests { use grovedb_version::version::{v3::GROVE_V3, GroveVersion}; use crate::batch::key_info::KeyInfo::KnownKey; + use crate::batch::{ GroveOp, KeyInfoPath, NonMerkTreeMeta, QualifiedGroveDbOp, SubelementsDeletionBehavior, }; use crate::reference_path::ReferencePathType; use crate::tests::{common::EMPTY_PATH, make_empty_grovedb, make_test_grovedb, TEST_LEAF}; + use crate::DisplacedValue; use crate::{Element, Error}; // =================================================================== @@ -465,6 +467,7 @@ mod tests { root_key: None, aggregate_data: AggregateData::NoAggregateData, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let dbg = format!("{:?}", internal_op); assert!(dbg.contains("Replace Tree Hash and Root Key")); @@ -482,6 +485,7 @@ mod tests { not_summed: false, not_counted_or_summed: false, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let dbg = format!("{:?}", internal_op2); assert!(dbg.contains("Insert Tree Hash and Root Key")); @@ -492,6 +496,7 @@ mod tests { path: KeyInfoPath::from_known_path([b"p".as_ref()]), key: Some(KnownKey(b"k".to_vec())), op: GroveOp::ReplaceNonMerkTreeRoot { hash, meta }, + displaced_value: DisplacedValue::MayBeParticipant, }; let dbg = format!("{:?}", internal_op3); assert!(dbg.contains("Replace Non-Merk Tree Root")); @@ -513,6 +518,7 @@ mod tests { non_counted: false, }, + displaced_value: DisplacedValue::MayBeParticipant, }; let dbg = format!("{:?}", internal_op4); assert!(dbg.contains("Insert Non-Merk Tree")); diff --git a/grovedb/src/tests/bidirectional_references_tests.rs b/grovedb/src/tests/bidirectional_references_tests.rs index 57948e9b9..916606569 100644 --- a/grovedb/src/tests/bidirectional_references_tests.rs +++ b/grovedb/src/tests/bidirectional_references_tests.rs @@ -3,8 +3,9 @@ //! `bidirectional_references::handling`, and the query paths over the new //! element family. -use crate::batch::BatchApplyOptions; -use crate::BackwardReferencesPolicy; +use crate::operations::delete::ClearOptions; +use crate::DisplacedValue; +use crate::TransactionArg; use grovedb_path::SubtreePath; use grovedb_version::version::GroveVersion; @@ -19,7 +20,7 @@ use crate::{ fn flag_on() -> Option { Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }) } @@ -53,6 +54,72 @@ fn db_with_bwr_item() -> TempGroveDb { db } +/// A bidirectional reference to an absolute position, for a referrer that +/// lives in a different subtree than its target. +fn absolute_bidi(path: Vec>, cascade: bool, max_hop: Option) -> Element { + Element::BidirectionalReference( + BidirectionalReference { + forward_reference_path: ReferencePathType::AbsolutePathReference(path), + backward_references: Vec::new(), + cascade_on_update: cascade, + max_hop, + }, + None, + ) +} + +/// A fresh `tree` at `[parent.., name]` holding `key => element`. +fn nested( + db: &TempGroveDb, + parent: &[&[u8]], + name: &[u8], + tree: Element, + key: &[u8], + element: Element, + transaction: TransactionArg, + grove_version: &GroveVersion, +) { + db.insert(parent, name, tree, None, transaction, grove_version) + .unwrap() + .unwrap(); + let mut child: Vec<&[u8]> = parent.to_vec(); + child.push(name); + db.insert( + child.as_slice(), + key, + element, + None, + transaction, + grove_version, + ) + .unwrap() + .unwrap(); +} + +/// Empty the subtree at `path` through the one route that reads nothing and +/// is therefore trusted with a `NotParticipant` claim it cannot check: a raw +/// `clear_subtree`. Registrations pointing at, or held by, the removed +/// element are deliberately left stale; that inconsistency is what the +/// regressions below reproduce. +fn raw_clear( + db: &TempGroveDb, + path: &[&[u8]], + transaction: TransactionArg, + grove_version: &GroveVersion, +) { + db.clear_subtree( + path, + Some(ClearOptions { + check_for_subtrees: false, + displaced_value: DisplacedValue::NotParticipant, + ..Default::default() + }), + transaction, + grove_version, + ) + .unwrap(); +} + #[test] fn bidi_reference_must_target_backward_references_element() { let grove_version = GroveVersion::latest(); @@ -365,7 +432,7 @@ fn cascade_requires_opt_in() { &[TEST_LEAF], b"value", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -545,7 +612,7 @@ fn override_checks_apply_under_the_flag() { b"value", Element::new_item_allowing_bidirectional_references(b"nope".to_vec()), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, validate_insertion_does_not_override: true, ..Default::default() }), @@ -602,7 +669,7 @@ fn delete_with_flag_handles_trees() { &[TEST_LEAF], b"empty", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -638,7 +705,7 @@ fn delete_with_flag_handles_trees() { &[TEST_LEAF], b"full", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, allow_deleting_non_empty_trees: false, deleting_non_empty_trees_returns_error: true, ..Default::default() @@ -653,7 +720,7 @@ fn delete_with_flag_handles_trees() { &[TEST_LEAF], b"full", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, allow_deleting_non_empty_trees: false, deleting_non_empty_trees_returns_error: false, ..Default::default() @@ -897,29 +964,30 @@ fn sum_queries_resolve_backward_references_sum_items() { )); // …and an edge that falls out of budget AFTER insertion (its target - // evolved into a reference through an explicit Skip overwrite) hits the - // budget on the read. - db.insert( + // evolved into a reference behind a raw clear, the trusted route that + // reads nothing) hits the budget on the read. + nested( + &db, &[TEST_LEAF, b"sums"], + b"s3s", + Element::empty_sum_tree(), b"s3", Element::new_sum_item_allowing_bidirectional_references(9), None, - None, grove_version, - ) - .unwrap() - .unwrap(); + ); db.insert( &[TEST_LEAF, b"sums"], b"capped", - Element::BidirectionalReference( - BidirectionalReference { - forward_reference_path: ReferencePathType::SiblingReference(b"s3".to_vec()), - backward_references: Vec::new(), - cascade_on_update: true, - max_hop: Some(1), - }, - None, + absolute_bidi( + vec![ + TEST_LEAF.to_vec(), + b"sums".to_vec(), + b"s3s".to_vec(), + b"s3".to_vec(), + ], + true, + Some(1), ), None, None, @@ -927,14 +995,19 @@ fn sum_queries_resolve_backward_references_sum_items() { ) .unwrap() .unwrap(); + raw_clear(&db, &[TEST_LEAF, b"sums", b"s3s"], None, grove_version); db.insert( - &[TEST_LEAF, b"sums"], + &[TEST_LEAF, b"sums", b"s3s"], b"s3", - Element::new_reference_with_sum_item(ReferencePathType::SiblingReference(b"s".to_vec()), 0), - Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + Element::new_reference_with_sum_item( + ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"sums".to_vec(), + b"s".to_vec(), + ]), + 0, + ), + None, None, grove_version, ) @@ -1146,17 +1219,38 @@ fn reinserting_an_identical_edge_is_a_no_op() { #[test] fn propagation_skips_and_cleans_origins_removed_without_bookkeeping() { // An origin removed through a path that performs no backward-references - // bookkeeping (here: a batch delete, which is rejected only for ops - // CARRYING the element family, not for ops touching participants) - // leaves a dangling slot on its target. Later flagged updates must not - // fail on it: the slot is skipped and lazily cleaned. + // bookkeeping (here: a raw `clear_subtree` declared `NotParticipant`, + // the trusted claim that reads nothing) leaves a dangling slot on its + // target. Later flagged updates must not fail on it: the slot is + // skipped and lazily cleaned. let grove_version = GroveVersion::latest(); let db = db_with_bwr_item(); db.insert( &[TEST_LEAF], + b"origins", + Element::empty_tree(), + None, + None, + grove_version, + ) + .unwrap() + .unwrap(); + db.insert( + &[TEST_LEAF, b"origins"], b"origin", - sibling_bidi(b"value", true), + Element::BidirectionalReference( + BidirectionalReference { + forward_reference_path: ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"value".to_vec(), + ]), + backward_references: Vec::new(), + cascade_on_update: true, + max_hop: None, + }, + None, + ), None, None, grove_version, @@ -1164,20 +1258,16 @@ fn propagation_skips_and_cleans_origins_removed_without_bookkeeping() { .unwrap() .unwrap(); - // Batch-delete the origin: no backward-references bookkeeping runs. - db.apply_batch( - vec![crate::batch::QualifiedGroveDbOp::delete_op( - vec![TEST_LEAF.to_vec()], - b"origin".to_vec(), - )], - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, + // Raw-clear the origin's subtree: no backward-references bookkeeping runs. + db.clear_subtree( + &[TEST_LEAF, b"origins"], + Some(ClearOptions { + displaced_value: DisplacedValue::NotParticipant, ..Default::default() }), None, grove_version, ) - .unwrap() .unwrap(); // A flagged update of the target now encounters the dangling slot — @@ -1220,7 +1310,7 @@ fn delete_with_flag_rejects_rows_of_indexed_primaries() { &[TEST_LEAF, b"pcit"], b"row", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -1701,7 +1791,7 @@ fn flagged_inserts_enforce_tree_shape_guards() { let opts = Some(InsertOptions { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: true, - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }); assert!(matches!( @@ -1731,16 +1821,30 @@ fn flagged_inserts_enforce_tree_shape_guards() { .is_err()); } -/// Reads that follow a reference whose target was removed by an explicit Skip -/// write surface the dedicated corrupted-reference error. +/// Reads that follow a reference whose target was removed without +/// bookkeeping surface the dedicated corrupted-reference error. #[test] fn dangling_bidirectional_reference_reads_report_corruption() { let grove_version = GroveVersion::latest(); - let db = db_with_bwr_item(); + let db = make_test_grovedb(grove_version); + nested( + &db, + &[TEST_LEAF], + b"targets", + Element::empty_tree(), + b"value", + Element::new_item_allowing_bidirectional_references(b"hello".to_vec()), + None, + grove_version, + ); db.insert( &[TEST_LEAF], b"ref", - sibling_bidi(b"value", true), + absolute_bidi( + vec![TEST_LEAF.to_vec(), b"targets".to_vec(), b"value".to_vec()], + true, + None, + ), None, None, grove_version, @@ -1748,20 +1852,9 @@ fn dangling_bidirectional_reference_reads_report_corruption() { .unwrap() .unwrap(); - // Explicit Skip delete skips all bookkeeping — allowed, consistency - // forfeited. - db.delete( - &[TEST_LEAF], - b"value", - Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - grove_version, - ) - .unwrap() - .unwrap(); + // The target vanishes behind a raw clear, which reads nothing and so + // cannot maintain the referrer: consistency forfeited. + raw_clear(&db, &[TEST_LEAF, b"targets"], None, grove_version); assert!(matches!( db.get(&[TEST_LEAF], b"ref", None, grove_version).unwrap(), @@ -1798,30 +1891,19 @@ fn propagation_cleans_dangling_referrers_on_chained_references() { ) .unwrap() .unwrap(); - db.insert( + nested( + &db, &[TEST_LEAF], + b"heads", + Element::empty_tree(), b"a", - sibling_bidi(b"b", true), - None, + absolute_bidi(vec![TEST_LEAF.to_vec(), b"b".to_vec()], true, None), Some(&tx), grove_version, - ) - .unwrap() - .unwrap(); + ); // Remove the chain head without bookkeeping. - db.delete( - &[TEST_LEAF], - b"a", - Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - Some(&tx), - grove_version, - ) - .unwrap() - .unwrap(); + raw_clear(&db, &[TEST_LEAF, b"heads"], Some(&tx), grove_version); // Flagged update of the end target propagates through `b`, which finds // its referrer `a` dangling and lazily drops the entry. @@ -1864,12 +1946,7 @@ fn propagation_cleans_dangling_referrers_on_chained_references() { fn retargeting_tolerates_targets_rewritten_without_bookkeeping() { let grove_version = GroveVersion::latest(); let db = make_test_grovedb(grove_version); - for (key, value) in [ - (b"t1".as_slice(), b"one".as_slice()), - (b"t2", b"two"), - (b"t3", b"three"), - (b"t4", b"four"), - ] { + for (key, value) in [(b"t2".as_slice(), b"two".as_slice()), (b"t4", b"four")] { db.insert( &[TEST_LEAF], key, @@ -1881,10 +1958,36 @@ fn retargeting_tolerates_targets_rewritten_without_bookkeeping() { .unwrap() .unwrap(); } + // t1 and t3 live in their own subtrees so they can be rewritten behind + // the raw clear, the trusted route that reads nothing. + nested( + &db, + &[TEST_LEAF], + b"t1s", + Element::empty_tree(), + b"t1", + Element::new_item_allowing_bidirectional_references(b"one".to_vec()), + None, + grove_version, + ); + nested( + &db, + &[TEST_LEAF], + b"t3s", + Element::empty_tree(), + b"t3", + Element::new_item_allowing_bidirectional_references(b"three".to_vec()), + None, + grove_version, + ); db.insert( &[TEST_LEAF], b"r1", - sibling_bidi(b"t1", true), + absolute_bidi( + vec![TEST_LEAF.to_vec(), b"t1s".to_vec(), b"t1".to_vec()], + true, + None, + ), None, None, grove_version, @@ -1894,7 +1997,11 @@ fn retargeting_tolerates_targets_rewritten_without_bookkeeping() { db.insert( &[TEST_LEAF], b"r2", - sibling_bidi(b"t3", true), + absolute_bidi( + vec![TEST_LEAF.to_vec(), b"t3s".to_vec(), b"t3".to_vec()], + true, + None, + ), None, None, grove_version, @@ -1902,16 +2009,14 @@ fn retargeting_tolerates_targets_rewritten_without_bookkeeping() { .unwrap() .unwrap(); - // t1 overwritten by a PLAIN item (no backward-references support) via - // an explicit Skip write; retargeting r1 finds nothing to clean. + // t1 rewritten as a PLAIN item (no backward-references support) behind + // the raw clear; retargeting r1 finds nothing to clean. + raw_clear(&db, &[TEST_LEAF, b"t1s"], None, grove_version); db.insert( - &[TEST_LEAF], + &[TEST_LEAF, b"t1s"], b"t1", Element::new_item(b"plain".to_vec()), - Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, None, grove_version, ) @@ -1928,16 +2033,14 @@ fn retargeting_tolerates_targets_rewritten_without_bookkeeping() { .unwrap() .unwrap(); - // t3 overwritten by a FRESH backward-references item (empty referrer - // list) via an explicit Skip write; retargeting r2 finds its entry gone. + // t3 rewritten as a FRESH backward-references item (empty referrer + // list) behind the raw clear; retargeting r2 finds its entry gone. + raw_clear(&db, &[TEST_LEAF, b"t3s"], None, grove_version); db.insert( - &[TEST_LEAF], + &[TEST_LEAF, b"t3s"], b"t3", Element::new_item_allowing_bidirectional_references(b"fresh".to_vec()), - Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, None, grove_version, ) @@ -3000,31 +3103,30 @@ fn per_edge_max_hop_is_enforced_on_reads() { )); // An edge can still fall OUT of budget after insertion: head points at - // a family item within budget, then an explicit Skip overwrite turns that - // target into a plain reference — reads must now hit the budget. - db.insert( + // a family item within budget, then that target evolves into a plain + // reference behind a raw clear (the trusted route that reads nothing); + // reads must now hit the budget. + nested( + &db, &[TEST_LEAF], + b"mids", + Element::empty_tree(), b"mid_evolved", Element::new_item_allowing_bidirectional_references(b"m".to_vec()), None, - None, grove_version, - ) - .unwrap() - .unwrap(); + ); db.insert( &[TEST_LEAF], b"head", - Element::BidirectionalReference( - BidirectionalReference { - forward_reference_path: ReferencePathType::SiblingReference( - b"mid_evolved".to_vec(), - ), - backward_references: Vec::new(), - cascade_on_update: true, - max_hop: Some(1), - }, - None, + absolute_bidi( + vec![ + TEST_LEAF.to_vec(), + b"mids".to_vec(), + b"mid_evolved".to_vec(), + ], + true, + Some(1), ), None, None, @@ -3032,14 +3134,15 @@ fn per_edge_max_hop_is_enforced_on_reads() { ) .unwrap() .unwrap(); + raw_clear(&db, &[TEST_LEAF, b"mids"], None, grove_version); db.insert( - &[TEST_LEAF], + &[TEST_LEAF, b"mids"], b"mid_evolved", - Element::new_reference(ReferencePathType::SiblingReference(b"value".to_vec())), - Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + Element::new_reference(ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"value".to_vec(), + ])), + None, None, grove_version, ) @@ -3236,7 +3339,7 @@ fn automatic_delete_handles_specialized_descendants() { let options = DeleteOptions { allow_deleting_non_empty_trees: true, deleting_non_empty_trees_returns_error: false, - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }; db.delete(&[TEST_LEAF], b"outer", Some(options), None, grove_version) @@ -3627,10 +3730,7 @@ fn bidi_insert_rejects_undersized_max_hop() { None, ), )], - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, - ..Default::default() - }), + None, None, grove_version, ) @@ -3647,27 +3747,26 @@ fn bidi_insert_rejects_undersized_max_hop() { fn proof_generation_respects_bidi_max_hop() { let grove_version = GroveVersion::latest(); let db = db_with_bwr_item(); - db.insert( + // head -> mid within budget, then mid evolves into a plain reference + // behind a raw clear (the trusted route that reads nothing): the chain + // now needs two hops. + nested( + &db, &[TEST_LEAF], + b"mids", + Element::empty_tree(), b"mid", Element::new_item_allowing_bidirectional_references(b"m".to_vec()), None, - None, grove_version, - ) - .unwrap() - .unwrap(); + ); db.insert( &[TEST_LEAF], b"head", - Element::BidirectionalReference( - BidirectionalReference { - forward_reference_path: ReferencePathType::SiblingReference(b"mid".to_vec()), - backward_references: Vec::new(), - cascade_on_update: true, - max_hop: Some(1), - }, - None, + absolute_bidi( + vec![TEST_LEAF.to_vec(), b"mids".to_vec(), b"mid".to_vec()], + true, + Some(1), ), None, None, @@ -3675,14 +3774,15 @@ fn proof_generation_respects_bidi_max_hop() { ) .unwrap() .unwrap(); + raw_clear(&db, &[TEST_LEAF, b"mids"], None, grove_version); db.insert( - &[TEST_LEAF], + &[TEST_LEAF, b"mids"], b"mid", - Element::new_reference(ReferencePathType::SiblingReference(b"value".to_vec())), - Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + Element::new_reference(ReferencePathType::AbsolutePathReference(vec![ + TEST_LEAF.to_vec(), + b"value".to_vec(), + ])), + None, None, grove_version, ) @@ -3924,10 +4024,7 @@ fn retarget_rejects_upstream_max_hop_violation() { b"b".to_vec(), sibling_bidi(b"d", true), )], - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, - ..Default::default() - }), + None, None, grove_version, ) @@ -3966,7 +4063,7 @@ fn cascade_removes_the_physical_referrer_record() { &[TEST_LEAF], b"value", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -4084,7 +4181,7 @@ fn cascade_forwards_the_sectioned_removal_callback() { SubtreePath::from(&[TEST_LEAF]), b"value", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, @@ -4117,43 +4214,32 @@ fn cascade_forwards_the_sectioned_removal_callback() { ); } -/// A registration whose referrer was removed by an explicit Skip write is stale +/// A registration whose referrer was removed without bookkeeping is stale /// bookkeeping: it must neither require consent nor block deleting its /// former target. #[test] fn stale_nonconsenting_registration_does_not_block_target_deletion() { let grove_version = GroveVersion::latest(); let db = db_with_bwr_item(); - db.insert( + nested( + &db, &[TEST_LEAF], + b"refs", + Element::empty_tree(), b"ref", - sibling_bidi(b"value", false), - flag_on(), + absolute_bidi(vec![TEST_LEAF.to_vec(), b"value".to_vec()], false, None), None, grove_version, - ) - .unwrap() - .unwrap(); - // Explicit Skip delete of the non-consenting referrer leaves its + ); + // Removing the non-consenting referrer behind a raw clear leaves its // registration dangling on `value`. - db.delete( - &[TEST_LEAF], - b"ref", - Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - grove_version, - ) - .unwrap() - .unwrap(); + raw_clear(&db, &[TEST_LEAF, b"refs"], None, grove_version); db.delete( &[TEST_LEAF], b"value", Some(DeleteOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), None, diff --git a/grovedb/src/tests/clear_append_tree_tests.rs b/grovedb/src/tests/clear_append_tree_tests.rs index 4b61e432a..79b5f6041 100644 --- a/grovedb/src/tests/clear_append_tree_tests.rs +++ b/grovedb/src/tests/clear_append_tree_tests.rs @@ -15,7 +15,7 @@ //! commitment stale, nothing propagated — is pinned for replay //! compatibility. -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use grovedb_version::version::{v3::GROVE_V3, GroveVersion}; use crate::{ @@ -561,7 +561,7 @@ fn clear_ordinary_tree_with_subtrees_under_grove_v3_option_branches() { .clear_subtree( [b"tree".as_ref()].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: false, trying_to_clear_with_subtrees_returns_error: false, @@ -577,7 +577,7 @@ fn clear_ordinary_tree_with_subtrees_under_grove_v3_option_branches() { .clear_subtree( [b"tree".as_ref()].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: true, trying_to_clear_with_subtrees_returns_error: false, diff --git a/grovedb/src/tests/commitment_tree_cost_bound_tests.rs b/grovedb/src/tests/commitment_tree_cost_bound_tests.rs index f9ed819b8..96e7898d3 100644 --- a/grovedb/src/tests/commitment_tree_cost_bound_tests.rs +++ b/grovedb/src/tests/commitment_tree_cost_bound_tests.rs @@ -101,7 +101,6 @@ fn try_average_case_estimate( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, None), }, @@ -111,7 +110,6 @@ fn try_average_case_estimate( KeyInfoPath::from_known_owned_path(vec![b"pool".to_vec()]), EstimatedLayerInformation { tree_type: TreeType::CommitmentTree(chunk_power), - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(16, false), estimated_layer_sizes: AllItems(8, 312, None), }, @@ -554,7 +552,6 @@ fn test_commitment_tree_insert_estimated_covers_actual_with_large_flags() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(1, false), estimated_layer_sizes: AllSubtrees(4, NoSumTrees, Some(FLAGS_LEN as u32)), }, @@ -563,7 +560,6 @@ fn test_commitment_tree_insert_estimated_covers_actual_with_large_flags() { KeyInfoPath::from_known_owned_path(vec![b"pool".to_vec()]), EstimatedLayerInformation { tree_type: TreeType::CommitmentTree(CHUNK_POWER), - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(16, false), estimated_layer_sizes: AllItems(8, 312, None), }, diff --git a/grovedb/src/tests/coverage_misc_tests.rs b/grovedb/src/tests/coverage_misc_tests.rs index a97be8f39..9f0cd1e40 100644 --- a/grovedb/src/tests/coverage_misc_tests.rs +++ b/grovedb/src/tests/coverage_misc_tests.rs @@ -274,7 +274,6 @@ mod tests { let primary_path = KeyInfoPath::from_known_path([b"primary".as_ref()]); let layer = EstimatedLayerInformation { tree_type: TreeType::ProvableCountProvableSumIndexedTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(100), estimated_layer_sizes: sizes, }; diff --git a/grovedb/src/tests/delete_cost_estimation_tests.rs b/grovedb/src/tests/delete_cost_estimation_tests.rs index a09dcb630..86d0ae798 100644 --- a/grovedb/src/tests/delete_cost_estimation_tests.rs +++ b/grovedb/src/tests/delete_cost_estimation_tests.rs @@ -21,7 +21,6 @@ use crate::{ fn normal_layer_info() -> EstimatedLayerInformation { EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(100), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees(8, Default::default(), None), } diff --git a/grovedb/src/tests/delete_indexed_tree_tests.rs b/grovedb/src/tests/delete_indexed_tree_tests.rs index 8fa7a5675..91fc47101 100644 --- a/grovedb/src/tests/delete_indexed_tree_tests.rs +++ b/grovedb/src/tests/delete_indexed_tree_tests.rs @@ -13,7 +13,7 @@ #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_element::indexed::IndexAxis; use grovedb_path::SubtreePath; use grovedb_storage::{ @@ -86,7 +86,7 @@ mod tests { deleting_non_empty_trees_returns_error: false, base_root_storage_is_free: true, validate_tree_at_path_exists: false, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, } } diff --git a/grovedb/src/tests/direct_insert_indexed_tests.rs b/grovedb/src/tests/direct_insert_indexed_tests.rs index f24b33e40..c8e12a936 100644 --- a/grovedb/src/tests/direct_insert_indexed_tests.rs +++ b/grovedb/src/tests/direct_insert_indexed_tests.rs @@ -12,7 +12,7 @@ #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_element::indexed::IndexAxis; use grovedb_version::version::GroveVersion; @@ -520,7 +520,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, } } diff --git a/grovedb/src/tests/estimated_costs_average_case_tests.rs b/grovedb/src/tests/estimated_costs_average_case_tests.rs index 6201de8e2..79bd6ff38 100644 --- a/grovedb/src/tests/estimated_costs_average_case_tests.rs +++ b/grovedb/src/tests/estimated_costs_average_case_tests.rs @@ -24,7 +24,6 @@ use crate::{ fn normal_layer_info() -> EstimatedLayerInformation { EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(100), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees(8, Default::default(), None), } @@ -34,7 +33,6 @@ fn normal_layer_info() -> EstimatedLayerInformation { fn sum_tree_layer_info() -> EstimatedLayerInformation { EstimatedLayerInformation { tree_type: TreeType::SumTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(50), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees(8, Default::default(), None), } @@ -44,7 +42,6 @@ fn sum_tree_layer_info() -> EstimatedLayerInformation { fn items_layer_info() -> EstimatedLayerInformation { EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(200), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 100, None), } @@ -674,7 +671,6 @@ fn test_average_case_replace_tree_sum_vs_normal() { fn indexed_layer_info(tt: TreeType) -> EstimatedLayerInformation { EstimatedLayerInformation { tree_type: tt, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(50), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees(8, Default::default(), None), } diff --git a/grovedb/src/tests/flat_drop_tests.rs b/grovedb/src/tests/flat_drop_tests.rs index dc67dd6b6..b1a49d014 100644 --- a/grovedb/src/tests/flat_drop_tests.rs +++ b/grovedb/src/tests/flat_drop_tests.rs @@ -20,8 +20,8 @@ //! corrupted-reference error rather than resolving to stale data. mod tests { - use crate::batch::BatchApplyOptions; - use crate::BackwardReferencesPolicy; + + use crate::DisplacedValue; use grovedb_costs::OperationCost; use grovedb_merk::tree_type::TreeType; use grovedb_storage::{ @@ -123,7 +123,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, grove_version, ) @@ -154,7 +154,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, grove_version, ) @@ -182,11 +182,9 @@ mod tests { b"flat".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::DropFlat, - )], - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + ) + .with_displaced_value(DisplacedValue::NotParticipant)], + None, None, version, ) @@ -215,7 +213,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"item", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, grove_version ) @@ -234,7 +232,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, &GROVE_V3 ) @@ -261,7 +259,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, Some(&tx), grove_version, ) @@ -308,7 +306,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, Some(&tx), grove_version, ) @@ -343,7 +341,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, Some(&tx), grove_version, ) @@ -376,7 +374,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, Some(&tx), grove_version, ) @@ -484,7 +482,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"cidx", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, grove_version, ) @@ -552,7 +550,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF, b"sums"].as_ref(), b"flat_sums", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, grove_version, ) @@ -588,19 +586,12 @@ mod tests { b"flat".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::DropFlat, - ), + ) + .with_displaced_value(DisplacedValue::NotParticipant), ]; - db.apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - grove_version, - ) - .unwrap() - .expect("apply batch"); + db.apply_batch(ops, None, None, grove_version) + .unwrap() + .expect("apply batch"); assert!(matches!( db.get_raw([TEST_LEAF].as_ref().into(), b"flat", None, grove_version) @@ -632,18 +623,11 @@ mod tests { b"flat".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::DropFlat, - )]; - db.apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - Some(&tx), - grove_version, ) - .unwrap() - .expect("apply batch in tx"); + .with_displaced_value(DisplacedValue::NotParticipant)]; + db.apply_batch(ops, None, Some(&tx), grove_version) + .unwrap() + .expect("apply batch in tx"); db.commit_transaction(tx).unwrap().expect("commit"); assert!(data_namespace_key_count(&db, prefix) > 0); @@ -665,18 +649,10 @@ mod tests { b"flat".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::DropFlat, - )]; + ) + .with_displaced_value(DisplacedValue::NotParticipant)]; assert!(matches!( - db.apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - &GROVE_V3 - ) - .unwrap(), + db.apply_batch(ops, None, None, &GROVE_V3).unwrap(), Err(Error::VersionError(_)) )); assert!(db @@ -720,18 +696,11 @@ mod tests { b"cidx".to_vec(), TreeType::ProvableCountIndexedTree, SubelementsDeletionBehavior::DropFlat, - )]; - db.apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - grove_version, ) - .unwrap() - .expect("apply batch"); + .with_displaced_value(DisplacedValue::NotParticipant)]; + db.apply_batch(ops, None, None, grove_version) + .unwrap() + .expect("apply batch"); assert_grove_verifies(&db, grove_version); assert_eq!(data_namespace_key_count(&db, primary_prefix), 0); @@ -750,18 +719,11 @@ mod tests { b"flat".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::DropFlat, - )]; - db.apply_operations_without_batching( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - grove_version, ) - .unwrap() - .expect("apply without batching"); + .with_displaced_value(DisplacedValue::NotParticipant)]; + db.apply_operations_without_batching(ops, None, None, grove_version) + .unwrap() + .expect("apply without batching"); assert!(matches!( db.get_raw([TEST_LEAF].as_ref().into(), b"flat", None, grove_version) @@ -804,7 +766,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, grove_version, ) @@ -860,7 +822,7 @@ mod tests { db.drop_flat_subtree( [TEST_LEAF].as_ref(), b"flat", - BackwardReferencesPolicy::Skip, + DisplacedValue::NotParticipant, None, grove_version, ) @@ -922,14 +884,14 @@ mod tests { b"flat".to_vec(), TreeType::NormalTree, SubelementsDeletionBehavior::DropFlat, - )]; + ) + .with_displaced_value(DisplacedValue::NotParticipant)]; let mut average_paths = HashMap::new(); average_paths.insert( KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(2, false), estimated_layer_sizes: AllSubtrees(32, NoSumTrees, None), }, @@ -938,7 +900,6 @@ mod tests { KeyInfoPath::from_known_owned_path(vec![TEST_LEAF.to_vec()]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLevel(4, false), estimated_layer_sizes: AllSubtrees(32, NoSumTrees, None), }, @@ -946,10 +907,7 @@ mod tests { let average = GroveDb::estimated_case_operations_for_batch( AverageCaseCostsType(average_paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -968,10 +926,7 @@ mod tests { let worst = GroveDb::estimated_case_operations_for_batch( WorstCaseCostsType(worst_paths), ops.clone(), - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), + None, |_cost, _old_flags, _new_flags| Ok(false), |_flags, _removed_key_bytes, _removed_value_bytes| { Ok((NoStorageRemoval, NoStorageRemoval)) @@ -982,15 +937,7 @@ mod tests { .expect("worst case estimate"); let actual = db - .apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - grove_version, - ) + .apply_batch(ops, None, None, grove_version) .cost_as_result() .expect("apply batch"); diff --git a/grovedb/src/tests/misc_coverage_tests.rs b/grovedb/src/tests/misc_coverage_tests.rs index 5e7c815a4..45a977f1c 100644 --- a/grovedb/src/tests/misc_coverage_tests.rs +++ b/grovedb/src/tests/misc_coverage_tests.rs @@ -519,7 +519,6 @@ fn wipe_db() { fn avg_items_layer_info() -> EstimatedLayerInformation { EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(100), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, None), } @@ -612,7 +611,6 @@ fn batch_average_case_mixed_operations_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(50), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -626,7 +624,6 @@ fn batch_average_case_mixed_operations_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"new_tree".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(0), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -772,7 +769,6 @@ fn batch_worst_case_gte_average_case() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(100), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, None), }, @@ -864,7 +860,6 @@ fn batch_average_case_insert_tree_cost_actual_comparison() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(0), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 7, @@ -1410,7 +1405,6 @@ fn batch_average_case_sum_tree_insert() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 12, @@ -1433,7 +1427,6 @@ fn batch_average_case_sum_tree_insert() { KeyInfoPath(vec![KeyInfo::KnownKey(b"sum_tree_key".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::SumTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(0), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 8, None), }, @@ -1543,7 +1536,6 @@ fn batch_average_case_delete_tree_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(20), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 10, @@ -1556,7 +1548,6 @@ fn batch_average_case_delete_tree_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(20), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 10, @@ -1862,7 +1853,6 @@ fn batch_average_case_replace_with_tree() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -1875,7 +1865,6 @@ fn batch_average_case_replace_with_tree() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -1915,7 +1904,6 @@ fn batch_average_case_patch_item_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -1928,7 +1916,6 @@ fn batch_average_case_patch_item_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(50), estimated_layer_sizes: EstimatedLayerSizes::AllItems(9, 32, None), }, @@ -1970,7 +1957,6 @@ fn batch_average_case_refresh_reference_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -1983,7 +1969,6 @@ fn batch_average_case_refresh_reference_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(50), estimated_layer_sizes: EstimatedLayerSizes::AllItems(7, 64, None), }, @@ -2018,7 +2003,6 @@ fn batch_average_case_replace_sum_item_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -2041,7 +2025,6 @@ fn batch_average_case_replace_sum_item_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::SumTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(100), estimated_layer_sizes: EstimatedLayerSizes::AllItems(32, 8, None), }, @@ -2076,7 +2059,6 @@ fn batch_average_case_replace_sum_tree_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -2099,7 +2081,6 @@ fn batch_average_case_replace_sum_tree_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(20), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 16, @@ -2150,7 +2131,6 @@ fn batch_average_case_insert_into_sum_tree() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(5), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 10, @@ -2173,7 +2153,6 @@ fn batch_average_case_insert_into_sum_tree() { KeyInfoPath(vec![KeyInfo::KnownKey(b"sum_parent".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::SumTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(0), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -2448,7 +2427,6 @@ fn batch_average_case_delete_sum_tree_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -2471,7 +2449,6 @@ fn batch_average_case_delete_sum_tree_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(20), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 16, @@ -2904,7 +2881,6 @@ fn batch_average_case_insert_count_tree_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 14, @@ -2927,7 +2903,6 @@ fn batch_average_case_insert_count_tree_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"count_tree_key".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::CountTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(0), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, None), }, @@ -3004,7 +2979,6 @@ fn batch_average_case_insert_only_item_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(5), estimated_layer_sizes: EstimatedLayerSizes::AllItems(15, 17, None), }, @@ -3086,7 +3060,6 @@ fn batch_average_case_insert_if_not_exists_item_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(5), estimated_layer_sizes: EstimatedLayerSizes::AllItems(15, 17, None), }, @@ -3211,7 +3184,6 @@ fn batch_average_case_delete_in_subtree_cost() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 4, @@ -3224,7 +3196,6 @@ fn batch_average_case_delete_in_subtree_cost() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(50), estimated_layer_sizes: EstimatedLayerSizes::AllItems(14, 32, None), }, @@ -3265,7 +3236,6 @@ fn batch_average_case_insert_reference_element() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(5), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 4, @@ -3278,7 +3248,6 @@ fn batch_average_case_insert_reference_element() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(20), estimated_layer_sizes: EstimatedLayerSizes::AllItems(11, 64, None), }, @@ -3316,7 +3285,6 @@ fn batch_average_case_replace_item_with_flags() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 4, @@ -3329,7 +3297,6 @@ fn batch_average_case_replace_item_with_flags() { KeyInfoPath(vec![KeyInfo::KnownKey(b"leaf".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(50), estimated_layer_sizes: EstimatedLayerSizes::AllItems(12, 32, Some(7)), }, @@ -3476,7 +3443,6 @@ fn batch_average_case_insert_big_sum_tree() { KeyInfoPath(vec![]), EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(10), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 7, @@ -3499,7 +3465,6 @@ fn batch_average_case_insert_big_sum_tree() { KeyInfoPath(vec![KeyInfo::KnownKey(b"big_sum".to_vec())]), EstimatedLayerInformation { tree_type: TreeType::BigSumTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::ApproximateElements(0), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, None), }, diff --git a/grovedb/src/tests/mod.rs b/grovedb/src/tests/mod.rs index b8df68f0f..919359748 100644 --- a/grovedb/src/tests/mod.rs +++ b/grovedb/src/tests/mod.rs @@ -1175,7 +1175,8 @@ pub fn make_deep_tree_with_sum_trees_mixed_with_items(grove_version: &GroveVersi } mod general_tests { - use crate::batch::BatchApplyOptions; + + use crate::DisplacedValue; use batch::QualifiedGroveDbOp; use grovedb_merk::{ element::get::ElementFetchFromStorageExtensions, proofs::query::SubqueryBranch, @@ -4977,7 +4978,7 @@ mod general_tests { None, ), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), Some(&transaction), @@ -4999,7 +5000,7 @@ mod general_tests { None, ), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), Some(&transaction), @@ -5021,7 +5022,7 @@ mod general_tests { None, ), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), Some(&transaction), @@ -5041,7 +5042,7 @@ mod general_tests { b"value", Element::new_item_allowing_bidirectional_references(b"not hello >:(".to_vec()), Some(InsertOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, ..Default::default() }), Some(&transaction), @@ -5092,12 +5093,12 @@ mod general_tests { } } - /// Without opt-in bookkeeping, every batch entry point rejects the - /// backward-references family and leaves preceding valid writes uncommitted. + /// Partial batches cannot plan reference maintenance, so both of their + /// segments reject the backward-references family and leave preceding + /// valid writes uncommitted; a full batch plans the family by default. #[test] - fn backward_references_elements_rejected_in_batches() { + fn backward_references_elements_rejected_in_partial_batches() { let grove_version = GroveVersion::latest(); - let db = make_test_grovedb(grove_version); let elements = [ Element::new_item_allowing_bidirectional_references(b"v".to_vec()), @@ -5114,6 +5115,7 @@ mod general_tests { ]; for element in elements { + let db = make_test_grovedb(grove_version); let valid_op = |key: &[u8]| { QualifiedGroveDbOp::insert_or_replace_op( vec![TEST_LEAF.to_vec()], @@ -5121,28 +5123,17 @@ mod general_tests { Element::new_item(b"valid".to_vec()), ) }; - let rejected_op = QualifiedGroveDbOp::insert_or_replace_op( + let family_op = QualifiedGroveDbOp::insert_or_replace_op( vec![TEST_LEAF.to_vec()], b"k".to_vec(), element.clone(), ); - // Exercise ordinary batch rejection, initial partial rejection, - // and rejection after a successful initial partial segment. - for phase in ["ordinary", "initial", "continuation"] { + // Exercise initial partial rejection and rejection after a + // successful initial partial segment. + for phase in ["initial", "continuation"] { let before = db.root_hash(None, grove_version).unwrap().unwrap(); - let ops = vec![valid_op(b"a_valid"), rejected_op.clone()]; + let ops = vec![valid_op(b"a_valid"), family_op.clone()]; let result = match phase { - "ordinary" => db - .apply_batch( - ops, - Some(BatchApplyOptions { - backward_references_policy: BackwardReferencesPolicy::Skip, - ..Default::default() - }), - None, - grove_version, - ) - .unwrap(), "initial" => db .apply_partial_batch( ops, @@ -5182,6 +5173,38 @@ mod general_tests { .unwrap() .is_empty()); } + + // A full batch plans the family by default: the item registers, + // while the sum item (TEST_LEAF is not a sum tree) and the + // reference (its target `x` does not exist) are held to their + // own rules, never refused as unsupported. + let result = db + .apply_batch( + vec![valid_op(b"a_valid"), family_op], + None, + None, + grove_version, + ) + .unwrap(); + if !matches!(element, Element::ItemWithBackwardsReferences(..)) { + assert!( + !matches!(result, Ok(()) | Err(Error::NotSupported(_))), + "the family payload is refused by its own rules, not as unsupported: \ + {result:?}" + ); + } else { + result.expect("a full batch plans the family by default"); + assert_eq!( + db.get([TEST_LEAF].as_ref(), b"k", None, grove_version) + .unwrap() + .unwrap(), + element + ); + } + assert!(db + .verify_grovedb(None, true, true, grove_version) + .unwrap() + .is_empty()); } } } diff --git a/grovedb/src/tests/nested_indexed_secondary_cleanup_tests.rs b/grovedb/src/tests/nested_indexed_secondary_cleanup_tests.rs index be3fd2fe6..884d8a83d 100644 --- a/grovedb/src/tests/nested_indexed_secondary_cleanup_tests.rs +++ b/grovedb/src/tests/nested_indexed_secondary_cleanup_tests.rs @@ -101,14 +101,15 @@ mod tests { ), }; result.value.expect("delete child"); - // V4 Maintain includes the participant scan: +312 loaded bytes - // for direct delete; +2 seeks, +467 loaded bytes and +1 hash for - // full/partial batches. Keep these default costs visible. + // A V4 direct delete under the default `MayBeParticipant` scans + // the subtree for participants to maintain (+312 loaded bytes). + // Batches check the claim on the cleanup walk they make anyway, + // so their V4 cost is the plain removal. Keep these visible. let (seek_count, storage_loaded_bytes, hash_node_calls) = match route { DeleteRoute::Direct if gv.protocol_version <= 3 => (21, 2415, 14), DeleteRoute::Direct => (20, 2652, 13), _ if gv.protocol_version <= 3 => (13, 1135, 8), - _ => (18, 2466, 13), + _ => (16, 1999, 12), }; let expected_cost = OperationCost { seek_count, diff --git a/grovedb/src/tests/operations_coverage_tests.rs b/grovedb/src/tests/operations_coverage_tests.rs index e3190bc66..1188cb3f5 100644 --- a/grovedb/src/tests/operations_coverage_tests.rs +++ b/grovedb/src/tests/operations_coverage_tests.rs @@ -3,7 +3,7 @@ #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_merk::proofs::{query::query_item::QueryItem, Query}; use grovedb_version::version::GroveVersion; @@ -555,7 +555,7 @@ mod tests { .clear_subtree( [TEST_LEAF, b"to_clear"].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: false, trying_to_clear_with_subtrees_returns_error: true, @@ -1562,7 +1562,7 @@ mod tests { validate_insertion_does_not_override: true, validate_insertion_does_not_override_tree: true, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), None, grove_version, @@ -1615,7 +1615,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: true, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), None, grove_version, @@ -2603,7 +2603,7 @@ mod tests { let result = db.clear_subtree( [TEST_LEAF, b"parent"].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: false, trying_to_clear_with_subtrees_returns_error: true, @@ -2653,7 +2653,7 @@ mod tests { .clear_subtree( [TEST_LEAF, b"parent2"].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: false, trying_to_clear_with_subtrees_returns_error: false, @@ -2713,7 +2713,7 @@ mod tests { .clear_subtree( [TEST_LEAF, b"parent3"].as_ref(), Some(ClearOptions { - backward_references_policy: BackwardReferencesPolicy::Maintain, + displaced_value: DisplacedValue::MayBeParticipant, check_for_subtrees: true, allow_deleting_subtrees: true, trying_to_clear_with_subtrees_returns_error: true, @@ -3172,7 +3172,7 @@ mod tests { validate_insertion_does_not_override: true, validate_insertion_does_not_override_tree: true, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), None, grove_version, @@ -3215,7 +3215,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: true, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), None, grove_version, @@ -3248,7 +3248,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: false, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), None, grove_version, diff --git a/grovedb/src/tests/ordinary_replacement_cost_tests.rs b/grovedb/src/tests/ordinary_replacement_cost_tests.rs index 5eb1bdb6b..07b84c46d 100644 --- a/grovedb/src/tests/ordinary_replacement_cost_tests.rs +++ b/grovedb/src/tests/ordinary_replacement_cost_tests.rs @@ -25,7 +25,7 @@ //! //! GROVE_V3 is live, so its legacy figures are pinned here. -use crate::BackwardReferencesPolicy; +use crate::DisplacedValue; use grovedb_costs::{ storage_cost::{ removal::StorageRemovedBytes::{BasicStorageRemoval, NoStorageRemoval}, @@ -68,7 +68,7 @@ fn overwrite_options() -> InsertOptions { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, } } @@ -113,7 +113,8 @@ fn replace_cost( vec![b"tree".to_vec()], KEY.to_vec(), after, - )], + ) + .with_displaced_value(DisplacedValue::NotParticipant)], None, Some(&tx), grove_version, diff --git a/grovedb/src/tests/partial_batch_consistency_tests.rs b/grovedb/src/tests/partial_batch_consistency_tests.rs index 92eb45670..458f3bedf 100644 --- a/grovedb/src/tests/partial_batch_consistency_tests.rs +++ b/grovedb/src/tests/partial_batch_consistency_tests.rs @@ -25,6 +25,8 @@ mod tests { Element, Error, }; + use crate::DisplacedValue; + // =================================================================== // 1. Callback returning duplicate operations should be rejected // =================================================================== @@ -112,6 +114,7 @@ mod tests { root_key: None, aggregate_data: AggregateData::NoAggregateData, }, + displaced_value: DisplacedValue::MayBeParticipant, }; Ok(vec![internal_op]) }, diff --git a/grovedb/src/tests/provable_count_indexed_tree_tests.rs b/grovedb/src/tests/provable_count_indexed_tree_tests.rs index 3562438d8..29cc8ea85 100644 --- a/grovedb/src/tests/provable_count_indexed_tree_tests.rs +++ b/grovedb/src/tests/provable_count_indexed_tree_tests.rs @@ -19,7 +19,7 @@ #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_version::version::GroveVersion; use crate::IndexedAxisEntrySliceExt; @@ -1553,7 +1553,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }; db.insert( [TEST_LEAF].as_ref(), @@ -1610,7 +1610,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }; let res = db .insert([TEST_LEAF].as_ref(), b"pcit", tampered, Some(opts), None, v) @@ -1910,7 +1910,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }); let result = db .insert( diff --git a/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs b/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs index 3e15ca3b1..414d09dc7 100644 --- a/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs +++ b/grovedb/src/tests/provable_count_provable_sum_indexed_tree_tests.rs @@ -16,7 +16,7 @@ #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_element::indexed::IndexAxis; use grovedb_version::version::GroveVersion; @@ -2058,7 +2058,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }; db.insert( [TEST_LEAF].as_ref(), @@ -2111,7 +2111,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }; let res = db .insert( @@ -2492,7 +2492,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }); let result = db .insert( diff --git a/grovedb/src/tests/provable_sum_indexed_tree_tests.rs b/grovedb/src/tests/provable_sum_indexed_tree_tests.rs index 9723bee19..19b1d2e7e 100644 --- a/grovedb/src/tests/provable_sum_indexed_tree_tests.rs +++ b/grovedb/src/tests/provable_sum_indexed_tree_tests.rs @@ -16,7 +16,7 @@ #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_version::version::GroveVersion; use crate::IndexedAxisEntrySliceExt; @@ -1578,7 +1578,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }; db.insert( [TEST_LEAF].as_ref(), @@ -1634,7 +1634,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }; let res = db .insert([TEST_LEAF].as_ref(), b"psit", tampered, Some(opts), None, v) diff --git a/grovedb/src/tests/verify_grovedb_indexed_tests.rs b/grovedb/src/tests/verify_grovedb_indexed_tests.rs index a9f5adffd..83343033c 100644 --- a/grovedb/src/tests/verify_grovedb_indexed_tests.rs +++ b/grovedb/src/tests/verify_grovedb_indexed_tests.rs @@ -6,7 +6,7 @@ #[cfg(test)] mod tests { - use crate::BackwardReferencesPolicy; + use crate::DisplacedValue; use grovedb_element::indexed::IndexAxis; use grovedb_merk::element::{ delete::ElementDeleteFromStorageExtensions, get::ElementFetchFromStorageExtensions, @@ -1419,7 +1419,7 @@ mod tests { validate_insertion_does_not_override: false, validate_insertion_does_not_override_tree: false, base_root_storage_is_free: true, - backward_references_policy: BackwardReferencesPolicy::Skip, + displaced_value: DisplacedValue::NotParticipant, }), None, &grovedb_version::version::v3::GROVE_V3, diff --git a/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v0.rs b/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v0.rs index df3686f39..f542a1f30 100644 --- a/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v0.rs +++ b/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v0.rs @@ -31,7 +31,6 @@ pub(super) fn add_average_case_merk_propagate_v0( tree_type, estimated_layer_count, estimated_layer_sizes, - may_contain_backward_references: _, } = input; let levels = estimated_layer_count.estimate_levels(); nodes_updated += levels; diff --git a/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v1.rs b/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v1.rs index 786afaf0c..7cb5972a2 100644 --- a/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v1.rs +++ b/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v1.rs @@ -33,7 +33,6 @@ pub(super) fn add_average_case_merk_propagate_v1( tree_type, estimated_layer_count, estimated_layer_sizes, - may_contain_backward_references: _, } = input; let levels = estimated_layer_count.estimate_levels(); nodes_updated += levels; diff --git a/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v2.rs b/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v2.rs index 7f82dd889..a36decb81 100644 --- a/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v2.rs +++ b/merk/src/estimated_costs/average_case_costs/add_average_case_merk_propagate/v2.rs @@ -38,7 +38,6 @@ pub(super) fn add_average_case_merk_propagate_v2( tree_type, estimated_layer_count, estimated_layer_sizes, - may_contain_backward_references: _, } = input; let levels = estimated_layer_count.estimate_levels(); nodes_updated += levels; diff --git a/merk/src/estimated_costs/average_case_costs/mod.rs b/merk/src/estimated_costs/average_case_costs/mod.rs index f19c10663..c681142f3 100644 --- a/merk/src/estimated_costs/average_case_costs/mod.rs +++ b/merk/src/estimated_costs/average_case_costs/mod.rs @@ -224,19 +224,6 @@ pub struct EstimatedLayerInformation { pub estimated_layer_count: EstimatedLayerCount, /// Estimated layer sizes pub estimated_layer_sizes: EstimatedLayerSizes, - /// Whether this layer may hold backward-reference participants - /// (`ItemWithBackwardsReferences`, `SumItemWithBackwardsReferences`, - /// `ItemWithSumItemWithBackwardsReferences`, `BidirectionalReference`). - /// - /// The estimator cannot see stored state. Under the default - /// `BackwardReferencesPolicy::Maintain`, a plain write or delete can owe - /// maintenance only for the participant it displaces, so that - /// displaced-state fan-out is charged only in layers that declare it. - /// Undeclared layers estimate plain writes exactly as `Skip` would; an - /// op that itself writes a participant is charged from the op regardless. - /// Declaring the layers that hold participants is the caller's - /// responsibility. - pub may_contain_backward_references: bool, } impl EstimatedLayerInformation {} @@ -500,7 +487,6 @@ mod tests { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(1, false), estimated_layer_sizes: EstimatedLayerSizes::AllItems(5, 20, None), }; @@ -520,7 +506,6 @@ mod tests { fn test_add_average_case_merk_propagate_all_items_updates_cost() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, Some(2)), }; @@ -542,7 +527,6 @@ mod tests { fn test_propagate_v0_all_subtrees() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllSubtrees( 8, @@ -561,7 +545,6 @@ mod tests { fn test_propagate_v0_all_items() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, Some(2)), }; @@ -576,7 +559,6 @@ mod tests { fn test_propagate_v0_all_reference() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllReference(8, 24, Some(2)), }; @@ -591,7 +573,6 @@ mod tests { fn test_propagate_v0_mix() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: Some((8, EstimatedSumTrees::NoSumTrees, Some(4), 2)), @@ -616,7 +597,6 @@ mod tests { fn test_propagate_v1_all_reference() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllReference(8, 24, Some(2)), }; @@ -631,7 +611,6 @@ mod tests { fn test_propagate_v1_mix() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: Some((8, EstimatedSumTrees::NoSumTrees, Some(4), 2)), @@ -923,13 +902,11 @@ mod tests { let layer_count = EstimatedLayerCount::EstimatedLevel(3, false); let plain_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, Some(2)), }; let sum_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::AllItemsWithSumItem(8, 32, Some(2)), }; @@ -961,13 +938,11 @@ mod tests { let layer_count = EstimatedLayerCount::EstimatedLevel(3, false); let plain_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::AllReference(8, 24, Some(1)), }; let sum_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::AllReferencesWithSumItem(8, 24, Some(1)), }; @@ -990,7 +965,6 @@ mod tests { let layer_count = EstimatedLayerCount::EstimatedLevel(3, false); let base = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: None, @@ -1002,7 +976,6 @@ mod tests { }; let with_sum = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: None, @@ -1388,7 +1361,6 @@ mod tests { fn test_propagate_v0_all_items_with_sum_item() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllItemsWithSumItem(8, 32, Some(2)), }; @@ -1399,7 +1371,6 @@ mod tests { // sum-bearing cost strictly exceeds the plain-AllItems cost. let plain_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, Some(2)), }; @@ -1415,7 +1386,6 @@ mod tests { fn test_propagate_v0_all_references_with_sum_item() { let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllReferencesWithSumItem(8, 24, Some(1)), }; @@ -1424,7 +1394,6 @@ mod tests { let plain_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(3, false), estimated_layer_sizes: EstimatedLayerSizes::AllReference(8, 24, Some(1)), }; @@ -1445,7 +1414,6 @@ mod tests { let layer_count = EstimatedLayerCount::EstimatedLevel(3, false); let base = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: Some((8, EstimatedSumTrees::NoSumTrees, Some(4), 1)), @@ -1457,7 +1425,6 @@ mod tests { }; let with_sum = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: Some((8, EstimatedSumTrees::NoSumTrees, Some(4), 1)), @@ -1507,7 +1474,6 @@ mod tests { let layer_count = EstimatedLayerCount::EstimatedLevel(3, false); let mix = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: None, @@ -1519,7 +1485,6 @@ mod tests { }; let all_items = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, Some(2)), }; @@ -1547,7 +1512,6 @@ mod tests { let layer_count = EstimatedLayerCount::EstimatedLevel(3, false); let layer_info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: Some((8, EstimatedSumTrees::NoSumTrees, Some(4), 2)), @@ -1600,7 +1564,6 @@ mod tests { for layer in cases { let info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: layer, }; @@ -1631,7 +1594,6 @@ mod tests { .add_average_case_merk_propagate = 99; let info = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: EstimatedLayerCount::EstimatedLevel(2, false), estimated_layer_sizes: EstimatedLayerSizes::AllItems(8, 32, None), }; @@ -1675,7 +1637,6 @@ mod tests { let layer_count = EstimatedLayerCount::EstimatedLevel(3, false); let base = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: None, @@ -1687,7 +1648,6 @@ mod tests { }; let with_sum = EstimatedLayerInformation { tree_type: TreeType::NormalTree, - may_contain_backward_references: false, estimated_layer_count: layer_count, estimated_layer_sizes: EstimatedLayerSizes::Mix { subtrees_size: None, diff --git a/merk/src/estimated_costs/worst_case_costs.rs b/merk/src/estimated_costs/worst_case_costs.rs index 4fe07daa3..b13f8f659 100644 --- a/merk/src/estimated_costs/worst_case_costs.rs +++ b/merk/src/estimated_costs/worst_case_costs.rs @@ -51,27 +51,6 @@ pub enum WorstCaseLayerInformation { MaxElementsNumber(u32), /// Number of levels NumberOfLevels(u32), - /// `MaxElementsNumber`, additionally declaring that the layer may hold - /// backward-reference participants; see - /// [`EstimatedLayerInformation::may_contain_backward_references`](super::average_case_costs::EstimatedLayerInformation::may_contain_backward_references). - MaxElementsNumberWithBackwardReferences(u32), - /// `NumberOfLevels`, additionally declaring that the layer may hold - /// backward-reference participants. - NumberOfLevelsWithBackwardReferences(u32), -} - -#[cfg(feature = "minimal")] -impl WorstCaseLayerInformation { - /// Whether the caller declared that this layer may hold backward-reference - /// participants, so worst-case estimation charges the displaced-state - /// fan-out for plain writes and deletes into it. - pub fn may_contain_backward_references(&self) -> bool { - matches!( - self, - WorstCaseLayerInformation::MaxElementsNumberWithBackwardReferences(_) - | WorstCaseLayerInformation::NumberOfLevelsWithBackwardReferences(_) - ) - } } #[cfg(feature = "minimal")] @@ -205,16 +184,14 @@ pub fn add_worst_case_merk_propagate( let mut nodes_updated = 0; // Propagation requires to recompute and write hashes up to the root let levels = match input { - WorstCaseLayerInformation::MaxElementsNumber(n) - | WorstCaseLayerInformation::MaxElementsNumberWithBackwardReferences(n) => { + WorstCaseLayerInformation::MaxElementsNumber(n) => { if *n == u32::MAX { 32 } else { ((*n + 1) as f32).log2().ceil() as u32 } } - WorstCaseLayerInformation::NumberOfLevels(n) - | WorstCaseLayerInformation::NumberOfLevelsWithBackwardReferences(n) => *n, + WorstCaseLayerInformation::NumberOfLevels(n) => *n, }; nodes_updated += levels; @@ -303,28 +280,6 @@ mod tests { assert_eq!(cost.hash_node_calls, 68); } - #[test] - fn test_backward_reference_declaration_does_not_change_propagation() { - for (plain, declared) in [ - ( - WorstCaseLayerInformation::MaxElementsNumber(16), - WorstCaseLayerInformation::MaxElementsNumberWithBackwardReferences(16), - ), - ( - WorstCaseLayerInformation::NumberOfLevels(3), - WorstCaseLayerInformation::NumberOfLevelsWithBackwardReferences(3), - ), - ] { - let mut plain_cost = OperationCost::default(); - add_worst_case_merk_propagate(&mut plain_cost, &plain).unwrap(); - let mut declared_cost = OperationCost::default(); - add_worst_case_merk_propagate(&mut declared_cost, &declared).unwrap(); - assert_eq!(plain_cost, declared_cost); - assert!(!plain.may_contain_backward_references()); - assert!(declared.may_contain_backward_references()); - } - } - #[test] fn test_is_empty_tree_except_cost_helpers() { let mut worst_cost = OperationCost::default();