feat!: declare per operation whether the displaced value may be a backward-reference participant

CHANGELOG.md

@@ -17,9 +17,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   updating a referenced element propagates the new hash along every chain,
   and deleting/overwriting it cascades the chains away (each affected
   reference must opt in via `cascade_on_update`). Maintenance is automatic
-  on V4, with `BackwardReferencesPolicy::Skip` as an explicit opt-out; batch
-  maintenance is gated by the new `apply_batch.backward_references_maintenance`
-  version slot. The referrer list is stored on the element itself under a
+  on V4, and every operation declares what it displaces (`DisplacedValue`,
+  see Changed); batch maintenance is gated by the new
+  `apply_batch.backward_references_maintenance` version slot. The referrer list is stored on the element itself under a
   two-layer hash (`combine(inner, backrefs)`), so registering a referrer
   never re-hashes what existing referrers committed to; public reads return
   the stripped element, and proofs authenticate these elements through the
@@ -43,16 +43,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   capacity, the ceiling for writes that cannot see the element they
   displace, ≤10-hop chains, 1 referrer per reference) while pre-V4
   estimation stays byte-stable for replay. See
-  `adr/bidirectional_references.md`. `clear_subtree` now exposes the same policy:
-  default Maintain refuses participant-containing subtrees before mutation.
-  `drop_flat_subtree` adds a required policy argument, and both it and batch
-  `DropFlat` require explicit Skip to preserve O(1) cost. Recursive deletions
-  under Maintain include participant-scan costs in the V4 cost pins. Ordinary
-  batches that touch no participants retain their original executor semantics.
-  Estimation charges the displaced-participant fan-out only in layers that
-  declare it (`EstimatedLayerInformation::may_contain_backward_references`,
-  or the `*WithBackwardReferences` worst-case variants); undeclared layers
-  estimate plain writes exactly as `Skip` does.
+  `adr/bidirectional_references.md`. `clear_subtree` exposes the same
+  declaration: `MayBeParticipant` refuses participant-containing subtrees
+  before mutation, `NotParticipant` is trusted for a raw clear.
+  `drop_flat_subtree` takes the declaration as a required argument, and both
+  it and batch `DropFlat` require `NotParticipant` to preserve O(1) cost.
+  Ordinary batches that touch no participants retain their original executor
+  semantics. Estimation charges the displaced-participant fan-out only for
+  ops declared `MayBeParticipant`.
 - **BREAKING**: Added `add_parent_tree_on_subquery` feature to PathQuery (#379)
   - New field in `Query` struct: `add_parent_tree_on_subquery: bool`
   - When set to `true`, parent tree elements (like CountTree or SumTree) are included in query results when performing subqueries
@@ -61,21 +59,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Updated proof verification logic to handle parent tree inclusion
 
 ### Changed
-- **BREAKING**: `EstimatedLayerInformation` gains
-  `may_contain_backward_references: bool` (declare `false` for layers that
-  never hold backward-reference participants), and `WorstCaseLayerInformation`
-  gains `MaxElementsNumberWithBackwardReferences` and
-  `NumberOfLevelsWithBackwardReferences`. Under the default `Maintain` policy
-  the estimators charge the displaced-participant fan-out and delete probe
-  only in declared layers, so ordinary V4 estimates no longer inflate for
-  every write.
-- **BREAKING**: Replace `propagate_backward_references` in insert, delete,
-  and batch options with `backward_references_policy` (`Maintain` by default,
-  or explicit `Skip`). V4 observes old values through retained Merk nodes so
-  ordinary mutations need no separate old-value fetch for classification.
-  Partial batches reject displaced participants; subtree removal/replacement
-  refuses unsupported descendant maintenance before commit. Earlier protocol
-  versions retain their historical behavior.
+- **BREAKING**: Replace `propagate_backward_references` with a per-operation
+  declaration of the stored value an operation displaces,
+  `DisplacedValue::{MayBeParticipant, NotParticipant}`, on `InsertOptions`,
+  `DeleteOptions`, `ClearOptions` and every `QualifiedGroveDbOp`
+  (`with_displaced_value`; `MayBeParticipant` is the default everywhere).
+  `BatchApplyOptions` carries no backward-references policy. V4 has one write
+  path: the displaced value is read for the write anyway, so
+  `MayBeParticipant` maintains a participant it finds and `NotParticipant`
+  refuses the operation before anything commits; where nothing reads the
+  contents (a flat drop, a raw `clear_subtree`, replacing a populated
+  subtree, a live recursive delete) `NotParticipant` is trusted. A batch
+  `DeleteTree` is never pre-scanned: `DontCheckWithNoCleanup` declares that
+  the batch's own deletes emptied the subtree, `Error` and `Skip` verify
+  that at apply time, and `DeleteChildren` checks the declaration on the
+  cleanup walk it makes anyway, refusing a participant the batch does not
+  explicitly delete. A delete-up-tree chain and a batch recursive removal
+  therefore cost the plain removal on V4. Partial batches still refuse
+  participant mutations, and earlier protocol versions retain their
+  historical behavior.
+- **BREAKING**: `EstimatedLayerInformation::may_contain_backward_references`
+  and the `*WithBackwardReferences` variants of `WorstCaseLayerInformation`
+  are removed. The estimators charge the displaced-participant fan-out and
+  delete probe per op declared `MayBeParticipant` instead of per layer; ops
+  that write a participant themselves are charged from the op regardless.
 - Bumped the GroveDB workspace crates and their internal dependency requirements
   to **6.0.0** for the public API changes since 5.0.1. This package version is
   independent of the existing `GroveVersion` runtime compatibility versions.

adr/bidirectional_references.md

@@ -51,17 +51,18 @@ chain origin. When such behavior is required, a different type of element should
 Moreover, these types are incompatible, which will be discussed in the "Rules" section.
 
 On `GROVE_V4`, ordinary inserts, replacements, deletes, and full batches
-maintain backward references automatically. Callers do not need to predict
-whether a plain operation will displace a participant. `InsertOptions`,
-`DeleteOptions`, `ClearOptions`, and `BatchApplyOptions` expose `backward_references_policy`,
-whose default is `BackwardReferencesPolicy::Maintain`.
-
-`BackwardReferencesPolicy::Skip` deliberately disables maintenance for an
-operation. It permits dangling references and stale hashes; it is not a hint
-that a key is known to have no references. Inserting a bidirectional reference
-through the live API still registers its edge even with `Skip`. Full batches
-with `Skip` reject family payloads, while allowing ordinary mutations that
-intentionally bypass maintenance.
+maintain backward references automatically, and every operation declares
+what it displaces: `InsertOptions`, `DeleteOptions`, `ClearOptions` and each
+`QualifiedGroveDbOp` carry a `DisplacedValue`, `MayBeParticipant` by default.
+GroveDB reads the displaced value for the write anyway, so for a keyed
+operation the declaration only decides what happens when that value takes
+part in backward references: `MayBeParticipant` maintains the references,
+`NotParticipant` refuses the operation before anything commits. Where nothing
+reads the contents — a flat drop, a raw `clear_subtree`, the replacement of a
+populated subtree, a live recursive delete — `NotParticipant` is trusted and
+leaves any participant's registrations stale, exactly like the storage it
+strands. There is no policy that skips maintenance on a value known to
+participate. Inserting a bidirectional reference always registers its edge.
 
 ## Versioning and scope
 
@@ -90,16 +91,26 @@ Current limitations:
   specialized/indexed descendants. Remove those descendants first.
 - Live participant maintenance below an indexed primary requires a full
   batch; the reference cache refuses that propagation before commit.
-- Recursive delete and subtree replacement inspect descendants under `Maintain`.
-  Those scans add reads and are charged in the V4 default cost tests.
-- Flat drop retains its O(1) contract. Standalone `drop_flat_subtree` requires
-  an explicit policy argument; it and batch `DropFlat` reject `Maintain`
-  before scanning. Use `Skip` to acknowledge stale or dangling registrations,
-  or use recursive delete when maintenance is required.
-- `clear_subtree` defaults to `Maintain`: it scans and refuses a subtree
+- `Delete` of a populated tree and subtree replacement inspect descendants
+  when declared `MayBeParticipant`; those scans add reads and are charged in
+  the V4 default cost tests. A batch `DeleteTree` is never pre-scanned:
+  `DontCheckWithNoCleanup` declares that the batch's own deletes emptied the
+  subtree, `Error` and `Skip` verify that at apply time, and `DeleteChildren`
+  (with the defensive `Error`/`Skip` sweeps) checks the declaration on the
+  cleanup walk it makes anyway, refusing a participant the batch does not
+  explicitly delete. A delete-up-tree chain and a batch recursive removal
+  therefore cost the plain removal under either declaration.
+- Flat drop retains its O(1) contract. Standalone `drop_flat_subtree` takes
+  the declaration as a required argument; it and batch `DropFlat` refuse
+  `MayBeParticipant` before reading anything and trust `NotParticipant`. Use
+  recursive delete when maintenance is required.
+- `clear_subtree` under `MayBeParticipant` scans and refuses a subtree
   containing participants before making any mutation, including with a caller
-  transaction. Delete the participants through the normal API first. Explicit
-  `ClearOptions::backward_references_policy = Skip` permits a raw clear.
+  transaction; delete the participants through the normal API first.
+  `NotParticipant` is trusted for a raw clear. A live recursive `delete`
+  declared `NotParticipant` likewise trusts its contents: a clear runs one
+  such delete per nested subtree inside the caller's transaction, where a
+  refusal part-way through could not be undone.
 
 Ordinary full batches keep their original operation set when neither stored
 nor incoming values participate in references. Preparation retains the Merks
@@ -116,13 +127,16 @@ transaction's original subtree contents. Cross-segment conflict checks prevent
 those committed-state inspections from overlooking changes staged by the first
 segment. A refusal discards the storage batch and preserves the caller's
 transaction. These scans have real costs, pinned alongside the full-batch costs;
-this API does not promise a no-scan recursive removal.
+this API does not promise a no-scan recursive removal, only that `DeleteTree`
+removals add no reads of their own: their contents are checked on the cleanup
+walk.
 
 ## Rules
 
 Next, we’ll go over the rules and limitations for using bidirectional references.
 
-These rules apply by default; `BackwardReferencesPolicy::Skip` explicitly opts out.
+These rules always apply; `DisplacedValue::NotParticipant` is a checked claim, not an
+opt-out.
 
 An 'Element with backward references' refers to `ItemWithBackwardsReferences`,
 `SumItemWithBackwardsReferences`, `ItemWithSumItemWithBackwardsReferences`, and
@@ -149,7 +163,8 @@ registered element) and the node growth registrations can inflict on a target.
 insertion.__ Public reads enforce the declared budget deterministically, so an edge whose
 chain is already longer than its declaration would never resolve; the write path rejects
 such dead edges instead of persisting them. (An edge can still fall out of budget later —
-e.g. its target is overwritten into a plain reference with `BackwardReferencesPolicy::Skip` — and
+e.g. its target is overwritten into a plain reference behind a raw clear declared
+`NotParticipant` — and
 reads then return `ReferenceLimit`.)
 - __Both ends of a bidirectional edge must sit at most 32 subtree levels deep__
 (`MAX_BACKWARD_REFERENCES_GROVE_DEPTH`, enforced at registration). Every later derived
@@ -226,15 +241,13 @@ estimation is preserved byte-for-byte for replay of historical admission
 decisions.
 
 The estimator cannot see stored state, so the bound for a write that may
-displace a participant is a per-layer declaration:
-`EstimatedLayerInformation::may_contain_backward_references` (average case)
-and the `*WithBackwardReferences` variants of `WorstCaseLayerInformation`.
-Undeclared layers charge no displaced-state fan-out and no delete probe —
-their `Maintain` estimates equal `Skip` estimates byte-for-byte — while ops
-that themselves write a participant (family items, bidirectional references)
-are charged from the op regardless. Declaring the layers that hold
-participants is the caller's responsibility; an undeclared layer that does
-hold them under-estimates cascades.
+displace a participant follows the op's own declaration: an op declared
+`MayBeParticipant` charges the displaced-state fan-out and the delete probe,
+an op declared `NotParticipant` charges the plain write alone, and ops that
+themselves write a participant (family items, bidirectional references) are
+charged from the op regardless. Declaring correctly is the caller's
+responsibility; the apply path refuses a false `NotParticipant` claim rather
+than running an unpriced cascade.
 
 Live writes use the same preparation observer. Ordinary values retain the
 existing parent and indexed-tree propagation; participating values reuse the
@@ -310,7 +323,7 @@ Previous read: [Merk cache](./merk_cache.md).
 Deletion or an update of an element with backward references triggers a cascade hash
 update or a deletion, both of which alter the state of affected subtrees, leading to
 regular hash propagation to ancestor subtrees up to the GroveDB root. In short, operations
-under `BackwardReferencesPolicy::Maintain` (the V4 default) can trigger updates across
+on V4 can trigger updates across
 several subtrees simultaneously.
 
 Thus, there are two ongoing propagations:

docs/book/src/batch-operations.md

@@ -255,27 +255,35 @@ replacement or delete displaces a registered element. Preparation observes
 old values and retains their Merk nodes for execution and storage accounting.
 The existing consent and batch-conflict rules still apply.
 
-Set `BatchApplyOptions { backward_references_policy:
-BackwardReferencesPolicy::Skip, ..Default::default() }` only when deliberately
-bypassing maintenance. This can leave stale hashes or dangling references.
-Partial batches refuse participant mutations; use a full batch for reference
-maintenance. Recursive removal of a subtree containing participants is
-supported by live `delete`; see the bidirectional references ADR for scope.
-
-Under `Maintain`, ordinary batches retain their executor semantics when no old
-or new value participates in backward references. Reference planner conflict
-rules apply only to batches that touch participants. Recursive subtree deletion
-and replacement inspect descendants and incur additional read costs.
-
-Cost estimation follows the same split. Layers whose
-`EstimatedLayerInformation` sets `may_contain_backward_references` (or use
-the `WithBackwardReferences` worst-case variants) charge the
-displaced-participant fan-out for plain writes and deletes; undeclared layers
-estimate them exactly as `Skip` would. Ops that write a participant
-themselves are always charged.
-
-`SubelementsDeletionBehavior::DropFlat` requires explicit
-`BatchApplyOptions::backward_references_policy = BackwardReferencesPolicy::Skip`.
-`Maintain` refuses flat drop before scanning, preserving its O(1) contract.
-Partial batches refuse participant mutations in either segment; their subtree
-inspection occurs before commit and can also incur recursive read costs.
+Every `QualifiedGroveDbOp` declares what it displaces: `MayBeParticipant`
+(the default) maintains a participant the write lands on, `NotParticipant`
+refuses the write if the stored value turns out to participate. The
+declaration is checked from the value the batch reads for the write anyway,
+so it never costs a read; declare `NotParticipant` through
+`with_displaced_value` on ops whose positions are known to hold no
+participants. Partial batches refuse participant mutations; use a full batch
+for reference maintenance. Recursive removal of a subtree containing
+participants is supported by live `delete`; see the bidirectional references
+ADR for scope.
+
+Ordinary batches retain their executor semantics when no old or new value
+participates in backward references. Reference planner conflict rules apply
+only to batches that touch participants. A `Delete` of a populated tree or a
+tree replacement declared `MayBeParticipant` inspects the descendants and
+incurs additional read costs. A `DeleteTree` is never pre-scanned:
+`DontCheckWithNoCleanup` declares that the batch's own deletes emptied the
+subtree, `Error` and `Skip` verify that at apply time, and `DeleteChildren`
+checks the declaration on the cleanup walk it already makes, refusing a
+participant the batch does not explicitly delete. A delete-up-tree chain and
+a recursive removal therefore cost the plain removal.
+
+Cost estimation follows the same split: an op declared `MayBeParticipant`
+charges the displaced-participant fan-out for its plain write or delete, an
+op declared `NotParticipant` estimates the plain write alone, and ops that
+write a participant themselves are always charged.
+
+`SubelementsDeletionBehavior::DropFlat` requires the op to declare
+`DisplacedValue::NotParticipant`; `MayBeParticipant` is refused before
+reading anything, preserving the O(1) contract. Partial batches refuse
+participant mutations in either segment; their inspection of tree
+replacements occurs before commit and can also incur recursive read costs.

docs/crates/grovedb.md

@@ -127,7 +127,7 @@ pub fn insert<B: AsRef<[u8]>>(
 #### Delete Operations
 - **delete**: Standard deletion
 - **delete_up_tree**: Remove empty parents recursively
-- **clear_subtree**: Bulk deletion; V4 Maintain scans and refuses backward-reference participants. Explicit Skip permits raw clearing.
+- **clear_subtree**: Bulk deletion; on V4, `MayBeParticipant` scans and refuses backward-reference participants, `NotParticipant` is trusted for a raw clear.
 
 #### Query System
 
@@ -434,8 +434,10 @@ GroveDB is designed with several core principles:
 - Performance optimizations
 
 
-On Grove V4, insert, delete, and full-batch options default to
-`BackwardReferencesPolicy::Maintain`. Old-value preparation retains fetched
-Merk nodes for the mutation. `BackwardReferencesPolicy::Skip` is an explicit
-opt-out that permits stale or dangling references. See
+On Grove V4, every insert, delete and batch operation declares what it
+displaces (`DisplacedValue`, `MayBeParticipant` by default). Old-value
+preparation retains fetched Merk nodes for the mutation, so the declaration
+is checked for free: `MayBeParticipant` maintains a participant it finds,
+`NotParticipant` refuses the operation. Only routes that read nothing (flat
+drop, raw clear) trust the claim. See
 [the bidirectional references design](../../adr/bidirectional_references.md).