feat(platform)!: require fee history for storage refunds and credit their recorded owners

book/src/fees/overview.md

@@ -49,8 +49,9 @@ in `FeeStorageVersion`:
 | `storage_seek_cost` | 2,000 | Cost of a single disk seek |
 
 Storage fees are **refundable**: when data is deleted, a portion of the original
-storage fee is returned to the identity that paid it (see [Refunds](#refunds)
-below).
+storage fee becomes a refund for the owner recorded in the stored bytes' storage
+flags, which is not always the identity that paid the fee (see
+[Refunds](#refunds) below).
 
 ### Processing Fees
 
@@ -219,6 +220,38 @@ out to proposers.
 There is a **dust limit**: refunds below 32 bytes worth of storage credits are
 discarded to prevent micro-refund spam.
 
+### Fee history and refund ownership (protocol version 15 onward)
+
+A refund is priced with the fee history of the block that removes the bytes:
+the `previous_fee_versions` map platform state carries, which the epoch change
+hook extends whenever the fee version number changes. `Drive::calculate_fee`
+v1 (`DRIVE_VERSION_V10`) consults that history for every owner-attributed
+storage removal, on every fee version number, and returns an internal error
+when a caller passes none. Earlier generations priced fee version number 1
+against an empty history, so a caller that forgot the history silently
+refunded at the first generation's storage rates; from protocol version 15
+that omission halts instead of mispricing. Every shipped schedule shares fee
+version number 1 and the same storage rates, so the credits themselves are
+unchanged for every shipped input.
+
+Refunds follow the recorded owner in the element's storage flags.
+`Drive::credit_storage_refunds_to_owners_operations` credits each owner that
+has a balance element without consulting any key or permission, so a frozen
+but existing owner still receives its bookkeeping refund. Two shares of a
+refund never reach a balance and are reported for the caller instead: the
+part that clears an owner's negative credit (identity debt, which lives
+outside the credit sum trees) and the part whose owner has no balance element
+(the native stand-in for a wiped owner). The caller moves both into the
+current epoch's processing pool with a single pool write and records every
+refund against its storage epoch in the pending epoch refunds, so the credit
+conservation check stays balanced. This primitive is the settlement step
+block lifecycle paths that remove owner-attributed bytes are meant to use in
+the block that removes them; at protocol version 15 the vote poll end cleanup
+does not yet price or settle its refunds, and wiring it up is a separate
+change. The protocol 12 schema migration, which shrank stored contracts
+without refunding the stripped bytes, ran once at that activation and is the
+recorded historical exception; it replays exactly as executed.
+
 ## Epoch-Based Fee Distribution
 
 Fees do not go directly to the block proposer. Instead, they accumulate in
@@ -292,6 +325,15 @@ Fee versions are stored in the `FEE_VERSIONS` array and looked up by number. The
 `uses_version_fee_multiplier_permille` field allows a global scaling factor
 (permille = divide by 1000; a value of 1000 means no change).
 
+`fee_version_number` keys the persisted fee history that refunds are priced
+against. A schedule that changes storage rates needs a new number, because the
+refund code resolves the schedule for an epoch through the history and (in
+generations before protocol version 15) shortcut number 1 to the first
+generation's rates. `FEE_VERSION1` and `FEE_VERSION2` share number 1 because
+only a non-storage group changed between them; a test in `rs-drive`'s fee
+operation module pins that every shipped schedule keeps the first generation's
+storage rates.
+
 ## Key Source Files
 
 | File | Contents |

packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/ranked_index_e2e_tests.rs

@@ -67,12 +67,15 @@ use dpp::block::block_info::BlockInfo;
 use dpp::data_contract::accessors::v0::DataContractV0Getters;
 use dpp::data_contract::document_type::random_document::CreateRandomDocument;
 use dpp::document::{Document, DocumentV0Getters, DocumentV0Setters};
+use dpp::fee::default_costs::CachedEpochIndexFeeVersions;
 use dpp::platform_value::Value;
 use dpp::prelude::DataContract;
 use dpp::tests::json_document::json_document_to_contract;
+use dpp::version::fee::FeeVersion;
 use dpp::version::PlatformVersion;
 use grovedb::element::indexed::AVG_FIXED_POINT_SCALE;
 use grovedb::Element;
+use std::collections::BTreeMap;
 
 /// The one index property every doctype in the fixture ranks by.
 const GROUP_PROPERTY: &str = "restaurantId";
@@ -1331,6 +1334,10 @@ fn estimated_and_actual_update_fees(
         .document_type_for_name(document_type_name)
         .unwrap_or_else(|_| panic!("{document_type_name} doctype exists"));
     let storage_flags = Some(Cow::Owned(StorageFlags::SingleEpoch(0)));
+    // The replaced element carries epoch flags, so pricing its removal needs
+    // the fee history of the removing block, as every production caller
+    // passes.
+    let fee_history: CachedEpochIndexFeeVersions = BTreeMap::from([(0, FeeVersion::first())]);
 
     let run = |apply: bool| {
         drive
@@ -1344,7 +1351,7 @@ fn estimated_and_actual_update_fees(
                 storage_flags.clone(),
                 None,
                 pv,
-                None,
+                Some(&fee_history),
             )
             .unwrap_or_else(|e| {
                 panic!("expected the {document_type_name} update (apply={apply}) to succeed: {e}")

packages/rs-drive/src/drive/contract/migration/strip_unknown_document_schema_properties.rs

@@ -214,3 +214,204 @@ impl Drive {
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::util::storage_flags::StorageFlags;
+    use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure;
+    use dpp::block::block_info::BlockInfo;
+    use dpp::data_contract::accessors::v0::DataContractV0Getters;
+    use dpp::identity::accessors::{IdentityGettersV0, IdentitySettersV0};
+    use dpp::identity::Identity;
+    use dpp::platform_value::Value;
+    use dpp::tests::json_document::json_document_to_contract_with_ids;
+    use dpp::version::PlatformVersion;
+
+    const OWNER_BALANCE: u64 = 5_000_000;
+
+    /// The protocol 12 schema migration rewrote every stored user contract
+    /// whose document schemas carried top-level properties the v1 document
+    /// meta-schema forbids. It kept each element's storage flags and applied
+    /// the rewrite with a discarded cost vector, so the storage fee share of
+    /// the stripped bytes was never refunded to the contract owner and no
+    /// pending epoch refund was recorded. That is what every node executed
+    /// at the protocol 12 activation, and replay from genesis must reproduce
+    /// it byte for byte: the stripped byte counts were never recorded, so a
+    /// retroactive settlement is impossible, and the migration never runs
+    /// again, so there is nothing at a later version left to correct. The
+    /// storage refund invariant that applies from protocol version 15 names
+    /// this migration as its recorded historical exception; this test pins
+    /// the shipped behaviour so the exception stays exactly what it was.
+    #[test]
+    fn should_keep_the_protocol_12_schema_strip_frozen_without_refunding_stripped_bytes() {
+        let platform_version = PlatformVersion::get(12).expect("protocol version 12");
+        let drive = setup_drive_with_initial_state_structure(Some(platform_version));
+        let transaction = drive.grove.start_transaction();
+
+        // The contract owner exists with a balance, so a refund would be
+        // observable as a balance change.
+        let mut owner =
+            Identity::random_identity(2, Some(12), platform_version).expect("expected an identity");
+        owner.set_balance(OWNER_BALANCE);
+        drive
+            .add_new_identity(
+                owner.clone(),
+                false,
+                &BlockInfo::default(),
+                true,
+                Some(&transaction),
+                platform_version,
+            )
+            .expect("expected to insert the owner");
+        drive
+            .add_to_system_credits(OWNER_BALANCE, Some(&transaction), platform_version)
+            .expect("expected to record the system credits");
+
+        // A user contract stored the way protocol version 12 stores it: the
+        // element carries the owner's storage flags.
+        let contract = json_document_to_contract_with_ids(
+            "tests/supporting_files/contract/family/family-contract.json",
+            None,
+            Some(owner.id()),
+            false,
+            platform_version,
+        )
+        .expect("expected the family contract");
+        drive
+            .insert_contract(
+                &contract,
+                BlockInfo::default(),
+                true,
+                Some(&transaction),
+                platform_version,
+            )
+            .expect("expected to insert the contract");
+
+        let contract_id = contract.id();
+        let contract_path = contract_root_path(contract_id.as_slice());
+        let stored_element = drive
+            .grove_get_raw(
+                (&contract_path).into(),
+                &[0],
+                DirectQueryType::StatefulDirectQuery,
+                Some(&transaction),
+                &mut vec![],
+                &platform_version.drive,
+            )
+            .expect("expected to read the contract element")
+            .expect("expected the contract element to exist");
+        let (clean_bytes, flags) = match stored_element {
+            Element::Item(bytes, flags) => (bytes, flags),
+            other => panic!("expected an item, got {:?}", other),
+        };
+        let owner_flags = StorageFlags::from_element_flags_ref(
+            flags
+                .as_ref()
+                .expect("a user contract carries storage flags"),
+        )
+        .expect("expected valid flags")
+        .expect("expected owner flags");
+        assert_eq!(
+            owner_flags.owner_id(),
+            Some(&owner.id().to_buffer()),
+            "the contract bytes are attributed to the owner"
+        );
+
+        // Rewrite the stored bytes the way a pre-v12 contract could look:
+        // with a top-level document schema property the v1 meta-schema does
+        // not allow. The element keeps its flags and grows.
+        let bincode_config = bincode::config::standard()
+            .with_big_endian()
+            .with_no_limit();
+        let (mut serialization_format, _): (DataContractInSerializationFormat, usize) =
+            bincode::borrow_decode_from_slice(&clean_bytes, bincode_config)
+                .expect("expected to decode the stored contract");
+        let person_schema = serialization_format
+            .document_schemas_mut()
+            .get_mut("person")
+            .expect("expected the person document type");
+        person_schema
+            .insert("legacyUnknownProperty".to_string(), Value::Bool(true))
+            .expect("expected to add the unknown property");
+        let inflated_bytes = bincode::encode_to_vec(&serialization_format, bincode_config)
+            .expect("expected to encode the inflated contract");
+        assert!(inflated_bytes.len() > clean_bytes.len());
+        drive
+            .grove_insert(
+                (&contract_path).into(),
+                &[0],
+                Element::Item(inflated_bytes.clone(), flags.clone()),
+                Some(&transaction),
+                None,
+                &mut vec![],
+                &platform_version.drive,
+            )
+            .expect("expected to store the inflated contract");
+        drive.cache.data_contracts.clear();
+
+        let pending_refunds_before = drive
+            .fetch_pending_epoch_refunds(Some(&transaction), &platform_version.drive)
+            .expect("expected the pending refunds");
+        assert!(pending_refunds_before.is_empty());
+
+        // The migration, exactly as the protocol 12 activation ran it.
+        drive
+            .strip_unknown_document_schema_properties(&transaction, &platform_version.drive)
+            .expect("expected the migration to succeed");
+
+        let migrated_element = drive
+            .grove_get_raw(
+                (&contract_path).into(),
+                &[0],
+                DirectQueryType::StatefulDirectQuery,
+                Some(&transaction),
+                &mut vec![],
+                &platform_version.drive,
+            )
+            .expect("expected to read the migrated element")
+            .expect("expected the migrated element to exist");
+        let (migrated_bytes, migrated_flags) = match migrated_element {
+            Element::Item(bytes, flags) => (bytes, flags),
+            other => panic!("expected an item, got {:?}", other),
+        };
+        assert!(
+            migrated_bytes.len() < inflated_bytes.len(),
+            "the migration strips the unknown property, so the element shrinks"
+        );
+        assert_eq!(
+            migrated_bytes, clean_bytes,
+            "the migration restores the bytes the contract had without the property"
+        );
+        assert_eq!(
+            migrated_flags, flags,
+            "the element keeps the owner's storage flags byte for byte"
+        );
+
+        // The stripped bytes were owner-paid, yet nothing was refunded: the
+        // owner's balance and the pending epoch refunds are untouched and the
+        // credit sum stays balanced.
+        assert_eq!(
+            drive
+                .fetch_identity_balance(
+                    owner.id().to_buffer(),
+                    Some(&transaction),
+                    platform_version
+                )
+                .expect("expected the owner's balance"),
+            Some(OWNER_BALANCE)
+        );
+        assert!(drive
+            .fetch_pending_epoch_refunds(Some(&transaction), &platform_version.drive)
+            .expect("expected the pending refunds")
+            .is_empty());
+        let total = drive
+            .calculate_total_credits_balance(Some(&transaction), &platform_version.drive)
+            .expect("expected the total credits balance");
+        assert!(
+            total.ok().expect("expected a well-formed balance"),
+            "no credits move during the migration: {:?}",
+            total
+        );
+    }
+}

packages/rs-drive/src/drive/document/update/mod.rs

@@ -3476,7 +3476,7 @@ mod tests {
                 storage_flags,
                 None,
                 platform_version,
-                None,
+                Some(&EPOCH_CHANGE_FEE_VERSION_TEST),
             )
             .expect("key-changing update on aggregate index must succeed");