From 329506a54bffe09fdb71a84fdb228d7ba55a0bf8 Mon Sep 17 00:00:00 2001 From: Quanzheng Long Date: Fri, 6 Nov 2026 21:30:52 -0800 Subject: [PATCH] Migrate the history of runs that predate their segment state in server-zd Runs that started before their partition offloaded history, or while offload was off, have no segment state. Once offload was on, window pruning stopped for them because nothing above the uploaded seq may be deleted, and none of their history reached object storage. The first command that leaves such a run running while its partition offloads now starts the migration. The open segment starts at the run's first seq and event 1, and each command of the run seals at most one segment of it, cut from history metadata at whole seqs. A segment ends at the first seq where it holds two history records and reaches the target bytes, 10,000 superseded values, or 10,000 history rows, or at the command's own seq, which ends the migration. Two records per segment let the migration gain on the run. Each segment holds the rows still stored in its seqs, so seqs that window pruning deleted earlier are holes inside the reported pruned range. Event ranges come from metadata, with records that predate last_assigned_event_id decoded instead. RunRecord field 64, history_migration_started_seq, marks a migrating run; its open segment then tracks no bytes. Closed runs are not migrated. A run that closes while migrating seals one more segment at the close and keeps the rest local until purge. Turning offload off ends a migration, and its unsealed rest joins the gap of commands applied while offload is off. Tests cover the migration in apply, a restored testdata/v1 recording with running, closed, and migrating legacy runs, the uploader, and the simulation, which now requires at most one sealed segment per run and command. testdata/v1 is regenerated with its generator. Refs #1325 --- docs/reference/server/configuration.mdx | 14 +- server-zd/crates/dexzd-apply/README.md | 32 +- .../src/command_handlers/run_cursor.rs | 4 +- .../dexzd-apply/src/history_offload/mod.rs | 1 + .../src/history_offload/segment_migration.rs | 299 ++++ .../src/history_offload/segment_seal.rs | 65 +- .../dexzd-apply/tests/history_migration.rs | 1218 +++++++++++++++++ .../dexzd-apply/tests/history_segments.rs | 22 - .../tests/history_migration_upload_tests.rs | 252 ++++ .../dexzd-engine/src/effects/tests/mod.rs | 1 + .../tests/format_compat/golden_replay.rs | 2 +- .../tests/format_compat/workload.rs | 3 + .../format_compat/workload/legacy_history.rs | 629 +++++++++ .../dexzd-simulation/src/apply_checker.rs | 19 +- .../dexzd-simulation/src/apply_simulation.rs | 6 +- .../history_migration_invariants.rs | 154 +++ .../invariants/history_segment_invariants.rs | 24 +- .../dexzd-simulation/src/invariants/mod.rs | 3 + server-zd/crates/dexzd-simulation/src/lib.rs | 4 +- .../tests/apply_invariants.rs | 5 + server-zd/proto/dexzd/v1/state.proto | 3 + server-zd/testdata/v1/README.md | 7 + .../node-a/0000000001/00000000000000000118 | Bin 0 -> 298 bytes .../node-a/0000000001/00000000000000000119 | Bin 0 -> 382 bytes .../node-a/0000000001/00000000000000000120 | Bin 0 -> 350 bytes .../node-a/0000000001/00000000000000000121 | Bin 0 -> 384 bytes .../node-a/0000000001/00000000000000000122 | Bin 0 -> 350 bytes .../node-a/0000000001/00000000000000000123 | Bin 0 -> 381 bytes .../node-a/0000000001/00000000000000000124 | Bin 0 -> 351 bytes .../node-a/0000000001/00000000000000000125 | Bin 0 -> 380 bytes .../node-a/0000000001/00000000000000000126 | Bin 0 -> 352 bytes .../node-a/0000000001/00000000000000000127 | Bin 0 -> 382 bytes .../node-a/0000000001/00000000000000000128 | Bin 0 -> 354 bytes .../node-a/0000000001/00000000000000000129 | Bin 0 -> 385 bytes .../node-a/0000000001/00000000000000000130 | Bin 0 -> 362 bytes .../node-a/0000000001/00000000000000000131 | Bin 0 -> 388 bytes .../node-a/0000000001/00000000000000000132 | Bin 0 -> 364 bytes .../node-a/0000000001/00000000000000000133 | Bin 0 -> 384 bytes .../node-a/0000000001/00000000000000000134 | Bin 0 -> 365 bytes .../node-a/0000000001/00000000000000000135 | Bin 0 -> 384 bytes .../node-a/0000000001/00000000000000000136 | Bin 0 -> 366 bytes .../node-a/0000000001/00000000000000000137 | Bin 0 -> 387 bytes .../node-a/0000000001/00000000000000000138 | Bin 0 -> 363 bytes .../node-a/0000000001/00000000000000000139 | Bin 0 -> 386 bytes .../node-a/0000000001/00000000000000000140 | Bin 0 -> 363 bytes .../node-a/0000000001/00000000000000000141 | Bin 0 -> 382 bytes .../node-a/0000000001/00000000000000000142 | Bin 0 -> 364 bytes .../node-a/0000000001/00000000000000000143 | Bin 0 -> 386 bytes .../node-a/0000000001/00000000000000000144 | Bin 0 -> 367 bytes .../node-a/0000000001/00000000000000000145 | Bin 0 -> 387 bytes .../node-a/0000000001/00000000000000000146 | Bin 0 -> 367 bytes .../node-a/0000000001/00000000000000000147 | Bin 0 -> 388 bytes .../node-a/0000000001/00000000000000000148 | Bin 0 -> 367 bytes .../node-a/0000000001/00000000000000000149 | Bin 0 -> 295 bytes .../node-a/0000000001/00000000000000000150 | Bin 0 -> 394 bytes .../node-a/0000000001/00000000000000000151 | Bin 0 -> 355 bytes .../node-a/0000000001/00000000000000000152 | Bin 0 -> 385 bytes .../node-a/0000000001/00000000000000000153 | Bin 0 -> 355 bytes .../node-a/0000000001/00000000000000000154 | Bin 0 -> 205 bytes .../node-a/0000000001/00000000000000000155 | Bin 0 -> 302 bytes .../node-a/0000000001/00000000000000000156 | Bin 0 -> 389 bytes .../node-a/0000000001/00000000000000000157 | Bin 0 -> 364 bytes .../node-a/0000000001/00000000000000000158 | Bin 0 -> 385 bytes .../node-a/0000000001/00000000000000000159 | Bin 0 -> 364 bytes .../node-a/0000000001/00000000000000000160 | Bin 0 -> 387 bytes .../node-a/0000000001/00000000000000000161 | Bin 0 -> 362 bytes .../node-a/0000000001/00000000000000000162 | Bin 0 -> 387 bytes .../node-a/0000000001/00000000000000000163 | Bin 0 -> 366 bytes .../node-a/0000000001/00000000000000000164 | Bin 0 -> 385 bytes .../node-a/0000000001/00000000000000000165 | Bin 0 -> 363 bytes .../node-a/0000000001/00000000000000000166 | Bin 0 -> 390 bytes .../node-a/0000000001/00000000000000000167 | Bin 0 -> 364 bytes .../node-a/0000000001/00000000000000000168 | Bin 0 -> 393 bytes .../node-a/0000000001/00000000000000000169 | Bin 0 -> 369 bytes .../node-a/0000000001/00000000000000000170 | Bin 0 -> 387 bytes .../node-a/0000000001/00000000000000000171 | Bin 0 -> 365 bytes .../node-a/0000000001/00000000000000000172 | Bin 0 -> 390 bytes .../node-a/0000000001/00000000000000000173 | Bin 0 -> 366 bytes .../node-a/0000000001/00000000000000000174 | Bin 0 -> 392 bytes .../node-a/0000000001/00000000000000000175 | Bin 0 -> 367 bytes .../node-a/0000000001/00000000000000000176 | Bin 0 -> 391 bytes .../node-a/0000000001/00000000000000000177 | Bin 0 -> 368 bytes .../node-a/0000000001/00000000000000000178 | Bin 0 -> 390 bytes .../node-a/0000000001/00000000000000000179 | Bin 0 -> 367 bytes .../node-a/0000000001/00000000000000000180 | Bin 0 -> 389 bytes .../node-a/0000000001/00000000000000000181 | Bin 0 -> 366 bytes .../node-a/0000000001/00000000000000000182 | Bin 0 -> 390 bytes .../node-a/0000000001/00000000000000000183 | Bin 0 -> 366 bytes .../node-a/0000000001/00000000000000000184 | Bin 0 -> 390 bytes .../node-a/0000000001/00000000000000000185 | Bin 0 -> 370 bytes .../node-a/0000000001/00000000000000000186 | Bin 0 -> 173 bytes .../node-a/0000000001/00000000000000000187 | Bin 0 -> 389 bytes .../node-a/0000000001/00000000000000000188 | Bin 0 -> 368 bytes server-zd/testdata/v1/state.txt | 381 +++++- 94 files changed, 3088 insertions(+), 60 deletions(-) create mode 100644 server-zd/crates/dexzd-apply/src/history_offload/segment_migration.rs create mode 100644 server-zd/crates/dexzd-apply/tests/history_migration.rs create mode 100644 server-zd/crates/dexzd-engine/src/effects/tests/history_migration_upload_tests.rs create mode 100644 server-zd/crates/dexzd-engine/tests/format_compat/workload/legacy_history.rs create mode 100644 server-zd/crates/dexzd-simulation/src/invariants/history_migration_invariants.rs create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000118 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000119 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000120 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000121 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000122 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000123 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000124 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000125 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000126 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000127 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000128 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000129 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000130 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000131 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000132 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000133 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000134 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000135 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000136 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000137 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000138 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000139 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000140 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000141 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000142 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000143 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000144 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000145 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000146 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000147 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000148 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000149 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000150 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000151 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000152 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000153 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000154 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000155 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000156 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000157 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000158 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000159 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000160 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000161 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000162 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000163 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000164 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000165 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000166 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000167 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000168 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000169 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000170 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000171 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000172 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000173 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000174 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000175 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000176 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000177 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000178 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000179 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000180 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000181 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000182 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000183 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000184 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000185 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000186 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000187 create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000188 diff --git a/docs/reference/server/configuration.mdx b/docs/reference/server/configuration.mdx index 64f1a1abfd..e465b41031 100644 --- a/docs/reference/server/configuration.mdx +++ b/docs/reference/server/configuration.mdx @@ -354,10 +354,22 @@ pruning then deletes local history only after its segment is uploaded. Each run records **history.segmentTargetBytes** when it starts, so a new value applies only to runs that start after the node restarts with it. +Runs that already exist when offload turns on migrate, including runs that +started while it was off. The first command that leaves such a run running seals +the oldest part of its local history. Each later command of the run seals one +more segment, until a segment reaches the command itself. The run then continues +like a run that started with offload on. Events that the history window removed +before the migration stay removed. A run that is already closed is not migrated; +deleting it after its retention deletes its local history. A run that closes +during its migration keeps the rest of its history on local disk until it is +deleted. + Turning offload off stops sealing. History of segments sealed earlier stays on local disk until each segment is uploaded; then pruning returns to the window rule. Commands applied while offload is off belong to no segment, so turning it -on again starts a new segment after them. +on again starts a new segment after them. Turning offload off also ends every +migration; the rest of each such run's history follows the window rule like +those commands. The lease holder uploads each sealed segment to the primary object store, under the **history/** prefix of the bucket that holds the commit log. It uploads a diff --git a/server-zd/crates/dexzd-apply/README.md b/server-zd/crates/dexzd-apply/README.md index d538e78814..be1687c858 100644 --- a/server-zd/crates/dexzd-apply/README.md +++ b/server-zd/crates/dexzd-apply/README.md @@ -52,7 +52,8 @@ superseded inside it, or when the command closed the run. Sealing writes a SEALE with a superseded seq inside the range, and the run's `HistoryUploadPending` index row, and emits `HistorySegmentSealed`. A value revived later keeps its row, so the segment's content never changes. Runs that started while offload was off -keep no segment state. While the partition does not offload, apply seals nothing, +keep no segment state until they migrate; see History migration. While the +partition does not offload, apply seals nothing, and each command of a run with segment state moves the open segment past itself. The seqs and events of those commands belong to no segment, and their local rows follow window pruning, so a segment opened after offload is turned on again never @@ -74,6 +75,35 @@ run state limit. Run purge deletes the pending index row with the run. Retention segment, as it does for an unacknowledged outbox message, so purging never abandons an upload. +## History migration + +A run that started while its partition did not offload history has no segment +state. The first command that leaves such a run running while its partition +offloads starts the migration: it sets `history_migration_started_seq` to its +seq and opens the segment at the run's first seq and event 1. While the run +migrates, the open segment holds every unsealed seq up to the current command +and tracks no bytes, and each command of the run seals one segment of it. + +A migration segment ends at a whole seq: the first one where it holds at least +two history records and reaches the target bytes, 10,000 superseded values, or +10,000 history rows. Otherwise it ends at the command's own seq, which ends the +migration. Cutting it reads one bounded page of history metadata and of table G. +Because segments end only at whole seqs, a migration step is bounded by the +largest single command in the migrated range, as sealing is: one seq whose +command superseded 30,000 values seals all of them in one step. +Two records per segment let the migration gain on the run, since each command +adds one record. The segment holds the rows still stored in its seqs, so seqs +that window pruning deleted earlier are holes inside the reported pruned event +range. Its event range comes from the metadata `last_assigned_event_id`; records +written before that field are decoded instead. Pruning keeps every row above U +while the partition offloads, so nothing is deleted before its segment uploads. + +A closed run never starts migrating, so retention purges its local history +directly. A run that closes during its migration seals one more segment at its +closing command and then keeps the rest local until purge; a time travel that +reopens it continues the migration. Turning offload off ends a migration, and the +unsealed rest joins the gap of commands applied while offload is off. + ## Value and storage accounting Each run tracks encoded Flow and History row bytes, including keys. Live and superseded value bytes use the encoded value sizes in birth records; garbage rows identify the superseded portion. Apply updates these totals from each command's writes and deletes. diff --git a/server-zd/crates/dexzd-apply/src/command_handlers/run_cursor.rs b/server-zd/crates/dexzd-apply/src/command_handlers/run_cursor.rs index b3171878ee..48e3f38dd9 100644 --- a/server-zd/crates/dexzd-apply/src/command_handlers/run_cursor.rs +++ b/server-zd/crates/dexzd-apply/src/command_handlers/run_cursor.rs @@ -17,7 +17,7 @@ use crate::command_handlers::{ CommandScope, HandlerOutput, ReplayedCommandCounters, ReplayedCommandOutput, }; use crate::history; -use crate::history_offload::segment_seal::seal_history_segment_if_due; +use crate::history_offload::segment_migration::seal_or_migrate_history_segment; use crate::projections::history_event_links::{ HistoryEventIndex, PendingHistoryEvents, StepEventLink, resolve_step_event_links, with_recorded_event_ids, @@ -283,7 +283,7 @@ impl RunCursor { ); search_document::refresh_search_document(transaction, &self, scope)?; history::after_command(transaction, &mut self, window_events)?; - seal_history_segment_if_due(transaction, &mut self, scope.now_micros, has_closed_run)?; + seal_or_migrate_history_segment(transaction, &mut self, scope.now_micros, has_closed_run)?; self.rows.put_run_record(transaction, &self.record); if scope.is_replay { reconcile_run_storage(transaction, &mut self)?; diff --git a/server-zd/crates/dexzd-apply/src/history_offload/mod.rs b/server-zd/crates/dexzd-apply/src/history_offload/mod.rs index 6087a5db20..debd6703fd 100644 --- a/server-zd/crates/dexzd-apply/src/history_offload/mod.rs +++ b/server-zd/crates/dexzd-apply/src/history_offload/mod.rs @@ -9,5 +9,6 @@ //! Offloading run history to object storage, gated by a committed partition mode. pub(crate) mod offload_mode; +pub(crate) mod segment_migration; pub(crate) mod segment_seal; pub(crate) mod segment_upload; diff --git a/server-zd/crates/dexzd-apply/src/history_offload/segment_migration.rs b/server-zd/crates/dexzd-apply/src/history_offload/segment_migration.rs new file mode 100644 index 0000000000..49208caadd --- /dev/null +++ b/server-zd/crates/dexzd-apply/src/history_offload/segment_migration.rs @@ -0,0 +1,299 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Migrating history a run recorded before it had segment state (plan section 5.17). + +use std::collections::BTreeMap; + +use dexzd_keyspace::row_keys::history_record_key; +use dexzd_keyspace::{ColumnFamily, RunTable, run_table_prefix}; +use dexzd_protocol::dex::FlowStatus; +use dexzd_protocol::dexzd::v1::{HistoryRecord, HistoryRowMetadata, RunRecord, ValueGarbageRecord}; + +use crate::ApplyError; +use crate::command_handlers::run_cursor::RunCursor; +use crate::history::genesis_seq; +use crate::history_offload::offload_mode::is_history_offload_enabled; +use crate::history_offload::segment_seal::{ + MAX_SEGMENT_SUPERSEDED_VALUES, SealedSegmentBounds, garbage_identity, + open_segment_after_command, seal_history_segment_if_due, segment_target_bytes, + write_sealed_segment, +}; +use crate::state_access::FlowTransaction; + +/// Bounds the history rows of one migration segment, so cutting it reads a bounded page. +const MAX_MIGRATION_SEGMENT_HISTORY_ROWS: usize = 10_000; +/// FlowStart records event 1 of every run. +const FIRST_EVENT_ID: i64 = 1; + +/// Seals a due open segment, or starts or continues migrating older history, sealing at most one. +/// +/// `has_closed_run` tells whether this command closed the run; `now_micros` is its proposal time. +pub(crate) fn seal_or_migrate_history_segment( + transaction: &mut FlowTransaction<'_>, + cursor: &mut RunCursor, + now_micros: i64, + has_closed_run: bool, +) -> Result<(), ApplyError> { + let record = &cursor.record; + if record.history_open_segment_first_seq == 0 { + // Closed runs never start: migrating them only uploads history that purge deletes next. + if is_running(record) && !record.is_purging && is_history_offload_enabled(transaction)? { + start_migration(transaction, cursor)?; + migrate_one_segment(transaction, cursor, now_micros)?; + } + return Ok(()); + } + if record.history_migration_started_seq == 0 { + return seal_history_segment_if_due(transaction, cursor, now_micros, has_closed_run); + } + if !is_history_offload_enabled(transaction)? { + // The unsealed rest joins the gap of commands applied while offload is off. + cursor.record.history_migration_started_seq = 0; + return seal_history_segment_if_due(transaction, cursor, now_micros, has_closed_run); + } + if record.is_purging || !(has_closed_run || is_running(record)) { + return Ok(()); + } + migrate_one_segment(transaction, cursor, now_micros) +} + +/// Opens the segment state of a run without one at its first seq and event 1. +fn start_migration( + transaction: &FlowTransaction<'_>, + cursor: &mut RunCursor, +) -> Result<(), ApplyError> { + let first_seq = genesis_seq(transaction, cursor.rows)?; + let record = &mut cursor.record; + record.history_migration_started_seq = cursor.seq; + record.history_open_segment_first_seq = first_seq; + record.history_open_segment_first_event_id = FIRST_EVENT_ID; + record.history_open_segment_bytes = 0; + record.history_open_segment_superseded_value_count = 0; + Ok(()) +} + +/// Seals the next migration segment; the migration ends once a segment reaches this command. +fn migrate_one_segment( + transaction: &mut FlowTransaction<'_>, + cursor: &mut RunCursor, + now_micros: i64, +) -> Result<(), ApplyError> { + let bounds = cut_migration_segment(transaction, cursor)?; + write_sealed_segment(transaction, cursor, &bounds, now_micros)?; + if bounds.last_seq == cursor.seq { + cursor.record.history_migration_started_seq = 0; + open_segment_after_command(cursor); + } else { + cursor.record.history_open_segment_first_seq = bounds.last_seq + 1; + cursor.record.history_open_segment_first_event_id = bounds.last_event_id + 1; + } + Ok(()) +} + +/// Stored history of one seq inside the migration range. +#[derive(Default)] +struct SeqHistory { + bytes: u64, + history_rows: usize, + superseded_values: u64, + record: Option, +} + +/// What a migration segment holds so far. +#[derive(Default)] +struct SegmentTotals { + bytes: u64, + history_rows: usize, + superseded_values: u64, + records: usize, +} + +impl SegmentTotals { + fn add(&mut self, seq_history: &SeqHistory) { + self.bytes = self.bytes.saturating_add(seq_history.bytes); + self.history_rows += seq_history.history_rows; + self.superseded_values += seq_history.superseded_values; + self.records += usize::from(seq_history.record.is_some()); + } + + /// Two records at least, so each command seals more history than it adds. + fn is_complete(&self, target_bytes: u64) -> bool { + self.records >= 2 + && (self.bytes >= target_bytes + || self.superseded_values >= MAX_SEGMENT_SUPERSEDED_VALUES + || self.history_rows >= MAX_MIGRATION_SEGMENT_HISTORY_ROWS) + } +} + +/// Returns the bounds of the next migration segment, which starts at the open segment and ends +/// at a whole seq: the first one where the segment holds two history records and reaches the +/// target bytes, the superseded value limit, or the history row limit; otherwise this command's +/// seq. When a bounded page ends first, the segment ends with that page or its second record. +fn cut_migration_segment( + transaction: &FlowTransaction<'_>, + cursor: &RunCursor, +) -> Result { + let first_seq = cursor.record.history_open_segment_first_seq; + let first_event_id = cursor.record.history_open_segment_first_event_id; + let current_seq = cursor.seq; + let mut seqs = BTreeMap::::new(); + let metadata_complete_through = scan_history_metadata(transaction, cursor, &mut seqs)?; + let garbage_complete_through = scan_superseded_values(transaction, cursor, &mut seqs)?; + let complete_through = metadata_complete_through.min(garbage_complete_through); + // The command's own record is stored, so the range holds at least one record. + let second_record_seq = seqs + .iter() + .filter(|(_, seq_history)| seq_history.record.is_some()) + .map(|(seq, _)| *seq) + .nth(1) + .unwrap_or(current_seq); + let target_bytes = segment_target_bytes(&cursor.record); + let mut totals = SegmentTotals::default(); + let cut_seq = seqs + .iter() + .take_while(|(seq, _)| **seq <= complete_through) + .find_map(|(seq, seq_history)| { + totals.add(seq_history); + totals.is_complete(target_bytes).then_some(*seq) + }); + // Whole seqs can pass the limits; the command that recorded such a seq wrote as many rows. + let last_seq = cut_seq.unwrap_or_else(|| complete_through.max(second_record_seq)); + let last_event_id = if last_seq == current_seq { + cursor.record.next_event_id.max(1) - 1 + } else { + newest_event_id_through(transaction, cursor, &seqs, last_seq, first_event_id - 1)? + }; + Ok(SealedSegmentBounds { + first_seq, + last_seq, + first_event_id, + last_event_id, + }) +} + +/// Adds the history records and checkpoints of the migration range to `seqs`, reading at most +/// one row more than the row limit. Returns the last seq whose rows were all read. +fn scan_history_metadata( + transaction: &FlowTransaction<'_>, + cursor: &RunCursor, + seqs: &mut BTreeMap, +) -> Result { + let rows = cursor.rows; + let prefix = run_table_prefix(rows.flow_hash, rows.run_id, RunTable::HistoryRowMetadata); + let limit = MAX_MIGRATION_SEGMENT_HISTORY_ROWS + 1; + let page = transaction.scan_messages::( + ColumnFamily::Flow, + &seq_bound(&prefix, cursor.record.history_open_segment_first_seq), + &seq_bound(&prefix, cursor.seq + 1), + limit, + "history metadata", + )?; + let is_truncated = page.len() == limit; + let mut last_seq = cursor.seq; + for (key, metadata) in page { + let suffix = key + .strip_prefix(prefix.as_slice()) + .filter(|suffix| suffix.len() == 13) + .ok_or_else(|| ApplyError::InvariantViolation { + detail: "history metadata key must have a 13-byte suffix".to_owned(), + })?; + let seq = u64::from_be_bytes(suffix[..8].try_into().expect("eight seq bytes")); + last_seq = seq; + let seq_history = seqs.entry(seq).or_default(); + seq_history.bytes = seq_history.bytes.saturating_add(metadata.row_size_bytes); + seq_history.history_rows += 1; + if suffix[8] == RunTable::HistoryRecord as u8 { + seq_history.record = Some(metadata); + } + } + Ok(if is_truncated { + last_seq - 1 + } else { + cursor.seq + }) +} + +/// Adds the values superseded inside the migration range to `seqs`, reading at most one more +/// than the superseded value limit. Returns the last seq whose values were all read. +fn scan_superseded_values( + transaction: &FlowTransaction<'_>, + cursor: &RunCursor, + seqs: &mut BTreeMap, +) -> Result { + let rows = cursor.rows; + let prefix = run_table_prefix(rows.flow_hash, rows.run_id, RunTable::ValueGarbage); + let limit = + usize::try_from(MAX_SEGMENT_SUPERSEDED_VALUES).expect("the value limit fits in usize") + 1; + let page = transaction.scan_messages::( + ColumnFamily::Flow, + &seq_bound(&prefix, cursor.record.history_open_segment_first_seq), + &seq_bound(&prefix, cursor.seq + 1), + limit, + "value garbage", + )?; + let is_truncated = page.len() == limit; + let mut last_seq = cursor.seq; + for (key, garbage) in page { + let (seq, _) = garbage_identity(&key, &prefix)?; + last_seq = seq; + let seq_history = seqs.entry(seq).or_default(); + seq_history.bytes = seq_history.bytes.saturating_add(garbage.size_bytes); + seq_history.superseded_values += 1; + } + Ok(if is_truncated { + last_seq - 1 + } else { + cursor.seq + }) +} + +/// Returns the newest event ID that the records in `seqs` through `last_seq` assigned, or +/// `before_range` when none did. +/// +/// Records older than `last_assigned_event_id` are decoded; events of pruned records then widen +/// the next range. +fn newest_event_id_through( + transaction: &FlowTransaction<'_>, + cursor: &RunCursor, + seqs: &BTreeMap, + last_seq: u64, + before_range: i64, +) -> Result { + let mut newest = before_range; + for (seq, seq_history) in seqs.range(..=last_seq) { + let Some(metadata) = &seq_history.record else { + continue; + }; + if metadata.last_assigned_event_id > 0 { + newest = newest.max(metadata.last_assigned_event_id); + continue; + } + let key = history_record_key(cursor.rows.flow_hash, cursor.rows.run_id, *seq); + let record = transaction + .get_message::(ColumnFamily::History, &key, "history record")? + .ok_or_else(|| ApplyError::InvariantViolation { + detail: format!("history record {seq} has metadata but no row"), + })?; + if let Some(event_id) = record.events.iter().map(|event| event.event_id).max() { + newest = newest.max(event_id); + } + } + Ok(newest) +} + +/// Returns the first key of `seq` in a run table whose keys start with an eight-byte seq. +fn seq_bound(prefix: &[u8], seq: u64) -> Vec { + let mut key = prefix.to_vec(); + key.extend_from_slice(&seq.to_be_bytes()); + key +} + +fn is_running(record: &RunRecord) -> bool { + record.status == FlowStatus::Running as i32 +} diff --git a/server-zd/crates/dexzd-apply/src/history_offload/segment_seal.rs b/server-zd/crates/dexzd-apply/src/history_offload/segment_seal.rs index 0da191a39d..1f2657e6f9 100644 --- a/server-zd/crates/dexzd-apply/src/history_offload/segment_seal.rs +++ b/server-zd/crates/dexzd-apply/src/history_offload/segment_seal.rs @@ -21,7 +21,8 @@ use dexzd_keyspace::row_keys::{ }; use dexzd_keyspace::{ColumnFamily, RunTable, ValueId, run_prefix, run_table_prefix}; use dexzd_protocol::dexzd::v1::{ - HistorySegmentRecord, HistorySegmentState, HistorySegmentValueRecord, ValueGarbageRecord, + HistorySegmentRecord, HistorySegmentState, HistorySegmentValueRecord, RunRecord, + ValueGarbageRecord, }; use prost::Message; @@ -77,14 +78,7 @@ pub(crate) fn seal_history_segment_if_due( record.history_open_segment_superseded_value_count = record .history_open_segment_superseded_value_count .saturating_add(additions.superseded_values); - let target_bytes = match record - .settings - .unwrap_or_default() - .history_segment_target_bytes - { - 0 => DEFAULT_HISTORY_SEGMENT_TARGET_BYTES, - configured => configured, - }; + let target_bytes = segment_target_bytes(record); let is_due = has_closed_run || record.history_open_segment_bytes >= target_bytes || record.history_open_segment_superseded_value_count >= MAX_SEGMENT_SUPERSEDED_VALUES; @@ -144,11 +138,37 @@ fn seal_open_segment( transaction: &mut FlowTransaction<'_>, cursor: &mut RunCursor, now_micros: i64, +) -> Result<(), ApplyError> { + let bounds = SealedSegmentBounds { + first_seq: cursor.record.history_open_segment_first_seq, + last_seq: cursor.seq, + first_event_id: cursor.record.history_open_segment_first_event_id, + last_event_id: cursor.record.next_event_id.max(1) - 1, + }; + write_sealed_segment(transaction, cursor, &bounds, now_micros)?; + open_segment_after_command(cursor); + Ok(()) +} + +/// Seq and event ID range of a segment that apply seals. +pub(crate) struct SealedSegmentBounds { + pub(crate) first_seq: u64, + pub(crate) last_seq: u64, + pub(crate) first_event_id: i64, + /// Smaller than `first_event_id` when the seqs assigned no event. + pub(crate) last_event_id: i64, +} + +/// Writes the SEALED segment, its value list, and the pending index row, then emits the effect. +pub(crate) fn write_sealed_segment( + transaction: &mut FlowTransaction<'_>, + cursor: &mut RunCursor, + bounds: &SealedSegmentBounds, + now_micros: i64, ) -> Result<(), ApplyError> { let rows = cursor.rows; - let first_seq = cursor.record.history_open_segment_first_seq; - let last_seq = cursor.seq; - let next_event_id = cursor.record.next_event_id.max(1); + let first_seq = bounds.first_seq; + let last_seq = bounds.last_seq; let garbage_prefix = run_table_prefix(rows.flow_hash, rows.run_id, RunTable::ValueGarbage); let mut start = garbage_prefix.clone(); start.extend_from_slice(&first_seq.to_be_bytes()); @@ -181,8 +201,8 @@ fn seal_open_segment( &HistorySegmentRecord { first_seq, last_seq, - first_event_id: cursor.record.history_open_segment_first_event_id, - last_event_id: next_event_id - 1, + first_event_id: bounds.first_event_id, + last_event_id: bounds.last_event_id, state: HistorySegmentState::Sealed as i32, sealed_at_micros: now_micros, superseded_value_count, @@ -196,14 +216,25 @@ fn seal_open_segment( history_upload_pending_index_key(partition, rows.flow_hash, rows.run_id), Vec::new(), ); - open_segment_after_command(cursor); let run_id = cursor.run_id(); cursor.add_effect(ApplyEffect::HistorySegmentSealed { run_id }); Ok(()) } +/// Returns the run's segment target bytes; runs older than the setting record zero. +pub(crate) fn segment_target_bytes(record: &RunRecord) -> u64 { + match record + .settings + .unwrap_or_default() + .history_segment_target_bytes + { + 0 => DEFAULT_HISTORY_SEGMENT_TARGET_BYTES, + configured => configured, + } +} + /// Opens an empty segment whose seqs and events start right after the command's. -fn open_segment_after_command(cursor: &mut RunCursor) { +pub(crate) fn open_segment_after_command(cursor: &mut RunCursor) { let first_seq = cursor.seq + 1; let record = &mut cursor.record; record.history_open_segment_first_seq = first_seq; @@ -213,7 +244,7 @@ fn open_segment_after_command(cursor: &mut RunCursor) { } /// Returns the superseded seq and value ID of a value garbage key. -fn garbage_identity(key: &[u8], prefix: &[u8]) -> Result<(u64, ValueId), ApplyError> { +pub(crate) fn garbage_identity(key: &[u8], prefix: &[u8]) -> Result<(u64, ValueId), ApplyError> { let suffix = key .strip_prefix(prefix) .filter(|suffix| suffix.len() == 20) diff --git a/server-zd/crates/dexzd-apply/tests/history_migration.rs b/server-zd/crates/dexzd-apply/tests/history_migration.rs new file mode 100644 index 0000000000..07bafa1198 --- /dev/null +++ b/server-zd/crates/dexzd-apply/tests/history_migration.rs @@ -0,0 +1,1218 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Migrating the history of runs that started before their partition offloaded history. + +use std::collections::{BTreeMap, BTreeSet}; + +use dexzd_apply::read_model::{get_history_events, read_history_segment_content}; +use dexzd_apply::{ + ApplyContext, ApplyEffect, ApplyOutcome, FlowStateReader, StateReadError, StateRow, StateWrite, + apply_flow_command, +}; +use dexzd_history_segment::{ + HistorySegmentContent, decode_history_segment, encode_history_segment, +}; +use dexzd_keyspace::row_keys::{ + history_record_key, history_row_metadata_key, history_segment_value_prefix, + history_upload_pending_index_key, prefix_range, run_named_key, run_singleton_key, +}; +use dexzd_keyspace::{ + ColumnFamily, IndexKind, PartitionHasher, RunId, RunTable, TenantId, ValueId, run_prefix, + run_table_prefix, +}; +use dexzd_protocol::dex::{ + AttributeWrite, ChannelMessage, FlowResetType, FlowTimeoutPolicy, GetHistoryEventsResponse, + PublishToChannelRequest, ResetFlowRequest, SetAttributesRequest, StartFlowRequest, + StopFlowRequest, StopType, Value, WorkerTarget, value, +}; +use dexzd_protocol::dexzd::v1::{ + AttributeRecord, AttributesSetCommand, ChannelPublishCommand, CommandReply, FlowCommand, + FlowStartCommand, FlowStopCommand, HistoryRecord, HistoryRowMetadata, HistorySegmentRecord, + HistorySegmentState, HistorySegmentUploadedCommand, HistorySegmentValueRecord, + RetentionTimerId, RunPurgeCommand, RunRecord, RunSettings, SetHistoryOffloadModeCommand, + TimeTravelCommand, TimerId, TimersFiredCommand, command_reply, flow_command::Command, + stored_value, timer_id, +}; +use prost::Message; + +const SECOND: i64 = 1_000_000; +/// Most commands one migration may take in these tests before it must have caught up. +const MAX_MIGRATION_COMMANDS: usize = 60; + +#[derive(Default, Clone, PartialEq, Eq, Debug)] +struct MemoryState { + rows: BTreeMap<(ColumnFamily, Vec), Vec>, +} + +impl FlowStateReader for MemoryState { + fn get( + &self, + column_family: ColumnFamily, + key: &[u8], + ) -> Result>, StateReadError> { + Ok(self.rows.get(&(column_family, key.to_vec())).cloned()) + } + + fn scan( + &self, + column_family: ColumnFamily, + start: &[u8], + end: &[u8], + limit: usize, + ) -> Result, StateReadError> { + Ok(self + .rows + .range((column_family, start.to_vec())..(column_family, end.to_vec())) + .take(limit) + .map(|((_, key), value)| (key.clone(), value.clone())) + .collect()) + } +} + +impl MemoryState { + fn commit(&mut self, outcome: &ApplyOutcome) { + for write in &outcome.writes { + match write { + StateWrite::Put { + column_family, + key, + value, + } => { + self.rows + .insert((*column_family, key.clone()), value.clone()); + } + StateWrite::Delete { column_family, key } + | StateWrite::SingleDelete { column_family, key } => { + self.rows.remove(&(*column_family, key.clone())); + } + StateWrite::DeleteRange { + column_family, + start, + end, + } => { + let doomed: Vec<_> = self + .rows + .range((*column_family, start.clone())..(*column_family, end.clone())) + .map(|(key, _)| key.clone()) + .collect(); + for key in doomed { + self.rows.remove(&key); + } + } + } + } + } +} + +struct Harness { + state: MemoryState, + context: ApplyContext, + now_micros: i64, + flow_id: String, + run_id: Vec, + settings: RunSettings, + request_count: u32, +} + +impl Harness { + fn new(settings: RunSettings) -> Self { + let flow_id = "migrating-flow"; + let hasher = PartitionHasher::new(&[7; 16], 1024).expect("valid partition hasher"); + let flow_hash = hasher.flow_hash(TenantId::DEFAULT, flow_id); + Self { + state: MemoryState::default(), + context: ApplyContext { + partition: hasher.partition_of(flow_hash), + flow_hash, + }, + now_micros: 1_800_000_000 * SECOND, + flow_id: flow_id.to_owned(), + run_id: Vec::new(), + settings, + request_count: 0, + } + } + + /// A run that started while its partition did not offload, with `commands` commands since. + fn legacy(settings: RunSettings, commands: usize) -> Self { + let mut harness = Self::new(settings); + harness.start(); + for round in 0..commands { + harness.apply_round(&format!("legacy{round}")); + } + assert_eq!(harness.run_record().history_open_segment_first_seq, 0); + harness + } + + fn command(&mut self, body: Command) -> FlowCommand { + self.now_micros += 1000; + FlowCommand { + flow_id: self.flow_id.clone(), + run_id: if matches!( + body, + Command::FlowStart(_) | Command::SetHistoryOffloadMode(_) + ) { + Vec::new() + } else { + self.run_id.clone() + }, + expected_run_attempt: 0, + epoch: 1, + behavior_version: 1, + proposed_at_micros: self.now_micros, + command_id: (self.now_micros as u128).to_be_bytes().to_vec(), + tenant_id: 0, + is_transactional_rpc_in_flight: false, + command: Some(body), + } + } + + /// Applies twice on the committed state, requires identical outcomes, and commits. + fn apply(&mut self, body: Command) -> ApplyOutcome { + let command = self.command(body); + let outcome = + apply_flow_command(&self.context, &self.state, &command).expect("apply succeeds"); + let again = + apply_flow_command(&self.context, &self.state, &command).expect("apply succeeds"); + assert_eq!(outcome, again, "apply must be deterministic"); + self.state.commit(&outcome); + outcome + } + + fn set_offload_mode(&mut self, is_enabled: bool) { + let outcome = self.apply(Command::SetHistoryOffloadMode( + SetHistoryOffloadModeCommand { + enabled: is_enabled, + partition: u32::from(self.context.partition.0), + }, + )); + assert!(is_ok(&outcome.reply), "{:?}", outcome.reply); + } + + fn start(&mut self) -> ApplyOutcome { + let outcome = self.apply(Command::FlowStart(FlowStartCommand { + new_run_id: vec![9; 16], + request: Some(StartFlowRequest { + flow_id: self.flow_id.clone(), + flow_type: "MigratingFlow".to_owned(), + request_id: "start-1".to_owned(), + ..StartFlowRequest::default() + }), + settings: Some(self.settings), + resolved_timeout_policy: FlowTimeoutPolicy::Fail as i32, + worker_target: Some(WorkerTarget { is_pull_mode: true }), + ..FlowStartCommand::default() + })); + let Some(command_reply::Reply::StartFlow(reply)) = &outcome.reply.reply else { + panic!("the run starts: {:?}", outcome.reply); + }; + self.run_id = reply.run_id.clone(); + outcome + } + + /// Replaces the `profile` Attribute when `tag` ends with an even digit and publishes to a + /// Channel otherwise, so records alternate between superseding a value and adding an event. + fn apply_round(&mut self, tag: &str) -> ApplyOutcome { + let is_write = tag + .bytes() + .last() + .is_some_and(|digit| (digit - b'0').is_multiple_of(2)); + if is_write { + self.set_attribute("profile", large_text(tag, 300)) + } else { + self.publish("updates", &large_text(tag, 200)) + } + } + + fn set_attribute(&mut self, key: &str, text: String) -> ApplyOutcome { + self.set_attributes(&[(key.to_owned(), text)]) + } + + fn set_attributes(&mut self, values: &[(String, String)]) -> ApplyOutcome { + self.request_count += 1; + let request_id = format!("write-{}", self.request_count); + let outcome = self.apply(Command::AttributesSet(AttributesSetCommand { + request: Some(SetAttributesRequest { + attributes: values + .iter() + .map(|(key, text)| AttributeWrite { + key: key.clone(), + value: Some(string_value(text)), + ..AttributeWrite::default() + }) + .collect(), + request_id: request_id.clone(), + ..SetAttributesRequest::default() + }), + hard_delete_objects: Vec::new(), + request_fingerprint: request_id.into_bytes(), + })); + assert!(is_ok(&outcome.reply), "{:?}", outcome.reply); + outcome + } + + fn publish(&mut self, channel_name: &str, text: &str) -> ApplyOutcome { + self.request_count += 1; + let outcome = self.apply(Command::ChannelPublish(ChannelPublishCommand { + request: Some(PublishToChannelRequest { + messages: vec![ChannelMessage { + channel_name: channel_name.to_owned(), + value: Some(string_value(text)), + message_id: format!("message-{}", self.request_count), + }], + ..PublishToChannelRequest::default() + }), + })); + assert!(is_ok(&outcome.reply), "{:?}", outcome.reply); + outcome + } + + fn stop(&mut self) -> ApplyOutcome { + let outcome = self.apply(Command::FlowStop(FlowStopCommand { + request: Some(StopFlowRequest { + flow_id: self.flow_id.clone(), + stop_type: StopType::Cancel as i32, + ..StopFlowRequest::default() + }), + })); + assert!(is_ok(&outcome.reply), "{:?}", outcome.reply); + outcome + } + + fn travel_to_seq(&mut self, seq: u64) -> ApplyOutcome { + let timestamp_micros = self.record_metadata(seq).timestamp_micros; + let outcome = self.apply(Command::TimeTravel(TimeTravelCommand { + should_restore_running_as_pending: false, + request: Some(ResetFlowRequest { + flow_id: self.flow_id.clone(), + reset_type: FlowResetType::HistoryEventTime as i32, + reason: "test".to_owned(), + skip_writes_reapply: true, + history_event_time: (i128::from(timestamp_micros) * 1000).to_string(), + ..ResetFlowRequest::default() + }), + })); + assert!( + matches!( + outcome.reply.reply, + Some(command_reply::Reply::TimeTravel(_)) + ), + "{:?}", + outcome.reply + ); + outcome + } + + fn fire_retention(&mut self) -> ApplyOutcome { + self.now_micros = self.run_record().retention_deadline_micros; + self.apply(Command::TimersFired(TimersFiredCommand { + timers: vec![TimerId { + run_attempt: 0, + kind: Some(timer_id::Kind::Retention(RetentionTimerId {})), + }], + })) + } + + /// Acknowledges the oldest SEALED segment with the checksum of its encoded content. + fn acknowledge_oldest_sealed(&mut self) -> Option { + let uploaded_through_seq = self.run_record().history_uploaded_through_seq; + let segment = self + .segments() + .into_iter() + .find(|segment| segment.last_seq > uploaded_through_seq)?; + let encoded = encode_history_segment(&self.segment_content(segment.first_seq)) + .expect("the content encodes"); + let outcome = self.apply(Command::HistorySegmentUploaded( + HistorySegmentUploadedCommand { + first_seq: segment.first_seq, + last_seq: segment.last_seq, + content_checksum: encoded.content_checksum.to_vec(), + object_size_bytes: encoded.bytes.len() as u64, + }, + )); + assert!(is_ok(&outcome.reply), "{:?}", outcome.reply); + assert_eq!( + self.run_record().history_uploaded_through_seq, + segment.last_seq + ); + Some(segment) + } + + fn acknowledge_every_sealed(&mut self) { + while self.acknowledge_oldest_sealed().is_some() {} + assert!(!self.has_upload_pending_index_row()); + } + + /// Applies rounds until the run's migration caught up, requiring exactly one sealed segment + /// per command. Returns the commands the migration took. + fn migrate_to_completion(&mut self, tag: &str) -> usize { + for command_count in 1..=MAX_MIGRATION_COMMANDS { + let outcome = self.apply_round(&format!("{tag}{command_count}")); + assert_eq!( + sealed_effect_count(&outcome), + 1, + "every command of a migrating run seals one segment" + ); + assert_segment_chain(self); + if self.run_record().history_migration_started_seq == 0 { + return command_count; + } + } + panic!("the migration did not catch up within {MAX_MIGRATION_COMMANDS} commands"); + } + + fn run(&self) -> RunId { + RunId::from_slice(&self.run_id).expect("the harness has a run") + } + + fn get( + &self, + column_family: ColumnFamily, + key: &[u8], + ) -> Option { + self.state + .rows + .get(&(column_family, key.to_vec())) + .map(|bytes| RowMessage::decode(bytes.as_slice()).expect("row decodes")) + } + + fn run_record(&self) -> RunRecord { + let key = run_singleton_key(self.context.flow_hash, self.run(), RunTable::RunRecord); + self.get(ColumnFamily::Flow, &key) + .expect("the run record exists") + } + + fn last_seq(&self) -> u64 { + self.run_record().next_run_seq - 1 + } + + fn record_metadata(&self, seq: u64) -> HistoryRowMetadata { + self.get( + ColumnFamily::Flow, + &history_row_metadata_key( + self.context.flow_hash, + self.run(), + seq, + RunTable::HistoryRecord, + 0, + ), + ) + .unwrap_or_else(|| panic!("history record {seq} has metadata")) + } + + /// Seqs of the history records stored locally, ascending. + fn stored_record_seqs(&self) -> Vec { + let prefix = run_table_prefix(self.context.flow_hash, self.run(), RunTable::HistoryRecord); + let (start, end) = prefix_range(&prefix); + self.state + .rows + .range((ColumnFamily::History, start)..(ColumnFamily::History, end)) + .map(|((_, key), _)| { + u64::from_be_bytes(key[prefix.len()..].try_into().expect("eight-byte seq")) + }) + .collect() + } + + /// Event IDs of the history records stored locally. + fn stored_event_ids(&self) -> BTreeSet { + self.stored_record_seqs() + .into_iter() + .flat_map(|seq| { + let key = history_record_key(self.context.flow_hash, self.run(), seq); + let record: HistoryRecord = self + .get(ColumnFamily::History, &key) + .expect("the record is stored"); + record.events.into_iter().map(|event| event.event_id) + }) + .collect() + } + + fn segments(&self) -> Vec { + self.table_rows::(RunTable::HistorySegment) + .into_values() + .collect() + } + + fn segment_values(&self, first_seq: u64) -> BTreeMap { + let prefix = history_segment_value_prefix(self.context.flow_hash, self.run(), first_seq); + let (start, end) = prefix_range(&prefix); + self.state + .rows + .range((ColumnFamily::Flow, start)..(ColumnFamily::Flow, end)) + .map(|((_, key), value)| { + ( + ValueId::from_key_suffix(&key[prefix.len()..]).expect("value ID suffix"), + HistorySegmentValueRecord::decode(value.as_slice()).expect("row decodes"), + ) + }) + .collect() + } + + fn segment_content(&self, first_seq: u64) -> HistorySegmentContent { + let content = read_history_segment_content( + &self.state, + self.context.flow_hash, + self.run(), + first_seq, + ) + .expect("the segment content reads") + .expect("the segment exists"); + let encoded = encode_history_segment(&content).expect("the content encodes"); + assert_eq!( + decode_history_segment(&encoded.bytes).expect("the object decodes"), + content + ); + content + } + + fn table_rows( + &self, + table: RunTable, + ) -> BTreeMap, RowMessage> { + let prefix = run_table_prefix(self.context.flow_hash, self.run(), table); + let (start, end) = prefix_range(&prefix); + self.state + .rows + .range((ColumnFamily::Flow, start)..(ColumnFamily::Flow, end)) + .map(|((_, key), value)| { + ( + key[prefix.len()..].to_vec(), + RowMessage::decode(value.as_slice()).expect("row decodes"), + ) + }) + .collect() + } + + fn has_upload_pending_index_row(&self) -> bool { + self.state.rows.contains_key(&( + ColumnFamily::PartitionIndex, + history_upload_pending_index_key( + self.context.partition, + self.context.flow_hash, + self.run(), + ), + )) + } + + fn attribute_value_id(&self, key: &str) -> ValueId { + let row_key = run_named_key(self.context.flow_hash, self.run(), RunTable::Attribute, key); + let record: AttributeRecord = self + .get(ColumnFamily::Flow, &row_key) + .expect("the Attribute exists"); + match record.value.and_then(|stored| stored.kind) { + Some(stored_value::Kind::TableValue(reference)) => { + let value_id = reference.value_id.expect("table values have an ID"); + ValueId { + run_seq: value_id.run_seq, + index_in_command: value_id.index_in_command, + } + } + other => panic!("Attribute {key} is not a table value: {other:?}"), + } + } + + fn history_page(&self) -> GetHistoryEventsResponse { + get_history_events( + &self.state, + self.context.flow_hash, + self.run(), + 0, + 1000, + &[], + 0, + usize::MAX, + ) + .expect("history reads") + } + + /// Clears the event ID fields of every history record's metadata, as rows of binaries + /// older than `last_assigned_event_id` store them. + fn clear_metadata_event_ids(&mut self) { + let prefix = run_table_prefix( + self.context.flow_hash, + self.run(), + RunTable::HistoryRowMetadata, + ); + for ((column_family, key), value) in &mut self.state.rows { + let is_record_metadata = *column_family == ColumnFamily::Flow + && key.starts_with(&prefix) + && key.get(prefix.len() + 8) == Some(&(RunTable::HistoryRecord as u8)); + if !is_record_metadata { + continue; + } + let metadata = HistoryRowMetadata::decode(value.as_slice()).expect("row decodes"); + *value = HistoryRowMetadata { + last_assigned_event_id: 0, + max_event_id: 0, + ..metadata + } + .encode_to_vec(); + } + } +} + +/// Settings whose window prunes before the migration: an eight-event window pruned one event at a +/// time, a checkpoint every two commands, and segments of about two records. +fn legacy_settings() -> RunSettings { + RunSettings { + history_window_events: 8, + history_prune_chunk_events: 1, + max_retained_superseded_bytes: 256 << 20, + retention_after_close_micros: 7 * 24 * 3600 * SECOND, + wire_inline_threshold_bytes: 100, + max_value_bytes: 16 << 20, + checkpoint_row_write_bytes_threshold: 1 << 20, + checkpoint_command_threshold: 2, + request_dedup_capacity: 1000, + max_run_state_bytes: 1 << 30, + max_active_step_executions: 10_000, + max_child_flow_result_bytes: 16 << 20, + history_segment_target_bytes: 1_500, + } +} + +fn string_value(text: &str) -> Value { + Value { + kind: Some(value::Kind::StringValue(text.to_owned())), + } +} + +/// A string long enough to be stored as a table value. +fn large_text(tag: &str, length: usize) -> String { + let mut text = format!("{tag}:"); + while text.len() < length { + text.push_str(tag); + } + text +} + +fn is_ok(reply: &CommandReply) -> bool { + matches!(reply.reply, Some(command_reply::Reply::Ok(_))) +} + +fn sealed_effect_count(outcome: &ApplyOutcome) -> usize { + outcome + .effects + .iter() + .filter(|effect| matches!(effect, ApplyEffect::HistorySegmentSealed { .. })) + .count() +} + +fn requests_purge(outcome: &ApplyOutcome) -> bool { + outcome + .effects + .iter() + .any(|effect| matches!(effect, ApplyEffect::RunPurgeRequested { .. })) +} + +/// Asserts that the segments are contiguous from seq 1 and event 1 up to the open segment, that +/// U names the last uploaded one, and that each lists every value superseded inside it. +fn assert_segment_chain(harness: &Harness) { + let record = harness.run_record(); + let mut next_seq = 1; + let mut next_event_id = 1; + for segment in harness.segments() { + assert_eq!(segment.first_seq, next_seq, "{segment:?}"); + assert!(segment.first_seq <= segment.last_seq, "{segment:?}"); + assert_eq!(segment.first_event_id, next_event_id, "{segment:?}"); + assert_segment_state(harness, &segment, &record); + next_seq = segment.last_seq + 1; + next_event_id = segment.last_event_id + 1; + } + assert_eq!(record.history_open_segment_first_seq, next_seq); + assert_eq!(record.history_open_segment_first_event_id, next_event_id); +} + +fn assert_segment_state(harness: &Harness, segment: &HistorySegmentRecord, record: &RunRecord) { + let state = if segment.last_seq <= record.history_uploaded_through_seq { + HistorySegmentState::Uploaded + } else { + HistorySegmentState::Sealed + }; + assert_eq!(segment.state, state as i32, "{segment:?}"); + let values = harness.segment_values(segment.first_seq); + assert_eq!(segment.superseded_value_count, values.len() as u64); + assert!( + values + .values() + .all(|row| (segment.first_seq..=segment.last_seq).contains(&row.superseded_seq)) + ); + if state == HistorySegmentState::Sealed { + let content = harness.segment_content(segment.first_seq); + let stored: Vec = harness + .stored_record_seqs() + .into_iter() + .filter(|seq| (segment.first_seq..=segment.last_seq).contains(seq)) + .collect(); + let held: Vec = content.records.iter().map(|row| row.seq).collect(); + assert_eq!(held, stored, "a SEALED segment holds its stored records"); + assert_eq!(content.values.len(), values.len()); + } +} + +/// Asserts that a history read returns exactly the stored events and reports every other event +/// ID since event 1 as pruned. +fn assert_history_reads_match_stored_events(harness: &Harness) { + let page = harness.history_page(); + let returned: BTreeSet = page.events.iter().map(|event| event.event_id).collect(); + assert_eq!(returned, harness.stored_event_ids()); + let pruned = page + .pruned_event_id_range + .map_or(0..=0, |range| range.start_event_id..=range.end_event_id); + let next_event_id = harness.run_record().next_event_id; + for event_id in 1..next_event_id { + assert_ne!( + returned.contains(&event_id), + pruned.contains(&event_id), + "event {event_id} is returned or reported pruned, never both or neither" + ); + } +} + +/// Seqs of history records that window pruning deleted, between genesis and the newest one. +fn pruned_record_seqs(harness: &Harness) -> BTreeSet { + let stored: BTreeSet = harness.stored_record_seqs().into_iter().collect(); + (1..=harness.last_seq()) + .filter(|seq| !stored.contains(seq)) + .collect() +} + +#[test] +fn a_running_legacy_run_migrates_one_segment_per_command_until_caught_up() { + let mut harness = Harness::legacy(legacy_settings(), 30); + let pruned_before = pruned_record_seqs(&harness); + assert!( + !pruned_before.is_empty(), + "the window pruned before offload" + ); + assert!(!pruned_before.contains(&1), "FlowStart stays"); + let pruned_range = harness + .history_page() + .pruned_event_id_range + .expect("pruned events are reported"); + assert_history_reads_match_stored_events(&harness); + + harness.set_offload_mode(true); + let first = harness.apply_round("migrate0"); + let record = harness.run_record(); + assert_eq!(sealed_effect_count(&first), 1); + assert_eq!(record.history_migration_started_seq, harness.last_seq()); + assert_eq!(record.history_open_segment_bytes, 0); + assert_segment_chain(&harness); + let commands = 1 + harness.migrate_to_completion("migrate"); + assert!(commands >= 3, "the legacy history spans several segments"); + + let segments = harness.segments(); + let (last, migrated) = segments + .split_last() + .expect("the migration sealed segments"); + assert_eq!( + last.last_seq, + harness.last_seq(), + "it ends at the catching-up command" + ); + for segment in migrated { + let content = harness.segment_content(segment.first_seq); + assert!(content.records.len() >= 2, "{segment:?} holds two records"); + assert!( + content + .records + .iter() + .all(|row| !pruned_before.contains(&row.seq)), + "pruned records stay holes" + ); + } + assert_eq!( + harness.history_page().pruned_event_id_range, + Some(pruned_range), + "the migration deletes nothing, so earlier pruned events stay reported as pruned" + ); + assert_history_reads_match_stored_events(&harness); + + let after = harness.apply_round("after0"); + let record = harness.run_record(); + assert_eq!(record.history_migration_started_seq, 0); + assert_eq!( + sealed_effect_count(&after), + 0, + "the open segment follows its bytes" + ); + assert!(record.history_open_segment_bytes > 0); + + for round in 1..12 { + harness.acknowledge_every_sealed(); + harness.apply_round(&format!("after{round}")); + assert_history_reads_match_stored_events(&harness); + } + let pruned_after = harness + .history_page() + .pruned_event_id_range + .expect("pruning continues after uploads"); + assert!(pruned_after.end_event_id > pruned_range.end_event_id); +} + +#[test] +fn a_run_started_while_offload_was_off_migrates_once_it_is_on() { + let mut harness = Harness::new(legacy_settings()); + harness.set_offload_mode(true); + harness.set_offload_mode(false); + harness.start(); + for round in 0..20 { + harness.apply_round(&format!("off{round}")); + } + assert_eq!(harness.run_record().history_open_segment_first_seq, 0); + assert!(harness.segments().is_empty()); + + harness.set_offload_mode(true); + harness.migrate_to_completion("on"); + assert_eq!(harness.segments()[0].first_seq, 1); + harness.acknowledge_every_sealed(); + assert_segment_chain(&harness); + assert_history_reads_match_stored_events(&harness); +} + +#[test] +fn a_closed_legacy_run_is_not_migrated_and_its_purge_deletes_its_history() { + let mut harness = Harness::legacy(legacy_settings(), 12); + harness.stop(); + harness.set_offload_mode(true); + + let fired = harness.fire_retention(); + assert_eq!(sealed_effect_count(&fired), 0); + assert!(requests_purge(&fired), "nothing waits for an upload"); + let record = harness.run_record(); + assert_eq!(record.history_open_segment_first_seq, 0); + assert_eq!(record.history_migration_started_seq, 0); + assert!(harness.segments().is_empty()); + assert!(!harness.stored_record_seqs().is_empty()); + + let purge = harness.apply(Command::RunPurge(RunPurgeCommand {})); + assert!(is_ok(&purge.reply), "{:?}", purge.reply); + assert_every_run_row_purged(&harness); +} + +#[test] +fn a_run_that_closes_while_migrating_seals_once_more_then_keeps_the_rest_local() { + let mut harness = Harness::legacy(legacy_settings(), 30); + harness.set_offload_mode(true); + harness.apply_round("migrate0"); + let stop = harness.stop(); + assert_eq!( + sealed_effect_count(&stop), + 1, + "the closing command still migrates" + ); + let record = harness.run_record(); + assert_ne!( + record.history_migration_started_seq, 0, + "the history is too long to catch up by the close" + ); + let unsealed_from = record.history_open_segment_first_seq; + assert_segment_chain(&harness); + + let postponed = harness.fire_retention(); + assert_eq!( + sealed_effect_count(&postponed), + 0, + "a closed run stops migrating" + ); + assert!( + !requests_purge(&postponed), + "retention waits for the uploads" + ); + assert_eq!( + harness.run_record().history_open_segment_first_seq, + unsealed_from + ); + assert!( + harness + .stored_record_seqs() + .iter() + .any(|seq| *seq >= unsealed_from), + "the rest stays local" + ); + + harness.acknowledge_every_sealed(); + let fired = harness.fire_retention(); + assert_eq!(sealed_effect_count(&fired), 0); + assert!(requests_purge(&fired)); + let purge = harness.apply(Command::RunPurge(RunPurgeCommand {})); + assert!(is_ok(&purge.reply), "{:?}", purge.reply); + assert_every_run_row_purged(&harness); +} + +#[test] +fn time_travel_during_a_migration_restores_its_target_and_the_migration_continues() { + let mut harness = Harness::legacy(legacy_settings(), 30); + harness.set_offload_mode(true); + harness.apply_round("migrate0"); + assert_ne!(harness.run_record().history_migration_started_seq, 0); + let target_seq = harness.last_seq(); + assert_eq!(harness.attribute_value_id("profile").run_seq, target_seq); + harness.apply_round("migrate1"); + harness.apply_round("migrate2"); + + let travel = harness.travel_to_seq(target_seq); + let travel_seq = harness.last_seq(); + assert_eq!( + sealed_effect_count(&travel), + 1, + "the time travel migrates too" + ); + let record = harness.run_record(); + assert_eq!(record.current_attempt, 2); + assert_eq!(harness.attribute_value_id("profile").run_seq, target_seq); + assert_segment_chain(&harness); + + harness.migrate_to_completion("travelled"); + let segment = harness + .segments() + .into_iter() + .find(|segment| (segment.first_seq..=segment.last_seq).contains(&travel_seq)) + .expect("a segment holds the time travel"); + let content = harness.segment_content(segment.first_seq); + assert!(content.records.iter().any(|row| row.seq == travel_seq)); + assert!( + content + .checkpoints + .iter() + .any(|row| row.seq == travel_seq && row.attempt == 2), + "the new attempt's start checkpoint is offloaded with its command" + ); + harness.acknowledge_every_sealed(); + assert_segment_chain(&harness); +} + +#[test] +fn migration_event_ranges_do_not_need_last_assigned_event_ids() { + let mut intact = Harness::legacy(legacy_settings(), 30); + let mut older = Harness::legacy(legacy_settings(), 30); + older.clear_metadata_event_ids(); + for harness in [&mut intact, &mut older] { + harness.set_offload_mode(true); + harness.apply_round("migrate0"); + harness.migrate_to_completion("migrate"); + } + let ranges = |harness: &Harness| { + harness + .segments() + .into_iter() + .map(|segment| { + ( + segment.first_seq, + segment.last_seq, + segment.first_event_id, + segment.last_event_id, + ) + }) + .collect::>() + }; + assert!(ranges(&intact).len() >= 3); + assert_eq!(ranges(&older), ranges(&intact)); +} + +#[test] +fn commands_larger_than_the_target_still_let_the_migration_catch_up() { + let mut harness = Harness::legacy( + RunSettings { + history_window_events: 1_000, + history_segment_target_bytes: 100, + ..legacy_settings() + }, + 16, + ); + harness.set_offload_mode(true); + harness.apply_round("migrate0"); + let commands = 1 + harness.migrate_to_completion("migrate"); + assert!(commands <= 17, "each command seals at least two records"); + let segments = harness.segments(); + let (_, migrated) = segments.split_last().expect("segments"); + for segment in migrated { + assert_eq!( + harness.segment_content(segment.first_seq).records.len(), + 2, + "{segment:?} ends at its second record" + ); + } +} + +#[test] +fn the_superseded_value_limit_cuts_migration_segments() { + let keys: Vec = (0..6_000).map(|index| format!("item-{index}")).collect(); + let mut harness = Harness::new(RunSettings { + history_window_events: 1_000, + history_segment_target_bytes: 1 << 30, + ..legacy_settings() + }); + harness.start(); + for round in 0..4 { + let values: Vec<(String, String)> = keys + .iter() + .map(|key| (key.clone(), large_text(&format!("{key}-{round}"), 110))) + .collect(); + harness.set_attributes(&values); + } + let superseding_seqs: Vec = (3..=harness.last_seq()).collect(); + harness.set_offload_mode(true); + harness.apply_round("migrate0"); + harness.migrate_to_completion("migrate"); + + let segments = harness.segments(); + assert!(segments.len() >= 3, "{segments:?}"); + let first = &segments[0]; + assert_eq!( + first.last_seq, superseding_seqs[0], + "the limit cuts after 6,000 values" + ); + assert_eq!(first.superseded_value_count, 6_000); + let second = &segments[1]; + assert_eq!( + (second.first_seq, second.last_seq), + (superseding_seqs[1], superseding_seqs[2]), + "a segment ends at the seq that reaches the limit" + ); + assert_eq!(second.superseded_value_count, 12_000); + assert_segment_chain(&harness); +} + +#[test] +fn a_legacy_seq_past_the_superseded_value_limit_migrates_in_one_step() { + let keys: Vec = (0..12_000).map(|index| format!("item-{index}")).collect(); + let mut harness = Harness::new(RunSettings { + history_window_events: 1_000, + history_segment_target_bytes: 1 << 30, + ..legacy_settings() + }); + harness.start(); + for round in 0..2 { + let values: Vec<(String, String)> = keys + .iter() + .map(|key| (key.clone(), large_text(&format!("{key}-{round}"), 110))) + .collect(); + harness.set_attributes(&values); + } + let superseding_seq = harness.last_seq(); + let superseded: BTreeSet = harness + .table_rows::(RunTable::ValueGarbage) + .into_keys() + .filter(|suffix| suffix[..8] == superseding_seq.to_be_bytes()) + .map(|suffix| ValueId::from_key_suffix(&suffix[8..]).expect("value ID suffix")) + .collect(); + assert_eq!(superseded.len(), 12_000); + + harness.set_offload_mode(true); + harness.apply_round("migrate0"); + assert!( + harness + .segments() + .iter() + .all(|segment| segment.last_seq < superseding_seq), + "the first step stops before the seq past the limit" + ); + let step = harness.apply_round("migrate1"); + assert_eq!( + sealed_effect_count(&step), + 1, + "one step seals the whole seq" + ); + let segment = harness + .segments() + .into_iter() + .find(|segment| (segment.first_seq..=segment.last_seq).contains(&superseding_seq)) + .expect("a segment holds the seq"); + assert_eq!(segment.first_seq, superseding_seq); + let listed = harness.segment_values(segment.first_seq); + let listed_at_seq: BTreeSet = listed + .iter() + .filter(|(_, row)| row.superseded_seq == superseding_seq) + .map(|(value_id, _)| *value_id) + .collect(); + assert_eq!( + listed_at_seq, superseded, + "every value of the seq is listed" + ); + assert_eq!(segment.superseded_value_count, listed.len() as u64); + assert_eq!( + harness.segment_content(segment.first_seq).values.len(), + listed.len() + ); + harness.migrate_to_completion("migrate"); + assert_segment_chain(&harness); +} + +#[test] +fn turning_offload_off_ends_a_migration_and_its_rest_becomes_a_gap() { + let mut harness = Harness::legacy(legacy_settings(), 30); + harness.set_offload_mode(true); + harness.apply_round("migrate0"); + let record = harness.run_record(); + assert_ne!(record.history_migration_started_seq, 0); + let gap_first_seq = record.history_open_segment_first_seq; + harness.acknowledge_every_sealed(); + let migrated = harness.segments(); + + harness.set_offload_mode(false); + let off = harness.apply_round("off0"); + assert_eq!(sealed_effect_count(&off), 0); + let record = harness.run_record(); + assert_eq!( + record.history_migration_started_seq, 0, + "the migration ended" + ); + assert_eq!( + record.history_open_segment_first_seq, + harness.last_seq() + 1 + ); + for round in 1..14 { + harness.apply_round(&format!("off{round}")); + } + let gap_last_seq = harness.last_seq(); + let gap_last_event_id = harness.run_record().next_event_id - 1; + let gap = gap_first_seq..=gap_last_seq; + assert!( + pruned_record_seqs(&harness) + .iter() + .any(|seq| gap.contains(seq)), + "pruning while offload is off deleted rows of the gap" + ); + assert_history_reads_match_stored_events(&harness); + + harness.set_offload_mode(true); + assert_gap_is_skipped(&mut harness, &migrated, &gap, gap_last_event_id); +} + +#[test] +fn reenabling_offload_skips_the_gap_and_reads_report_its_pruned_events() { + let mut harness = Harness::new(legacy_settings()); + harness.set_offload_mode(true); + harness.start(); + for round in 0..12 { + harness.apply_round(&format!("on{round}")); + harness.acknowledge_every_sealed(); + } + let sealed = harness.segments(); + assert!(!sealed.is_empty()); + let gap_first_seq = harness.run_record().history_open_segment_first_seq; + + harness.set_offload_mode(false); + for round in 0..14 { + harness.apply_round(&format!("off{round}")); + } + let gap_last_seq = harness.last_seq(); + let gap_last_event_id = harness.run_record().next_event_id - 1; + let gap = gap_first_seq..=gap_last_seq; + assert!( + pruned_record_seqs(&harness) + .iter() + .any(|seq| gap.contains(seq)), + "pruning while offload is off deleted rows of the gap" + ); + + harness.set_offload_mode(true); + assert_gap_is_skipped(&mut harness, &sealed, &gap, gap_last_event_id); +} + +/// With offload on again after the gap, seals segments until one follows the gap, then asserts +/// that no segment holds a gap seq and that reads stay consistent while the window passes it. +fn assert_gap_is_skipped( + harness: &mut Harness, + before_gap: &[HistorySegmentRecord], + gap: &std::ops::RangeInclusive, + gap_last_event_id: i64, +) { + let mut round = 0; + while harness.segments().len() == before_gap.len() { + harness.apply_round(&format!("again{round}")); + assert_history_reads_match_stored_events(harness); + round += 1; + assert!(round < 20, "a segment seals after offload is on again"); + } + let segments = harness.segments(); + assert_eq!(&segments[..before_gap.len()], before_gap); + let reopened = &segments[before_gap.len()]; + assert_eq!(reopened.first_seq, gap.end() + 1); + assert_eq!(reopened.first_event_id, gap_last_event_id + 1); + // Content stays inside a segment's seqs, so disjoint seqs keep gap rows out of every object. + for segment in &segments { + assert!(segment.last_seq < *gap.start() || segment.first_seq > *gap.end()); + assert!( + harness + .segment_values(segment.first_seq) + .values() + .all(|row| !gap.contains(&row.superseded_seq)) + ); + } + let reopened_content = harness.segment_content(reopened.first_seq); + assert!( + reopened_content + .records + .iter() + .all(|row| !gap.contains(&row.seq)) + ); + assert!( + reopened_content + .checkpoints + .iter() + .all(|row| !gap.contains(&row.seq)) + ); + assert!( + harness + .stored_record_seqs() + .iter() + .any(|seq| gap.contains(seq)) + ); + for round in 0..30 { + harness.acknowledge_every_sealed(); + harness.apply_round(&format!("passing{round}")); + assert_history_reads_match_stored_events(harness); + } + let record = harness.run_record(); + assert!(record.history_uploaded_through_seq > *gap.end()); + assert!( + record.window_start_seq > *gap.end(), + "the window passed the gap" + ); + let pruned = harness + .history_page() + .pruned_event_id_range + .expect("pruned events are reported"); + assert!(pruned.end_event_id >= gap_last_event_id); + assert!( + harness + .stored_record_seqs() + .iter() + .all(|seq| *seq == 1 || !gap.contains(seq)), + "the window rule pruned the gap" + ); +} + +fn assert_every_run_row_purged(harness: &Harness) { + assert!( + harness + .state + .rows + .keys() + .all( + |(column_family, key)| *column_family != ColumnFamily::PartitionIndex + || key.get(2) != Some(&(IndexKind::HistoryUploadPending as u8)) + ) + ); + let run_prefix = run_prefix(harness.context.flow_hash, harness.run()); + assert!( + harness + .state + .rows + .keys() + .all( + |(column_family, key)| *column_family == ColumnFamily::PartitionIndex + || !key.starts_with(&run_prefix) + ), + "the purge deleted every row of the run" + ); +} diff --git a/server-zd/crates/dexzd-apply/tests/history_segments.rs b/server-zd/crates/dexzd-apply/tests/history_segments.rs index 9d8ab37374..22762d41b9 100644 --- a/server-zd/crates/dexzd-apply/tests/history_segments.rs +++ b/server-zd/crates/dexzd-apply/tests/history_segments.rs @@ -1088,28 +1088,6 @@ fn reenabling_offload_opens_a_fresh_segment_after_the_commands_applied_while_off harness.encode_segment(reopened.first_seq); } -#[test] -fn runs_started_before_offload_keep_no_segment_state_and_keep_their_history() { - let mut harness = Harness::new(small_window_settings(1)); - harness.start(); - harness.set_offload_mode(true); - for round in 0..12 { - let outcome = harness.set_attribute("profile", large_text(&format!("v{round}"), 300)); - assert_eq!(sealed_effect_count(&outcome), 0); - } - assert_eq!(harness.run_record().history_open_segment_first_seq, 0); - assert!(harness.segments().is_empty()); - assert_eq!( - harness.history_record_count(), - harness.last_seq() as usize, - "nothing is uploaded, so nothing is pruned" - ); - assert!( - harness.run_record().window_start_seq > 1, - "the window still moves" - ); -} - #[test] fn window_pruning_deletes_only_uploaded_rows() { let mut harness = Harness::offloading(small_window_settings(2_000)); diff --git a/server-zd/crates/dexzd-engine/src/effects/tests/history_migration_upload_tests.rs b/server-zd/crates/dexzd-engine/src/effects/tests/history_migration_upload_tests.rs new file mode 100644 index 0000000000..9ebc8cbf2b --- /dev/null +++ b/server-zd/crates/dexzd-engine/src/effects/tests/history_migration_upload_tests.rs @@ -0,0 +1,252 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Uploads of history that a run recorded before its partition offloaded history. + +use std::time::Duration; + +use dexzd_apply::FlowStateReader; +use dexzd_apply::read_model::read_history_segment_content; +use dexzd_history_segment::{ + decode_history_segment, encode_history_segment, history_run_object_key_prefix, + history_segment_object_key, +}; +use dexzd_keyspace::row_keys::prefix_range; +use dexzd_keyspace::{ColumnFamily, RunId, RunTable, TenantId, run_table_prefix}; +use dexzd_object_store::ObjectStore; +use dexzd_protocol::dex::{AttributeWrite, SetAttributesRequest, StopFlowRequest, StopType}; +use dexzd_protocol::dexzd::v1::{ + AttributesSetCommand, FlowCommand, FlowStopCommand, HistorySegmentRecord, HistorySegmentState, + RunSettings, SetHistoryOffloadModeCommand, flow_command, +}; +use prost::Message; + +use super::{EffectHarness, is_error_reply, settings, start_request, string_value}; +use crate::flow_coordination::CommitUrgency; + +/// Rounds of 100 ms that a wait allows; paused time makes them instant. +const WAIT_ROUNDS: usize = 3_000; +const FLOW_ID: &str = "legacy-order"; + +#[tokio::test(start_paused = true)] +async fn a_run_started_while_offload_was_off_uploads_all_its_history_once_offload_is_on() { + let harness = EffectHarness::new(); + harness.rebuild_all_partitions().await; + let run_id = harness.start_flow( + start_request(FLOW_ID), + RunSettings { + history_segment_target_bytes: 1, + ..settings() + }, + ); + for round in 0..8 { + write_notes(&harness, run_id, round); + } + assert_eq!( + harness + .run_record(FLOW_ID, run_id) + .history_open_segment_first_seq, + 0 + ); + assert!(segments(&harness, run_id).is_empty()); + + set_history_offload_mode(&harness, true); + let mut commands = 0; + loop { + write_notes(&harness, run_id, 8 + commands); + commands += 1; + if harness + .run_record(FLOW_ID, run_id) + .history_migration_started_seq + == 0 + { + break; + } + assert!(commands < 30, "the migration catches up"); + } + assert!(commands >= 2, "the recorded history took several commands"); + stop_run(&harness, run_id); + + wait_until(&harness, "every segment is uploaded", || { + let record = harness.run_record(FLOW_ID, run_id); + segments(&harness, run_id).iter().all(|segment| { + segment.state == HistorySegmentState::Uploaded as i32 + && segment.last_seq <= record.history_uploaded_through_seq + }) && record.history_uploaded_through_seq == record.next_run_seq - 1 + }) + .await; + assert_uploaded_chain_from_the_first_seq(&harness, run_id).await; + harness.shutdown().await; +} + +/// Requires contiguous segments from seq 1 and event 1, each stored as one object whose content +/// and checksum match committed state, and no other object of the run. +async fn assert_uploaded_chain_from_the_first_seq(harness: &EffectHarness, run_id: RunId) { + let run = harness.run_location(FLOW_ID, run_id); + let mut next_seq = 1; + let mut next_event_id = 1; + let mut expected_keys = Vec::new(); + for segment in segments(harness, run_id) { + assert_eq!( + (segment.first_seq, segment.first_event_id), + (next_seq, next_event_id), + "{segment:?}" + ); + next_seq = segment.last_seq + 1; + next_event_id = segment.last_event_id + 1; + let key = history_segment_object_key( + TenantId::DEFAULT, + run.flow_hash, + run_id, + segment.first_seq, + segment.last_seq, + ); + let object = harness + .durable + .objects + .get(&key) + .await + .expect("the in-memory store reads") + .unwrap_or_else(|| panic!("{key} exists")) + .bytes; + let content = read_history_segment_content( + &harness.flow_effect_operations.snapshot(), + run.flow_hash, + run_id, + segment.first_seq, + ) + .expect("the segment content reads") + .expect("the segment exists"); + assert_eq!( + decode_history_segment(&object).expect("the object decodes"), + content + ); + let encoded = encode_history_segment(&content).expect("committed content encodes"); + assert_eq!(segment.content_checksum, encoded.content_checksum.to_vec()); + expected_keys.push(key); + } + let prefix = history_run_object_key_prefix(TenantId::DEFAULT, run.flow_hash, run_id); + let stored_keys: Vec = harness + .durable + .objects + .list(&prefix, None, 1000) + .await + .expect("the in-memory store lists") + .objects + .into_iter() + .map(|object| object.key) + .collect(); + assert_eq!(stored_keys, expected_keys); +} + +fn write_notes(harness: &EffectHarness, run_id: RunId, round: usize) { + commit( + harness, + run_id, + flow_command::Command::AttributesSet(AttributesSetCommand { + request: Some(SetAttributesRequest { + flow_id: FLOW_ID.to_owned(), + attributes: vec![AttributeWrite { + key: "notes".to_owned(), + value: Some(string_value(&format!("v{round} {}", "x".repeat(200)))), + ..AttributeWrite::default() + }], + request_id: format!("write-{round}"), + ..SetAttributesRequest::default() + }), + hard_delete_objects: Vec::new(), + request_fingerprint: vec![1], + }), + ); +} + +fn stop_run(harness: &EffectHarness, run_id: RunId) { + commit( + harness, + run_id, + flow_command::Command::FlowStop(FlowStopCommand { + request: Some(StopFlowRequest { + flow_id: FLOW_ID.to_owned(), + stop_type: StopType::Terminate as i32, + ..StopFlowRequest::default() + }), + }), + ); +} + +fn commit(harness: &EffectHarness, run_id: RunId, body: flow_command::Command) { + let reply = harness + .flow_effect_operations + .commit( + &EffectHarness::flow(FLOW_ID), + FlowCommand { + flow_id: FLOW_ID.to_owned(), + run_id: run_id.as_bytes().to_vec(), + command: Some(body), + ..FlowCommand::default() + }, + CommitUrgency::Prompt, + ) + .expect("the command commits"); + assert!(!is_error_reply(&reply), "the command failed: {reply:?}"); +} + +/// Commits the offload mode to every partition, as a node with `history.offloadEnabled` does. +fn set_history_offload_mode(harness: &EffectHarness, is_enabled: bool) { + for (partition, flow_id) in harness + .hasher + .partition_command_flow_ids() + .into_iter() + .enumerate() + { + let reply = harness + .flow_effect_operations + .commit( + &EffectHarness::flow(&flow_id), + FlowCommand { + flow_id: flow_id.clone(), + command: Some(flow_command::Command::SetHistoryOffloadMode( + SetHistoryOffloadModeCommand { + enabled: is_enabled, + partition: u32::try_from(partition).expect("few partitions"), + }, + )), + ..FlowCommand::default() + }, + CommitUrgency::Prompt, + ) + .expect("the mode commits"); + assert!(!is_error_reply(&reply), "the mode failed: {reply:?}"); + } +} + +fn segments(harness: &EffectHarness, run_id: RunId) -> Vec { + let run = harness.run_location(FLOW_ID, run_id); + let prefix = run_table_prefix(run.flow_hash, run.run_id, RunTable::HistorySegment); + let (start, end) = prefix_range(&prefix); + harness + .flow_effect_operations + .snapshot() + .scan(ColumnFamily::Flow, &start, &end, usize::MAX) + .expect("memory reads never fail") + .into_iter() + .map(|(_, bytes)| HistorySegmentRecord::decode(bytes.as_slice()).expect("decodes")) + .collect() +} + +/// Waits until `condition` holds, letting tasks and paused time advance. +async fn wait_until(harness: &EffectHarness, what: &str, condition: impl Fn() -> bool) { + for _ in 0..WAIT_ROUNDS { + harness.settle().await; + if condition() { + return; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + panic!("timed out waiting until {what}"); +} diff --git a/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs b/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs index 75e5bf27f1..7ff157fa4c 100644 --- a/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs +++ b/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs @@ -14,6 +14,7 @@ mod deadline_tests; mod dispatch_tests; mod handout_tests; mod hard_delete_tests; +mod history_migration_upload_tests; mod history_upload_tests; mod keyword_index_tests; mod outbox_and_purge_tests; diff --git a/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs b/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs index 8cc025b76a..b416639eb9 100644 --- a/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs +++ b/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs @@ -29,7 +29,7 @@ use crate::testdata_tree::{ }; /// Digest of every file of each version, README included. Regenerate v1 before 1.0; afterward versions are immutable: add `v`. -const PINNED_VERSION_DIGESTS: &[(&str, &str)] = &[("v1", "7f19d73154a70a9ebc4b50879ece2457")]; +const PINNED_VERSION_DIGESTS: &[(&str, &str)] = &[("v1", "e96ae4da37f20885d85eeba5b4f30682")]; /// The restoring node starts this long after the last recorded command. pub(crate) const RESTART_DELAY_MICROS: i64 = 60 * 60 * 1_000_000; diff --git a/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs b/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs index 9263e0d501..e460d2b8cd 100644 --- a/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs +++ b/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs @@ -135,6 +135,7 @@ impl Workload<'_> { .await; self.set_history_offload_modes(&[PartitionId(0)], false) .await; + self.legacy_runs_record_history_without_offload().await; } /// `history-flow`, in partition 1, which offloads history: a replaced Attribute supersedes a @@ -2030,6 +2031,8 @@ pub(crate) async fn put_new_object(store: &InMemoryObjectStore, key: &str, body: ); } +mod legacy_history; + mod load_blobs_purge_tests { use std::sync::Arc; diff --git a/server-zd/crates/dexzd-engine/tests/format_compat/workload/legacy_history.rs b/server-zd/crates/dexzd-engine/tests/format_compat/workload/legacy_history.rs new file mode 100644 index 0000000000..c252b94505 --- /dev/null +++ b/server-zd/crates/dexzd-engine/tests/format_compat/workload/legacy_history.rs @@ -0,0 +1,629 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Runs recorded while their partition did not offload history, and their migration once a +//! node restored from the recording turns offload on (plan section 5.17). + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use dexzd_apply::read_model::{get_history_events, resolve_run_id}; +use dexzd_keyspace::{ValueId, run_prefix}; +use dexzd_protocol::dex::{FlowStatus, GetHistoryEventsResponse}; +use dexzd_protocol::dexzd::v1::{AttributeRecord, HistoryRecord, RunRecord, stored_value}; + +use super::*; +use crate::compat_node::{engine_node_config, start_engine_node}; +use crate::format_decoding::decode_objects; +use crate::golden_replay::RESTART_DELAY_MICROS; +use crate::testdata_tree::{OBJECTS_DIRECTORY, load_objects, read_files, testdata_versions}; + +/// Never offloads history in the recording. +const NEVER_OFFLOADED_PARTITION: PartitionId = PartitionId(2); +/// Turns offload on at the end of the recording, while `legacy-migrating` runs. +const LATE_OFFLOADED_PARTITION: PartitionId = PartitionId(3); +const RUNNING_FLOW_PREFIX: &str = "legacy-running"; +const CLOSED_FLOW_PREFIX: &str = "legacy-closed"; +const MIGRATING_FLOW_PREFIX: &str = "legacy-migrating"; +/// Enough commands for the eight-event window to prune and for several migration segments. +const LEGACY_ROUNDS: usize = 30; +/// Most commands a migration may take in these tests. +const MAX_MIGRATION_COMMANDS: usize = 60; +const RESTORE_SEED: u64 = 0x5eed_2026_1107; + +impl Workload<'_> { + /// `legacy-running`, `legacy-closed`, and `legacy-migrating`: runs with a small window + /// and small segments that record history while their partition does not offload. + pub(super) async fn legacy_runs_record_history_without_offload(&self) { + let running = self + .start_legacy_run( + RUNNING_FLOW_PREFIX, + NEVER_OFFLOADED_PARTITION, + legacy_run_settings(), + ) + .await; + for round in 0..LEGACY_ROUNDS { + self.apply_legacy_round(&running, round).await; + } + let closed = self + .start_legacy_run( + CLOSED_FLOW_PREFIX, + NEVER_OFFLOADED_PARTITION, + RunSettings { + retention_after_close_micros: 60 * SECOND_MICROS, + ..legacy_run_settings() + }, + ) + .await; + for round in 0..4 { + self.apply_legacy_round(&closed, round).await; + } + self.commit(&closed, cancel_command(&closed)).await; + let migrating = self + .start_legacy_run( + MIGRATING_FLOW_PREFIX, + LATE_OFFLOADED_PARTITION, + legacy_run_settings(), + ) + .await; + for round in 0..LEGACY_ROUNDS { + self.apply_legacy_round(&migrating, round).await; + } + self.set_history_offload_modes(&[LATE_OFFLOADED_PARTITION], true) + .await; + for round in LEGACY_ROUNDS..LEGACY_ROUNDS + 2 { + let outcome = self.apply_legacy_round(&migrating, round).await; + assert_eq!(sealed_segment_count(&outcome), 1, "the run migrates"); + } + } + + async fn start_legacy_run( + &self, + prefix: &str, + partition: PartitionId, + settings: RunSettings, + ) -> FlowRun { + let flow_id = self.legacy_flow_id(prefix, partition); + self.start_flow( + StartFlowRequest { + flow_id: flow_id.clone(), + flow_type: "LegacyFlow".to_owned(), + request_id: format!("start-{flow_id}"), + ..StartFlowRequest::default() + }, + settings, + ) + .await + } + + fn legacy_flow_id(&self, prefix: &str, partition: PartitionId) -> String { + (0..) + .map(|index| format!("{prefix}-{index}")) + .find(|flow_id| self.context(flow_id).partition == partition) + .expect("some Flow ID hashes to the partition") + } + + /// Replaces `profile` in even rounds and publishes to `updates` in odd rounds, so records + /// alternate between superseding a value and recording an event. + async fn apply_legacy_round(&self, run: &FlowRun, round: usize) -> ApplyOutcome { + let text = format!("{LONG_PROFILE_TEXT} Round {round}."); + if round.is_multiple_of(2) { + let request_id = format!("{}-write-{round}", run.flow_id); + let request = SetAttributesRequest { + flow_id: run.flow_id.clone(), + attributes: vec![attribute("profile", string_value(&text))], + request_id: request_id.clone(), + ..SetAttributesRequest::default() + }; + self.commit( + run, + Command::AttributesSet(AttributesSetCommand { + request_fingerprint: request_fingerprint(&request), + request: Some(request), + hard_delete_objects: Vec::new(), + }), + ) + .await + } else { + let message_id = format!("{}-message-{round}", run.flow_id); + self.commit( + run, + publish_command(vec![channel_message("updates", &message_id, &text)]), + ) + .await + } + } + + async fn legacy_run(&self, prefix: &str, partition: PartitionId) -> FlowRun { + let flow_id = self.legacy_flow_id(prefix, partition); + let flow_hash = self.context(&flow_id).flow_hash; + let view = self + .node + .committer() + .committed_view() + .expect("open a committed view"); + let lookup_flow_id = flow_id.clone(); + let run_id = tokio::task::spawn_blocking(move || { + resolve_run_id(&view, flow_hash, TenantId::DEFAULT, &lookup_flow_id, "") + .expect("the Flow directory reads") + .expect("the recorded Flow has a run") + }) + .await + .expect("the read task completes"); + FlowRun { flow_id, run_id } + } + + async fn run_record_of(&self, run: &FlowRun) -> RunRecord { + let mut records = self.run_rows::(run, RunTable::RunRecord).await; + assert_eq!(records.len(), 1, "{} has one run record", run.flow_id); + records.remove(0) + } + + async fn history_segments(&self, run: &FlowRun) -> Vec { + self.run_rows::(run, RunTable::HistorySegment) + .await + } + + /// Committed history records of the run in the History column family, by seq. + async fn stored_history_records(&self, run: &FlowRun) -> BTreeMap { + let flow_hash = self.context(&run.flow_id).flow_hash; + let prefix = run_table_prefix(flow_hash, run.run_id, RunTable::HistoryRecord); + let (start, end) = prefix_range(&prefix); + let view = self + .node + .committer() + .committed_view() + .expect("open a committed view"); + let rows = tokio::task::spawn_blocking(move || { + view.scan(ColumnFamily::History, &start, &end, usize::MAX) + .expect("scan committed history") + }) + .await + .expect("the scan task completes"); + rows.into_iter() + .map(|(_, value)| { + let record = HistoryRecord::decode(value.as_slice()).expect("records decode"); + (record.seq, record) + }) + .collect() + } + + async fn history_events(&self, run: &FlowRun) -> GetHistoryEventsResponse { + let flow_hash = self.context(&run.flow_id).flow_hash; + let run_id = run.run_id; + let view = self + .node + .committer() + .committed_view() + .expect("open a committed view"); + tokio::task::spawn_blocking(move || { + get_history_events(&view, flow_hash, run_id, 0, 1000, &[], 0, usize::MAX) + .expect("history reads") + }) + .await + .expect("the read task completes") + } + + async fn segment_content(&self, run: &FlowRun, first_seq: u64) -> HistorySegmentContent { + let flow_hash = self.context(&run.flow_id).flow_hash; + let run_id = run.run_id; + let view = self + .node + .committer() + .committed_view() + .expect("open a committed view"); + tokio::task::spawn_blocking(move || { + read_history_segment_content(&view, flow_hash, run_id, first_seq) + .expect("the segment content reads") + .expect("the segment exists") + }) + .await + .expect("the read task completes") + } + + /// Acknowledges every SEALED segment oldest first, as the uploader would. + async fn acknowledge_sealed_segments(&self, run: &FlowRun) { + loop { + let uploaded_through_seq = self.run_record_of(run).await.history_uploaded_through_seq; + let Some(segment) = self + .history_segments(run) + .await + .into_iter() + .find(|segment| segment.last_seq > uploaded_through_seq) + else { + return; + }; + let content = self.segment_content(run, segment.first_seq).await; + let encoded = encode_history_segment(&content).expect("the segment encodes"); + self.commit( + run, + Command::HistorySegmentUploaded(HistorySegmentUploadedCommand { + first_seq: segment.first_seq, + last_seq: segment.last_seq, + content_checksum: encoded.content_checksum.to_vec(), + object_size_bytes: encoded.bytes.len() as u64, + }), + ) + .await; + } + } + + /// Applies rounds from `first_round` until the migration caught up, requiring one sealed + /// segment per command. Returns the commands it took. + async fn migrate_to_completion(&self, run: &FlowRun, first_round: usize) -> usize { + for command_count in 1..=MAX_MIGRATION_COMMANDS { + let outcome = self + .apply_legacy_round(run, first_round + command_count) + .await; + assert_eq!(sealed_segment_count(&outcome), 1, "one segment per command"); + let record = self.run_record_of(run).await; + assert_segment_chain(&self.history_segments(run).await, &record); + if record.history_migration_started_seq == 0 { + return command_count; + } + } + panic!("the migration did not catch up within {MAX_MIGRATION_COMMANDS} commands"); + } +} + +fn legacy_run_settings() -> RunSettings { + RunSettings { + history_window_events: 8, + history_prune_chunk_events: 1, + checkpoint_command_threshold: 2, + history_segment_target_bytes: 1_500, + ..workload_run_settings() + } +} + +fn cancel_command(run: &FlowRun) -> Command { + Command::FlowStop(FlowStopCommand { + request: Some(StopFlowRequest { + flow_id: run.flow_id.clone(), + reason: "legacy run closed".to_owned(), + stop_type: StopType::Cancel as i32, + ..StopFlowRequest::default() + }), + }) +} + +fn sealed_segment_count(outcome: &ApplyOutcome) -> usize { + outcome + .effects + .iter() + .filter(|effect| matches!(effect, ApplyEffect::HistorySegmentSealed { .. })) + .count() +} + +/// Requires segments contiguous from seq 1 and event 1, followed by the open segment. +fn assert_segment_chain(segments: &[HistorySegmentRecord], record: &RunRecord) { + let mut next_seq = 1; + let mut next_event_id = 1; + for segment in segments { + assert_eq!( + (segment.first_seq, segment.first_event_id), + (next_seq, next_event_id), + "{segment:?}" + ); + assert!(segment.first_seq <= segment.last_seq, "{segment:?}"); + next_seq = segment.last_seq + 1; + next_event_id = segment.last_event_id + 1; + } + assert_eq!( + ( + record.history_open_segment_first_seq, + record.history_open_segment_first_event_id + ), + (next_seq, next_event_id) + ); +} + +fn profile_value_seq(attributes: &[AttributeRecord]) -> u64 { + let [profile] = attributes else { + panic!("the legacy run holds only its profile: {attributes:?}"); + }; + match profile + .value + .as_ref() + .and_then(|stored| stored.kind.as_ref()) + { + Some(stored_value::Kind::TableValue(reference)) => { + let value_id = reference + .value_id + .as_ref() + .expect("table values have an ID"); + ValueId { + run_seq: value_id.run_seq, + index_in_command: value_id.index_in_command, + } + .run_seq + } + other => panic!("the profile is not a table value: {other:?}"), + } +} + +/// A node restored from the newest recorded testdata version, driven with workload helpers. +struct RestoredNode { + node: EngineNode, + clock: Arc, + ids: Arc, + object_store: Arc, + hasher: PartitionHasher, +} + +impl RestoredNode { + async fn start() -> Self { + let version = testdata_versions().pop().expect("testdata holds a version"); + let decoded = decode_objects(&read_files(&version.directory.join(OBJECTS_DIRECTORY))); + let object_store = Arc::new(load_objects(&version.directory).await); + let recorded_node = decoded + .manifest + .nodes + .values() + .next() + .expect("the manifest names the recording node"); + let last_command_micros = decoded + .commands + .iter() + .map(|command| command.proposed_at_micros) + .max() + .expect("the log holds commands"); + let clock = Arc::new(ScriptedClock::starting_at( + last_command_micros + RESTART_DELAY_MICROS, + )); + let ids = Arc::new(SeededIdSource::new(RESTORE_SEED, clock.clone())); + let config = engine_node_config( + &recorded_node.node_id, + recorded_node.incarnation, + decoded.cluster.partition_count, + ); + let node = + start_engine_node(&config, object_store.clone(), clock.clone(), ids.clone()).await; + let hasher = PartitionHasher::new( + &decoded.cluster.partition_hash_key, + decoded.cluster.partition_count, + ) + .expect("the recorded cluster has a valid partition hasher"); + Self { + node, + clock, + ids, + object_store, + hasher, + } + } + + fn workload(&self) -> Workload<'_> { + Workload { + node: &self.node, + clock: &self.clock, + ids: &self.ids, + object_store: &self.object_store, + hasher: &self.hasher, + } + } + + async fn shutdown(self) { + self.node + .shutdown() + .await + .expect("the node releases its lease"); + } +} + +#[tokio::test(start_paused = true)] +async fn a_recorded_running_run_migrates_one_segment_per_command_once_offload_is_on() { + let restored = RestoredNode::start().await; + let workload = restored.workload(); + let run = workload + .legacy_run(RUNNING_FLOW_PREFIX, NEVER_OFFLOADED_PARTITION) + .await; + let record = workload.run_record_of(&run).await; + assert_eq!(record.status, FlowStatus::Running as i32); + assert_eq!(record.history_open_segment_first_seq, 0, "no segment state"); + let stored_before = workload.stored_history_records(&run).await; + let pruned_seqs: BTreeSet = (1..record.next_run_seq) + .filter(|seq| !stored_before.contains_key(seq)) + .collect(); + assert!(!pruned_seqs.is_empty(), "the window pruned before offload"); + let before = workload.history_events(&run).await; + let pruned_range = before + .pruned_event_id_range + .expect("pruned events are reported"); + + workload + .set_history_offload_modes(&[NEVER_OFFLOADED_PARTITION], true) + .await; + let first = workload.apply_legacy_round(&run, LEGACY_ROUNDS).await; + assert_eq!(sealed_segment_count(&first), 1); + let record = workload.run_record_of(&run).await; + assert_eq!( + record.history_migration_started_seq, + record.next_run_seq - 1 + ); + let commands = 1 + workload.migrate_to_completion(&run, LEGACY_ROUNDS).await; + assert!(commands >= 3, "the recorded history spans several segments"); + + let segments = workload.history_segments(&run).await; + let stored = workload.stored_history_records(&run).await; + for segment in &segments { + let content = workload.segment_content(&run, segment.first_seq).await; + let held: Vec = content.records.iter().map(|row| row.seq).collect(); + let expected: Vec = stored + .range(segment.first_seq..=segment.last_seq) + .map(|(seq, _)| *seq) + .collect(); + assert_eq!(held, expected, "{segment:?} holds the stored records"); + assert!(held.iter().all(|seq| !pruned_seqs.contains(seq))); + } + assert_eq!(segments[0].first_seq, 1); + workload.acknowledge_sealed_segments(&run).await; + + let after = workload.history_events(&run).await; + assert_eq!( + after.pruned_event_id_range, + Some(pruned_range), + "events pruned before the migration are still reported as pruned" + ); + let before_ids: Vec = before.events.iter().map(|event| event.event_id).collect(); + let after_ids: Vec = after.events.iter().map(|event| event.event_id).collect(); + assert!( + after_ids.starts_with(&before_ids), + "the migration keeps every event" + ); + assert!( + after_ids + .iter() + .all(|event_id| *event_id == 1 || *event_id > pruned_range.end_event_id) + ); + restored.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn a_recorded_closed_run_is_not_migrated_and_its_purge_deletes_its_local_history() { + let restored = RestoredNode::start().await; + let workload = restored.workload(); + let run = workload + .legacy_run(CLOSED_FLOW_PREFIX, NEVER_OFFLOADED_PARTITION) + .await; + let record = workload.run_record_of(&run).await; + assert_eq!(record.status, FlowStatus::Canceled as i32); + assert!(!workload.stored_history_records(&run).await.is_empty()); + + workload + .set_history_offload_modes(&[NEVER_OFFLOADED_PARTITION], true) + .await; + let fired = workload.fire_due_timers(&run, SECOND_MICROS).await; + assert_eq!(sealed_segment_count(&fired), 0); + assert!( + fired + .effects + .iter() + .any(|effect| matches!(effect, ApplyEffect::RunPurgeRequested { .. })), + "retention waits for no upload" + ); + let record = workload.run_record_of(&run).await; + assert_eq!(record.history_open_segment_first_seq, 0); + assert!(workload.history_segments(&run).await.is_empty()); + + workload + .commit(&run, Command::RunPurge(RunPurgeCommand {})) + .await; + let flow_hash = workload.context(&run.flow_id).flow_hash; + let (start, end) = prefix_range(&run_prefix(flow_hash, run.run_id)); + let view = restored + .node + .committer() + .committed_view() + .expect("open a committed view"); + let remaining = tokio::task::spawn_blocking(move || { + [ + ColumnFamily::Flow, + ColumnFamily::History, + ColumnFamily::Value, + ] + .into_iter() + .map(|column_family| { + view.scan(column_family, &start, &end, usize::MAX) + .expect("scan committed rows") + .len() + }) + .sum::() + }) + .await + .expect("the scan task completes"); + assert_eq!(remaining, 0, "the purge deleted the run's local history"); + restored.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn time_travel_during_the_migration_of_a_recorded_run_is_correct() { + let restored = RestoredNode::start().await; + let workload = restored.workload(); + let run = workload + .legacy_run(RUNNING_FLOW_PREFIX, NEVER_OFFLOADED_PARTITION) + .await; + workload + .set_history_offload_modes(&[NEVER_OFFLOADED_PARTITION], true) + .await; + workload.apply_legacy_round(&run, LEGACY_ROUNDS).await; + let record = workload.run_record_of(&run).await; + assert_ne!(record.history_migration_started_seq, 0); + let target_seq = record.next_run_seq - 1; + let attributes = workload + .run_rows::(&run, RunTable::Attribute) + .await; + assert_eq!(profile_value_seq(&attributes), target_seq); + let target_timestamp_micros = + workload.stored_history_records(&run).await[&target_seq].timestamp_micros; + workload.apply_legacy_round(&run, LEGACY_ROUNDS + 1).await; + workload.apply_legacy_round(&run, LEGACY_ROUNDS + 2).await; + + let travel = workload + .commit( + &run, + Command::TimeTravel(TimeTravelCommand { + should_restore_running_as_pending: false, + request: Some(ResetFlowRequest { + flow_id: run.flow_id.clone(), + reset_type: FlowResetType::HistoryEventTime as i32, + reason: "travel during the migration".to_owned(), + skip_writes_reapply: true, + history_event_time: (i128::from(target_timestamp_micros) * 1000).to_string(), + ..ResetFlowRequest::default() + }), + }), + ) + .await; + assert!( + matches!( + travel.reply.reply, + Some(command_reply::Reply::TimeTravel(ref reply)) if reply.run_attempt == 2 + ), + "{:?}", + travel.reply + ); + assert_eq!( + sealed_segment_count(&travel), + 1, + "the time travel migrates too" + ); + let travel_seq = workload.run_record_of(&run).await.next_run_seq - 1; + let attributes = workload + .run_rows::(&run, RunTable::Attribute) + .await; + assert_eq!( + profile_value_seq(&attributes), + target_seq, + "the travel restored the profile" + ); + + workload + .migrate_to_completion(&run, LEGACY_ROUNDS + 2) + .await; + let segment = workload + .history_segments(&run) + .await + .into_iter() + .find(|segment| (segment.first_seq..=segment.last_seq).contains(&travel_seq)) + .expect("a segment holds the time travel"); + let content = workload.segment_content(&run, segment.first_seq).await; + assert!(content.records.iter().any(|row| row.seq == travel_seq)); + assert!( + content + .checkpoints + .iter() + .any(|row| row.seq == travel_seq && row.attempt == 2), + "the second attempt's start checkpoint is offloaded with the time travel" + ); + workload.acknowledge_sealed_segments(&run).await; + let record = workload.run_record_of(&run).await; + assert_eq!(record.current_attempt, 2); + assert_segment_chain(&workload.history_segments(&run).await, &record); + restored.shutdown().await; +} diff --git a/server-zd/crates/dexzd-simulation/src/apply_checker.rs b/server-zd/crates/dexzd-simulation/src/apply_checker.rs index 7a8d2a1340..b173d0f732 100644 --- a/server-zd/crates/dexzd-simulation/src/apply_checker.rs +++ b/server-zd/crates/dexzd-simulation/src/apply_checker.rs @@ -22,10 +22,11 @@ use prost::Message; use crate::invalid_client_inputs::{check_invalid_client_input_rejected, invalid_client_input}; use crate::invariants::{ - CheckerMemory, CloseBarrierCounts, InvariantViolation, OffloadGapsByRun, WindowPointState, - check_close_barrier, check_deletions_were_uploaded, check_flow, check_fresh_snapshot, - check_open_segment_transitions, check_outbox_writes, check_sealed_segment_encodings, - check_window_value_content, remember_ack, remember_window_points, + CheckerMemory, CloseBarrierCounts, HistoryMigrationCounts, InvariantViolation, + OffloadGapsByRun, WindowPointState, check_close_barrier, check_deletions_were_uploaded, + check_flow, check_fresh_snapshot, check_history_migration, check_open_segment_transitions, + check_outbox_writes, check_sealed_segment_encodings, check_window_value_content, remember_ack, + remember_window_points, }; use crate::memory_state::MemoryFlowState; use crate::state_view::{FlowView, RunSelection}; @@ -51,6 +52,7 @@ pub struct ApplyChecker { history_offload_off_time_travels: usize, /// Every run an admitted transactional RPC ever held, for checks of recovered stores. ever_rpc_held_runs: BTreeSet, + history_migration_counts: HistoryMigrationCounts, } /// When the checker retained one window point, in applied command counts. @@ -93,6 +95,7 @@ impl ApplyChecker { close_barrier_counts: CloseBarrierCounts::default(), history_offload_off_time_travels: 0, ever_rpc_held_runs: BTreeSet::new(), + history_migration_counts: HistoryMigrationCounts::default(), } } @@ -189,6 +192,9 @@ impl ApplyChecker { &mut self.memory.rpc_held_runs, )?; self.close_barrier_counts.merge(&close_barrier_counts); + let migration_counts = + check_history_migration(&command.flow_id, &outcome, &previous_records, &view)?; + self.history_migration_counts.merge(&migration_counts); self.ever_rpc_held_runs .extend(self.memory.rpc_held_runs.iter().copied()); let pruned_points = remember_window_points( @@ -237,6 +243,11 @@ impl ApplyChecker { self.close_barrier_counts } + /// History migrations that started, caught up, or ended because offload turned off. + pub fn history_migration_counts(&self) -> HistoryMigrationCounts { + self.history_migration_counts + } + /// Every run whose pending close an admitted transactional RPC held at some point. pub(crate) fn ever_rpc_held_runs(&self) -> &BTreeSet { &self.ever_rpc_held_runs diff --git a/server-zd/crates/dexzd-simulation/src/apply_simulation.rs b/server-zd/crates/dexzd-simulation/src/apply_simulation.rs index 6468fb6abc..85e8e265c3 100644 --- a/server-zd/crates/dexzd-simulation/src/apply_simulation.rs +++ b/server-zd/crates/dexzd-simulation/src/apply_simulation.rs @@ -15,7 +15,7 @@ use prost::Message; use xxhash_rust::xxh3::xxh3_128; use crate::apply_checker::ApplyChecker; -use crate::invariants::{CloseBarrierCounts, InvariantViolation}; +use crate::invariants::{CloseBarrierCounts, HistoryMigrationCounts, InvariantViolation}; use crate::model::{ClosedErasePhase, CommandGenerator}; /// Replays allowed while shrinking one failure. @@ -49,6 +49,8 @@ pub struct SimulationSummary { pub history_segments_sealed: usize, /// TimeTravel commands applied to a run with segment state while offload was off. pub history_offload_off_time_travels: usize, + /// History migrations of runs that predate their segment state. + pub history_migrations: HistoryMigrationCounts, } impl SimulationSummary { @@ -66,6 +68,7 @@ impl SimulationSummary { self.permission_mapping_commands += other.permission_mapping_commands; self.history_segments_sealed += other.history_segments_sealed; self.history_offload_off_time_travels += other.history_offload_off_time_travels; + self.history_migrations.merge(&other.history_migrations); for (kind, count) in &other.applied_by_kind { *self.applied_by_kind.entry(kind).or_default() += count; } @@ -233,6 +236,7 @@ pub fn run_seed( summary.size_rejections = checker.size_rejections().clone(); summary.close_barrier = checker.close_barrier_counts(); summary.history_offload_off_time_travels = checker.history_offload_off_time_travels(); + summary.history_migrations = checker.history_migration_counts(); summary.final_row_count = checker.state().row_count(); Ok(summary) } diff --git a/server-zd/crates/dexzd-simulation/src/invariants/history_migration_invariants.rs b/server-zd/crates/dexzd-simulation/src/invariants/history_migration_invariants.rs new file mode 100644 index 0000000000..459d68b459 --- /dev/null +++ b/server-zd/crates/dexzd-simulation/src/invariants/history_migration_invariants.rs @@ -0,0 +1,154 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Migration of history that runs recorded before they had segment state (plan section 5.17): +//! bounded work per command, and segment state that matches the migration. + +use std::collections::BTreeMap; + +use dexzd_apply::{ApplyEffect, ApplyOutcome}; +use dexzd_keyspace::RunId; +use dexzd_protocol::dex::FlowStatus; +use dexzd_protocol::dexzd::v1::RunRecord; + +use crate::invariants::InvariantViolation; +use crate::state_view::{FlowView, RunView}; + +/// Migrations that started, caught up, and ended because offload turned off. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct HistoryMigrationCounts { + /// Runs without segment state that started migrating. + pub started: usize, + /// Migrations whose last segment reached the command that sealed it. + pub completed: usize, + /// Migrations that ended because their partition stopped offloading. + pub abandoned: usize, +} + +impl HistoryMigrationCounts { + /// Adds another count to this one. + pub fn merge(&mut self, other: &Self) { + self.started += other.started; + self.completed += other.completed; + self.abandoned += other.abandoned; + } +} + +/// Returns whether the command at `seq` must start or continue the run's migration: it took a +/// run seq while offload was on and left a run running whose history predates its segment state. +pub(crate) fn must_migrate( + previous: &RunRecord, + current: &RunRecord, + view: &FlowView, + seq: u64, +) -> bool { + let has_older_history = + previous.history_open_segment_first_seq == 0 || previous.history_migration_started_seq != 0; + view.is_history_offload_enabled + && has_older_history + && current.next_run_seq == seq + 1 + && current.status == FlowStatus::Running as i32 + && !current.is_purging +} + +/// Returns whether the command at `seq` started or continued the run's migration while offload +/// was on: it sealed one segment at the open segment's start, and the open segment moved right +/// after that segment, which holds two history records, or after the command once it caught up. +pub(crate) fn is_migration_step( + previous: &RunRecord, + current: &RunRecord, + run: &RunView, + view: &FlowView, + seq: u64, +) -> bool { + let (first_seq, first_event_id) = if previous.history_open_segment_first_seq == 0 { + ( + run.attempts.get(&1).map_or(1, |attempt| attempt.start_seq), + 1, + ) + } else if previous.history_migration_started_seq != 0 { + ( + previous.history_open_segment_first_seq, + previous.history_open_segment_first_event_id, + ) + } else { + return false; + }; + let Some(segment) = run.history_segments.get(&first_seq) else { + return false; + }; + let has_caught_up = segment.last_seq == seq + && current.history_migration_started_seq == 0 + && current.history_open_segment_first_seq == seq + 1 + && current.history_open_segment_first_event_id == current.next_event_id.max(1); + let segment_records = run + .history + .iter() + .filter(|(record_seq, _)| (segment.first_seq..=segment.last_seq).contains(record_seq)) + .count(); + let has_moved_after_segment = segment.last_seq < seq + && segment_records >= 2 + && current.history_migration_started_seq != 0 + && current.history_open_segment_first_seq == segment.last_seq + 1 + && current.history_open_segment_first_event_id == segment.last_event_id + 1; + view.is_history_offload_enabled + && segment.first_event_id == first_event_id + && current.history_open_segment_bytes == 0 + && current.history_open_segment_superseded_value_count == 0 + && (has_caught_up || has_moved_after_segment) +} + +/// Requires at most one sealed segment per run and command (H11), and an open segment without +/// tracked bytes while a run migrates (H12). Returns the migrations the command moved. +pub(crate) fn check_history_migration( + flow_id: &str, + outcome: &ApplyOutcome, + previous_records: &BTreeMap, + view: &FlowView, +) -> Result { + let mut seals = BTreeMap::::new(); + for effect in &outcome.effects { + if let ApplyEffect::HistorySegmentSealed { run_id } = effect { + *seals.entry(*run_id).or_default() += 1; + } + } + if let Some((run_id, count)) = seals.iter().find(|(_, count)| **count > 1) { + return Err(InvariantViolation::new( + "H11-one-seal-per-command", + format!("Flow {flow_id} run {run_id} sealed {count} segments in one command"), + )); + } + let mut counts = HistoryMigrationCounts::default(); + for (run_id, previous) in previous_records { + let Some(record) = view.runs.get(run_id).and_then(|run| run.record.as_ref()) else { + continue; + }; + if record.history_migration_started_seq != 0 + && (record.history_open_segment_bytes != 0 + || record.history_open_segment_superseded_value_count != 0) + { + return Err(InvariantViolation::new( + "H12-migration-segment-state", + format!("Flow {flow_id} run {run_id} tracks open segment bytes while migrating"), + )); + } + let was_migrating = previous.history_migration_started_seq != 0; + let is_migrating = record.history_migration_started_seq != 0; + let has_started = previous.history_open_segment_first_seq == 0 + && record.history_open_segment_first_seq != 0; + counts.started += usize::from(has_started); + if (was_migrating || has_started) && !is_migrating { + if view.is_history_offload_enabled { + counts.completed += 1; + } else { + counts.abandoned += 1; + } + } + } + Ok(counts) +} diff --git a/server-zd/crates/dexzd-simulation/src/invariants/history_segment_invariants.rs b/server-zd/crates/dexzd-simulation/src/invariants/history_segment_invariants.rs index 39f7ec163f..9ea6d73fda 100644 --- a/server-zd/crates/dexzd-simulation/src/invariants/history_segment_invariants.rs +++ b/server-zd/crates/dexzd-simulation/src/invariants/history_segment_invariants.rs @@ -20,6 +20,7 @@ use dexzd_protocol::dexzd::v1::{ FlowCommand, HistorySegmentState, RunRecord, flow_command::Command, }; +use crate::invariants::history_migration_invariants::{is_migration_step, must_migrate}; use crate::invariants::{InvariantViolation, RunContext}; use crate::memory_state::MemoryFlowState; use crate::state_view::{FlowView, RunView}; @@ -75,17 +76,21 @@ pub(crate) fn check_open_segment_transitions( record.history_open_segment_superseded_value_count, ) }; + let seq = previous.next_run_seq.max(1); + let is_migration_required = must_migrate(previous, current, view, seq); if previous.history_open_segment_first_seq == 0 { - if current.history_open_segment_first_seq != 0 { + if (current.history_open_segment_first_seq != 0 || is_migration_required) + && !is_migration_step(previous, current, run, view, seq) + { return Err(violation( "H9-open-segment-transition", &context, - "an existing run without segment state gained it", + "an existing run without segment state gained it without migrating, or a \ + running one did not start migrating", )); } continue; } - let seq = previous.next_run_seq.max(1); if current.next_run_seq != seq + 1 { if open(current) != open(previous) { return Err(violation( @@ -126,11 +131,20 @@ pub(crate) fn check_open_segment_transitions( let is_kept = current.history_open_segment_first_seq == first_seq && current.history_open_segment_first_event_id == previous.history_open_segment_first_event_id; - if !(is_kept || (moved_after_command && sealed_now)) { + let has_migration_step = is_migration_step(previous, current, run, view, seq); + let is_valid = if is_migration_required { + has_migration_step + } else { + is_kept || (moved_after_command && sealed_now) || has_migration_step + }; + if !is_valid { return Err(violation( "H9-open-segment-transition", &context, - format!("seq {seq} moved the open segment without sealing it"), + format!( + "seq {seq} moved the open segment without sealing it, or a running \ + migration made no progress" + ), )); } } diff --git a/server-zd/crates/dexzd-simulation/src/invariants/mod.rs b/server-zd/crates/dexzd-simulation/src/invariants/mod.rs index e3604c247c..59971d1b4e 100644 --- a/server-zd/crates/dexzd-simulation/src/invariants/mod.rs +++ b/server-zd/crates/dexzd-simulation/src/invariants/mod.rs @@ -11,6 +11,7 @@ mod attribute_sync_invariants; mod directory_invariants; mod dispatch_queue_invariants; +mod history_migration_invariants; mod history_segment_invariants; mod lock_invariants; mod search_order_invariants; @@ -28,6 +29,8 @@ pub use attribute_sync_invariants::CloseBarrierCounts; pub(crate) use attribute_sync_invariants::{ AttributeSyncAcks, check_close_barrier, check_outbox_writes, remember_ack, }; +pub use history_migration_invariants::HistoryMigrationCounts; +pub(crate) use history_migration_invariants::check_history_migration; pub(crate) use history_segment_invariants::{ OffloadGapsByRun, check_deletions_were_uploaded, check_open_segment_transitions, check_sealed_segment_encodings, diff --git a/server-zd/crates/dexzd-simulation/src/lib.rs b/server-zd/crates/dexzd-simulation/src/lib.rs index 21f6e5cdc0..994493fd9f 100644 --- a/server-zd/crates/dexzd-simulation/src/lib.rs +++ b/server-zd/crates/dexzd-simulation/src/lib.rs @@ -38,8 +38,8 @@ pub use invalid_client_inputs::{ check_invalid_client_input_rejected, invalid_client_input, }; pub use invariants::{ - CheckerMemory, CloseBarrierCounts, GUARDED_ATTRIBUTE_KEYS, InvariantViolation, check_flow, - check_fresh_snapshot, + CheckerMemory, CloseBarrierCounts, GUARDED_ATTRIBUTE_KEYS, HistoryMigrationCounts, + InvariantViolation, check_flow, check_fresh_snapshot, }; pub use memory_state::MemoryFlowState; pub use model::{ diff --git a/server-zd/crates/dexzd-simulation/tests/apply_invariants.rs b/server-zd/crates/dexzd-simulation/tests/apply_invariants.rs index 1b952a71d1..27c8e0dd6a 100644 --- a/server-zd/crates/dexzd-simulation/tests/apply_invariants.rs +++ b/server-zd/crates/dexzd-simulation/tests/apply_invariants.rs @@ -144,6 +144,11 @@ fn apply_keeps_every_invariant_over_random_command_sequences() { total.history_segments_sealed > 0, "the seed set never sealed a history segment: {total:#?}" ); + let migrations = total.history_migrations; + assert!( + migrations.started > 0 && migrations.completed > 0 && migrations.abandoned > 0, + "the seed set never started, completed, and abandoned a history migration: {total:#?}" + ); assert!( total.history_offload_off_time_travels > 0, "the seed set never time traveled a run while its partition did not offload: {total:#?}" diff --git a/server-zd/proto/dexzd/v1/state.proto b/server-zd/proto/dexzd/v1/state.proto index 0333a6e6ef..78f9b3ef5a 100644 --- a/server-zd/proto/dexzd/v1/state.proto +++ b/server-zd/proto/dexzd/v1/state.proto @@ -164,6 +164,9 @@ message RunRecord { uint64 history_open_segment_superseded_value_count = 62; // Last seq of the newest uploaded history segment (U); zero before the first upload. uint64 history_uploaded_through_seq = 63; + // Seq that started migrating older history; zero when not migrating. The open segment then + // tracks no bytes. + uint64 history_migration_started_seq = 64; // First event ID the open history segment can hold: the run's next event ID when it opened. int64 history_open_segment_first_event_id = 66; } diff --git a/server-zd/testdata/v1/README.md b/server-zd/testdata/v1/README.md index 884a0849a1..f516e8e398 100644 --- a/server-zd/testdata/v1/README.md +++ b/server-zd/testdata/v1/README.md @@ -79,6 +79,13 @@ Then the node took a snapshot. After the snapshot: run's only history segment with that value listed, and a HistorySegmentUploaded acknowledgment marks the segment uploaded and removes the run's pending index row. - Partition 0 disables history offload, which deletes its mode record. +- `legacy-running` and `legacy-closed`, in partition 2, which never offloads history, run with an + eight-event window, a checkpoint every two commands, and 1,500-byte segments. Thirty commands + that alternately replace a long `profile` Attribute and publish to a Channel let the window + prune `legacy-running`, which stays running without segment state. `legacy-closed` applies four + such commands and is canceled; its 60-second retention timer stays pending. +- `legacy-migrating`, in partition 3, records the same thirty commands. Then partition 3 enables + history offload, and each of the run's next two commands seals one migration segment. Every `FlowCommand` kind of behavior version 1 appears in the log. Hand-out commands outside `handout-sync` and `handout-travel` carry no task ID, as an older log does. Log cleanup did not run, diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000118 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000118 new file mode 100644 index 0000000000000000000000000000000000000000..ebb74c15f7d7fe174da9a07aa0ecb27b82fc90be GIT binary patch literal 298 zcmZ>9jfisb@porraA9Cz;1Xb5ED;mHQ17?Oc51Mfz?ls~YZepB2`{NibRoG@I-(xu=n3S5^Pbb&`|ZEhjDxW`Pt1MvyEE hgS%m6A+Jxt)ti1AobR8jXV1*axR=h<2ayR#IyNB6Kq!PF%d!kdq%a5;=hXea z<>u-plDrR8Xqg#E6fGK)Z z@nnhsqNpOz{ntw8r~_Z$clfzv&#ay5?A?VN@%3cu^|#K_kQ6qbkg5OFy(S1sBHCZ{_fXVF}z|?MVK<=U4u^@$9E}q5}i~6VPB( cL9eq3Zd0^s54xMTnR*^!HEj6q;N#qfOfzw;NB{r; literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000120 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000120 new file mode 100644 index 0000000000000000000000000000000000000000..f13d4d99f3c55e1d24751eca3f8ec69b4bb8f66d GIT binary patch literal 350 zcmV-k0ipgxMOa!yOixDv07C!(016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zcrXDv76<|m0T@vr{Q)SL0x+z(o~G{Nt?x@n;GwXE^P-spD77#B09d650OC(MHDGPh z0pt*sP{FD!hD&39RA%yqVvFXD>^(&^38cG8OG{K1!?E((&DV^mCC-dXM4F)#W8aqQ z+MDwc6-8N=g%@ECl|I&@!&rGCXj~|g%C%@2lFlfpbQj>j?l^F4BQfgY$R+Z10EkA48saETxB|_ wG>*)*D^K>cb1G6~K}7=5MFRj6-e90X-AU1^J^auT9LD6;=wHEncX*Z?g`aVrH~;_u literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000121 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000121 new file mode 100644 index 0000000000000000000000000000000000000000..76ea84af223e2ded7173b116e9fa8d1dd5ec9b47 GIT binary patch literal 384 zcmV-`0e}8PMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zc`yMw7YG6n0T@yso&qSk0x+z`#(?F=^s(LnNbthHh4Z4B11PmG{Qy|42mt2LKsYcZ zHwS?K8H5(1@Q8fYnX%*_tQa?l9jhfhZo8o&Sp zksvt$F90h5E}s(#1jmSsb>KV!rH@mFX$625Igt%aAq-$lfyoKRwrv}bXlWEM&#C+O z7zrf37FiWla!~^mO5CB#_3}4JMcu&J^sm0ho^SduK|K4vyZ(q+x}t}sMYOAbAy7q; zE5cN9CaP2f&;8$(?$HU(?00BgGHF(>I(vsgwy-_1OOAz eU{pb`vk7ifv}zA@H?B%FkpvN0`@7bBYJ{y=&Z5}> literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000122 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000122 new file mode 100644 index 0000000000000000000000000000000000000000..9b3a032b170536d1ad1b140f23dccc9171473bd5 GIT binary patch literal 350 zcmV-k0ipgxMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zdN2Vx7YG6n0T@ys`~fJK0x<0Ojqr|#+|aoJNPy(dh4Z4B11PmG{Qy{<2LR#^IW%BR z(*a;(P;mF{XOc@e4IK!>2kVw>+k1*=GLSB5HaAhBK2l>B#vrPsOPs0GXaPbgW8c>F z?9KUzNs=gQAy?_JR8<}9H>r6$`>^G<+$?;@I`Q6jrH`|CEj9a{$!|SSY9udT>ny}P zT*|UH8jYR(_Rg@>F@8i7whi%9@tjEWJrek{H7>d3dlE7vu@W4tnnq4F_0? z$`?2QE&wY4H=)HESPTJAHnV`x1^|!>CT3IDSA}# zM2aktBqh)N-<96c3cl=jXkBt=R!w#G-h^ynd-C=A+~(4d8aZJWX_JF>tWN{23>5(jPL|=vZ4?Hd<$ z?alcKYMQERAy)~pm1SM~O>W-KK5V%yISb#hR=oFJ>0>QMY&H9z$!|eWZX_?i^I3^` zxRhmcIy8Rv+q=V7&-gXv?_r$#4qZvNzeUd*wt`$`YpdzU?w2im_c`we7nRYo28G+E}yX<$l-ylH>pY literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000125 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000125 new file mode 100644 index 0000000000000000000000000000000000000000..f5c13d1edcc607aa60e65ecd39cc36be92cbfa4c GIT binary patch literal 380 zcmV-?0fYWTMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zeJ}w!7YG6n0T@ysngS@g0x+wHq>Q-m&*O#xNU+EJh4Z4B11PmG{Qy{v2mt2LKQ>Tr zbC51xr5~KccIaVY&pXJNorN*4iwH@IT0V;OCMkq912_m}V+!|SqXvyd%Z~L#YQae6 zIRGvID*z`f8z~9_B?1h|bpS>lE*UQ-q%0;xHZXxOfGPzeBo~^dX+ENrS-?84?%!jg zZFU{9DoW&{FjPt0pv(61H&{d6z>@lBzvIq#{hu4t{%@{7B95-uk?Bz7>{kd>G31IU zRcwhRjnDnxo!(Ummfm;h`D78bAM5N53Yo%d%Gd34pj$(3a5etaxD<-!t4%WARh-nZchpB(o!!pY@+@SqT_P*&`e#(I@42&<~D%s#HNVJ{3z*k`iicJ^P(ZTZ>vj#c8l?@ArR=&;x9izd4rJ=u}Ge2{1# z=HF75{Q=P^+Gp+yd)>pXDS!Xs+=J*!y8SGMuCUkRDjTk*AG@7r-@E^LGc;1wG%oYh zy@(n>HIk~8P_5KPRt`2Mr41B|fe8%M0FXFTDlSq)u!d<{5aO}PP6$3gC^#IBK}WM( y5G=@%xwbX(IQWo-nlP$KV9E#s02AI|ph4Y9(W*U5Y5EC+Bn)6()j$B9D2B1Mx2k_k*a6hxK*k&%gQ+cX{0${=8#SNHHS z5jVXCSrrv>Q2`W5+}~w;`CF`@Uf_&+Xy0Sccm1Cn)Bf+Szd@ERxshp5?d(fnD*18A zl}egq$wklo;hpZ)2hPCn5%kF-YCqQ5`x`j|Y|7W|ci*iew>X=Dv*vvM_2#A(34}sc0c>P6i+<*44rK)*mqqC+jfv&^mCYX$jn-{Y(bG>05r=G3~Emr~?E56VPB( cL9eq3Zd0^s51{5KXuK^|r3)!`cnAHfv>s=tsQ>@~ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000128 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000128 new file mode 100644 index 0000000000000000000000000000000000000000..de27bcf559e0b6eda875a0284ba75ce823eb5098 GIT binary patch literal 354 zcmV-o0iFItMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zfB`T8Iu{565CIrdApZdmwFdy=&p9?= zPtpOvXjnB!5eQoUmCQH z8yV`_oAVOXG*y{}T&2TSmX)yITZ5C4hOMQb#N;B&vI7w}T^1^hBXjKvq0>QQYy$vP-e90X-AU1^Juom%qV3>8Hy-Z3 IB8s9l6;b7vDF6Tf literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000131 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000131 new file mode 100644 index 0000000000000000000000000000000000000000..9854102c553e4568b672004d0ce6fa0d60d1f5cd GIT binary patch literal 388 zcmV-~0ek*LMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zg8?uBIu{565CIrdAfW;%yaF)6qUNsm)WrGN0Z5RJ*oO0>nFA=bFZ}>m#RvcfkwG~? zMKuSnRT+KAg4C8<1uU>HESS17hq?NG+i8BxcI zq-`CF`CQ!qw7wBIr4yZ+COY5zCZk0DJgdD7{xa`r_6 zlw`4F4JC18X(rG8;ho-94aUIl5%kF;YD3o9+Z(w9Y|7W|bKb2Zw>bNPvEF?C_2#A( z3=A$4F*ynx5F}rD5-T7NmG$vled~`GLIU>jF!j)8FqUNw+@cLl2EXZBe#(r_<4ZI7}w9G>*)*tLb(-Z;((&^ zvJltolP13{LFtja{19m;=HXJ74GPgn+Hda++=u8&x&hh7y7uP$$TFy| zY_pK7jMxjq684+)yq$g6a$ABHzGJO;@4M2+T9nuW094*!ph4Y9(W*Tc_us8cZ;R&H Ktd@^X?W4lE*QK%m literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000133 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000133 new file mode 100644 index 0000000000000000000000000000000000000000..b416e40623a1b76a8988c998f1e7c6bfa0f64e94 GIT binary patch literal 384 zcmV-`0e}8PMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zg#j=DIu-~55CIrcAf5s!yaF)4oR*2Sru+N?NU+fMhV!DC11PmG{Qy|42ml7rKsQit zb8vG~MC-o0{5;hGdHc~{k--G0#qO1x)ygxG-Xw*v#xMlOf^ecQ>DO^AaFFM=8@hk_ z0y+RL04o5efsc_i91j{b7#;u*5t)t|7ZXi3NG?zzfe>K`PEs~Fjyo8U*2V$*yt;>v zjk@`D$f_umixQwp0om4F~`RkUTa} zZ_)uYJ5#B!9w4jXSTskSQ1cZG0Xwr_<+g4cqw7-z%9W^h{Dz~Kw@&y>2wM}Tt<^%X&jkrTcpp^ zF3M1oMl}gc8v*?-ioUQHf^loq+mGjWd(+4SP0Oe9Dr$V_EhaxiK z;8F_L?2jhDB|+(ty!_B;CFbE$mJJHgINERT414{mr3e59kw7;v zB{v5Rx-byjQK#WS3F&85!{R?j=vXb;(sY!IAiYTn;R__5Gz@MdZH^wm_!17xvs(#w zO@cW9E&wY4oy!7kc=B*__3}4Jy`Erf`d7bW&o}*-8_)i4uKyE5tSF*sk8<@h2B>&q zMHwmrh$NKH{oj?|Q47ZGcW7O*XjV{l_6~)NVSDoR`dsMJkQm6$k(Z&^$Ll zEdVC~Cm6#|hFi!93wJlqANU}+qgYg?qw z(;m7|6Gk-+Oc??FEr#x}7vw5i+f6@qzs|yU-}8oW8G*k-YrRXhHLEDa-bJXdLag_lLdS;n$SEhjH#dbS2%IjCJkJ`3dWw zx-^-ETxG;w7`CwABluv;Y7A literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000137 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000137 new file mode 100644 index 0000000000000000000000000000000000000000..60167093d4777dcd13720d82974b8a52c89f726a GIT binary patch literal 387 zcmV-}0et>MMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zi2*PHIu-~55CIrcAfW;%yaF)Cl9=U;x4EBdNbuCNhx4MD11PmG{Qy|S2ml6=K{-I> zHV0rS+1N1*dq$@NNP@jzXCM*lu8$1&7P(n1O~35kDGm1!2*$3?wjWZfzOW6ycGmnr zgD87C04@M404Xe}hZG123Im1&3g`%#*(hm|=|qIc1}Y{7@TI^6C1cyRjYzyS30ddV z{eD&w$*xD2Rh?kfKy?b-p5FLdU~T%X-!bT${!03QlWb5@g&(e?^lnudJi(J0-=AQHm z8YzN>iU0sioXg?htw?Mx?BmJW(hsu(j`s2J^jjxb>vR;j!5WhMeb2A_mE+k@FR23r h094RmR6(z^32sxgY7by0j@(i*xg?|@EeB)UxP|yxt&ji! literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000138 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000138 new file mode 100644 index 0000000000000000000000000000000000000000..8bc95e8f3041fb18d52d929f6a75589021fd0267 GIT binary patch literal 363 zcmV-x0hInkMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= ziUBYIIu-~55CIrcAc+DfngTHW>8|j}{^rP8NPzajhx4MD11PmG{Qy`62ml69JT_2o z(gENa;Sn4TVo`&%kZ`!3MkPRpM)sc2n4q@1Ny}}nq$)tPgmUo$19J#&2}nX1wv+*p5sNI#3_#oklVE8anQL34&eI;c zP}4>=2}~RT{Vj&xuovViTiZ=PcE8TTci;1Va2}N-Vdm+B6g7ZyC6zOw+=(l#kO~%1 z0@v(|Cchm)$&tMLk7z;ja4F07d}tKyw|9rV-r?7jzlU+|KXfJCnv8Yr&G`xIpt>}f zgK% literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000139 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000139 new file mode 100644 index 0000000000000000000000000000000000000000..72a36c949176814674f2b766f38fdef194489995 GIT binary patch literal 386 zcmV-|0e${NMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zivchJIu{565CIrdAfEy#yaF)K=H-{fh^Oh90Z5RS(ueb+nFA=bFZ}>mwFm$P5kWXW z9~gn&1SCmc84B8WD| z4^j3y04@M401p@o4giP4L&Ib@0U{*D$wphf}QpUd^~H%LJZ!P@k%zQ>zy`Y$=2{oh^xL>$2)$<-j*)yEL1 zA_^8uRXoK}CeQufmG03B*6ep^U23ZY|~_Y>4zC6LEAJu{i_tLZ5snOSUr;M_x#FVJD&aY%jg6E gRM22lL9eq3Zd0^s5Bl7=$xNs=!wMjIVZ=JilfzS?oB#j- literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000140 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000140 new file mode 100644 index 0000000000000000000000000000000000000000..2069a2028b5a28c5b6a5316eb0d8c023c05051df GIT binary patch literal 363 zcmV-x0hInkMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zi~%qKIu-~55CIrcAc+DfngTGOozcsIqRN?ENT9gkhx4MD11PmG{Qy`62ml69JT*{n z(g83_Qj4HQpb(2TUg@>cK0mKZBYRJ1OrUo+X}Qgn)JjAt1@2;-h&5h##H@Yc_PBF4 z04)G003@Ca3=0?90q6*c(SSh#!Q6pcf|L=3?Pb8^q$10*0uh(dAXpkl=Gs;X^t6vM z)U;7e0ux95EsoBx7vw5i+fF}rKhVN=KlFBR9+msh%+m)aY5?U{D#t>(7N>Ht08vr{ z*X)xfzb!%ek-U5nX(Q(0QkESO(MZ~F?+tq$#IGrT598d6=t{Z)8;*7D&3OsSpt?4k zgLPvNU+)Phx4MD11PmG{Qy{<2ml6AKsP{h z(gDN%==@1}LtSpnSeih@-Y8FGFCkve@~;_XosY3AcB#6;hU^%MW@ z0XhIK04e|>DN)K{f#bt#=|BJ-A~YK_s1pxNfLpLa(%`)ipr~YoLLq|@$mx8pjVE*0 zmuVEYmpGd$#i|CXQ+rZfTDR}b(jOL++|k|^)SumPwDR58-VxBlf*_rjrbhchU;|Ao zh}u9Ic%lfs@2*DP=EP!L|Km8WNm3Ez<&(ax;4Zf^ZH-jtxA)>97W;@}SN>j3E*97L zumrd;Jl4dNcohuC%ChTq_IDj1U|F`3J1Vi5$D^?Ks3nYj0Xe&RM22l cL9eq3Zd0^s4@l3vB1We<6fs)lteMSHpyjZv`2YX_ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000142 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000142 new file mode 100644 index 0000000000000000000000000000000000000000..271b3bcbb4022d5047a54ba6af7043c8ad5458be GIT binary patch literal 364 zcmV-y0h9hjMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zjsY+MIu-~55CIrcAc_JgngTGtgyoi(<>$wHNWl94hx4MD11PmG{Qy`E2ml6HgRjb@Q7La!tHVA zHUKREC;%7>C65;tr&}bv(;Zl|j3eh;)Z|@Cz{ll**e-Go_hv-VW0olg7_U62VWl&w& zW+7Mkuos3Q>^JFoJNvNZwgfGF$4c?uccqWD2(bYJ094*!ph4Y9(W*Ti`mwFm$P5kWUl zZ*xSjNv3Z4464SbY88HBELxc%T8WUPsO6&wZIVJ*8MOAR>%ehGw&wC5G)69p+C7Ly ziLu7`d1EfWh36cv`NI)VC!AXk7wrvI@(%N(|&#Qa* zSc#inldOtDxhMhZ6x`osd-+?eq<-LxdT8I{&v*Tw9Mk^qu3tiaTry*4g_T`2uXp*X?)Tts}QM8-laueE#+3rWFjK z3Q&!SC8W@?kPZ|y5d+rc;b491kJtf1=HW2)(26+IGzMm9S8sh5Is0R zMMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zkpVCPIu{565CIrdAfN&$yaF)9@wxBm)z`wa0Z8!Ot%&oYnFA=bFZ}>my$ApXQ9(FB zMKuSY$L=XG+&O>=$|H;pXoY>Daok2CM4FD0W0Bq@$yRxYDKN$f|I%po;mqg^!#1!v z1(Q8G04@M402QG|!X89yI$A_q00}8Thzbn=EjmP2ctM!}XA)LiJUEV9IEj@Nfq72d zzsE!$>2;{0CYEa=m`VZMpv(30H%LQG!P)e$zQ@}){g)q}{oh^xMI2qx#Oe_3>Te8C zG3AOfRJ@5Lh|m4smG03D&g^$+U9xCaPj&VNg{)zF^7Z;1=+clIlx@LTe_a0c=AP6B zc2rWceZUkPj2M7e2r&ZH)!|@$>4z9YL)PK&^si!YrfCw~VD(70-}5Vf?RfUnkD~(w h094RmR6(z^32sxgY7emC>xtxFq-jt|4Tj-N6U=Hut9JkZ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000146 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000146 new file mode 100644 index 0000000000000000000000000000000000000000..14482c2e6eb3ef3857422bd91176cff168cb0cc9 GIT binary patch literal 367 zcmV-#0g(PgMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zk^wLQIu-~55CIrcAdLbjngTG{^t$u3hP%g5NPz#mi1VVE11PmG{Qy`c2ml69Jvcz+ zCLIG8>uM^Wib^d$&VP<)#t*vI+!ZyzfB+ zarZX>EdVG04=W}TbmXEF&;bHK<0L=^<`CQ#polDNE(9bc7+IDXgg8tl!O}P~*S1Qa zr+tK>rj2SEm^cFcEsox>7vw5i+fF}rKhVN=KlJuEkIH>y=IMhKRe*9Tm1Ch?i%WU& za2PlN*X)xfzb!%ek-U5nX(#64QkESO(MZ~F?+$w%#IGrT598d6=t{Z)8;y1C&3Ovz zpt?4igY@D)VRr8Bok_ir)!74lAl1$F^B; zMhOEt051S502>4dI3vp=LR{ex1&o{o_@FTf(WHW8gP9Zu@uuJYDw1F`x_YpY|7W|ci*iew>TSuv*vvM_2#A( z3>*jmk(P^wQBIA^r@}&p#kxWutZ)4h`y^x`5K|BB1ZSGYz%5$RWbm85<+mNv{u*j@ i1OQafU{pb`vk7ifv}z9ylX8zKN5Let_0(3sklQLls;Kz@ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000148 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000148 new file mode 100644 index 0000000000000000000000000000000000000000..646cfd2e9b4c93c4e9889b72a322d6b56d8cf7f6 GIT binary patch literal 367 zcmV-#0g(PgMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zlmRdSIu{565CIrdAdCVingTHGn8o{+$-unj0Z5?1+lcd`nFA=bFZ}>m9S8sh5Ir|Q zBz@ zo^}z2nl`FwVB!eHGk-YrSXeH+1QkD%0(Ky;~?+%Oo!>=iS598d2=t{Z)8_jj?&H0Jz zpu#qqg9jfisb@porraA9Cz;1Xb5ED;mHQ17?Oc51Mfz?ls~Y^wF+de!lH(M)EIt@?i%BA&7`T;aOP$iXCa zfsu6)92}TD zUa$v%q*x**_$7dZGwyI}F-b653G~QT2o@MGaGZAaE>q?gyBAJe9LxeK3XC9G76v!# f!i#*~GrEFj1gtT$aM{0{F`Rkr>VM0eKV|>`%9dv9 literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000150 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000150 new file mode 100644 index 0000000000000000000000000000000000000000..2706cda7e56e25d29d372a048120a78dbb999e92 GIT binary patch literal 394 zcmZ>9jfisb@porraAja%;1Xb5ED;mHQ17?Oc51Mfz?ls~Ya e@%CiS%ZmiA$c5$4H@#?e$!&|{yYyI-kEs9~#*cUa literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000151 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000151 new file mode 100644 index 0000000000000000000000000000000000000000..ec5d45f204ac171e02eaa1f58c4fcae16ef8d55e GIT binary patch literal 355 zcmV-p0i6CsMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zmjN&VIu{565CIrdAbmy$1l&4>~qb zPtyS~OJoUzDpt(UW{m|Y+ZS6>VBL-E35_kaEH#pwcnLVV(NN)wdAE5ADzrar9t4PP z5gO`PwDuX)!2~rd=~gEgR#gL^tDMNTFS~Nri{*Q68*hGZH&Sd;Ftq*7(#RcIIh8Vf z)-i^?~EABn}GCIrq?1pm!|@9fJqrCiTFV1gm`QKJ|{6APw1t336Coj0!&ysEIWIu{565CIrdAf5s!w*oN8x46}(=J%qV0Z8D*n2GbEnFA=bFZ}>mtq1_t&_Fpb zB_|!QqR`&JBb3A$-1@;W%1BmD7c!O;LiQZ`o`!9fq$PeOGq&C!sGN0;0T@{Wa{BsVG$7@81FfLvPWjJ#-2h%k6285IU4(DM;t z^0smsmeS(s9D&7Axi|xym6ngXTrc;%$^C&dNxRx79`$E;)L6dz+gk#zE*SDTH@Dgs z0v1HMpo|4=U`ua)cQ?}0oZ!s%KT9L0WHnXF^r(+5>?L1Xdqds%@xAy6&aTnem!2i!<$HD*0R#Zk f&vaBl+mi{?q7c*`unQ!my$1l&4>~tc zPtyU=BB0I85H3vyF(L@lTCp`?Sj?M{J)yCsmZe5=6K{5Av|T~Ih`;IkLU{dQ^B@@Q zEdm?rShV&Sl)(fyEa_Gw7*-VnpR1k7wlBMK*Nf$QZW?cXZ#PnGN-(tj&eF&pSv!?7 zeb(`Xz2r-4U)h}ZZHzhn*&QvG@Ba0Mc%bQp zXzXv48C!t#RHnxwJ(rhK5uy+GAO@x7)4{>XfJk6GPC_r$LwG!q05G!hp%7W%Og1bE zN*;kmfs(gXvnY=hv14Ym8KlaY#$?t% BnkWDO literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000154 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000154 new file mode 100644 index 0000000000000000000000000000000000000000..a9317f64377b12070d8acf2f7f9385789b3e5728 GIT binary patch literal 205 zcmZ>9jfisb@porraAsg&;1Xb5ED;mHQ17?Oc51Mfz?ls~Y>C+4BC&qczI~rnybqhJsLW;cYa9jfisb@porraAja%;1Xb5ED;mHQ17?Oc51Mfz?ls~Y^aX5?U& zy2r>RoRgZKm|Urwo0(peSdy8Su4`xpG>_2$rpF3oaa?m%`OhAu@8+L67H?d{xQvmj zsiAJeoU2@4E-ccu60#BE@^Q<_FOQNeE=epZ(Umgd-~hU$0SFis8V>ZepB2`{NibRo@ZI7s-CbGFzWBphwKua2FFJ7n{gk4> l2$E%C@G_oQ$m?Ihx@m4&@STTO_6t=0G)-k^T|X%7GZ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000156 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000156 new file mode 100644 index 0000000000000000000000000000000000000000..162c43ee6aeaac87d7b8d21a7316e5d65332419f GIT binary patch literal 389 zcmV;00eb#KMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zoB=QaIu-~65CIrcAfo~(!vZkZuhEsX-TjjqNTABqiSweF11PmG{Qy|i2mlrlLOVd^ zCLKHbA^-9C27(NCI%te{%PhrvdLG0ZkVf`|#%_<^mXO@0w*D1mld9t){AI^Vjtko3 zVMjFE1v>yQ04o46Rl&tnvvExy&M7qmvRV<*AOqwNmTIA~RwxsW4v&k25!mrOG+-Q^ zH#Rp%SSb{_cLvrb<>Cx%R00=Wu9y3y**$_YNxj+}fBLgKhFHG)+nWNOE{O8Hn_KM+ z0Sl@w_+mjBhyoD+AkA;@M#@bK&TRj)G;&E+P^C;S`WVAr@};#m(VZWk6hFb)Lp1iK z=gVn1s*WVUM*>ShiNq-8gusS$E)Dyg{b2|TnZ}ZOwSqHEbKsLID6;L#uH1EF`JSG} j0|Wrm&}39WuM-MyS+r^ofpY23$M*pp>oT2!mYOmQXuqcJ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000157 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000157 new file mode 100644 index 0000000000000000000000000000000000000000..89befa8f21df85f321b8d4f06b3f90f7e08f3775 GIT binary patch literal 364 zcmV-y0h9hjMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zodGZbIu{575CIrdAc+Dfo&qq+hwbQq;PKF*0Z6dxm1qc8ZP&_tJ z#U>rJhy-Cn(YcupMHh$Q%7S&-wXl}#DWVlaYm-gV5*HK!Xb=VBvy;3*(Vkb^xcoVL zi#7l*04M;U4-3|~EFKVm4V08nvh*se`@v z3_(=kkSqd~Pv6MQ8J2>)&5nEZW%t7tzWdqR!FeS2xzN`KDYk&fg-A{$aw87JgHIL; z9!LTd?Q;7&qQlD$!J{Yu literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000158 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000158 new file mode 100644 index 0000000000000000000000000000000000000000..90baabb2215a672e7affa7251154a4f7061bae78 GIT binary patch literal 385 zcmV-{0e=2OMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zo&hicIu-~65CIrcAfEy#!vZj@)RCZ>k;(Z)NWhHriSweF11PmG{Qy|C2mlrlK{-Hg z(*bks))IknH}MF)4bvB_E}&b2{u$-V1r+`{Y|~bKa#Pe`nMrWlP literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000159 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000159 new file mode 100644 index 0000000000000000000000000000000000000000..3f1a1cf729d59f4fdb935515cea4cbe663c4faf5 GIT binary patch literal 364 zcmV-y0h9hjMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zp8+rdIu{575CIrdAc+Dfo&qrVjobLR&Xk#o0Z7oTgNpN_nFA=bFZ}>m1qc8ZP&_t3 z#Wo#R#RVum0|hGw;iqit#JKa&SoTfGUVXItjCOC5OO93>jd6OF-gUkB5Lom`B+n-RRMLG|a4seeI$nW=K|zvqUf6PGq|F88 zF$NHYBdQEgy5A<|4Ln7v)s|fKXE(e!zWdl4;&r0eS?n*26`BC_M4~qWJ(3slU_ubE zaY6%=?Q@nw-o$EE%Jdn>6!wuXt&L@Oa{LY6fu~;-_NC{?Y5CdZdbtA^=@MG|jD!$P z7mie`5j;`U0)ML(WZREjxhux;JvGgn-`a&#P6-YJ0Mp)Nph4YX(W*UmkWP%}`Fbg& KuW&B=z**fE!kL`_ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000160 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000160 new file mode 100644 index 0000000000000000000000000000000000000000..eb8582703916636a4dcb0087dc2841278f56a470 GIT binary patch literal 387 zcmV-}0et>MMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zpaC!eIu{575CIrdAfN&$!vZk%uGhN7#Dmb~0Z8D^l8W=9nFA=bFZ}>my$ApnQ9(IC z#V8#m9ps%uMz(n5zUCbDS8Sv^Lh~>3$)2#b+oRhOlKq!UC05{vr!{g2d=|5S0PuE? z0^DpDI{+^LD*zr6N+E+DISwE|#)F>+0$pus3$Az^C&2R5U+RX>nZ2z+~@<~=urA#mSIKp1?rL{THogd#6?_upB8vD}o z<+L0f3#|l?63PWffu^->pAZbj$$n>lnT9~)u%upfU~StD_@)YqZ2Ph+cjZ{Vrzi0Q h0MpQ9R6(y33U67oY7Y#)YIKHClsJ14=Ng#1-125ar$+z) literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000161 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000161 new file mode 100644 index 0000000000000000000000000000000000000000..cab070d8632269b35f8b7550cf6fbd2db80f6d35 GIT binary patch literal 362 zcmV-w0hRtlMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zp#d-fIu-~65CIrcAcz7eo&qrNjPLd3+N#z#Nbv8Wiu0nG11PmG{Qy|~2LKijJT+i# z(gBnjv_+P0GlWG{dRP^=7%L(Z+sK|oyK6{ovPoK^%-2s*PyQ z1eEP}EQRce6;mnGYaM>Dk9=wEOuMtg=fEd0b&SHk^!zw2U%Ol{ch5(1dOHKRoo${ I#FlE4jU^SF1^@s6 literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000162 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000162 new file mode 100644 index 0000000000000000000000000000000000000000..917cf161e9edb32b63a2de6933780ce72177d685 GIT binary patch literal 387 zcmV-}0et>MMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zq5&`gIu-~65CIrcAfW;%!vZktiHFOv+5MwSNPv;Ciu0nG11PmG{Qy|S2mlt5K{+sw z(*YoqMrt*ma!EjL0sr*;p-%>_t#J`apEjyv3olEJWIt5{QV1rrbi$@vki?>(95M&z zfA0}H051S502aW9!V2S}W8!2^sTR|UqqIeY$eb)y;$oSgJV+&th=Pu&xi$$|SLv1A zQHpE?lHEfD7Dfci2w+A6_b%JZeGv`4fwfsd+Z}WIv|ECBzWdy}6Hiz~T@O*tc13^{ zRai{1B1;T;$WRH*Z}7_6Q4echKm4lX(X=01x!wzT;``)VZFAnGAzvi#z}i7o_O0hn z`f(&ECo2OR9tM?1dI7mS5*loi{n8FIg9B~Dvw{|awQXDAi|j}8*Pm^;TKmpq zAW<5QRI3q8Rn-E2s~gC+AG>l_jN^N17H@uQ7g9MTI0FFF-ejOb-C@zHJ@>ZS5|C8u MDuRs#C!T%y7fr{S9smFU literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000164 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000164 new file mode 100644 index 0000000000000000000000000000000000000000..e9cd820c7733514af31612c0328bf51a70d73bd8 GIT binary patch literal 385 zcmV-{0e=2OMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zqyaDiIu-~65CIrcAfEy#!vZk*rQyi9meZttNTActiu0nG11PmG{Qy|C2mlrlK{!xv z(gDxUq^*7c&LI#hSLd|vDpU|x?UF_|ien4CyAhJxY(hOy)s3y{K?JKpIuUcT6Im$K z>=ilyF90k6D~TB5upg-7qd}=$Hc(VNG>J+q(gxE|}t+L^sn%F0m)&bt2BRmrAlL$-4L=;Mj*lW(=XjV}H8C3yzUE~>I`J%7@U z0|X@Tv~rP;4va&9RiJUQE)Vug`@;qdG7rz{*$2)vO@UvsA<12Tw&k`P&o}if9s~f> f&}39WuM-MyS+r^owcYx)W1y`ke%YoHZusaBi>{)v literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000165 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000165 new file mode 100644 index 0000000000000000000000000000000000000000..c4478add18fe4a8130383e4d6eb197acfe8d24a7 GIT binary patch literal 363 zcmV-x0hInkMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zr2#MjIu-~65CIrcAc+Dfo&qq$rKiu<)T@wuNU-$Yiu0nG11PmG{Qy`62mlsPJT_2m z(*Z!rObK8cj_9b4;dY{dhUj7}(TZ4vG@qA1&N@SQI8bI_uqz4kckmqrt&>_Jb zbOMy^bCyEp#G0v;=`)Te>?2=V8_VwG_#6BKQ@<$eOV5wf^0UkJatD{`5?cGsR3K3o zj#R4{rmAXzzts$6+mBtjE5`9XwTd^twF{}75gY>m)81sDLET}|sy#2eL4>aYb8_AG JT2#TjzNanwoK*k- literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000166 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000166 new file mode 100644 index 0000000000000000000000000000000000000000..0e76aa4f44c724a5bc3dd7bccfbb2b08122fdf90 GIT binary patch literal 390 zcmV;10eSvJMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zrU5VkIu-~65CIrcAfy5)#R4$mjDv^!x!QqGNWhiuiu0nG11PmG{Qy|q2mm%wLODQ1 zH3#)e2hkW&>2y`WQU6$YT~nboJ0CNn%aW8_1PAB`GQ0NlYaYfH2Zx9MqaL{5IeI-rV~YDvcl2ffl-hWJs$?ls@LZE z)n-^DZC39HC{C&B3~*oq&gI6r-$C2?LrZ!;{I#=p`#-g&{_mUrL@cS|$;Ugp;TH)| zQKgDCR6NBN00{;G-~L~ox7v!9-h1fX*&~WU`1W-!?2zZQ^Toe$*@o@l^oW+O-TC=? zJGEX1HAY%o2No0?2$78o8HpKBbzzog`=ND!ATO-x{V+w#>sn|BF=)2;P9OK9~hEml~HOROu9*_w}%15Yr1&^-@6gr7WHasdUE*=CrUH}Hcnk1bUww#=)gF*RC zCaQQO3qa))=jP@KOhvlYmVEVRH^ex;``Me}X-JO)V}D_s*e=lvk)BBOMjl864?vWl zC9vj}>?2=VJJas$_#FHKQ^zRmOV5wf^0mwLatD?*T|#TmnFu7x zvN+PMHZWCH2Yjw-AlrWI%3U*#@3~RD`Mq68<#gZ!02JS3ph4YX(W*TVcCl^qNl}R8 KT0ztS9l3>P8k#Hs literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000168 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000168 new file mode 100644 index 0000000000000000000000000000000000000000..94982e5c208bc420478b962aa6ba2dd23cd8208d GIT binary patch literal 393 zcmV;40e1dGMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zr~xnmIu{575CIrdAf^H+#R4$y=iSAsjHI;i0Z8E2jEnQ4nFA=bFZ}>m?Faxi(Ly^g zJ_k=V9Tu`MClzc;|q3iv5q{x?KMSMv&D znFhF4Yri`HF90k6E~^cP6NHneL$vu|0pM7B*|ZYw&k`Q n&o>1x9s~ds(PUIXuM-MyS+r^oWk4+@rWnS^d$E|ryD~^c!TYAV literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000169 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000169 new file mode 100644 index 0000000000000000000000000000000000000000..fc83026a83b196b3c3fec3aa6b2b873c368b662e GIT binary patch literal 369 zcmV-%0gnDeMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zsR1wnIu-~65CIrcAddnlpaL+d&g#d?h_U!lNbvZZi}RwH11PmG{Qy`s2mm(FJvlJt zHXQ(5_!N*)X_Sy07@7Nr;Rlg0F8+AZ_JqdnquaenF8MdWqYVsDMpnrHTt&*&mW=ghH_SM```Y{AZA{N2V}D_+-~yyaGQAS%nYMMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zssS(oIu-~65CIrcAfW;%#R4#lley`v;?aOCNPwEFi}RwH11PmG{Qy|S2mm&bK{-Hg z(*c&8d>WR(@!ELJ3^0T7z3PE%zEwBcD2^>FZ8swOr^W)cARrRekF>w+xbS?!Hw|cj z^&SB_051S503ZT@4-`}Qj*kW;6|+%-I%$z1{)44dD~lz>#B|c2I2ef=jzb6ID!sBf zYJrtTl4DYVl}WiY0vVOu-P`+SkdpY}*{q*!?w@AbEk~c+%^SBwL|svY!z8-dz6h`) z$rW3yxQZv&;3a+T@5;*2&@-<&^s0PmT9K`<+=Z|3c>Y$~TX$)A1|>s0<587;E1xIz z9x$N{Yo+r*0D&A43NEUF&E@G}d})VR!b0ZheAdrSJkzv>GiXKfUh{1Gw%aE+{V+HL h02I+=R6(y33U67oY7b8e7Tzhq;B4Xm4F~`>kUTYD zZPEeVAb7lsd9___z z2BHc`vIJB#0_RB05txGW(vFPvW%ttarNlOBtA|s*BH|+2h)WO}XjD@GURIo8Q;I8D#?p0{|4?WS~LaVbQ8RQV%!q0Z5?S$&2%%nFA=bFZ}>m)d&DK5kfgY zkJ16pXe|R|NEVVPOp{kw!L(W6RoY3CaHR(aCW5R8WJsXO1F1c^45( zY+=1eI{+^LEC3>^*|Neg5oCNY8YvbFlN1yeIgvkF>ZEm{Y=D?3Djp0vVg~bcG_TSt z+oKU$DI~p*2CmC0!w6hR0vBDjm-{3-x&m*rg0?;W^l7&Y@qG8YH$yySkpugPcD5w~ zuK1Bfl`FR5$U%rI&2R9^>QM*YzJB;s$)#yQwsO5FWQgyRZ?)ZtE)Dr4`32q{s$Pyhe` literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000173 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000173 new file mode 100644 index 0000000000000000000000000000000000000000..a50a05200a6666b25783a1b29197871b6835a222 GIT binary patch literal 366 zcmV-!0g?VhMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= ztpP9rIu{575CIrdAd3PhpaL-br{Jr>r@oY@0Z6d@*o*U`nFA=bFZ}>m6$k(}&^$L# zZPNizBdU7P)#hNzY-f2h8XZ`5aqGjkwkPcGKAYW}X>D!0lf&=e8Mt~zVPATFoR+^`u9rKPi@Jo?UPBp7 z)#c(yx$3}`WhL;tih*qVu`74gIKC$*@#Z&oA(iui0|NjQ-(;Xc-C@zHJrz?}oA;U3 M6m*?#B}{cYK2FS>1ONa4 literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000174 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000174 new file mode 100644 index 0000000000000000000000000000000000000000..24415b5e6529596c1eb947d0aaff64e1eb547e91 GIT binary patch literal 392 zcmV;30eAjHMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zt^qIsIu{575CIrdAf*B*#R4$%lacI=^8Kl#0Z71}=!^5BnFA=bFZ}>m}@qG8Yw?rJ#q6z3J+S$$s z$l{6?Z?gD`E(IGh0nKml%IZ-H=DvRTRmrPqO}279D&&jrlW(=%p)L*iDtQRzeyXx> zJ%7@V10@xK7LtmT8O*{4AhHx9KH5mM{n8G%0|ag);#omE!CWZBz*kw5mfEvIIrs8iPCMeU`4hqq^*Gf literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000175 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000175 new file mode 100644 index 0000000000000000000000000000000000000000..2b6714c05709a5d2d5d257006d2c521ffabb43e9 GIT binary patch literal 367 zcmV-#0g(PgMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zuK_RtIu-~65CIrcAdLbjpaL+VkCE{3AbMz+)$ki z%4-Ou=rdIWqL1&U<_cUz%GH*P^=CKCIKKPZo8fUx&%!SU9Oiqn2WlE)_y}7 zOcmzhNV(d;m1QCDyPAP)`>`u`)i}N3@F|r6BK2vVy?FL zq_E!X7drqi04xBXmXHGum{focZsW0<*~nnAwg`&MWvi9e#iAmyxWJJ3h{TR(K_T-h zy|O(@k)=q|yQ$#PoM0P{CM7RjwwL=TYI*}}vx2rg=K8c-l6b!R-TNVaU=hS~6YXqQ z2wZUli!N7eMUxj$OwDib%IeVw*1mrDRmrJoN49c(DddUolW(=%l`akWD0v6gPO7qR zJ%7@V0}KNMClnQf01edyP{I+(bg(Uy{n8FIgamC1@vNYQU~St(e3TtY{`#{mx9xbo lDR_DW02I+=R6(y33U67oY7b>pYGI@x&w(2=%BEmjvIu{575CIrdAdLbjpaL+jjmB?tgEP(3$5 z#Wo$h#xbxWB9z>Cnx6z*;gB&T(S2<~_QcTc|FnCPTrz2z5XzWHl>*!K;wvJ+bx{kj zZ@V`DE&wP19}5@Yb5%g06N?K9lj<2fzNQcffT$#+Lt>)hfq>%)To9~D(s^Oa*_k{T zl;;dY6p&Q0P&NYRCg%xEMY`3NeD!BH#5lhD*_+{ENKfNpe_@2!0-_fpJ(1{*ybtRE z;-HY(1eEP}mO}Q#s;QLewT?ITkuR;CX?J#f4t{~DV-)tK=f`RJ+U0t=0~Thwgx0<@ z6-bnY;YhdIz*JQo@VTmiZ2Pe*cg;Ay=N9qi_jVzbGs6J`02JS3ph4YX(W*W5_@TW> O44{1mg|^YER6GfP-=ui} literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000178 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000178 new file mode 100644 index 0000000000000000000000000000000000000000..23000e4459a84d1053e1210bd93de455045f8f57 GIT binary patch literal 390 zcmV;10eSvJMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zvH>swIu-~65CIrcAfy5)#R4$Dp~&a7)aj}`NPwfIjPs(I11PmG{Qy|q2mm%wLOL+z zHXVSuDm?KZ#2`Rmnxt3CBa`A`UE7BJ(hf0$gKXP)*3U|CrfCa&(0U|${n?h=ay;Mk k(|7~`6wzcm9S8t65Ir_f zZPNjeW)Qe;g%vOrbko*_w1bVxtgmilPYmt;j5lqPOJ2}Iv_;nXcGI#!QGm2Kr1Ea< z6*mAb04M+*BTotFs(eBw6&xQH)-!l~Q6bTZG`iUEOi(xwXgmQ6!T}}eys+iuP#p}) zZw4gFXQ~8HG6Lu3<_TOyy499^^=CK4IKKPYd*Okmhhee5Fh*sg&upjv?$LUs^lU?(Fy+yaHFpDC|qmkJIwC%k^>xmnsI#3P2$b(?LsPN0|x^D6yIc^LET}|sy%L?o>p2N NVmWIvb$xIXIgx+Wo<#rv literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000180 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000180 new file mode 100644 index 0000000000000000000000000000000000000000..3fd343245127c7d052bc20aaa894e0ad52915f3a GIT binary patch literal 389 zcmV;00eb#KMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zv;i;yIu{575CIrdAff^&#R4$;>h<@CneyuM0Z5?b!;JHynFA=bFZ}>m%?JQC(Lpyr zb92f6byvoonyBDS4Duj_dF90k6H6R!c6s;r^qH#V>G##iJ5=mYcu`M zlBQ{+oqqwDjwn6*S)bXK(C!Ie$LBwY!|6N^yS;7pRc!9 z3wA3M0uTrU1|$X-2T?_*fyHBGq3PNFI0p)@EY$UeSfVcsQ)n0Qh&K3MANOOftG^9P j9s~ds(PUIXuM-MyS+r^ooBP^#2q!4YM2MqeAh@?5ng6RY literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000181 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000181 new file mode 100644 index 0000000000000000000000000000000000000000..4b4735c70859d71ef62abe782ade7b89a338ddca GIT binary patch literal 366 zcmV-!0g?VhMOa!yOixDv07U=*016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zwE-{zIu{575CIrdAd3PhpaL-0?cUSUue9B>0Z6ch(~R??nFA=bFZ}>m6$k(}&^$Ik zZ_@$B@tEp>iIIH^0&|Yb11vizpYimq?Fo(DN3?sBTrxE#zY;Jgh?lr-8C=p7Fq!JoUR9HL^cszj&!T}}eys+i$P@fFS za|WafXu!SU9Oiqmxj88*1kg( zOx2~~NV(d;m1SN0u5KXPe(cI!HIDDeNxb>ZT}b70-~#{@-(;Xc-C@zHJ4 z(g9GBOwaCmYdi%U|0PlKs25h8PX9uAN?68#bIKEh2}^ zjSW`zKmD0&}y1wmtszX}1*deD}NeLcGx8Dtn1`woL$7 z)X*Xd7GJSr1B4OHZ}7_M(FW$ee)v_%sA)sCa{VY|()Y=?+U`b|hWwMf0&^Et*|(lQ z>Bj*9BBm49Mm18Ag>)zq4nP^r!`XgmhZ{iw=i%_Ipk-iAqv`NZHYEA$&$isQBhM=NYKdfjPs(I11PmG{Qy`U2mm$^JvLBn z(*dWdDvesrG-H#%mBAObR&Dh{_eBcy-6;4K}!HyaTaujZQGzIKw2C^0sY%6 zHvldGC;%ZK-AU-WfJ7=8DI+869X!4$(P%|N+4!)yz?dlTcmo&(1WM9*VawT}J{Xke z5J(o#bP2G80_P^@4O~UK)s}qqXE($+zWdqx;b}~d17m+-^lSjqBbi=_^h}`Tv&)AF^;^>PQA40Q>ueTOob zDoutX-D(6^mX*Ng>ISmy$FAHpC~riNZ{#!jq{?J11PmG{Qy|q2mm%wLOL+z zC>^R$wYW(ey(q@2PaHSSve7IB9&GIpLNmH3$GU&^&qif$(XMGbtxX{~WY|ZDdbq?AmafY?79!t$zhZg>S|!Ia^bd!-vcw z4QO`>HvldGC;%VtB8BA2j7TpW9U3l{J8*nG5(|K-CB(!6BLkwK#0_W^5T2y-!j|*% zL>QF!5KNd6l@Y*<1P4ve7r2Uat1bEJ&u)lueD||=g9D+Qh{pcHD6s`hPK9zSlPfut z(uoH}#YTXY?T0LdEQ+;LDbs5qjY3)qAGsFkMGjMf`!oKwUI4xhhTrYQ=&AWuw z{__l|inHNJx9Y%^WhL-IwFBAqV^{8)aeU8h;?3{vLMrD22Lk{U-(;Xc-C@zHJpxV| Q_8Vk(7nqH8Aq1CI=*H@vfB*mh literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000186 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000186 new file mode 100644 index 0000000000000000000000000000000000000000..eedcf53a0e78a5947d28d6c5671decc1e56c1d30 GIT binary patch literal 173 zcmZ>9jfisb@porraAII!;1Xb5ED;mHQ17?Oc51Mfz?ls~Yzf(ntXZaySXek11(;R( T+GZbqxTKz2%BA#N=(ZXF5Y;;+ literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000187 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000187 new file mode 100644 index 0000000000000000000000000000000000000000..6440aa1702c51c9768991efdef00a1db52b557de GIT binary patch literal 389 zcmV;00eb#KMOa!yOixDv07L))016NRqZnLJ0Dn%ZJ(W{R5X`U=25xU;Wi4SC0U!Y= zy8$o(Iu-~65CIrcAfo~(#R4#+@!XM;_2RxwNRZ90jq{?J11PmG{Qy|i2mm$_LOL+z zHXRW5p8_SJ)OxVhD~2^ZR}l#bst?V$Eo~IXmi6{q%eDVOi_RkXp@cc8g+P>LBi4pl z!o+%yI{+>KEC8SIBM|bifjVs)krMz96qgSzDn#C}(@Uy?AxYurFj*X!M2pj~fOVB# z*&W5eN+H>OWKd~Vt_^{PB^O<`m-{3-x&vpkezrT_`m|eyc)t7GTOy8Jv4ryxy%uXD2w*GzLD&f+TzW*_PXKJm2)w jcmx0x(PUIXuM-MyS+r^o?^_w@$cmB?tgEP(3$L z#U>qCnORCub?6!oX9td^f0yV$=(g6#o+8?{5!z&vw8RBPgabe*RohA48m|D>+qnEW zdy6*!E&wP1A|s0s(RBfZ0BDS?d{pkh@kNDF0!S(xAQu=J5d|VQfKjk0N#}(vCx`lA zP<}%oSwPc8z~T|_%*_}0MY`3NeD!BH#5lhD+1tTwOs)fCe_^E90wiZLITXo_90`I% z34?qm0nhe5mO}o-s;QLeH4jhNN4~Unrrqh`ciJwHy%*Dlw~9ae_Agw~!z z8BEX0aHLy(;8#@%{7%(Cw*A