From 16bc6509033962cfbd047ea63a4f97c0c655ae8a Mon Sep 17 00:00:00 2001 From: Quanzheng Long Date: Fri, 6 Nov 2026 21:49:56 -0800 Subject: [PATCH] Upload sealed history segments from the server-zd engine While a partition offloads history, the lease holder now uploads every committed SEALED history segment to the primary object store under history/ and proposes the HistorySegmentUploaded acknowledgments in seq order. - One upload task per run (TrackedWork::HistoryUpload). The committed HistorySegmentSealed effect starts it, partition rebuild starts it from the HistoryUploadPending index after a restart, takeover, or snapshot restore, and a periodic rescan restarts stopped work. - Each segment is read from committed state and encoded on a blocking thread, then written with CreateOnly. An existing object that fully decodes, on a blocking thread, with the same content checksum is acknowledged. Any other existing object is replaced with If-Match, logged as an error, and counted. - Encoded bytes live only together with an upload slot, so no object bytes are held while an acknowledgment or a backoff waits, and memory stays bounded by history.uploadConcurrency (default 16). - Every history object call ends when the node stops, so no request outlives its owner. - Retryable object storage failures retry with backoff and never delay commands. A failure without a retry, such as a permanent rejection or corrupt state, is logged and counted, and every trigger then skips the run until the next rescan allows it again. Nothing is acknowledged meanwhile. Uploads stop without a retry once the run is gone or purging. - The rescan logs, counts, and skips a run whose rows cannot be read, so the other runs of its partition are still rescanned. - Uploads continue after the offload mode is turned off, so every sealed segment is eventually acknowledged. - Retention postpones a closed run while it has a SEALED segment. The purge of a run that kept segment state waits one minute, so the client side of every earlier write has ended, then deletes the run's objects by catalog key, sweeps the run's history/ prefix, and retries a permanent rejection each minute before RunPurge. A write the store had already accepted can still leak an object of the purged run; a later orphan sweep is to remove it. - New metrics: upload backlog segments, bytes, and oldest seal age; uploads by outcome; uploaded bytes; checksum mismatches; rescan errors; and the history object kind of object store requests. - The format compatibility workload acknowledges the short-lived run's sealed segment before its retention fires, and testdata/v1 is regenerated with its generator. Refs #1325 --- docs/reference/server/configuration.mdx | 22 + docs/reference/server/metrics.mdx | 13 + server-zd/Cargo.lock | 4 + server-zd/README.md | 9 +- server-zd/config/dex-server-zd.example.yaml | 1 + server-zd/crates/dexzd-apply/README.md | 4 +- .../dexzd-apply/src/cross_flow/run_purge.rs | 21 +- .../dexzd-apply/tests/history_segments.rs | 83 +- server-zd/crates/dexzd-e2e-tests/Cargo.toml | 3 + .../tests/dev_server/harness.rs | 16 + .../tests/dev_server/history_uploads.rs | 212 +++ .../dexzd-e2e-tests/tests/dev_server/main.rs | 1 + server-zd/crates/dexzd-engine/Cargo.toml | 2 +- server-zd/crates/dexzd-engine/README.md | 4 + .../src/effects/history_uploads/mod.rs | 47 + .../history_uploads/run_history_objects.rs | 175 +++ .../effects/history_uploads/segment_upload.rs | 470 +++++++ .../effects/history_uploads/upload_rescan.rs | 282 ++++ .../crates/dexzd-engine/src/effects/mod.rs | 49 +- .../src/effects/object_deletions.rs | 8 +- .../src/effects/partition_rebuild.rs | 19 + .../src/effects/task_queue/mod.rs | 5 + .../src/effects/tests/history_upload_tests.rs | 1245 +++++++++++++++++ .../dexzd-engine/src/effects/tests/mod.rs | 9 +- .../tests/format_compat/golden_replay.rs | 2 +- .../tests/format_compat/workload.rs | 23 +- .../crates/dexzd-metrics/src/fixed_labels.rs | 83 ++ .../crates/dexzd-metrics/src/lag_gauges.rs | 139 ++ server-zd/crates/dexzd-metrics/src/lib.rs | 69 +- .../dexzd-metrics/src/object_requests.rs | 19 +- server-zd/crates/dexzd-server/Cargo.toml | 1 + .../dexzd-server/src/component_configs.rs | 4 + .../dexzd-server/src/config/engine_config.rs | 9 + .../crates/dexzd-server/src/config/tests.rs | 28 + .../dexzd-server/src/config/validation.rs | 7 + .../crates/dexzd-server/src/server_node.rs | 1 + .../tests/serve_process/harness.rs | 12 + .../tests/serve_process/history_uploads.rs | 241 ++++ .../dexzd-server/tests/serve_process/main.rs | 1 + .../tests/serve_process/metrics.rs | 2 +- .../src/engine_simulation/simulated_node.rs | 2 + server-zd/testdata/v1/README.md | 6 +- .../node-a/0000000001/00000000000000000070 | Bin 180 -> 201 bytes .../node-a/0000000001/00000000000000000071 | Bin 175 -> 179 bytes .../node-a/0000000001/00000000000000000072 | Bin 342 -> 175 bytes .../node-a/0000000001/00000000000000000073 | Bin 210 -> 341 bytes .../node-a/0000000001/00000000000000000074 | Bin 212 -> 210 bytes .../node-a/0000000001/00000000000000000075 | Bin 211 -> 213 bytes .../node-a/0000000001/00000000000000000076 | Bin 212 -> 211 bytes .../node-a/0000000001/00000000000000000077 | Bin 210 -> 213 bytes .../node-a/0000000001/00000000000000000078 | Bin 213 -> 210 bytes .../node-a/0000000001/00000000000000000079 | Bin 277 -> 213 bytes .../node-a/0000000001/00000000000000000080 | Bin 199 -> 277 bytes .../node-a/0000000001/00000000000000000081 | Bin 211 -> 198 bytes .../node-a/0000000001/00000000000000000082 | Bin 231 -> 210 bytes .../node-a/0000000001/00000000000000000083 | Bin 297 -> 230 bytes .../node-a/0000000001/00000000000000000084 | Bin 198 -> 298 bytes .../node-a/0000000001/00000000000000000085 | Bin 211 -> 199 bytes .../node-a/0000000001/00000000000000000086 | Bin 287 -> 210 bytes .../node-a/0000000001/00000000000000000087 | Bin 198 -> 286 bytes .../node-a/0000000001/00000000000000000088 | Bin 210 -> 198 bytes .../node-a/0000000001/00000000000000000089 | Bin 259 -> 210 bytes .../node-a/0000000001/00000000000000000090 | Bin 210 -> 258 bytes .../node-a/0000000001/00000000000000000091 | Bin 245 -> 210 bytes .../node-a/0000000001/00000000000000000092 | Bin 314 -> 245 bytes .../node-a/0000000001/00000000000000000093 | Bin 204 -> 314 bytes .../node-a/0000000001/00000000000000000094 | Bin 317 -> 204 bytes .../node-a/0000000001/00000000000000000095 | Bin 205 -> 317 bytes .../node-a/0000000001/00000000000000000096 | Bin 213 -> 204 bytes .../node-a/0000000001/00000000000000000097 | Bin 216 -> 213 bytes .../node-a/0000000001/00000000000000000098 | Bin 205 -> 215 bytes .../node-a/0000000001/00000000000000000099 | Bin 213 -> 204 bytes .../node-a/0000000001/00000000000000000100 | Bin 323 -> 213 bytes .../node-a/0000000001/00000000000000000101 | Bin 213 -> 324 bytes .../node-a/0000000001/00000000000000000102 | Bin 238 -> 213 bytes .../node-a/0000000001/00000000000000000103 | Bin 246 -> 238 bytes .../node-a/0000000001/00000000000000000104 | Bin 220 -> 246 bytes .../node-a/0000000001/00000000000000000105 | Bin 214 -> 224 bytes .../node-a/0000000001/00000000000000000106 | Bin 253 -> 214 bytes .../node-a/0000000001/00000000000000000107 | Bin 170 -> 252 bytes .../node-a/0000000001/00000000000000000108 | Bin 312 -> 170 bytes .../node-a/0000000001/00000000000000000109 | Bin 234 -> 312 bytes .../node-a/0000000001/00000000000000000110 | Bin 179 -> 234 bytes .../node-a/0000000001/00000000000000000111 | Bin 301 -> 180 bytes .../node-a/0000000001/00000000000000000112 | Bin 364 -> 300 bytes .../node-a/0000000001/00000000000000000113 | Bin 383 -> 364 bytes .../node-a/0000000001/00000000000000000114 | Bin 198 -> 383 bytes .../node-a/0000000001/00000000000000000115 | Bin 203 -> 198 bytes .../node-a/0000000001/00000000000000000116 | Bin 165 -> 203 bytes .../node-a/0000000001/00000000000000000117 | Bin 0 -> 164 bytes server-zd/testdata/v1/state.txt | 400 +++--- 91 files changed, 3500 insertions(+), 253 deletions(-) create mode 100644 server-zd/crates/dexzd-e2e-tests/tests/dev_server/history_uploads.rs create mode 100644 server-zd/crates/dexzd-engine/src/effects/history_uploads/mod.rs create mode 100644 server-zd/crates/dexzd-engine/src/effects/history_uploads/run_history_objects.rs create mode 100644 server-zd/crates/dexzd-engine/src/effects/history_uploads/segment_upload.rs create mode 100644 server-zd/crates/dexzd-engine/src/effects/history_uploads/upload_rescan.rs create mode 100644 server-zd/crates/dexzd-engine/src/effects/tests/history_upload_tests.rs create mode 100644 server-zd/crates/dexzd-server/tests/serve_process/history_uploads.rs create mode 100644 server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000117 diff --git a/docs/reference/server/configuration.mdx b/docs/reference/server/configuration.mdx index 9ae19d0e08..64f1a1abfd 100644 --- a/docs/reference/server/configuration.mdx +++ b/docs/reference/server/configuration.mdx @@ -338,6 +338,7 @@ disk or fail Steps. | --- | --- | --- | | **history.offloadEnabled** | false | Experimental. Whether partitions offload run history to object storage. | | **history.segmentTargetBytes** | 4194304 (4 MiB) | Experimental. Since 0.1.21. Bytes of history records, checkpoints, and superseded values after which a run seals its open history segment for upload. From 1 byte to 268435456 (256 MiB). | +| **history.uploadConcurrency** | 16 | Experimental. Since 0.1.21. Most sealed history segments the lease holder reads, encodes, and uploads at once. From 1 to 1024. Each upload in flight holds about one encoded segment in memory. | The setting applies to every partition, not to each run. When a node takes the lease, it records the value in every partition whose recorded value differs, @@ -358,6 +359,27 @@ local disk until each segment is uploaded; then pruning returns to the window rule. Commands applied while offload is off belong to no segment, so turning it on again starts a new segment after them. +The lease holder uploads each sealed segment to the primary object store, under +the **history/** prefix of the bucket that holds the commit log. It uploads a +run's segments oldest first and records each upload before the next one. A +segment's content depends only on committed state. A new lease holder therefore +writes the same object again and accepts an existing object with the same +content. An existing object with different content is replaced; the node logs an +error and increments **dexzd_history_upload_checksum_mismatches_total**. + +Uploads never delay commands. While object storage fails them, they retry +with backoff, sealed segments stay on local disk, and the upload backlog metrics +grow; see [Metrics](/reference/server/metrics). A permanent rejection, such as +denied access, is retried once a minute instead. Uploads of segments sealed +earlier continue after **history.offloadEnabled** is turned off. Retention +postpones the deletion of a closed run until its sealed segments are uploaded. +Deleting a run also deletes every object under its **history/** prefix. That +deletion starts a minute after the run's deletion begins, so that uploads an +earlier lease holder abandoned have ended on its side. An upload that object +storage accepted just before can still leave an object of the deleted run; no +read returns it. Do not configure a bucket lifecycle rule that expires objects +under **history/**. + ## Limits {#limits} server-zd enforces these limits. A request that a limit rejects fails at once; diff --git a/docs/reference/server/metrics.mdx b/docs/reference/server/metrics.mdx index 799f27daad..c64de57bde 100644 --- a/docs/reference/server/metrics.mdx +++ b/docs/reference/server/metrics.mdx @@ -35,6 +35,7 @@ No series has a Flow, Flow type, partition, tenant, or key label. | Timers | **dexzd_timer_lag_seconds** | | Snapshots | **dexzd_snapshots_total** with **outcome**; **dexzd_snapshot_last_duration_seconds** | | Local disk | **dexzd_local_storage_bytes** | +| History offload (experimental) | **dexzd_history_upload_backlog_segments**; **dexzd_history_upload_backlog_bytes**; **dexzd_history_upload_oldest_sealed_age_seconds**; **dexzd_history_uploads_total** with **outcome**; **dexzd_history_upload_bytes_total**; **dexzd_history_upload_checksum_mismatches_total**; **dexzd_history_upload_rescan_errors_total** | | Attribute Store sync | See [Monitoring](/reference/server/attribute-store-sync#attribute-store-monitoring) | | External search projection | See [Monitoring](/reference/server/search#search-projection-monitoring) | @@ -78,6 +79,18 @@ Worker polls use one persistent stream, so there is no empty-poll count. **event="delivered"**, every task frame sent, including RPC task frames. A returned attempt that Dex hands out again counts again. Long-poll waits count toward **flow_read** latency. +The history offload gauges come from a rescan of every partition each minute. +**dexzd_history_upload_backlog_segments** counts sealed history segments that +are not uploaded yet. **dexzd_history_upload_backlog_bytes** is their +uncompressed size. **outcome** of **dexzd_history_uploads_total** is one of +**created**, **already_present** (a repeated upload found the same content), +**replaced** (an existing object had other content), each counted once the +upload is recorded, and **failed**. A transient object storage failure is +retried with backoff; any other failure is retried at the next rescan. Alert when +**dexzd_history_upload_oldest_sealed_age_seconds** stays above 900, or when +**dexzd_history_upload_checksum_mismatches_total** or +**dexzd_history_upload_rescan_errors_total** rises. + ~~~text # Failed API calls by class and code sum by (rpc_class, code) (rate(dexzd_api_requests_total{code!="ok"}[5m])) diff --git a/server-zd/Cargo.lock b/server-zd/Cargo.lock index f3743bba3e..50e76a3355 100644 --- a/server-zd/Cargo.lock +++ b/server-zd/Cargo.lock @@ -1180,8 +1180,11 @@ dependencies = [ name = "dexzd-e2e-tests" version = "0.1.0" dependencies = [ + "dexzd-history-segment", + "dexzd-keyspace", "dexzd-protocol", "dexzd-server", + "dexzd-storage", "futures", "prost", "prost-types", @@ -1379,6 +1382,7 @@ dependencies = [ "dexzd-clock", "dexzd-cluster-metadata", "dexzd-engine", + "dexzd-history-segment", "dexzd-keyspace", "dexzd-log", "dexzd-metrics", diff --git a/server-zd/README.md b/server-zd/README.md index 059375e42a..807f34f70d 100644 --- a/server-zd/README.md +++ b/server-zd/README.md @@ -151,7 +151,7 @@ Every label value is from a closed set. No series has a Flow, Flow type, partiti | `dexzd_commit_records_total` | counter | — | Committed commands | | `dexzd_commit_bytes_total` | counter | — | Committed batch bytes | | `dexzd_commit_latency_seconds` | histogram, buckets 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5 | — | Batch seal to durable acknowledgement | -| `dexzd_object_store_requests_total` | counter | `operation`: put, get, list, delete; `object_kind`: log, snapshot, manifest, lease, other | Object storage requests; multipart upload counts once as put | +| `dexzd_object_store_requests_total` | counter | `operation`: put, get, list, delete; `object_kind`: log, snapshot, manifest, lease, history, other | Object storage requests; multipart upload counts once as put | | `dexzd_object_store_request_errors_total` | counter | `operation`: put, get, list, delete | Failed requests after SDK retries | | `dexzd_object_store_put_bytes_total` | counter | — | Uploaded bytes | | `dexzd_snapshots_total` | counter | `outcome`: ok, error | Snapshot attempts | @@ -189,6 +189,13 @@ Every label value is from a closed set. No series has a Flow, Flow type, partiti | `dexzd_search_initial_loads_total` | counter | — | Partition loads started, including loads of progress that cannot continue; the log gives the reason | | `dexzd_search_projection_mode_skipped_partitions_total` | counter | — | Stopped partitions that kept their previous search projection mode at startup | | `dexzd_history_offload_mode_skipped_partitions_total` | counter | — | Stopped partitions that kept their previous history offload mode at startup | +| `dexzd_history_upload_backlog_segments` | gauge | — | Sealed history segments not yet uploaded, summed over each partition's last rescan | +| `dexzd_history_upload_backlog_bytes` | gauge | — | Uncompressed bytes of the history rows and superseded values of `dexzd_history_upload_backlog_segments` | +| `dexzd_history_upload_oldest_sealed_age_seconds` | gauge | — | Age at scrape time of the oldest sealed history segment not yet uploaded, 0 when none | +| `dexzd_history_uploads_total` | counter | `outcome`: created, already_present, replaced, failed | History segment uploads, counted once acknowledged; failed attempts retry with backoff when transient, otherwise at the next rescan | +| `dexzd_history_upload_bytes_total` | counter | — | Bytes of the history segment objects this node created or replaced | +| `dexzd_history_upload_checksum_mismatches_total` | counter | — | Existing history segment objects whose content differed from committed state and were replaced | +| `dexzd_history_upload_rescan_errors_total` | counter | — | Runs or index rows that the history upload rescan could not read; the log names each and the rescan continues | | `dexzd_search_initial_load_documents_total` | counter | — | Documents written by initial SQL visibility loads | Worker polling uses a persistent stream, so there is no empty poll outcome. The stream event metric counts each task delivery and each stream close. diff --git a/server-zd/config/dex-server-zd.example.yaml b/server-zd/config/dex-server-zd.example.yaml index 5a61d5cdfb..a8942194a5 100644 --- a/server-zd/config/dex-server-zd.example.yaml +++ b/server-zd/config/dex-server-zd.example.yaml @@ -84,6 +84,7 @@ history: # Experimental: keep false until the release notes announce history offload. offloadEnabled: false segmentTargetBytes: 4194304 + uploadConcurrency: 16 values: wireInlineThresholdBytes: 100 maxValueBytes: 16777216 diff --git a/server-zd/crates/dexzd-apply/README.md b/server-zd/crates/dexzd-apply/README.md index e3dbf0cf8d..d538e78814 100644 --- a/server-zd/crates/dexzd-apply/README.md +++ b/server-zd/crates/dexzd-apply/README.md @@ -70,7 +70,9 @@ at or below U while the partition offloads history and, whatever the mode, while the run has a SEALED segment. So turning offload off never deletes sealed content before its acknowledgment; once every SEALED segment is acknowledged, pruning returns to the window rule. Segment rows count as history bytes, never toward the -run state limit. Run purge deletes the pending index row with the run. +run state limit. Run purge deletes the pending index row with the run. Retention postpones a closed run while it has a SEALED +segment, as it does for an unacknowledged outbox message, so purging never +abandons an upload. ## Value and storage accounting diff --git a/server-zd/crates/dexzd-apply/src/cross_flow/run_purge.rs b/server-zd/crates/dexzd-apply/src/cross_flow/run_purge.rs index 37e5ba9fa5..d5f0cb1f9e 100644 --- a/server-zd/crates/dexzd-apply/src/cross_flow/run_purge.rs +++ b/server-zd/crates/dexzd-apply/src/cross_flow/run_purge.rs @@ -74,7 +74,10 @@ pub(crate) fn on_retention_fired( cursor: &mut RunCursor, scope: &CommandScope<'_>, ) -> Result<(), ApplyError> { - if has_unacknowledged_outbox(transaction, cursor.rows)? { + // Purging stops uploads, so a sealed segment would never reach object storage. + if has_unacknowledged_outbox(transaction, cursor.rows)? + || has_sealed_history_segment(transaction, cursor.rows)? + { postpone_retention(transaction, cursor, scope.now_micros); return Ok(()); } @@ -261,6 +264,22 @@ fn has_unacknowledged_outbox( .any(|(_, message)| message.acknowledged_at_micros == 0)) } +/// Returns whether the run has a SEALED history segment; its upload pending index row exists +/// exactly while one remains. +fn has_sealed_history_segment( + transaction: &FlowTransaction<'_>, + rows: RunRows, +) -> Result { + let key = history_upload_pending_index_key( + transaction.context().partition, + rows.flow_hash, + rows.run_id, + ); + Ok(transaction + .get(ColumnFamily::PartitionIndex, &key)? + .is_some()) +} + fn remove_timer_indexes( transaction: &mut FlowTransaction<'_>, rows: RunRows, diff --git a/server-zd/crates/dexzd-apply/tests/history_segments.rs b/server-zd/crates/dexzd-apply/tests/history_segments.rs index 887cc337f9..9d8ab37374 100644 --- a/server-zd/crates/dexzd-apply/tests/history_segments.rs +++ b/server-zd/crates/dexzd-apply/tests/history_segments.rs @@ -1251,37 +1251,82 @@ impl Harness { } #[test] -fn purging_a_run_deletes_its_upload_pending_index_row() { +fn retention_waits_for_every_sealed_segment_then_the_purge_deletes_every_row() { let mut harness = harness_with_three_sealed_segments(); harness.stop(); assert!(harness.has_upload_pending_index_row()); - let record = harness.run_record(); - let retention = record.retention_deadline_micros; + let retention = harness.run_record().retention_deadline_micros; assert!(retention > 0); harness.now_micros = retention; - let fired = harness.apply(Command::TimersFired( - dexzd_protocol::dexzd::v1::TimersFiredCommand { - timers: vec![dexzd_protocol::dexzd::v1::TimerId { - run_attempt: 0, - kind: Some(dexzd_protocol::dexzd::v1::timer_id::Kind::Retention( - dexzd_protocol::dexzd::v1::RetentionTimerId {}, - )), - }], - }, + let postponed = harness.apply(retention_fired()); + assert!(!requests_purge(&postponed), "{:?}", postponed.effects); + let record = harness.run_record(); + assert!(!record.is_purging, "purging would stop the uploads"); + assert!(record.retention_deadline_micros > retention); + + for segment in harness.segments() { + if segment.state == HistorySegmentState::Sealed as i32 { + let acknowledged = harness.ack(segment.first_seq, segment.last_seq); + assert!(is_ok(&acknowledged.reply), "{:?}", acknowledged.reply); + } + } + assert!(!harness.has_upload_pending_index_row()); + harness.now_micros = record.retention_deadline_micros; + let fired = harness.apply(retention_fired()); + assert!(requests_purge(&fired), "{:?}", fired.effects); + let purge = harness.apply(Command::RunPurge( + dexzd_protocol::dexzd::v1::RunPurgeCommand {}, )); - assert!( - fired - .effects - .iter() - .any(|effect| matches!(effect, ApplyEffect::RunPurgeRequested { .. })), - "{:?}", - fired.effects + assert!(is_ok(&purge.reply), "{:?}", purge.reply); + assert_every_run_row_purged(&harness); +} + +#[test] +fn purging_a_run_deletes_its_upload_pending_index_row() { + let mut harness = harness_with_three_sealed_segments(); + harness.stop(); + assert!(harness.has_upload_pending_index_row()); + // Retention never purges a run with a SEALED segment, so the test marks the purge itself. + let key = run_singleton_key( + harness.context.flow_hash, + harness.run(), + RunTable::RunRecord, ); + let purging = RunRecord { + is_purging: true, + ..harness.run_record() + }; + harness + .state + .rows + .insert((ColumnFamily::Flow, key), purging.encode_to_vec()); let purge = harness.apply(Command::RunPurge( dexzd_protocol::dexzd::v1::RunPurgeCommand {}, )); assert!(is_ok(&purge.reply), "{:?}", purge.reply); assert!(!harness.has_upload_pending_index_row()); + assert_every_run_row_purged(&harness); +} + +fn retention_fired() -> Command { + Command::TimersFired(dexzd_protocol::dexzd::v1::TimersFiredCommand { + timers: vec![dexzd_protocol::dexzd::v1::TimerId { + run_attempt: 0, + kind: Some(dexzd_protocol::dexzd::v1::timer_id::Kind::Retention( + dexzd_protocol::dexzd::v1::RetentionTimerId {}, + )), + }], + }) +} + +fn requests_purge(outcome: &ApplyOutcome) -> bool { + outcome + .effects + .iter() + .any(|effect| matches!(effect, ApplyEffect::RunPurgeRequested { .. })) +} + +fn assert_every_run_row_purged(harness: &Harness) { assert!( harness .state diff --git a/server-zd/crates/dexzd-e2e-tests/Cargo.toml b/server-zd/crates/dexzd-e2e-tests/Cargo.toml index 008ea5ce5e..c1e48cc58d 100644 --- a/server-zd/crates/dexzd-e2e-tests/Cargo.toml +++ b/server-zd/crates/dexzd-e2e-tests/Cargo.toml @@ -11,8 +11,11 @@ publish.workspace = true [dependencies] [dev-dependencies] +dexzd-history-segment.workspace = true +dexzd-keyspace.workspace = true dexzd-protocol.workspace = true dexzd-server = { workspace = true, features = ["test-util"] } +dexzd-storage.workspace = true futures.workspace = true prost.workspace = true prost-types.workspace = true diff --git a/server-zd/crates/dexzd-e2e-tests/tests/dev_server/harness.rs b/server-zd/crates/dexzd-e2e-tests/tests/dev_server/harness.rs index 54549e0565..21eb4c92ba 100644 --- a/server-zd/crates/dexzd-e2e-tests/tests/dev_server/harness.rs +++ b/server-zd/crates/dexzd-e2e-tests/tests/dev_server/harness.rs @@ -104,6 +104,22 @@ impl DevServer { WorkerTaskServiceClient::new(self.channel.clone()) } + /// Stops the server gracefully and returns its data directory, so a test can open the + /// stopped server's local store. + pub(crate) async fn shutdown_keeping_data(self) -> TempDir { + let Self { + server, + _data_directory: data_directory, + .. + } = self; + server + .expect("the server is shut down once") + .shutdown() + .await + .expect("the server stops cleanly"); + data_directory + } + /// Stops the server gracefully and returns how long the shutdown took. pub(crate) async fn shutdown(mut self) -> Duration { let server = self.server.take().expect("the server is shut down once"); diff --git a/server-zd/crates/dexzd-e2e-tests/tests/dev_server/history_uploads.rs b/server-zd/crates/dexzd-e2e-tests/tests/dev_server/history_uploads.rs new file mode 100644 index 0000000000..a1484fe6d7 --- /dev/null +++ b/server-zd/crates/dexzd-e2e-tests/tests/dev_server/history_uploads.rs @@ -0,0 +1,212 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! With history offload on, the dev server uploads a closed run's sealed history segment to its +//! embedded s3local store and acknowledges it in committed state. + +use std::io::Read; +use std::net::SocketAddr; +use std::time::Duration; + +use dexzd_history_segment::{decode_history_segment, decode_history_segment_header}; +use dexzd_keyspace::{ + ColumnFamily, IndexKind, PartitionId, RunId, RunTable, parse_run_key, partition_index_prefix, +}; +use dexzd_protocol::dex::{StepOptions, StopFlowRequest, StopType}; +use dexzd_protocol::dexzd::v1::{HistorySegmentRecord, HistorySegmentState, RunRecord}; +use dexzd_storage::{LocalReadView, RocksDbLocalStore, RocksDbStoreOptions, ScanDirection}; +use prost::Message; +use s3local::{ListObjectsRequest, ObjectReadRequest, Store}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; + +use super::harness::{DevServer, free_loopback_address, start_pull_flow}; +use super::string_value; + +const FLOW_ID: &str = "history-upload-flow"; +/// The bucket of a dev server's embedded object store. +const DEV_BUCKET: &str = "dex-dev"; +/// The partition count of a new `dev` data directory. +const DEV_PARTITION_COUNT: u16 = 32; +const UPLOAD_BUDGET: Duration = Duration::from_secs(30); +const CREATED_UPLOADS: &str = "dexzd_history_uploads_total{outcome=\"created\"} "; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_closed_runs_history_segment_uploads_to_the_embedded_object_store() { + let metrics = free_loopback_address(); + let overrides = format!("history:\n offloadEnabled: true\nmetrics:\n listen: '{metrics}'\n"); + let server = DevServer::start_with_config_overrides(Some(&overrides)).await; + let mut flow_client = server.flow_client(); + let started = start_pull_flow( + &mut flow_client, + FLOW_ID, + "OffloadedFlow", + "IdleStep", + string_value("idle"), + StepOptions::default(), + ) + .await; + let run_id: RunId = started.run_id.parse().expect("the run ID is canonical"); + flow_client + .stop_flow(StopFlowRequest { + flow_id: FLOW_ID.to_owned(), + run_id: started.run_id.clone(), + stop_type: StopType::Terminate as i32, + ..StopFlowRequest::default() + }) + .await + .expect("StopFlow closes the run"); + + let store = server.dev_object_store_for_test(); + let object_key = wait_for_history_object(&store).await; + assert!( + object_key.contains(&run_id.to_string()), + "the key names the run: {object_key}" + ); + let object = read_object(&store, &object_key); + let content = decode_history_segment(&object).expect("the uploaded object decodes"); + assert_eq!(content.identity.run_id, run_id); + assert_eq!(content.identity.first_seq, 1); + assert!(!content.records.is_empty()); + wait_for_created_uploads(metrics, 1).await; + + let data_directory = server.shutdown_keeping_data().await; + let local_store = RocksDbLocalStore::open( + data_directory.path().join("store"), + RocksDbStoreOptions::default(), + ) + .expect("the stopped server's store opens"); + let (segments, run_record) = stored_run_history(&local_store, run_id); + assert_eq!(segments.len(), 1, "{segments:?}"); + let segment = &segments[0]; + assert_eq!(segment.state, HistorySegmentState::Uploaded as i32); + assert_eq!(segment.last_seq, content.identity.last_seq); + let layout = decode_history_segment_header(&object, object.len() as u64) + .expect("the uploaded header decodes"); + assert_eq!(segment.content_checksum, layout.content_checksum().to_vec()); + assert_eq!(segment.object_size_bytes, object.len() as u64); + assert_eq!(run_record.history_uploaded_through_seq, segment.last_seq); + for partition in 0..DEV_PARTITION_COUNT { + let pending = local_store + .scan_prefix( + ColumnFamily::PartitionIndex, + &partition_index_prefix(PartitionId(partition), IndexKind::HistoryUploadPending), + ScanDirection::Forward, + 1, + ) + .expect("the index scans"); + assert!(pending.is_empty(), "partition {partition} still uploads"); + } +} + +/// Waits until the store holds one history object and returns its key. +async fn wait_for_history_object(store: &Store) -> String { + let listing = ListObjectsRequest { + prefix: "history/".to_owned(), + delimiter: String::new(), + max_keys: 100, + continuation_token: None, + start_after: None, + }; + let waited = tokio::time::timeout(UPLOAD_BUDGET, async { + loop { + let page = store + .list_objects(DEV_BUCKET, &listing) + .expect("the history objects can be listed"); + if let Some(object) = page.objects.first() { + assert_eq!(page.objects.len(), 1, "one run sealed one segment"); + return object.key.clone(); + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await; + waited.unwrap_or_else(|_| panic!("no history object appeared within {UPLOAD_BUDGET:?}")) +} + +fn read_object(store: &Store, key: &str) -> Vec { + let body = store + .read_object(DEV_BUCKET, key, &ObjectReadRequest::default()) + .expect("the history object reads"); + let mut bytes = Vec::new(); + body.file + .take(body.length) + .read_to_end(&mut bytes) + .expect("the history object body reads"); + bytes +} + +/// Waits until the node reports `count` created history uploads. +async fn wait_for_created_uploads(metrics: SocketAddr, count: u64) { + let mut last_seen = None; + let waited = tokio::time::timeout(UPLOAD_BUDGET, async { + loop { + last_seen = scrape_counter(metrics, CREATED_UPLOADS).await; + if last_seen == Some(count) { + return; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await; + waited.unwrap_or_else(|_| { + panic!("the node did not report {count} created uploads; last saw {last_seen:?}") + }); +} + +/// Returns the sample of the line that starts with `series`, or `None` while the listener is down. +async fn scrape_counter(metrics: SocketAddr, series: &str) -> Option { + let mut stream = tokio::net::TcpStream::connect(metrics).await.ok()?; + stream + .write_all(b"GET /metrics HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n") + .await + .ok()?; + let mut response = Vec::new(); + stream.read_to_end(&mut response).await.ok()?; + String::from_utf8(response) + .ok()? + .lines() + .find_map(|line| line.strip_prefix(series)) + .and_then(|value| value.trim().parse().ok()) +} + +/// Returns the run's history segment rows and its run record from a stopped node's store. +fn stored_run_history( + local_store: &RocksDbLocalStore, + run_id: RunId, +) -> (Vec, RunRecord) { + let mut segments = Vec::new(); + let mut run_record = None; + for (key, value) in local_store + .scan( + ColumnFamily::Flow, + None, + None, + ScanDirection::Forward, + usize::MAX, + ) + .expect("the Flow rows scan") + { + let Ok(parsed) = parse_run_key(&key) else { + continue; + }; + if parsed.run_id != run_id { + continue; + } + match parsed.table { + RunTable::HistorySegment => segments.push( + HistorySegmentRecord::decode(value.as_slice()).expect("the segment row decodes"), + ), + RunTable::RunRecord => { + run_record = + Some(RunRecord::decode(value.as_slice()).expect("the run record decodes")); + } + _ => {} + } + } + (segments, run_record.expect("the run record is stored")) +} diff --git a/server-zd/crates/dexzd-e2e-tests/tests/dev_server/main.rs b/server-zd/crates/dexzd-e2e-tests/tests/dev_server/main.rs index 41903dcd24..540c9fe977 100644 --- a/server-zd/crates/dexzd-e2e-tests/tests/dev_server/main.rs +++ b/server-zd/crates/dexzd-e2e-tests/tests/dev_server/main.rs @@ -17,6 +17,7 @@ mod flow_controls; mod hard_delete_closed; mod harness; mod history_paging; +mod history_uploads; mod queued_state; mod queued_time_travel; mod reserved_slots; diff --git a/server-zd/crates/dexzd-engine/Cargo.toml b/server-zd/crates/dexzd-engine/Cargo.toml index 0df9288b16..2b45c59a1f 100644 --- a/server-zd/crates/dexzd-engine/Cargo.toml +++ b/server-zd/crates/dexzd-engine/Cargo.toml @@ -14,6 +14,7 @@ bytes.workspace = true dexzd-apply.workspace = true dexzd-clock.workspace = true dexzd-cluster-metadata.workspace = true +dexzd-history-segment.workspace = true dexzd-keyspace.workspace = true dexzd-log.workspace = true dexzd-metrics.workspace = true @@ -42,7 +43,6 @@ test-util = [] [dev-dependencies] # Enables test-util for this crate's own integration tests. dexzd-engine = { workspace = true, features = ["test-util"] } -dexzd-history-segment.workspace = true tempfile.workspace = true tokio = { workspace = true, features = ["test-util"] } diff --git a/server-zd/crates/dexzd-engine/README.md b/server-zd/crates/dexzd-engine/README.md index 3b173f1827..30934345b0 100644 --- a/server-zd/crates/dexzd-engine/README.md +++ b/server-zd/crates/dexzd-engine/README.md @@ -4,6 +4,10 @@ Runtime that proposes, commits, and applies server-zd commands and runs their effects: dispatch, timers, deliveries, and recovery. +## History segment uploads + +The effect runner uploads each committed SEALED history segment to the primary object store and proposes `HistorySegmentUploaded`. One task per run uploads its segments oldest first. The committed `HistorySegmentSealed` effect, partition rebuild from the `HistoryUploadPending` index, and a periodic rescan start that task, so a restart, takeover, or snapshot restore never loses an upload. A segment's key and content come from committed state alone. A repeated upload therefore finds an equal content checksum and acknowledges the existing object; an object with another checksum is replaced with an If-Match write. Failed object storage calls retry with backoff and never delay commands; a permanent rejection waits for the next rescan. Every call ends when the node stops, so no history request outlives its owner. A run purge waits one minute, so the client side of every write that an earlier owner or an abandoned attempt started has ended. It then deletes the run's segment objects by their catalog keys, sweeps the run's `history/` prefix, and proposes `RunPurge`. Dropping a call does not fence a write that the object store already accepted: such a write can still land after the purge. It leaks an object of a purged run, whose run ID is never reused, so no read returns it. A periodic orphan sweep of `history/`, not yet implemented, is to remove these objects. + ## SubFlow hard-delete Attributes WaitFor results upload hard-delete values before the parent command enters the log. Object reference indices cover the parent Attribute writes, then each SubFlow's initial Attributes in condition order. Apply places the SubFlow references in its child-start outbox message without doing I/O. diff --git a/server-zd/crates/dexzd-engine/src/effects/history_uploads/mod.rs b/server-zd/crates/dexzd-engine/src/effects/history_uploads/mod.rs new file mode 100644 index 0000000000..b59c4398e1 --- /dev/null +++ b/server-zd/crates/dexzd-engine/src/effects/history_uploads/mod.rs @@ -0,0 +1,47 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! History offload on the leader: uploads sealed history segments and acknowledges them in seq +//! order, rescans the upload backlog, and deletes a purged run's segment objects. +//! +//! Apply decides each segment's range and content, so every owner of a partition writes the same +//! object at the same key. A repeated upload therefore finds an object with an equal content +//! checksum and acknowledges it without fencing. Uploads never block command commits: a failed +//! upload retries with backoff while the backlog gauges grow. +//! +//! Every object storage call ends when the node stops, so no history request outlives its owner. + +mod run_history_objects; +mod segment_upload; +mod upload_rescan; + +use std::future::Future; + +use dexzd_object_store::ObjectStoreResult; + +pub(crate) use run_history_objects::delete_run_history_objects; +pub(crate) use segment_upload::upload_sealed_history_segments; +pub(crate) use upload_rescan::run_history_upload_rescan; + +use crate::effects::{EffectContext, EffectError}; + +/// Runs one object storage call, abandoning it once the node stops. +/// +/// Returns the call's result, with a failure as [`EffectError::ObjectStore`], or +/// [`EffectError::Stopped`] when shutdown came first. +async fn until_stopped( + context: &EffectContext, + call: impl Future>, +) -> Result { + // Biased for reproducible simulation; shutdown fires once. + tokio::select! { + biased; + () = context.shutdown.cancelled() => Err(EffectError::Stopped), + result = call => result.map_err(EffectError::ObjectStore), + } +} diff --git a/server-zd/crates/dexzd-engine/src/effects/history_uploads/run_history_objects.rs b/server-zd/crates/dexzd-engine/src/effects/history_uploads/run_history_objects.rs new file mode 100644 index 0000000000..5e31fbfb2f --- /dev/null +++ b/server-zd/crates/dexzd-engine/src/effects/history_uploads/run_history_objects.rs @@ -0,0 +1,175 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Deletes a purging run's history segment objects before its RunPurge (plan section 5.12). + +use std::time::Duration; + +use dexzd_history_segment::{history_run_object_key_prefix, history_segment_object_key}; +use dexzd_keyspace::{ColumnFamily, RunTable, run_table_prefix}; +use dexzd_protocol::dexzd::v1::HistorySegmentRecord; + +use super::until_stopped; +use crate::effects::state_reads::{RunLocation, decode_row, read_run_record, scan_all}; +use crate::effects::{EffectContext, EffectError, RetryBackoff}; +use crate::flow_coordination::FlowAddress; + +/// Lets history writes still in flight end first, from this node's abandoned attempts or a +/// previous owner; object requests time out after 30 seconds by default. +const HISTORY_WRITE_SETTLE: Duration = Duration::from_secs(60); +/// How long a purge waits after object storage rejects a call permanently. +const PERMANENT_FAILURE_RECHECK: Duration = Duration::from_secs(60); +/// Most keys one listing page or batch delete takes. +const DELETE_BATCH_KEYS: usize = 1000; + +/// Deletes every history segment object of a purging run that kept segment state, then sweeps +/// the run's `history/` prefix for objects outside its catalog, such as those of a timeline that +/// an older restored snapshot replaced. +/// +/// Retention postpones while a segment is SEALED, so this node writes no object of the run any +/// more. Deletion starts only after [`HISTORY_WRITE_SETTLE`], so the client side of every write +/// that a previous owner or an abandoned attempt started has ended. Failed calls retry with +/// backoff, and permanent rejections every minute. +pub(crate) async fn delete_run_history_objects( + context: &EffectContext, + flow: &FlowAddress, + run: RunLocation, +) -> Result<(), EffectError> { + let Some(catalog_keys) = history_catalog_keys(context, flow, run)? else { + return Ok(()); + }; + // A write the store already accepted can still land later; a future orphan sweep removes it. + if !sleep_until_stopped(context, HISTORY_WRITE_SETTLE).await { + return Err(EffectError::Stopped); + } + for batch in catalog_keys.chunks(DELETE_BATCH_KEYS) { + delete_keys(context, run, batch).await?; + } + sweep_run_prefix(context, flow, run).await +} + +/// Returns the object key of every history segment row of the run, in any state, or `None` +/// when the run never kept segment state, so it never uploaded history. +fn history_catalog_keys( + context: &EffectContext, + flow: &FlowAddress, + run: RunLocation, +) -> Result>, EffectError> { + let reader = context + .flow_effect_operations + .flow_state_reader(flow, false)?; + let has_segment_state = read_run_record(reader.as_ref(), run)? + .is_some_and(|record| record.history_open_segment_first_seq != 0); + let prefix = run_table_prefix(run.flow_hash, run.run_id, RunTable::HistorySegment); + let keys = scan_all(reader.as_ref(), ColumnFamily::Flow, &prefix)? + .into_iter() + .map(|(key, value)| { + let segment: HistorySegmentRecord = decode_row("HistorySegment", &key, &value)?; + Ok(history_segment_object_key( + flow.tenant_id, + run.flow_hash, + run.run_id, + segment.first_seq, + segment.last_seq, + )) + }) + .collect::, EffectError>>()?; + Ok((has_segment_state || !keys.is_empty()).then_some(keys)) +} + +/// Lists the run's history prefix and deletes what it finds until a listing is empty. +async fn sweep_run_prefix( + context: &EffectContext, + flow: &FlowAddress, + run: RunLocation, +) -> Result<(), EffectError> { + let prefix = history_run_object_key_prefix(flow.tenant_id, run.flow_hash, run.run_id); + let max_keys = u32::try_from(DELETE_BATCH_KEYS).expect("the batch size fits in u32"); + let mut backoff = context.retry_backoff(); + loop { + let listed = until_stopped( + context, + context.history_object_store.list(&prefix, None, max_keys), + ) + .await; + let listing = match listed { + Ok(listing) => listing, + Err(EffectError::Stopped) => return Err(EffectError::Stopped), + Err(error) => { + wait_after_failure(context, &mut backoff, run, &error).await?; + continue; + } + }; + if listing.objects.is_empty() { + return Ok(()); + } + let keys: Vec = listing + .objects + .into_iter() + .map(|object| object.key) + .collect(); + delete_keys(context, run, &keys).await?; + } +} + +async fn delete_keys( + context: &EffectContext, + run: RunLocation, + keys: &[String], +) -> Result<(), EffectError> { + let mut backoff = context.retry_backoff(); + loop { + match until_stopped(context, context.history_object_store.delete_many(keys)).await { + Ok(()) => return Ok(()), + Err(EffectError::Stopped) => return Err(EffectError::Stopped), + Err(error) => wait_after_failure(context, &mut backoff, run, &error).await?, + } + } +} + +/// Waits before retrying a failed call: with backoff when it may pass, otherwise for the slow +/// recheck of a permanent rejection. +/// +/// Returns [`EffectError::Stopped`] when the node stops first. +async fn wait_after_failure( + context: &EffectContext, + backoff: &mut RetryBackoff, + run: RunLocation, + error: &EffectError, +) -> Result<(), EffectError> { + let has_waited = if error.is_retryable() { + tracing::warn!( + %error, + run_id = %run.run_id, + "deleting the run's history segment objects failed; retrying" + ); + backoff.wait(&context.shutdown).await + } else { + tracing::error!( + %error, + run_id = %run.run_id, + "object storage rejected deleting the run's history segment objects; retrying in a minute" + ); + sleep_until_stopped(context, PERMANENT_FAILURE_RECHECK).await + }; + if has_waited { + Ok(()) + } else { + Err(EffectError::Stopped) + } +} + +/// Sleeps for `delay`; returns false when the node stopped first. +async fn sleep_until_stopped(context: &EffectContext, delay: Duration) -> bool { + // Biased for reproducible simulation; shutdown fires once. + tokio::select! { + biased; + () = context.shutdown.cancelled() => false, + () = tokio::time::sleep(delay) => true, + } +} diff --git a/server-zd/crates/dexzd-engine/src/effects/history_uploads/segment_upload.rs b/server-zd/crates/dexzd-engine/src/effects/history_uploads/segment_upload.rs new file mode 100644 index 0000000000..e841b9204a --- /dev/null +++ b/server-zd/crates/dexzd-engine/src/effects/history_uploads/segment_upload.rs @@ -0,0 +1,470 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Uploads one run's sealed history segments, oldest first, and proposes each acknowledgment +//! (plan section 5.4). + +use std::collections::BTreeSet; + +use bytes::Bytes; +use dexzd_apply::ApplyError; +use dexzd_apply::FlowStateReader; +use dexzd_apply::read_model::{ + ReadModelError, oldest_sealed_history_segment, read_history_segment_content, +}; +use dexzd_history_segment::{ + HistorySegmentCodecError, decode_history_segment, decode_history_segment_header, + encode_history_segment, history_segment_object_key, +}; +use dexzd_keyspace::{RunId, TenantId}; +use dexzd_metrics::HistoryUploadOutcome; +use dexzd_object_store::{ + ObjectStoreError, ObjectStoreErrorKind, ObjectStoreOperation, WriteCondition, WriteOutcome, +}; +use dexzd_protocol::dexzd::v1::{ + FlowCommand, HistorySegmentUploadedCommand, command_reply, flow_command, +}; +use tokio::sync::OwnedSemaphorePermit; + +use super::until_stopped; +use crate::blocking_work::BlockingWorkFailure; +use crate::effects::state_reads::RunLocation; +use crate::effects::{EffectContext, EffectError}; +use crate::flow_coordination::{CommitUrgency, FlowAddress}; + +/// gRPC NOT_FOUND, which apply returns for a run that is gone or purging. +const GRPC_NOT_FOUND: i32 = 5; +/// Conditional writes one upload attempt makes before it backs off; each extra write means the +/// object changed between our read and write. +const MAX_CONDITIONAL_WRITES: usize = 4; + +/// One sealed segment encoded from committed state. It owns its upload slot, so encoded bytes +/// never outlive the slot and memory stays bounded by `history.uploadConcurrency`. +struct EncodedSealedSegment { + identity: SealedSegmentIdentity, + body: Bytes, + _upload_slot: OwnedSemaphorePermit, +} + +/// What names a sealed segment and its object; holds no object bytes. +struct SealedSegmentIdentity { + first_seq: u64, + last_seq: u64, + object_key: String, + content_checksum: [u8; 16], +} + +/// A segment whose object is stored, with how the upload ended; holds no object bytes. +struct StoredSealedSegment { + identity: SealedSegmentIdentity, + outcome: HistoryUploadOutcome, + object_size_bytes: u64, + /// Bytes this upload wrote; zero when the object already existed. + written_bytes: u64, +} + +/// Uploads the run's SEALED history segments in seq order and acknowledges each one, until +/// committed state has none left. Retryable object storage failures retry with backoff and +/// never block commands. +/// +/// A run whose upload failed without a retry is skipped by every trigger until the rescan +/// allows it again, so it is retried at most once per rescan. +/// +/// Returns `Ok` once no SEALED segment remains, the run is gone or purging, or the run waits +/// for the rescan after a failure. Returns an error when committed state cannot be read or +/// encoded, when object storage rejects a call permanently, or when an acknowledgment is +/// rejected or does not advance the run; nothing is acknowledged meanwhile. +pub(crate) async fn upload_sealed_history_segments( + context: &EffectContext, + flow: &FlowAddress, + run_id: RunId, +) -> Result<(), EffectError> { + if failed_runs(context).contains(&run_id) { + tracing::debug!(%run_id, "a failed history upload waits for the next rescan"); + return Ok(()); + } + let result = upload_run_segments(context, flow, run_id).await; + match &result { + Ok(()) => { + failed_runs(context).remove(&run_id); + } + Err(EffectError::Stopped) => {} + Err(_) => { + failed_runs(context).insert(run_id); + } + } + result +} + +/// Lets the next trigger retry `run_id` after a failure; the rescan calls it. +pub(super) fn allow_upload_retry(context: &EffectContext, run_id: RunId) { + failed_runs(context).remove(&run_id); +} + +fn failed_runs(context: &EffectContext) -> std::sync::MutexGuard<'_, BTreeSet> { + context + .history_upload_failed_runs + .lock() + .expect("the failed upload set lock is never poisoned") +} + +async fn upload_run_segments( + context: &EffectContext, + flow: &FlowAddress, + run_id: RunId, +) -> Result<(), EffectError> { + let run = context.run_location(flow, run_id); + let mut backoff = context.retry_backoff(); + let mut acknowledged_first_seq = None; + loop { + let upload_slot = tokio::select! { + biased; + () = context.shutdown.cancelled() => return Err(EffectError::Stopped), + permit = context.history_upload_slots.clone().acquire_owned() => { + permit.map_err(|_| EffectError::Stopped)? + } + }; + let segment = match encode_oldest_sealed_segment(context, flow, run, upload_slot).await { + Ok(Some(segment)) => segment, + Ok(None) => return Ok(()), + Err(EffectError::Stopped) => return Err(EffectError::Stopped), + Err(error) if error.is_retryable() => { + tracing::warn!(%error, %run_id, "reading a sealed history segment failed; retrying"); + if !backoff.wait(&context.shutdown).await { + return Err(EffectError::Stopped); + } + continue; + } + Err(error) => return Err(failed_until_rescan(context, run_id, None, error)), + }; + let object_key = segment.identity.object_key.clone(); + if acknowledged_first_seq == Some(segment.identity.first_seq) { + let error = EffectError::InvariantViolation { + detail: format!( + "history segment {}..={} of run {run_id} stays SEALED after its \ + acknowledgment applied", + segment.identity.first_seq, segment.identity.last_seq + ), + }; + return Err(failed_until_rescan( + context, + run_id, + Some(&object_key), + error, + )); + } + let stored = match store_segment_object(context, segment).await { + Ok(stored) => stored, + Err(EffectError::Stopped) => return Err(EffectError::Stopped), + Err(error) if error.is_retryable() => { + context + .metrics + .history_uploads + .record(HistoryUploadOutcome::Failed); + tracing::warn!( + %error, + %run_id, + object_key, + "uploading a history segment failed; retrying" + ); + if !backoff.wait(&context.shutdown).await { + return Err(EffectError::Stopped); + } + continue; + } + Err(error) => { + return Err(failed_until_rescan( + context, + run_id, + Some(&object_key), + error, + )); + } + }; + backoff = context.retry_backoff(); + let is_run_uploading = match acknowledge_segment(context, flow, run_id, &stored).await { + Ok(is_run_uploading) => is_run_uploading, + Err(EffectError::Stopped) => return Err(EffectError::Stopped), + Err(error) => { + return Err(failed_until_rescan( + context, + run_id, + Some(&object_key), + error, + )); + } + }; + // Recorded once the acknowledgment committed, so the counter shows finished uploads. + record_stored_segment(context, &stored); + if !is_run_uploading { + return Ok(()); + } + acknowledged_first_seq = Some(stored.identity.first_seq); + } +} + +/// Counts and logs an upload failure that waits for the next rescan, and returns its error. +fn failed_until_rescan( + context: &EffectContext, + run_id: RunId, + object_key: Option<&str>, + error: EffectError, +) -> EffectError { + context + .metrics + .history_uploads + .record(HistoryUploadOutcome::Failed); + tracing::error!( + %error, + %run_id, + object_key, + "a history segment upload failed without a retry; retrying at the next rescan" + ); + error +} + +/// Reads the run's oldest SEALED segment from committed state and encodes it on a blocking +/// thread, so zstd compression never stalls async tasks. +/// +/// Returns `None` when no SEALED segment remains or the run is gone or purging. Returns an +/// error, and writes nothing, when the segment's content cannot be read or encoded. +async fn encode_oldest_sealed_segment( + context: &EffectContext, + flow: &FlowAddress, + run: RunLocation, + upload_slot: OwnedSemaphorePermit, +) -> Result, EffectError> { + let reader = context + .flow_effect_operations + .flow_state_reader(flow, false)?; + let tenant_id = flow.tenant_id; + let encoding = context + .config + .blocking_work + .run_tracked(&context.tracker, move || { + encode_from_committed_state(reader.as_ref(), tenant_id, run) + }); + let encoded = match encoding.await { + Ok(result) => result?, + Err(BlockingWorkFailure::Panicked(payload)) => std::panic::resume_unwind(payload), + Err(BlockingWorkFailure::Cancelled) => return Err(EffectError::Stopped), + }; + Ok(encoded.map(|(identity, body)| EncodedSealedSegment { + identity, + body, + _upload_slot: upload_slot, + })) +} + +fn encode_from_committed_state( + reader: &dyn FlowStateReader, + tenant_id: TenantId, + run: RunLocation, +) -> Result, EffectError> { + let segment = match oldest_sealed_history_segment(reader, run.flow_hash, run.run_id) { + Ok(Some(segment)) => segment, + Ok(None) | Err(ReadModelError::RunNotFound { .. }) => return Ok(None), + Err(error) => return Err(read_model_error(error)), + }; + let content = + read_history_segment_content(reader, run.flow_hash, run.run_id, segment.first_seq) + .map_err(read_model_error)? + .ok_or_else(|| EffectError::InvariantViolation { + detail: format!( + "history segment {}..={} of run {} disappeared within one committed view", + segment.first_seq, segment.last_seq, run.run_id + ), + })?; + let encoded = encode_history_segment(&content) + .map_err(|error| encoding_error(run.run_id, segment.first_seq, &error))?; + let identity = SealedSegmentIdentity { + first_seq: segment.first_seq, + last_seq: segment.last_seq, + object_key: history_segment_object_key( + tenant_id, + run.flow_hash, + run.run_id, + segment.first_seq, + segment.last_seq, + ), + content_checksum: encoded.content_checksum, + }; + Ok(Some((identity, Bytes::from(encoded.bytes)))) +} + +/// Creates the segment's object, or confirms an existing one (plan R11), then releases the +/// encoded bytes and the upload slot before returning. +/// +/// An existing object that decodes with the same content checksum came from an earlier upload +/// of this segment. Any other existing object came from committed history that no longer +/// exists, such as an older restored snapshot, or from a determinism bug; committed state is +/// authoritative while the segment is SEALED, so it is replaced with an If-Match write. +/// +/// Returns the stored segment, the object storage error, which leaves no partial object, or +/// [`EffectError::Stopped`] when the node stops during a call. +async fn store_segment_object( + context: &EffectContext, + segment: EncodedSealedSegment, +) -> Result { + let store = &context.history_object_store; + let key = segment.identity.object_key.as_str(); + let body_size = segment.body.len() as u64; + let mut condition = WriteCondition::CreateOnly; + for _ in 0..MAX_CONDITIONAL_WRITES { + let is_replacement = matches!(condition, WriteCondition::ReplaceIfEtagMatches { .. }); + let written = + until_stopped(context, store.put(key, segment.body.clone(), condition)).await?; + if let WriteOutcome::Written { .. } = written { + let outcome = if is_replacement { + HistoryUploadOutcome::Replaced + } else { + HistoryUploadOutcome::Created + }; + return Ok(stored_segment(segment, outcome, body_size, body_size)); + } + let Some(existing) = until_stopped(context, store.get(key)).await? else { + condition = WriteCondition::CreateOnly; + continue; + }; + let existing_checksum = verified_content_checksum(context, existing.bytes).await?; + if existing_checksum == Some(segment.identity.content_checksum) { + let outcome = HistoryUploadOutcome::AlreadyPresent; + return Ok(stored_segment(segment, outcome, existing.object_size, 0)); + } + context.metrics.history_upload_checksum_mismatches.inc(); + tracing::error!( + object_key = key, + existing_object_size_bytes = existing.object_size, + "an existing history segment object differs from committed state; replacing it" + ); + condition = WriteCondition::ReplaceIfEtagMatches { + etag: existing.etag, + }; + } + Err(EffectError::ObjectStore(ObjectStoreError::new( + ObjectStoreOperation::Put, + key, + ObjectStoreErrorKind::Transient, + format!("the object changed during each of {MAX_CONDITIONAL_WRITES} conditional writes"), + ))) +} + +/// Drops the segment's bytes and upload slot, keeping what its acknowledgment needs. +fn stored_segment( + segment: EncodedSealedSegment, + outcome: HistoryUploadOutcome, + object_size_bytes: u64, + written_bytes: u64, +) -> StoredSealedSegment { + StoredSealedSegment { + identity: segment.identity, + outcome, + object_size_bytes, + written_bytes, + } +} + +/// Decodes an existing object completely on a blocking thread, under the caller's upload slot. +/// +/// Returns its content checksum, or `None` for an object that fails any block, value, trailer, +/// or content check. +async fn verified_content_checksum( + context: &EffectContext, + object: Bytes, +) -> Result, EffectError> { + let verification = context + .config + .blocking_work + .run_tracked(&context.tracker, move || { + decode_history_segment(&object).ok()?; + decode_history_segment_header(&object, object.len() as u64) + .ok() + .map(|layout| layout.content_checksum()) + }); + match verification.await { + Ok(checksum) => Ok(checksum), + Err(BlockingWorkFailure::Panicked(payload)) => std::panic::resume_unwind(payload), + Err(BlockingWorkFailure::Cancelled) => Err(EffectError::Stopped), + } +} + +fn record_stored_segment(context: &EffectContext, stored: &StoredSealedSegment) { + let metrics = &context.metrics; + metrics.history_uploads.record(stored.outcome); + metrics.history_upload_bytes.inc_by(stored.written_bytes); + tracing::debug!( + object_key = stored.identity.object_key, + outcome = ?stored.outcome, + object_size_bytes = stored.object_size_bytes, + "stored a history segment object" + ); +} + +/// Proposes the segment's acknowledgment and waits until it commits. +/// +/// Returns false when the run is gone or purging, which ends the run's uploads. A duplicate or +/// stale acknowledgment applies as a no-op and returns true. +async fn acknowledge_segment( + context: &EffectContext, + flow: &FlowAddress, + run_id: RunId, + stored: &StoredSealedSegment, +) -> Result { + let identity = &stored.identity; + let command = FlowCommand { + flow_id: flow.flow_id.clone(), + run_id: run_id.as_bytes().to_vec(), + tenant_id: flow.tenant_id.0, + command: Some(flow_command::Command::HistorySegmentUploaded( + HistorySegmentUploadedCommand { + first_seq: identity.first_seq, + last_seq: identity.last_seq, + content_checksum: identity.content_checksum.to_vec(), + object_size_bytes: stored.object_size_bytes, + }, + )), + ..FlowCommand::default() + }; + let reply = context + .propose_until_committed(flow, command, CommitUrgency::Batched) + .await + .ok_or(EffectError::Stopped)?; + match reply.reply { + Some(command_reply::Reply::Error(error)) if error.grpc_code == GRPC_NOT_FOUND => { + tracing::debug!(%run_id, "stopping history uploads of a run that is gone or purging"); + Ok(false) + } + Some(command_reply::Reply::Error(error)) => Err(EffectError::InvariantViolation { + detail: format!( + "HistorySegmentUploaded {}..={} of run {run_id} was rejected: {}", + identity.first_seq, identity.last_seq, error.message + ), + }), + _ => Ok(true), + } +} + +/// Maps a committed read failure; a failed store read stays retryable. +fn read_model_error(error: ReadModelError) -> EffectError { + match error { + ReadModelError::State(ApplyError::StateRead(error)) => EffectError::StateRead(error), + ReadModelError::State(ApplyError::CorruptState { table, key, detail }) => { + EffectError::CorruptState { table, key, detail } + } + error => EffectError::InvariantViolation { + detail: format!("reading a sealed history segment failed: {error}"), + }, + } +} + +/// Maps a segment encoding failure; committed content always encodes, so a failure is a bug. +fn encoding_error(run_id: RunId, first_seq: u64, error: &HistorySegmentCodecError) -> EffectError { + EffectError::InvariantViolation { + detail: format!("encoding history segment {first_seq} of run {run_id} failed: {error}"), + } +} diff --git a/server-zd/crates/dexzd-engine/src/effects/history_uploads/upload_rescan.rs b/server-zd/crates/dexzd-engine/src/effects/history_uploads/upload_rescan.rs new file mode 100644 index 0000000000..d5667eb391 --- /dev/null +++ b/server-zd/crates/dexzd-engine/src/effects/history_uploads/upload_rescan.rs @@ -0,0 +1,282 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! Periodic rescan of every owned partition's sealed history segments (plan section 5.15): it +//! publishes the upload backlog gauges and restarts the uploads of runs whose work stopped. A run +//! whose rows cannot be read is logged, counted, and skipped, so it never hides the others. + +use std::collections::BTreeMap; +use std::sync::Arc; + +use dexzd_apply::{ApplyEffect, FlowStateReader}; +use dexzd_keyspace::row_keys::{history_segment_key, history_segment_value_prefix}; +use dexzd_keyspace::{ + ColumnFamily, IndexKind, PartitionId, RunTable, TenantId, partition_index_prefix, + run_table_prefix, +}; +use dexzd_protocol::dexzd::v1::{ + HistoryRowMetadata, HistorySegmentRecord, HistorySegmentState, HistorySegmentValueRecord, +}; + +use super::segment_upload::allow_upload_retry; +use crate::blocking_work::BlockingWorkFailure; +use crate::effects::state_reads::{ + RunLocation, decode_row, parse_run_item_index_key, prefix_successor, read_run_record, scan_all, +}; +use crate::effects::{EffectContext, EffectError, RouterMessage}; +use crate::flow_coordination::FlowAddress; + +/// Uncompressed bytes of each sealed segment seen by the last scan, keyed by partition, run, and +/// first seq. A sealed segment's rows never change before its acknowledgment, so a size is +/// computed once. +type SealedSegmentSizes = BTreeMap<(PartitionId, RunLocation, u64), u64>; + +/// What one scan of a partition found. +#[derive(Default)] +struct PartitionUploadBacklog { + segments: u64, + bytes: u64, + oldest_sealed_at_micros: Option, + runs: Vec<(FlowAddress, RunLocation)>, + sizes: SealedSegmentSizes, + /// Index rows or runs that could not be read, with the run when its key parsed. + unreadable: Vec<(Option, EffectError)>, +} + +/// What one scan found for one run. +struct RunUploadBacklog { + flow: FlowAddress, + segments: u64, + bytes: u64, + oldest_sealed_at_micros: Option, + sizes: SealedSegmentSizes, +} + +/// Rescans every owned partition each `history_upload_rescan_interval` until shutdown, then +/// clears the backlog gauges. +pub(crate) async fn run_history_upload_rescan(context: Arc) { + let interval = context.config.history_upload_rescan_interval; + if interval.is_zero() { + return; + } + let mut ticker = tokio::time::interval(interval); + ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + let mut sizes = SealedSegmentSizes::new(); + loop { + // Biased for reproducible simulation; shutdown fires once. + tokio::select! { + biased; + () = context.shutdown.cancelled() => break, + _ = ticker.tick() => {} + } + let partitions = context.task_queue_memory().owned_partition_list(); + for partition in partitions { + match rescan_partition(&context, partition, &sizes).await { + Ok(backlog) => publish_partition_backlog(&context, partition, backlog, &mut sizes), + Err(EffectError::Stopped) => break, + Err(error) => { + tracing::warn!(%error, %partition, "rescanning sealed history segments failed"); + } + } + } + } + context.metrics.history_upload_backlog.clear(); +} + +/// Scans the partition's upload pending runs on a blocking thread. +async fn rescan_partition( + context: &EffectContext, + partition: PartitionId, + sizes: &SealedSegmentSizes, +) -> Result { + let reader = context.flow_effect_operations.committed_state_reader()?; + let known_sizes: SealedSegmentSizes = sizes + .iter() + .filter(|((size_partition, _, _), _)| *size_partition == partition) + .map(|(key, size)| (*key, *size)) + .collect(); + let scan = context + .config + .blocking_work + .run_tracked(&context.tracker, move || { + scan_partition_backlog(reader.as_ref(), partition, &known_sizes) + }); + match scan.await { + Ok(result) => result, + Err(BlockingWorkFailure::Panicked(payload)) => std::panic::resume_unwind(payload), + Err(BlockingWorkFailure::Cancelled) => Err(EffectError::Stopped), + } +} + +/// Publishes the partition's gauges, replaces its remembered sizes, and asks for the uploads of +/// its pending runs; the router ignores a request for a run whose upload already runs, except to +/// run it once more. +fn publish_partition_backlog( + context: &EffectContext, + partition: PartitionId, + backlog: PartitionUploadBacklog, + sizes: &mut SealedSegmentSizes, +) { + context.metrics.history_upload_backlog.set_partition( + usize::from(partition.0), + backlog.segments, + backlog.bytes, + backlog.oldest_sealed_at_micros, + ); + sizes.retain(|(size_partition, _, _), _| *size_partition != partition); + sizes.extend(backlog.sizes); + for (run, error) in backlog.unreadable { + context.metrics.history_upload_rescan_errors.inc(); + tracing::error!( + %error, + %partition, + run_id = run.map(|run| run.run_id.to_string()), + "the history upload rescan could not read a run; skipping it" + ); + } + for (flow, run) in backlog.runs { + allow_upload_retry(context, run.run_id); + context.report(RouterMessage::Effects { + flow, + effects: vec![ApplyEffect::HistorySegmentSealed { run_id: run.run_id }], + is_committed: true, + scheduled_at: None, + position: None, + }); + } +} + +fn scan_partition_backlog( + reader: &dyn FlowStateReader, + partition: PartitionId, + known_sizes: &SealedSegmentSizes, +) -> Result { + let mut backlog = PartitionUploadBacklog::default(); + let prefix = partition_index_prefix(partition, IndexKind::HistoryUploadPending); + for (key, _) in scan_all(reader, ColumnFamily::PartitionIndex, &prefix)? { + let run = match parse_run_item_index_key(partition, IndexKind::HistoryUploadPending, &key) { + Ok((run, _)) => run, + Err(error) => { + backlog.unreadable.push((None, error)); + continue; + } + }; + match scan_run_backlog(reader, partition, run, known_sizes) { + Ok(Some(run_backlog)) => { + backlog.segments += run_backlog.segments; + backlog.bytes = backlog.bytes.saturating_add(run_backlog.bytes); + backlog.oldest_sealed_at_micros = [ + backlog.oldest_sealed_at_micros, + run_backlog.oldest_sealed_at_micros, + ] + .into_iter() + .flatten() + .min(); + backlog.sizes.extend(run_backlog.sizes); + backlog.runs.push((run_backlog.flow, run)); + } + Ok(None) => {} + Err(error) => backlog.unreadable.push((Some(run), error)), + } + } + Ok(backlog) +} + +/// Returns what the run waits to upload, or `None` when the run is gone or purging. +fn scan_run_backlog( + reader: &dyn FlowStateReader, + partition: PartitionId, + run: RunLocation, + known_sizes: &SealedSegmentSizes, +) -> Result, EffectError> { + let Some(record) = read_run_record(reader, run)? else { + return Ok(None); + }; + if record.is_purging { + return Ok(None); + } + let mut run_backlog = RunUploadBacklog { + flow: FlowAddress { + tenant_id: TenantId(record.tenant_id), + flow_id: record.flow_id, + }, + segments: 0, + bytes: 0, + oldest_sealed_at_micros: None, + sizes: SealedSegmentSizes::new(), + }; + for segment in sealed_segments(reader, run, record.history_uploaded_through_seq)? { + let size_key = (partition, run, segment.first_seq); + let bytes = match known_sizes.get(&size_key) { + Some(bytes) => *bytes, + None => sealed_segment_bytes(reader, run, &segment)?, + }; + run_backlog.sizes.insert(size_key, bytes); + run_backlog.segments += 1; + run_backlog.bytes = run_backlog.bytes.saturating_add(bytes); + run_backlog.oldest_sealed_at_micros = Some( + run_backlog + .oldest_sealed_at_micros + .map_or(segment.sealed_at_micros, |oldest| { + oldest.min(segment.sealed_at_micros) + }), + ); + } + Ok(Some(run_backlog)) +} + +/// Returns the run's SEALED segments: every segment after the uploaded seq. +fn sealed_segments( + reader: &dyn FlowStateReader, + run: RunLocation, + uploaded_through_seq: u64, +) -> Result, EffectError> { + let start = history_segment_key( + run.flow_hash, + run.run_id, + uploaded_through_seq.saturating_add(1), + ); + let end = prefix_successor(&run_table_prefix( + run.flow_hash, + run.run_id, + RunTable::HistorySegment, + )); + let mut segments = Vec::new(); + for (key, value) in reader.scan(ColumnFamily::Flow, &start, &end, usize::MAX)? { + let segment: HistorySegmentRecord = decode_row("HistorySegment", &key, &value)?; + if segment.state == HistorySegmentState::Sealed as i32 { + segments.push(segment); + } + } + Ok(segments) +} + +/// Returns the uncompressed bytes the segment uploads: its history rows, as their metadata +/// records them, and its superseded values. +fn sealed_segment_bytes( + reader: &dyn FlowStateReader, + run: RunLocation, + segment: &HistorySegmentRecord, +) -> Result { + let metadata_prefix = run_table_prefix(run.flow_hash, run.run_id, RunTable::HistoryRowMetadata); + let mut start = metadata_prefix.clone(); + start.extend_from_slice(&segment.first_seq.to_be_bytes()); + let mut end = metadata_prefix; + end.extend_from_slice(&segment.last_seq.saturating_add(1).to_be_bytes()); + let mut bytes = 0_u64; + for (key, value) in reader.scan(ColumnFamily::Flow, &start, &end, usize::MAX)? { + let metadata: HistoryRowMetadata = decode_row("HistoryRowMetadata", &key, &value)?; + bytes = bytes.saturating_add(metadata.row_size_bytes); + } + let value_prefix = history_segment_value_prefix(run.flow_hash, run.run_id, segment.first_seq); + for (key, value) in scan_all(reader, ColumnFamily::Flow, &value_prefix)? { + let listed: HistorySegmentValueRecord = decode_row("HistorySegmentValue", &key, &value)?; + bytes = bytes.saturating_add(listed.size_bytes); + } + Ok(bytes) +} diff --git a/server-zd/crates/dexzd-engine/src/effects/mod.rs b/server-zd/crates/dexzd-engine/src/effects/mod.rs index f297754eb8..05f1a62a33 100644 --- a/server-zd/crates/dexzd-engine/src/effects/mod.rs +++ b/server-zd/crates/dexzd-engine/src/effects/mod.rs @@ -7,7 +7,7 @@ // SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 //! Runs the work that committed commands imply: method dispatch, timers, cross-Flow deliveries, -//! hard-delete object deletion, and retention purges. +//! hard-delete object deletion, history segment uploads, and retention purges. //! //! [`EffectRunner::handle`] only enqueues. One router task owns the scheduling state and //! spawns tracked tasks for the work. Everything here can be rebuilt from committed state with @@ -16,6 +16,7 @@ mod dispatch; mod flow_effect_operations; mod hard_delete_objects; +mod history_uploads; mod method_request; mod object_deletions; mod outbox_delivery; @@ -28,14 +29,14 @@ mod timer_service; #[cfg(test)] mod tests; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::sync::Arc; use std::time::Duration; use dexzd_apply::{ApplyEffect, StateReadError}; use dexzd_clock::WallClock; use dexzd_keyspace::{PartitionHasher, PartitionId, RunId}; -use dexzd_object_store::ObjectStoreError; +use dexzd_object_store::{ObjectStore, ObjectStoreError}; use dexzd_protocol::dexzd::v1::{ CommandReply, FlowCommand, LogPosition, PendingHardDeleteUpload, RunSettings, }; @@ -129,6 +130,13 @@ pub struct EffectsConfig { /// Default 100000. Deleted entries one periodic Heads seek may skip before it gives up on /// a queue until the next round. pub heads_max_skipped_deletions: u64, + /// Default 16, `history.uploadConcurrency`. Most sealed history segments this node reads, + /// encodes, and uploads at once. Values below 1 count as 1. + pub history_upload_concurrency: usize, + /// Default 60 seconds; zero disables. How often the uploader rescans every owned partition's + /// sealed history segments: the scan refreshes the upload backlog gauges and restarts the + /// upload of any run whose upload work stopped. + pub history_upload_rescan_interval: Duration, } impl Default for EffectsConfig { @@ -163,6 +171,8 @@ impl Default for EffectsConfig { blocking_work: BlockingWorkPlacement::BlockingThreadPool, heads_resync_interval: Duration::from_secs(5 * 60), heads_max_skipped_deletions: 100_000, + history_upload_concurrency: 16, + history_upload_rescan_interval: Duration::from_secs(60), } } } @@ -278,6 +288,7 @@ pub(crate) enum TrackedWork { HardDeleteObjects { run_id: RunId }, RunPurge { run_id: RunId }, PendingCloseCheck { run_id: RunId }, + HistoryUpload { run_id: RunId }, } /// Dependencies of the effect runner, passed once to [`EffectRunner::new`]. @@ -288,6 +299,9 @@ pub struct EffectDependencies { pub dispatcher: Arc, /// Hard-delete object operations. pub hard_delete_objects: HardDeleteObjects, + /// The primary object store, which holds the commit log; sealed history segments upload to + /// its `history/` prefix. + pub history_object_store: Arc, /// The cluster's partition hasher. pub partition_hasher: PartitionHasher, /// Owner wall clock; timers never fire before it reaches their fire time. @@ -309,6 +323,12 @@ pub(crate) struct EffectContext { pub(crate) flow_effect_operations: Arc, pub(crate) dispatcher: Arc, pub(crate) hard_delete_objects: HardDeleteObjects, + pub(crate) history_object_store: Arc, + /// Permits for history segment uploads; see [`EffectsConfig::history_upload_concurrency`]. + pub(crate) history_upload_slots: Arc, + /// Runs whose upload failed without a retry; only the rescan clears an entry, so every + /// trigger retries such a run at most once per rescan. + pub(crate) history_upload_failed_runs: std::sync::Mutex>, pub(crate) hasher: PartitionHasher, pub(crate) clock: Arc, pub(crate) validation_config: ValidationConfig, @@ -514,8 +534,8 @@ impl EffectRunner { } /// Rebuilds pending work of every run of `partition` from committed state: pending and - /// running invocations, timers, outbox messages, hard-delete deletions, retention purges, - /// and hard-delete uploads that never committed. + /// running invocations, timers, outbox messages, hard-delete deletions, history segment + /// uploads, retention purges, and hard-delete uploads that never committed. pub async fn rebuild_partition(&self, partition: PartitionId) -> Result<(), EffectError> { self.rebuild_partition_effect_count(partition) .await @@ -609,6 +629,11 @@ impl EffectTasks { .max_blocking_state_reads .clamp(1, Semaphore::MAX_PERMITS), )); + let history_upload_slots = Arc::new(Semaphore::new( + self.config + .history_upload_concurrency + .clamp(1, Semaphore::MAX_PERMITS), + )); let task_queue_memory = task_queue::TaskQueueMemory::new( self.dependencies.metrics.step_task_queue_entries.clone(), ); @@ -619,6 +644,9 @@ impl EffectTasks { flow_effect_operations, dispatcher: self.dependencies.dispatcher, hard_delete_objects: self.dependencies.hard_delete_objects, + history_object_store: self.dependencies.history_object_store, + history_upload_slots, + history_upload_failed_runs: std::sync::Mutex::new(BTreeSet::new()), hasher: self.dependencies.partition_hasher, clock: self.dependencies.clock, validation_config: ValidationConfig { @@ -640,6 +668,9 @@ impl EffectTasks { }); tracker.spawn(task_queue::run_claim_pump(Arc::clone(&context))); tracker.spawn(task_queue::run_heads_resync(Arc::clone(&context))); + tracker.spawn(history_uploads::run_history_upload_rescan(Arc::clone( + &context, + ))); #[cfg(test)] let test_context = Arc::clone(&context); let router = EffectRouter { @@ -833,8 +864,6 @@ impl EffectRouter { } // A speculative change may roll back; only its committed copy counts. ApplyEffect::DispatchQueueChanged { .. } => {} - // No uploader runs yet; the pending index row keeps the segment findable. - ApplyEffect::HistorySegmentSealed { .. } => {} effect if !is_committed => { tracing::debug!(?effect, "ignoring a speculative external effect"); } @@ -886,6 +915,9 @@ impl EffectRouter { ApplyEffect::PendingCloseHeldByRpcs { run_id } => { self.spawn_tracked(flow, TrackedWork::PendingCloseCheck { run_id }); } + ApplyEffect::HistorySegmentSealed { run_id } => { + self.spawn_tracked(flow, TrackedWork::HistoryUpload { run_id }); + } } } @@ -916,6 +948,9 @@ impl EffectRouter { pending_close_check::finalize_pending_close_after_rpcs(&context, &flow, *run_id) .await } + TrackedWork::HistoryUpload { run_id } => { + history_uploads::upload_sealed_history_segments(&context, &flow, *run_id).await + } }; if let Err(error) = result { tracing::warn!(%error, ?work, "effect work stopped with an error"); diff --git a/server-zd/crates/dexzd-engine/src/effects/object_deletions.rs b/server-zd/crates/dexzd-engine/src/effects/object_deletions.rs index 2a866bb005..9c5f7b2ddc 100644 --- a/server-zd/crates/dexzd-engine/src/effects/object_deletions.rs +++ b/server-zd/crates/dexzd-engine/src/effects/object_deletions.rs @@ -7,7 +7,8 @@ // SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 //! Object storage deletions that precede a command: hard-delete objects replaced by a committed -//! write, and a run's whole hard-delete prefix before its retention purge. +//! write, and a run's history segment objects and whole hard-delete prefix before its retention +//! purge. use dexzd_keyspace::row_keys::HARD_DELETE_PENDING_OBJECT; use dexzd_keyspace::{ColumnFamily, RunId, RunTable, run_table_prefix}; @@ -17,6 +18,7 @@ use dexzd_protocol::dexzd::v1::{ }; use crate::effects::hard_delete_objects::HardDeleteObjectOwner; +use crate::effects::history_uploads::delete_run_history_objects; use crate::effects::state_reads::{decode_row, read_run_record, scan_all}; use crate::effects::{EffectContext, EffectError}; use crate::flow_coordination::{CommitUrgency, FlowAddress}; @@ -89,7 +91,8 @@ pub(crate) async fn delete_pending_hard_delete_objects( } } -/// Deletes the run's hard-delete object prefix, then proposes `RunPurge`. +/// Deletes the run's history segment objects and its hard-delete object prefix, then proposes +/// `RunPurge`. /// /// Apply requests it only after deciding a purge that never postpones, so deletion is safe. pub(crate) async fn purge_run( @@ -107,6 +110,7 @@ pub(crate) async fn purge_run( if !is_purging { return Ok(()); } + delete_run_history_objects(context, flow, run).await?; let owner = HardDeleteObjectOwner { tenant_id: flow.tenant_id, flow_hash: run.flow_hash, diff --git a/server-zd/crates/dexzd-engine/src/effects/partition_rebuild.rs b/server-zd/crates/dexzd-engine/src/effects/partition_rebuild.rs index ed3242ac88..3701105395 100644 --- a/server-zd/crates/dexzd-engine/src/effects/partition_rebuild.rs +++ b/server-zd/crates/dexzd-engine/src/effects/partition_rebuild.rs @@ -58,6 +58,7 @@ pub(crate) async fn rebuild_partition( rebuild_pending_close_checks(reader.as_ref(), partition, &mut work)?; rebuild_outbox_messages(reader.as_ref(), partition, &mut work)?; rebuild_hard_delete_deletions(reader.as_ref(), partition, &mut work)?; + rebuild_history_uploads(reader.as_ref(), partition, &mut work)?; rebuild_run_purges(reader.as_ref(), partition, &mut work)?; let effect_count: usize = work .values() @@ -238,6 +239,24 @@ fn rebuild_hard_delete_deletions( Ok(()) } +/// Runs with SEALED history segments: a previous owner may have stopped before acknowledging +/// them, and a takeover or snapshot restore finds them only here. +fn rebuild_history_uploads( + reader: &dyn FlowStateReader, + partition: PartitionId, + work: &mut BTreeMap)>, +) -> Result<(), EffectError> { + let prefix = partition_index_prefix(partition, IndexKind::HistoryUploadPending); + for (key, _) in scan_all(reader, ColumnFamily::PartitionIndex, &prefix)? { + let (run, _) = parse_run_item_index_key(partition, IndexKind::HistoryUploadPending, &key)?; + if let Some((flow, _)) = flow_of_run(reader, run)? { + let effect = ApplyEffect::HistorySegmentSealed { run_id: run.run_id }; + add_effects(work, run, flow, vec![effect]); + } + } + Ok(()) +} + /// Runs whose retention expired and whose purge started before the restart. fn rebuild_run_purges( reader: &dyn FlowStateReader, diff --git a/server-zd/crates/dexzd-engine/src/effects/task_queue/mod.rs b/server-zd/crates/dexzd-engine/src/effects/task_queue/mod.rs index 41ab6ad689..3ef69de9ce 100644 --- a/server-zd/crates/dexzd-engine/src/effects/task_queue/mod.rs +++ b/server-zd/crates/dexzd-engine/src/effects/task_queue/mod.rs @@ -97,6 +97,11 @@ impl TaskQueueMemory { } } + /// Returns the partitions this node rebuilt and serves, in ascending order. + pub(crate) fn owned_partition_list(&self) -> Vec { + self.owned_partitions.iter().copied().collect() + } + /// Lowers the queue's smallest written key, so the next scan starts at or before it. pub(crate) fn note_written(&mut self, entry: &DispatchQueueEntry) { let slot = (entry.partition, entry.queue.clone()); diff --git a/server-zd/crates/dexzd-engine/src/effects/tests/history_upload_tests.rs b/server-zd/crates/dexzd-engine/src/effects/tests/history_upload_tests.rs new file mode 100644 index 0000000000..12e7f8ea2e --- /dev/null +++ b/server-zd/crates/dexzd-engine/src/effects/tests/history_upload_tests.rs @@ -0,0 +1,1245 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! History segment uploads with the real apply state machine: seq-ordered acknowledgments, +//! repeated uploads after a restart or restore, replaced objects, object storage outages, +//! retention, and purges. + +use std::sync::atomic::{AtomicU8, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use bytes::Bytes; +use dexzd_apply::FlowStateReader; +use dexzd_apply::read_model::read_history_segment_content; +use dexzd_history_segment::{ + HISTORY_SEGMENT_OBJECT_KEY_PREFIX, HistorySegmentContent, decode_history_segment, + encode_history_segment, history_run_object_key_prefix, history_segment_object_key, +}; +use dexzd_keyspace::row_keys::{ + history_segment_key, history_upload_pending_index_key, prefix_range, +}; +use dexzd_keyspace::{ColumnFamily, PartitionId, RunId, RunTable, TenantId, run_table_prefix}; +use dexzd_metrics::HistoryUploadOutcome; +use dexzd_object_store::{ + FaultAction, FaultInjector, InMemoryObjectStore, ObjectStore, ObjectStoreOperation, + WriteCondition, +}; +use dexzd_protocol::dex::{AttributeWrite, SetAttributesRequest, StopFlowRequest, StopType}; +use dexzd_protocol::dexzd::v1::{ + AttributesSetCommand, FlowCommand, FlowStopCommand, HistorySegmentRecord, HistorySegmentState, + RunSettings, SetHistoryOffloadModeCommand, command_reply, flow_command, +}; +use dexzd_storage::InMemoryLocalStore; +use prost::Message; +use tokio::sync::OwnedSemaphorePermit; + +use super::outbox_and_purge_tests::mark_purging; +use super::{ + DurableState, EffectHarness, RecordedCommand, SECOND_MICROS, is_error_reply, ok_reply, + settings, start_request, string_value, +}; +use crate::effects::EffectsConfig; +use crate::flow_coordination::CommitUrgency; + +/// Rounds of 100 ms that a wait allows; paused time makes them instant. +const WAIT_ROUNDS: usize = 3_000; +const SHORT_RETENTION: Duration = Duration::from_secs(60); +const RESCAN_INTERVAL: Duration = Duration::from_secs(1); + +/// What the store does with history segment writes and with batch deletes; every other call +/// proceeds. +#[derive(Default)] +struct HistoryWriteFaults { + mode: AtomicU8, + delete_mode: AtomicU8, +} + +impl HistoryWriteFaults { + const PROCEED: u8 = 0; + const FAIL: u8 = 1; + const APPLY_THEN_OUTCOME_UNKNOWN_ONCE: u8 = 2; + const DELAY: u8 = 3; + const FAIL_PERMANENTLY: u8 = 4; + /// Longer than every wait of the tests, as a write stuck in a slow network. + const DELAY_FOR_TEN_MINUTES: u8 = 5; + + fn set(&self, mode: u8) { + self.mode.store(mode, Ordering::SeqCst); + } + + fn set_deletes(&self, mode: u8) { + self.delete_mode.store(mode, Ordering::SeqCst); + } +} + +impl FaultInjector for HistoryWriteFaults { + fn action(&self, operation: ObjectStoreOperation, key: &str) -> FaultAction { + let mode = match operation { + ObjectStoreOperation::Put if key.starts_with(HISTORY_SEGMENT_OBJECT_KEY_PREFIX) => { + self.mode.load(Ordering::SeqCst) + } + ObjectStoreOperation::Delete => self.delete_mode.load(Ordering::SeqCst), + _ => Self::PROCEED, + }; + match mode { + Self::FAIL => FaultAction::FailBeforeEffect, + Self::APPLY_THEN_OUTCOME_UNKNOWN_ONCE => { + self.set(Self::PROCEED); + FaultAction::ApplyThenOutcomeUnknown + } + Self::DELAY => FaultAction::Delay(Duration::from_secs(10)), + Self::FAIL_PERMANENTLY => FaultAction::FailPermanently, + Self::DELAY_FOR_TEN_MINUTES => FaultAction::Delay(Duration::from_secs(600)), + _ => FaultAction::Proceed, + } + } +} + +fn durable_with_faults(faults: Arc) -> DurableState { + DurableState { + objects: Arc::new(InMemoryObjectStore::with_injector(faults)), + ..DurableState::new() + } +} + +fn rescan_config() -> EffectsConfig { + EffectsConfig { + history_upload_rescan_interval: RESCAN_INTERVAL, + ..EffectsConfig::default() + } +} + +/// Commits the enabled offload mode to every partition, as a node with `history.offloadEnabled` +/// does. +fn enable_history_offload(harness: &EffectHarness) { + set_history_offload_mode(harness, true); +} + +/// Commits the offload mode to every partition. +fn set_history_offload_mode(harness: &EffectHarness, is_enabled: bool) { + for (partition, flow_id) in harness + .hasher + .partition_command_flow_ids() + .into_iter() + .enumerate() + { + let partition = u32::try_from(partition).expect("few partitions"); + let reply = harness + .flow_effect_operations + .commit( + &EffectHarness::flow(&flow_id), + FlowCommand { + flow_id: flow_id.clone(), + command: Some(flow_command::Command::SetHistoryOffloadMode( + SetHistoryOffloadModeCommand { + enabled: is_enabled, + partition, + }, + )), + ..FlowCommand::default() + }, + CommitUrgency::Prompt, + ) + .expect("the mode commits"); + assert!( + !is_error_reply(&reply), + "setting the mode failed: {reply:?}" + ); + } +} + +/// Takes every upload slot, so uploads wait until the permits drop. +async fn hold_upload_slots(harness: &EffectHarness) -> OwnedSemaphorePermit { + let slots = u32::try_from(EffectsConfig::default().history_upload_concurrency) + .expect("the default concurrency fits"); + Arc::clone(&harness.effect_context().history_upload_slots) + .acquire_many_owned(slots) + .await + .expect("the upload slots stay open") +} + +/// Terminates the run; the closing command seals its open segment. +fn close_run(harness: &EffectHarness, flow_id: &str, run_id: RunId) { + let command = FlowCommand { + flow_id: flow_id.to_owned(), + run_id: run_id.as_bytes().to_vec(), + command: Some(flow_command::Command::FlowStop(FlowStopCommand { + request: Some(StopFlowRequest { + flow_id: flow_id.to_owned(), + stop_type: StopType::Terminate as i32, + ..StopFlowRequest::default() + }), + })), + ..FlowCommand::default() + }; + let reply = harness + .flow_effect_operations + .commit( + &EffectHarness::flow(flow_id), + command, + CommitUrgency::Prompt, + ) + .expect("stop commits"); + assert!(!is_error_reply(&reply), "stop failed: {reply:?}"); +} + +fn short_retention_settings() -> RunSettings { + RunSettings { + retention_after_close_micros: 60 * SECOND_MICROS, + ..settings() + } +} + +/// Returns the run's committed history segments in seq order. +fn segments(harness: &EffectHarness, flow_id: &str, run_id: RunId) -> Vec { + let run = harness.run_location(flow_id, run_id); + let prefix = run_table_prefix(run.flow_hash, run.run_id, RunTable::HistorySegment); + let (start, end) = prefix_range(&prefix); + harness + .flow_effect_operations + .snapshot() + .scan(ColumnFamily::Flow, &start, &end, usize::MAX) + .expect("memory reads never fail") + .into_iter() + .map(|(_, bytes)| HistorySegmentRecord::decode(bytes.as_slice()).expect("decodes")) + .collect() +} + +fn is_upload_pending(harness: &EffectHarness, flow_id: &str, run_id: RunId) -> bool { + let run = harness.run_location(flow_id, run_id); + let partition: PartitionId = harness.partition_of(flow_id); + harness + .flow_effect_operations + .snapshot() + .get( + ColumnFamily::PartitionIndex, + &history_upload_pending_index_key(partition, run.flow_hash, run.run_id), + ) + .expect("memory reads never fail") + .is_some() +} + +fn is_fully_uploaded(harness: &EffectHarness, flow_id: &str, run_id: RunId) -> bool { + let segments = segments(harness, flow_id, run_id); + !segments.is_empty() + && segments + .iter() + .all(|segment| segment.state == HistorySegmentState::Uploaded as i32) + && !is_upload_pending(harness, flow_id, run_id) +} + +fn object_key( + harness: &EffectHarness, + flow_id: &str, + run_id: RunId, + segment: &HistorySegmentRecord, +) -> String { + let run = harness.run_location(flow_id, run_id); + history_segment_object_key( + TenantId::DEFAULT, + run.flow_hash, + run_id, + segment.first_seq, + segment.last_seq, + ) +} + +/// Returns the content that committed state assigns to the segment. +fn committed_content( + harness: &EffectHarness, + flow_id: &str, + run_id: RunId, + first_seq: u64, +) -> HistorySegmentContent { + let run = harness.run_location(flow_id, run_id); + read_history_segment_content( + &harness.flow_effect_operations.snapshot(), + run.flow_hash, + run_id, + first_seq, + ) + .expect("the segment content reads") + .expect("the segment exists") +} + +async fn stored_object(harness: &EffectHarness, key: &str) -> Option { + harness + .durable + .objects + .get(key) + .await + .expect("the in-memory store reads") + .map(|content| content.bytes) +} + +async fn object_keys_with_prefix(harness: &EffectHarness, prefix: &str) -> Vec { + harness + .durable + .objects + .list(prefix, None, 1000) + .await + .expect("the in-memory store lists") + .objects + .into_iter() + .map(|object| object.key) + .collect() +} + +fn acknowledgments(harness: &EffectHarness) -> Vec { + harness.flow_effect_operations.commands_matching(|command| { + matches!(command, flow_command::Command::HistorySegmentUploaded(_)) + }) +} + +fn acknowledged_first_seq(recorded: &RecordedCommand) -> u64 { + match &recorded.command.command { + Some(flow_command::Command::HistorySegmentUploaded(acknowledgment)) => { + acknowledgment.first_seq + } + other => panic!("not an acknowledgment: {other:?}"), + } +} + +fn run_purges(harness: &EffectHarness) -> Vec { + harness + .flow_effect_operations + .commands_matching(|command| matches!(command, flow_command::Command::RunPurge(_))) +} + +/// Waits until `condition` holds, letting tasks and paused time advance. +async fn wait_until(harness: &EffectHarness, what: &str, condition: impl Fn() -> bool) { + for _ in 0..WAIT_ROUNDS { + harness.settle().await; + if condition() { + return; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + panic!("timed out waiting until {what}"); +} + +async fn wait_until_uploaded(harness: &EffectHarness, flow_id: &str, run_id: RunId) { + wait_until(harness, "every sealed segment is acknowledged", || { + is_fully_uploaded(harness, flow_id, run_id) + }) + .await; +} + +/// Checks the run's uploaded chain: contiguous segments, U at the last one, and one object per +/// segment whose content and checksum match committed state. +async fn assert_uploaded_chain(harness: &EffectHarness, flow_id: &str, run_id: RunId) { + let segments = segments(harness, flow_id, run_id); + for pair in segments.windows(2) { + assert_eq!(pair[1].first_seq, pair[0].last_seq + 1, "{segments:?}"); + } + let last = segments.last().expect("the run sealed a segment"); + assert_eq!( + harness + .run_record(flow_id, run_id) + .history_uploaded_through_seq, + last.last_seq + ); + let run = harness.run_location(flow_id, run_id); + let run_prefix = history_run_object_key_prefix(TenantId::DEFAULT, run.flow_hash, run_id); + let mut expected_keys = Vec::new(); + for segment in &segments { + let key = object_key(harness, flow_id, run_id, segment); + let object = stored_object(harness, &key) + .await + .unwrap_or_else(|| panic!("{key} exists")); + let content = committed_content(harness, flow_id, run_id, segment.first_seq); + assert_eq!( + decode_history_segment(&object).expect("the object decodes"), + content + ); + let encoded = encode_history_segment(&content).expect("committed content encodes"); + assert_eq!(segment.content_checksum, encoded.content_checksum.to_vec()); + assert_eq!(segment.object_size_bytes, object.len() as u64); + expected_keys.push(key); + } + assert_eq!( + object_keys_with_prefix(harness, &run_prefix).await, + expected_keys, + "one object per segment, without gaps or extra objects" + ); +} + +#[tokio::test(start_paused = true)] +async fn sealed_segments_upload_and_are_acknowledged_in_seq_order() { + let harness = EffectHarness::new(); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_id = harness.start_flow( + start_request("order-1"), + RunSettings { + history_segment_target_bytes: 1, + ..settings() + }, + ); + close_run(&harness, "order-1", run_id); + + wait_until_uploaded(&harness, "order-1", run_id).await; + assert_uploaded_chain(&harness, "order-1", run_id).await; + let segments = segments(&harness, "order-1", run_id); + assert!(segments.len() >= 2, "every command sealed: {segments:?}"); + let acknowledgments = acknowledgments(&harness); + assert_eq!( + acknowledgments + .iter() + .map(acknowledged_first_seq) + .collect::>(), + segments + .iter() + .map(|segment| segment.first_seq) + .collect::>(), + "one acknowledgment per segment, oldest first" + ); + for acknowledgment in &acknowledgments { + assert_eq!(acknowledgment.reply, ok_reply()); + assert_eq!(acknowledgment.urgency, CommitUrgency::Batched); + assert_eq!(acknowledgment.command.expected_run_attempt, 0); + } + let uploads = &harness.metrics.history_uploads; + assert_eq!( + uploads.count(HistoryUploadOutcome::Created), + segments.len() as u64 + ); + assert_eq!(uploads.count(HistoryUploadOutcome::Failed), 0); + assert_eq!( + harness.metrics.history_upload_bytes.get(), + segments + .iter() + .map(|segment| segment.object_size_bytes) + .sum::() + ); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn a_restarted_owner_reuploads_an_unacknowledged_segment_and_acknowledges_it() { + let durable = DurableState::new(); + let first = EffectHarness::with_config(durable.clone(), EffectsConfig::default()); + first.rebuild_all_partitions().await; + enable_history_offload(&first); + let run_id = first.start_flow(start_request("order-1"), settings()); + close_run(&first, "order-1", run_id); + // No later proposal commits, so the owner stops between the upload and its acknowledgment. + first + .flow_effect_operations + .unavailable_proposals + .store(usize::MAX, Ordering::SeqCst); + let segment = segments(&first, "order-1", run_id) + .pop() + .expect("the close sealed a segment"); + let key = object_key(&first, "order-1", run_id, &segment); + wait_until(&first, "the segment object exists", || { + durable.objects.call_count(ObjectStoreOperation::Put, &key) == 1 + }) + .await; + assert!(stored_object(&first, &key).await.is_some()); + assert!(is_upload_pending(&first, "order-1", run_id)); + assert_eq!( + first + .metrics + .history_uploads + .count(HistoryUploadOutcome::Created), + 0, + "an upload counts once its acknowledgment commits" + ); + first.shutdown().await; + + let second = EffectHarness::with_config(durable.clone(), EffectsConfig::default()); + second.rebuild_all_partitions().await; + wait_until_uploaded(&second, "order-1", run_id).await; + + assert_uploaded_chain(&second, "order-1", run_id).await; + let uploads = &second.metrics.history_uploads; + assert_eq!(uploads.count(HistoryUploadOutcome::AlreadyPresent), 1); + assert_eq!(uploads.count(HistoryUploadOutcome::Created), 0); + assert_eq!(second.metrics.history_upload_checksum_mismatches.get(), 0); + assert_eq!( + durable.objects.call_count(ObjectStoreOperation::Put, &key), + 2, + "the new owner's CreateOnly write found the object" + ); + let acknowledgments = acknowledgments(&second); + assert_eq!(acknowledgments.len(), 1); + assert_eq!(acknowledgments[0].reply, ok_reply()); + second.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn an_unknown_write_outcome_is_retried_and_finds_the_written_object() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::APPLY_THEN_OUTCOME_UNKNOWN_ONCE); + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig::default(), + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_id = harness.start_flow(start_request("order-1"), settings()); + close_run(&harness, "order-1", run_id); + + wait_until_uploaded(&harness, "order-1", run_id).await; + assert_uploaded_chain(&harness, "order-1", run_id).await; + let uploads = &harness.metrics.history_uploads; + assert_eq!(uploads.count(HistoryUploadOutcome::Failed), 1); + assert_eq!(uploads.count(HistoryUploadOutcome::AlreadyPresent), 1); + assert_eq!(acknowledgments(&harness).len(), 1); + harness.shutdown().await; +} + +/// Seeds the segment's key with `seeded_object` before the first upload, then checks that the +/// upload replaced it with committed content and acknowledged it. +async fn assert_seeded_object_is_replaced( + seeded_object: impl FnOnce(&HistorySegmentContent) -> Vec, +) { + let harness = EffectHarness::new(); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let upload_slots = hold_upload_slots(&harness).await; + let run_id = harness.start_flow(start_request("order-1"), settings()); + close_run(&harness, "order-1", run_id); + let segment = segments(&harness, "order-1", run_id) + .pop() + .expect("the close sealed a segment"); + let content = committed_content(&harness, "order-1", run_id, segment.first_seq); + let key = object_key(&harness, "order-1", run_id, &segment); + harness + .durable + .objects + .put( + &key, + Bytes::from(seeded_object(&content)), + WriteCondition::Unconditional, + ) + .await + .expect("the seeded object is written"); + drop(upload_slots); + + wait_until_uploaded(&harness, "order-1", run_id).await; + assert_uploaded_chain(&harness, "order-1", run_id).await; + let uploads = &harness.metrics.history_uploads; + assert_eq!(uploads.count(HistoryUploadOutcome::Replaced), 1); + assert_eq!(uploads.count(HistoryUploadOutcome::Created), 0); + assert_eq!(harness.metrics.history_upload_checksum_mismatches.get(), 1); + assert_eq!(acknowledgments(&harness).len(), 1); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn an_existing_object_with_other_content_is_replaced_then_acknowledged() { + assert_seeded_object_is_replaced(|content| { + let mut stale = content.clone(); + let record = stale.records.first_mut().expect("the segment has records"); + let last_byte = record.row.last_mut().expect("a record row is not empty"); + *last_byte ^= 0xff; + encode_history_segment(&stale) + .expect("the stale content encodes") + .bytes + }) + .await; +} + +#[tokio::test(start_paused = true)] +async fn an_existing_object_that_does_not_decode_is_replaced_then_acknowledged() { + assert_seeded_object_is_replaced(|_| b"not a history segment".to_vec()).await; +} + +#[tokio::test(start_paused = true)] +async fn an_object_store_outage_grows_the_backlog_without_blocking_commands_then_drains() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::FAIL); + let harness = + EffectHarness::with_config(durable_with_faults(Arc::clone(&faults)), rescan_config()); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let flow_ids = ["order-1", "order-2", "order-3"]; + let mut run_ids = Vec::new(); + for flow_id in flow_ids { + let run_id = harness.start_flow(start_request(flow_id), settings()); + close_run(&harness, flow_id, run_id); + run_ids.push(run_id); + } + tokio::time::sleep(RESCAN_INTERVAL * 3).await; + harness.settle().await; + + let backlog = &harness.metrics.history_upload_backlog; + assert_eq!(backlog.segments(), 3); + assert!(backlog.bytes() > 0); + let oldest_sealed_at = flow_ids + .iter() + .zip(&run_ids) + .flat_map(|(flow_id, run_id)| segments(&harness, flow_id, *run_id)) + .map(|segment| segment.sealed_at_micros) + .min(); + assert_eq!(backlog.oldest_sealed_at_micros(), oldest_sealed_at); + assert!( + harness + .metrics + .history_uploads + .count(HistoryUploadOutcome::Failed) + >= 3 + ); + assert!( + object_keys_with_prefix(&harness, HISTORY_SEGMENT_OBJECT_KEY_PREFIX) + .await + .is_empty() + ); + assert!(acknowledgments(&harness).is_empty()); + + // Commands keep committing during the outage, and the backlog keeps growing. + let late_run = harness.start_flow(start_request("order-late"), settings()); + close_run(&harness, "order-late", late_run); + let bytes_before = backlog.bytes(); + tokio::time::sleep(RESCAN_INTERVAL * 3).await; + harness.settle().await; + assert_eq!(backlog.segments(), 4); + assert!(backlog.bytes() > bytes_before); + + faults.set(HistoryWriteFaults::PROCEED); + for (flow_id, run_id) in flow_ids.iter().zip(&run_ids) { + wait_until_uploaded(&harness, flow_id, *run_id).await; + assert_uploaded_chain(&harness, flow_id, *run_id).await; + } + wait_until_uploaded(&harness, "order-late", late_run).await; + wait_until(&harness, "the backlog gauges drain", || { + backlog.segments() == 0 + }) + .await; + assert_eq!(backlog.bytes(), 0); + assert_eq!(backlog.oldest_sealed_at_micros(), None); + assert_eq!(acknowledgments(&harness).len(), 4); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn retention_waits_for_uploads_and_the_purge_deletes_the_runs_history_objects() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::FAIL); + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig::default(), + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_id = harness.start_flow( + start_request("order-1"), + RunSettings { + history_segment_target_bytes: 1, + ..short_retention_settings() + }, + ); + let kept_run = harness.start_flow(start_request("order-2"), settings()); + close_run(&harness, "order-1", run_id); + close_run(&harness, "order-2", kept_run); + let first_deadline = harness + .run_record("order-1", run_id) + .retention_deadline_micros; + + tokio::time::sleep(SHORT_RETENTION + Duration::from_secs(1)).await; + harness.settle().await; + let postponed = harness.run_record("order-1", run_id); + assert!( + !postponed.is_purging, + "a run with sealed segments is not purged" + ); + assert!(postponed.retention_deadline_micros > first_deadline); + assert!(run_purges(&harness).is_empty()); + + faults.set(HistoryWriteFaults::PROCEED); + wait_until_uploaded(&harness, "order-1", run_id).await; + wait_until_uploaded(&harness, "order-2", kept_run).await; + let run = harness.run_location("order-1", run_id); + let run_prefix = history_run_object_key_prefix(TenantId::DEFAULT, run.flow_hash, run_id); + let kept = harness.run_location("order-2", kept_run); + let kept_prefix = history_run_object_key_prefix(TenantId::DEFAULT, kept.flow_hash, kept_run); + assert_eq!( + object_keys_with_prefix(&harness, &run_prefix).await.len(), + segments(&harness, "order-1", run_id).len() + ); + let kept_keys = object_keys_with_prefix(&harness, &kept_prefix).await; + assert_eq!(kept_keys.len(), 1); + + tokio::time::sleep(Duration::from_secs(3600)).await; + // The purge waits for in-flight history writes to settle before it deletes. + wait_until(&harness, "the purge commits", || { + !run_purges(&harness).is_empty() + }) + .await; + let purges = run_purges(&harness); + assert_eq!(purges.len(), 1); + assert_eq!(purges[0].reply, ok_reply()); + assert!( + object_keys_with_prefix(&harness, &run_prefix) + .await + .is_empty(), + "the purge deleted every history object of the run" + ); + assert_eq!( + object_keys_with_prefix(&harness, &kept_prefix).await, + kept_keys + ); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn uploads_of_a_purging_run_stop_without_writing_or_retrying() { + let harness = EffectHarness::new(); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let upload_slots = hold_upload_slots(&harness).await; + let run_id = harness.start_flow(start_request("order-1"), settings()); + close_run(&harness, "order-1", run_id); + mark_purging(&harness, "order-1", run_id); + drop(upload_slots); + + tokio::time::sleep(Duration::from_secs(600)).await; + harness.settle().await; + let run = harness.run_location("order-1", run_id); + let run_prefix = history_run_object_key_prefix(TenantId::DEFAULT, run.flow_hash, run_id); + assert_eq!( + harness + .durable + .objects + .call_count(ObjectStoreOperation::Put, &run_prefix), + 0 + ); + assert!(acknowledgments(&harness).is_empty()); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn an_acknowledgment_that_finds_the_run_purging_ends_its_uploads() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::DELAY); + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig::default(), + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_id = harness.start_flow(start_request("order-1"), settings()); + close_run(&harness, "order-1", run_id); + let segment = segments(&harness, "order-1", run_id) + .pop() + .expect("the close sealed a segment"); + let key = object_key(&harness, "order-1", run_id, &segment); + wait_until(&harness, "the upload is in flight", || { + harness + .durable + .objects + .call_count(ObjectStoreOperation::Put, &key) + == 1 + }) + .await; + mark_purging(&harness, "order-1", run_id); + + wait_until(&harness, "the acknowledgment is rejected", || { + acknowledgments(&harness).len() == 1 + }) + .await; + let acknowledgment = &acknowledgments(&harness)[0]; + assert!( + matches!( + &acknowledgment.reply.reply, + Some(command_reply::Reply::Error(error)) if error.grpc_code == 5 + ), + "{:?}", + acknowledgment.reply + ); + tokio::time::sleep(Duration::from_secs(600)).await; + harness.settle().await; + assert_eq!(acknowledgments(&harness).len(), 1, "no retry"); + assert_eq!( + harness + .durable + .objects + .call_count(ObjectStoreOperation::Put, &key), + 1 + ); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn a_node_restored_from_a_snapshot_before_the_acknowledgments_finishes_the_uploads() { + let durable = DurableState::new(); + let first = EffectHarness::with_config(durable.clone(), EffectsConfig::default()); + first.rebuild_all_partitions().await; + enable_history_offload(&first); + let upload_slots = hold_upload_slots(&first).await; + let flow_ids = ["order-1", "order-2"]; + let run_ids: Vec = flow_ids + .iter() + .map(|flow_id| { + let run_id = first.start_flow(start_request(flow_id), settings()); + close_run(&first, flow_id, run_id); + run_id + }) + .collect(); + let snapshot = first.flow_effect_operations.committed_snapshot(); + drop(upload_slots); + for (flow_id, run_id) in flow_ids.iter().zip(&run_ids) { + wait_until_uploaded(&first, flow_id, *run_id).await; + } + first.shutdown().await; + + // The restored state predates the acknowledgments, but the objects exist. + let restored = DurableState { + state: Arc::new(Mutex::new(snapshot)), + objects: Arc::clone(&durable.objects), + local_store: Arc::new(InMemoryLocalStore::new()), + clock: Arc::clone(&durable.clock), + }; + let second = EffectHarness::with_config(restored, EffectsConfig::default()); + for (flow_id, run_id) in flow_ids.iter().zip(&run_ids) { + assert!(is_upload_pending(&second, flow_id, *run_id)); + } + second.rebuild_all_partitions().await; + for (flow_id, run_id) in flow_ids.iter().zip(&run_ids) { + wait_until_uploaded(&second, flow_id, *run_id).await; + assert_uploaded_chain(&second, flow_id, *run_id).await; + } + assert_eq!( + second + .metrics + .history_uploads + .count(HistoryUploadOutcome::AlreadyPresent), + 2 + ); + second.shutdown().await; +} + +/// Writes the Attribute `key` with a value long enough to live in the value table. +fn set_long_attribute( + harness: &EffectHarness, + flow_id: &str, + run_id: RunId, + key: &str, + text: &str, +) { + let command = FlowCommand { + flow_id: flow_id.to_owned(), + run_id: run_id.as_bytes().to_vec(), + command: Some(flow_command::Command::AttributesSet(AttributesSetCommand { + request: Some(SetAttributesRequest { + flow_id: flow_id.to_owned(), + attributes: vec![AttributeWrite { + key: key.to_owned(), + value: Some(string_value(&format!("{text} {}", "x".repeat(200)))), + ..AttributeWrite::default() + }], + request_id: uuid::Uuid::now_v7().to_string(), + ..SetAttributesRequest::default() + }), + hard_delete_objects: Vec::new(), + request_fingerprint: vec![1], + })), + ..FlowCommand::default() + }; + let reply = harness + .flow_effect_operations + .commit( + &EffectHarness::flow(flow_id), + command, + CommitUrgency::Prompt, + ) + .expect("the write commits"); + assert!(!is_error_reply(&reply), "the write failed: {reply:?}"); +} + +#[tokio::test(start_paused = true)] +async fn a_stopping_owner_abandons_its_pending_write_before_the_new_owner_purges_the_run() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::DELAY_FOR_TEN_MINUTES); + let durable = durable_with_faults(Arc::clone(&faults)); + let first = EffectHarness::with_config(durable.clone(), EffectsConfig::default()); + first.rebuild_all_partitions().await; + enable_history_offload(&first); + let run_id = first.start_flow(start_request("order-1"), short_retention_settings()); + close_run(&first, "order-1", run_id); + let segment = segments(&first, "order-1", run_id) + .pop() + .expect("the close sealed a segment"); + let key = object_key(&first, "order-1", run_id, &segment); + wait_until(&first, "the old owner's write is in flight", || { + durable.objects.call_count(ObjectStoreOperation::Put, &key) == 1 + }) + .await; + // The old owner stops, as at a takeover, before the store accepted its delayed write. + first.effect_context().shutdown.cancel(); + faults.set(HistoryWriteFaults::PROCEED); + + let second = EffectHarness::with_config(durable.clone(), EffectsConfig::default()); + second.rebuild_all_partitions().await; + wait_until_uploaded(&second, "order-1", run_id).await; + // An object outside the catalog, as an older timeline wrote it, under the run's prefix. + let run = second.run_location("order-1", run_id); + let stale_key = history_segment_object_key( + TenantId::DEFAULT, + run.flow_hash, + run_id, + segment.first_seq, + segment.last_seq + 7, + ); + second + .durable + .objects + .put( + &stale_key, + Bytes::from_static(b"stale"), + WriteCondition::Unconditional, + ) + .await + .expect("the stale object is written"); + tokio::time::sleep(SHORT_RETENTION + Duration::from_secs(1)).await; + wait_until(&second, "the purge commits", || { + !run_purges(&second).is_empty() + }) + .await; + let run_prefix = history_run_object_key_prefix(TenantId::DEFAULT, run.flow_hash, run_id); + assert!( + object_keys_with_prefix(&second, &run_prefix) + .await + .is_empty() + ); + + // Past the delay, the dropped call wrote nothing; a write the store accepted is not covered. + tokio::time::sleep(Duration::from_secs(15 * 60)).await; + second.settle().await; + assert!( + object_keys_with_prefix(&second, &run_prefix) + .await + .is_empty(), + "no write of the old owner landed after the purge" + ); + assert_eq!( + durable.objects.call_count(ObjectStoreOperation::Put, &key), + 2 + ); + first.shutdown().await; + second.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn uploads_hold_no_slot_or_object_bytes_while_their_acknowledgments_wait() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::DELAY); + let concurrency = 2; + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig { + history_upload_concurrency: concurrency, + ..EffectsConfig::default() + }, + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_count = 20; + for index in 0..run_count { + let flow_id = format!("order-{index}"); + let run_id = harness.start_flow(start_request(&flow_id), settings()); + close_run(&harness, &flow_id, run_id); + } + // No acknowledgment commits, so every run's upload waits on it. + harness + .flow_effect_operations + .unavailable_proposals + .store(usize::MAX, Ordering::SeqCst); + let started = tokio::time::Instant::now(); + wait_until(&harness, "every upload started", || { + harness + .durable + .objects + .call_count(ObjectStoreOperation::Put, HISTORY_SEGMENT_OBJECT_KEY_PREFIX) + == run_count + }) + .await; + // Each write takes 10 seconds, so two at a time start the last after at least 90 seconds. + assert!( + started.elapsed() >= Duration::from_secs(90), + "{:?}", + started.elapsed() + ); + tokio::time::sleep(Duration::from_secs(11)).await; + harness.settle().await; + assert_eq!( + object_keys_with_prefix(&harness, HISTORY_SEGMENT_OBJECT_KEY_PREFIX) + .await + .len(), + usize::try_from(run_count).expect("few runs") + ); + assert!(acknowledgments(&harness).is_empty()); + // Encoded bytes live only with an upload slot, so free slots mean no bytes are held. + assert_eq!( + harness + .effect_context() + .history_upload_slots + .available_permits(), + concurrency + ); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn a_segment_sealed_before_offload_turns_off_keeps_its_content_through_pruning() { + let harness = EffectHarness::new(); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let upload_slots = hold_upload_slots(&harness).await; + let run_id = harness.start_flow( + start_request("order-1"), + RunSettings { + history_segment_target_bytes: 1, + history_window_events: 4, + history_prune_chunk_events: 1, + ..settings() + }, + ); + for round in 0..3 { + set_long_attribute(&harness, "order-1", run_id, "notes", &format!("v{round}")); + } + let sealed = segments(&harness, "order-1", run_id); + assert!(sealed.len() >= 4, "each command sealed: {sealed:?}"); + // A previous owner stored every segment completely, but its acknowledgments were lost. + let mut seeded = Vec::new(); + for segment in &sealed { + let key = object_key(&harness, "order-1", run_id, segment); + let content = committed_content(&harness, "order-1", run_id, segment.first_seq); + let object = Bytes::from( + encode_history_segment(&content) + .expect("committed content encodes") + .bytes, + ); + harness + .durable + .objects + .put(&key, object.clone(), WriteCondition::CreateOnly) + .await + .expect("the complete object is written"); + seeded.push((key, object)); + } + + set_history_offload_mode(&harness, false); + for round in 3..40 { + set_long_attribute(&harness, "order-1", run_id, "notes", &format!("v{round}")); + } + assert_eq!( + segments(&harness, "order-1", run_id).len(), + sealed.len(), + "nothing seals while offload is off" + ); + drop(upload_slots); + + wait_until_uploaded(&harness, "order-1", run_id).await; + let uploads = &harness.metrics.history_uploads; + assert_eq!( + uploads.count(HistoryUploadOutcome::AlreadyPresent), + sealed.len() as u64 + ); + assert_eq!(uploads.count(HistoryUploadOutcome::Replaced), 0); + assert_eq!(harness.metrics.history_upload_checksum_mismatches.get(), 0); + for (key, object) in seeded { + assert_eq!(stored_object(&harness, &key).await, Some(object), "{key}"); + } + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn a_permanent_upload_rejection_waits_for_the_rescan_and_acknowledges_nothing() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::FAIL_PERMANENTLY); + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig::default(), + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_id = harness.start_flow(start_request("order-1"), settings()); + close_run(&harness, "order-1", run_id); + let puts = || { + harness + .durable + .objects + .call_count(ObjectStoreOperation::Put, HISTORY_SEGMENT_OBJECT_KEY_PREFIX) + }; + wait_until(&harness, "the first upload is rejected", || puts() == 1).await; + + tokio::time::sleep(Duration::from_secs(30)).await; + harness.settle().await; + assert_eq!( + puts(), + 1, + "a permanent rejection is not retried with backoff" + ); + assert_eq!( + harness + .metrics + .history_uploads + .count(HistoryUploadOutcome::Failed), + 1 + ); + tokio::time::sleep(Duration::from_secs(40)).await; + harness.settle().await; + assert_eq!(puts(), 2, "the rescan rechecks the run"); + assert!(acknowledgments(&harness).is_empty()); + assert!(is_upload_pending(&harness, "order-1", run_id)); + + faults.set(HistoryWriteFaults::PROCEED); + wait_until_uploaded(&harness, "order-1", run_id).await; + assert_uploaded_chain(&harness, "order-1", run_id).await; + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn a_permanent_deletion_rejection_holds_the_purge_until_deletion_succeeds() { + let faults = Arc::new(HistoryWriteFaults::default()); + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig::default(), + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_id = harness.start_flow(start_request("order-1"), short_retention_settings()); + close_run(&harness, "order-1", run_id); + wait_until_uploaded(&harness, "order-1", run_id).await; + faults.set_deletes(HistoryWriteFaults::FAIL_PERMANENTLY); + + tokio::time::sleep(SHORT_RETENTION + Duration::from_secs(200)).await; + harness.settle().await; + assert!(run_purges(&harness).is_empty(), "the purge waits"); + assert!(harness.run_record("order-1", run_id).is_purging); + let run = harness.run_location("order-1", run_id); + let run_prefix = history_run_object_key_prefix(TenantId::DEFAULT, run.flow_hash, run_id); + assert_eq!( + object_keys_with_prefix(&harness, &run_prefix).await.len(), + 1 + ); + let deletions = harness + .durable + .objects + .call_count(ObjectStoreOperation::Delete, ""); + assert!( + (1..=4).contains(&deletions), + "a permanent rejection is rechecked each minute: {deletions} deletions" + ); + + faults.set_deletes(HistoryWriteFaults::PROCEED); + wait_until(&harness, "the purge commits", || { + !run_purges(&harness).is_empty() + }) + .await; + assert_eq!(run_purges(&harness)[0].reply, ok_reply()); + assert!( + object_keys_with_prefix(&harness, &run_prefix) + .await + .is_empty() + ); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn a_run_the_rescan_cannot_read_never_hides_the_other_runs_of_its_partition() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::FAIL_PERMANENTLY); + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig::default(), + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let healthy_flow = "order-healthy"; + let partition = harness.partition_of(healthy_flow); + let broken_flow = (0..) + .map(|index| format!("order-broken-{index}")) + .find(|flow_id| harness.partition_of(flow_id) == partition) + .expect("some Flow ID shares the partition"); + let broken_run = harness.start_flow(start_request(&broken_flow), settings()); + close_run(&harness, &broken_flow, broken_run); + let healthy_run = harness.start_flow(start_request(healthy_flow), settings()); + close_run(&harness, healthy_flow, healthy_run); + wait_until(&harness, "both uploads were rejected", || { + harness + .durable + .objects + .call_count(ObjectStoreOperation::Put, HISTORY_SEGMENT_OBJECT_KEY_PREFIX) + == 2 + }) + .await; + // The broken run's segment row no longer decodes. + let broken_segment = segments(&harness, &broken_flow, broken_run) + .pop() + .expect("the close sealed a segment"); + let broken = harness.run_location(&broken_flow, broken_run); + harness + .flow_effect_operations + .state + .lock() + .expect("lock") + .rows + .insert( + ( + ColumnFamily::Flow, + history_segment_key(broken.flow_hash, broken_run, broken_segment.first_seq), + ), + vec![0xff; 3], + ); + faults.set(HistoryWriteFaults::PROCEED); + + wait_until_uploaded(&harness, healthy_flow, healthy_run).await; + assert_uploaded_chain(&harness, healthy_flow, healthy_run).await; + assert!(harness.metrics.history_upload_rescan_errors.get() >= 1); + assert!(is_upload_pending(&harness, &broken_flow, broken_run)); + harness.shutdown().await; +} + +#[tokio::test(start_paused = true)] +async fn seals_after_a_permanent_rejection_wait_for_the_rescan() { + let faults = Arc::new(HistoryWriteFaults::default()); + faults.set(HistoryWriteFaults::FAIL_PERMANENTLY); + let harness = EffectHarness::with_config( + durable_with_faults(Arc::clone(&faults)), + EffectsConfig::default(), + ); + harness.rebuild_all_partitions().await; + enable_history_offload(&harness); + let run_id = harness.start_flow( + start_request("order-1"), + RunSettings { + history_segment_target_bytes: 1, + ..settings() + }, + ); + let puts = || { + harness + .durable + .objects + .call_count(ObjectStoreOperation::Put, HISTORY_SEGMENT_OBJECT_KEY_PREFIX) + }; + wait_until(&harness, "the first upload is rejected", || puts() == 1).await; + + let sealed_before = segments(&harness, "order-1", run_id).len(); + for round in 0..10 { + set_long_attribute(&harness, "order-1", run_id, "notes", &format!("v{round}")); + } + assert!(segments(&harness, "order-1", run_id).len() >= sealed_before + 10); + tokio::time::sleep(Duration::from_secs(5)).await; + harness.settle().await; + assert_eq!( + puts(), + 1, + "new seals do not retry a rejected run before the rescan" + ); + tokio::time::sleep(Duration::from_secs(60)).await; + harness.settle().await; + assert_eq!(puts(), 2, "the rescan retries the run once"); + assert!(acknowledgments(&harness).is_empty()); + harness.shutdown().await; +} diff --git a/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs b/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs index d630141a2f..75e5bf27f1 100644 --- a/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs +++ b/server-zd/crates/dexzd-engine/src/effects/tests/mod.rs @@ -14,6 +14,7 @@ mod deadline_tests; mod dispatch_tests; mod handout_tests; mod hard_delete_tests; +mod history_upload_tests; mod keyword_index_tests; mod outbox_and_purge_tests; mod rebuild_tests; @@ -1423,6 +1424,7 @@ pub(super) struct EffectHarness { pub(super) clock: Arc, pub(super) stream_store: Arc, pub(super) attribute_sync_waker: AttributeSyncWaker, + pub(super) metrics: Arc, tasks: EffectTasksHandle, } @@ -1478,6 +1480,9 @@ impl EffectHarness { Arc::new(SystemEngineIdSource), ); let dispatcher = Arc::new(FakeDispatcher::new()); + let metrics = Arc::new(dexzd_metrics::NodeMetrics::unregistered_with_partitions( + usize::try_from(hasher.partition_count()).expect("a partition count fits in usize"), + )); let attribute_sync_waker = AttributeSyncWaker::new(); let stream_store = Arc::new( StreamStore::new(&StreamStoreConfig { @@ -1489,9 +1494,10 @@ impl EffectHarness { let (runner, tasks) = EffectRunner::new( &config, EffectDependencies { - metrics: Arc::new(dexzd_metrics::NodeMetrics::unregistered()), + metrics: Arc::clone(&metrics), dispatcher: Arc::clone(&dispatcher) as _, hard_delete_objects: hard_delete_objects.clone(), + history_object_store: Arc::clone(&durable.objects) as _, partition_hasher: hasher.clone(), clock: Arc::clone(&clock) as _, validation_config, @@ -1540,6 +1546,7 @@ impl EffectHarness { clock, stream_store, attribute_sync_waker, + metrics, tasks, } } diff --git a/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs b/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs index a4749fa8f6..8cc025b76a 100644 --- a/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs +++ b/server-zd/crates/dexzd-engine/tests/format_compat/golden_replay.rs @@ -29,7 +29,7 @@ use crate::testdata_tree::{ }; /// Digest of every file of each version, README included. Regenerate v1 before 1.0; afterward versions are immutable: add `v`. -const PINNED_VERSION_DIGESTS: &[(&str, &str)] = &[("v1", "12893d2ae03a13586a269bdbeccb43d8")]; +const PINNED_VERSION_DIGESTS: &[(&str, &str)] = &[("v1", "7f19d73154a70a9ebc4b50879ece2457")]; /// The restoring node starts this long after the last recorded command. pub(crate) const RESTART_DELAY_MICROS: i64 = 60 * 60 * 1_000_000; diff --git a/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs b/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs index 0e7199d897..9263e0d501 100644 --- a/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs +++ b/server-zd/crates/dexzd-engine/tests/format_compat/workload.rs @@ -15,7 +15,7 @@ use dexzd_engine::{ decode_hard_delete_value, default_run_settings, encode_hard_delete_value, hard_delete_placeholder_value, }; -use dexzd_history_segment::encode_history_segment; +use dexzd_history_segment::{HistorySegmentContent, encode_history_segment}; use dexzd_keyspace::row_keys::{decode_search_outbox_key, partition_index_range, prefix_range}; use dexzd_keyspace::{ ColumnFamily, IndexKind, PartitionHasher, PartitionId, RunId, RunTable, TenantId, @@ -183,17 +183,23 @@ impl Workload<'_> { }), ) .await; + let content = self.acknowledge_only_history_segment(&run).await; + assert_eq!(content.values.len(), 1, "the replaced value is listed"); + } + + /// Acknowledges the run's only history segment as an upload would, and returns its content. + async fn acknowledge_only_history_segment(&self, run: &FlowRun) -> HistorySegmentContent { let segments = self - .run_rows::(&run, RunTable::HistorySegment) + .run_rows::(run, RunTable::HistorySegment) .await; - assert_eq!(segments.len(), 1, "the stop sealed one segment"); + assert_eq!(segments.len(), 1, "the close sealed one segment"); let segment = segments[0].clone(); let view = self .node .committer() .committed_view() .expect("open a committed view"); - let flow_hash = self.context(&flow_id).flow_hash; + let flow_hash = self.context(&run.flow_id).flow_hash; let run_id = run.run_id; let first_seq = segment.first_seq; let content = tokio::task::spawn_blocking(move || { @@ -203,10 +209,9 @@ impl Workload<'_> { }) .await .expect("the read task completes"); - assert_eq!(content.values.len(), 1, "the replaced value is listed"); let encoded = encode_history_segment(&content).expect("the segment encodes"); self.commit( - &run, + run, Command::HistorySegmentUploaded(HistorySegmentUploadedCommand { first_seq: segment.first_seq, last_seq: segment.last_seq, @@ -215,6 +220,7 @@ impl Workload<'_> { }), ) .await; + content } /// Sets the history offload mode of each partition in `partitions`. @@ -1392,7 +1398,8 @@ impl Workload<'_> { .await; } - /// A short retention fires after close, and the purge deletes the run. + /// A short retention fires after close, once the run's sealed history segment is uploaded, + /// and the purge deletes the run. async fn purge_after_retention(&self) { let short_lived = self .start_flow( @@ -1419,6 +1426,8 @@ impl Workload<'_> { close_response(CloseDecisionType::ForceComplete, "done"), ) .await; + // Retention postpones while a segment is SEALED. + self.acknowledge_only_history_segment(&short_lived).await; let fired = self.fire_due_timers(&short_lived, 61 * SECOND_MICROS).await; assert!( fired diff --git a/server-zd/crates/dexzd-metrics/src/fixed_labels.rs b/server-zd/crates/dexzd-metrics/src/fixed_labels.rs index 89596c54b8..0e4252690a 100644 --- a/server-zd/crates/dexzd-metrics/src/fixed_labels.rs +++ b/server-zd/crates/dexzd-metrics/src/fixed_labels.rs @@ -205,6 +205,89 @@ impl EncodeMetric for AttributeSyncWriteFailures { } } +/// How one upload of a sealed history segment ended; every outcome but `Failed` is counted once +/// the upload's acknowledgment committed. +#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] +pub enum HistoryUploadOutcome { + /// The upload created the object. + Created, + /// An object with the same content checksum already existed, as a repeated upload finds. + AlreadyPresent, + /// An object with another content checksum existed; the upload replaced it. + Replaced, + /// An upload attempt failed. A transient object storage failure counts here and is retried + /// with backoff. A failure that is not retried with backoff also counts here and is retried + /// only at the next rescan: a permanent object storage rejection, a segment that cannot be + /// read or encoded from committed state, or a rejected acknowledgment. + Failed, +} + +label_value!(HistoryUploadOutcome { Created => "created", AlreadyPresent => "already_present", Replaced => "replaced", Failed => "failed" }); + +impl HistoryUploadOutcome { + const ALL: [Self; 4] = [ + Self::Created, + Self::AlreadyPresent, + Self::Replaced, + Self::Failed, + ]; + + const fn index(self) -> usize { + match self { + Self::Created => 0, + Self::AlreadyPresent => 1, + Self::Replaced => 2, + Self::Failed => 3, + } + } +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +struct HistoryUploadOutcomeLabel { + outcome: HistoryUploadOutcome, +} + +/// Fixed counters of history segment uploads, by outcome. +#[derive(Clone, Debug)] +pub struct HistoryUploadOutcomes { + values: Arc<[Counter; 4]>, +} + +impl Default for HistoryUploadOutcomes { + fn default() -> Self { + Self { + values: Arc::new(std::array::from_fn(|_| Counter::default())), + } + } +} + +impl HistoryUploadOutcomes { + /// Records one upload attempt. + pub fn record(&self, outcome: HistoryUploadOutcome) { + self.values[outcome.index()].inc(); + } + + /// Returns the upload attempts recorded with `outcome`. + pub fn count(&self, outcome: HistoryUploadOutcome) -> u64 { + self.values[outcome.index()].get() + } +} + +impl EncodeMetric for HistoryUploadOutcomes { + fn encode(&self, mut encoder: MetricEncoder) -> Result<(), std::fmt::Error> { + for outcome in HistoryUploadOutcome::ALL { + encoder + .encode_family(&HistoryUploadOutcomeLabel { outcome })? + .encode_counter::(&self.values[outcome.index()].get(), None)?; + } + Ok(()) + } + + fn metric_type(&self) -> MetricType { + MetricType::Counter + } +} + /// What this process does as a hot standby. #[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)] pub enum HotStandbyMetricState { diff --git a/server-zd/crates/dexzd-metrics/src/lag_gauges.rs b/server-zd/crates/dexzd-metrics/src/lag_gauges.rs index a8c5724412..6ff4234a9e 100644 --- a/server-zd/crates/dexzd-metrics/src/lag_gauges.rs +++ b/server-zd/crates/dexzd-metrics/src/lag_gauges.rs @@ -173,6 +173,131 @@ impl EncodeMetric for SearchOutboxOldestEntryAge { } } +/// Sealed history segments waiting for upload in each partition, as of the uploader's last scan. +#[derive(Clone, Debug)] +pub struct HistoryUploadBacklog { + partitions: Arc>, +} + +#[derive(Debug, Default)] +struct PartitionHistoryUploadBacklog { + segments: AtomicU64, + /// Uncompressed bytes of the waiting segments' history rows and superseded values. + bytes: AtomicU64, + /// Seal time of the oldest waiting segment, the sealing command's proposed_at_micros; 0 when none. + oldest_sealed_at_micros: AtomicI64, +} + +impl HistoryUploadBacklog { + pub(crate) fn new(partition_count: usize) -> Self { + Self { + partitions: Arc::new( + (0..partition_count) + .map(|_| PartitionHistoryUploadBacklog::default()) + .collect(), + ), + } + } + + /// Records what the uploader's last scan of `partition` found waiting: the segment count, + /// their uncompressed bytes, and the seal time of the oldest one, or `None` when none waits. + /// Ignores a partition outside the configured count. + pub fn set_partition( + &self, + partition: usize, + segments: u64, + bytes: u64, + oldest_sealed_at_micros: Option, + ) { + if let Some(backlog) = self.partitions.get(partition) { + backlog.segments.store(segments, Ordering::Relaxed); + backlog.bytes.store(bytes, Ordering::Relaxed); + backlog + .oldest_sealed_at_micros + .store(oldest_sealed_at_micros.unwrap_or(0), Ordering::Relaxed); + } + } + + /// Clears every partition, for a node that stopped uploading. + pub fn clear(&self) { + for partition in 0..self.partitions.len() { + self.set_partition(partition, 0, 0, None); + } + } + + /// Returns the segments waiting in every partition. + pub fn segments(&self) -> u64 { + self.partitions + .iter() + .map(|backlog| backlog.segments.load(Ordering::Relaxed)) + .fold(0, u64::saturating_add) + } + + /// Returns the uncompressed bytes waiting in every partition. + pub fn bytes(&self) -> u64 { + self.partitions + .iter() + .map(|backlog| backlog.bytes.load(Ordering::Relaxed)) + .fold(0, u64::saturating_add) + } + + /// Returns the seal time of the oldest segment waiting in any partition, or `None` when none + /// waits. + pub fn oldest_sealed_at_micros(&self) -> Option { + self.partitions + .iter() + .map(|backlog| backlog.oldest_sealed_at_micros.load(Ordering::Relaxed)) + .filter(|sealed_at| *sealed_at > 0) + .min() + } +} + +#[derive(Clone, Debug)] +pub struct HistoryUploadBacklogSegments(pub(crate) HistoryUploadBacklog); + +#[derive(Clone, Debug)] +pub struct HistoryUploadBacklogBytes(pub(crate) HistoryUploadBacklog); + +#[derive(Clone, Debug)] +pub struct HistoryUploadOldestSealedAge(pub(crate) HistoryUploadBacklog); + +impl EncodeMetric for HistoryUploadBacklogSegments { + fn encode(&self, mut encoder: MetricEncoder) -> Result<(), std::fmt::Error> { + encoder.encode_gauge(&i64::try_from(self.0.segments()).unwrap_or(i64::MAX)) + } + + fn metric_type(&self) -> MetricType { + MetricType::Gauge + } +} + +impl EncodeMetric for HistoryUploadBacklogBytes { + fn encode(&self, mut encoder: MetricEncoder) -> Result<(), std::fmt::Error> { + encoder.encode_gauge(&i64::try_from(self.0.bytes()).unwrap_or(i64::MAX)) + } + + fn metric_type(&self) -> MetricType { + MetricType::Gauge + } +} + +impl EncodeMetric for HistoryUploadOldestSealedAge { + fn encode(&self, mut encoder: MetricEncoder) -> Result<(), std::fmt::Error> { + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs_f64(); + let age = self.0.oldest_sealed_at_micros().map_or(0.0, |sealed_at| { + (now - sealed_at as f64 / 1_000_000.0).max(0.0) + }); + encoder.encode_gauge(&age) + } + + fn metric_type(&self) -> MetricType { + MetricType::Gauge + } +} + #[cfg(test)] mod tests { use super::*; @@ -189,4 +314,18 @@ mod tests { backlog.clear(); assert_eq!(backlog.entries().sum::(), 0); } + + #[test] + fn history_upload_backlog_totals_partitions_and_keeps_the_oldest_seal() { + let backlog = HistoryUploadBacklog::new(3); + backlog.set_partition(0, 2, 300, Some(50)); + backlog.set_partition(2, 1, 100, Some(20)); + backlog.set_partition(9, 100, 100, Some(1)); + assert_eq!(backlog.segments(), 3); + assert_eq!(backlog.bytes(), 400); + assert_eq!(backlog.oldest_sealed_at_micros(), Some(20)); + backlog.clear(); + assert_eq!(backlog.segments(), 0); + assert_eq!(backlog.oldest_sealed_at_micros(), None); + } } diff --git a/server-zd/crates/dexzd-metrics/src/lib.rs b/server-zd/crates/dexzd-metrics/src/lib.rs index 4f5ee475df..e92c620e08 100644 --- a/server-zd/crates/dexzd-metrics/src/lib.rs +++ b/server-zd/crates/dexzd-metrics/src/lib.rs @@ -35,15 +35,17 @@ use std::path::PathBuf; pub use api_requests::{ApiLatency, ApiRequests, GrpcCode, RpcClass, classify_rpc}; pub use commit_latency::CommitLatency; pub use fixed_labels::{ - AttributeSyncFailureKind, AttributeSyncWriteFailures, HotStandbyMetricState, - HotStandbyStateGauges, ObjectErrorCounters, OutcomeCounters, RoleGauges, TakeoverCounters, - TakeoverDurations, TakeoverStandbyMode, TakeoverStoreSource, + AttributeSyncFailureKind, AttributeSyncWriteFailures, HistoryUploadOutcome, + HistoryUploadOutcomes, HotStandbyMetricState, HotStandbyStateGauges, ObjectErrorCounters, + OutcomeCounters, RoleGauges, TakeoverCounters, TakeoverDurations, TakeoverStandbyMode, + TakeoverStoreSource, }; pub use histogram::AtomicHistogram; use histogram::{FLOW_LATENCY_BOUNDS, SCHEDULE_LATENCY_BOUNDS}; pub use lag_gauges::{ - SearchOutboxBacklog, SearchOutboxEntries, SearchOutboxMaxPartitionEntries, - SearchOutboxOldestEntryAge, TimerLag, + HistoryUploadBacklog, HistoryUploadBacklogBytes, HistoryUploadBacklogSegments, + HistoryUploadOldestSealedAge, SearchOutboxBacklog, SearchOutboxEntries, + SearchOutboxMaxPartitionEntries, SearchOutboxOldestEntryAge, TimerLag, }; pub use object_requests::{ObjectKind, ObjectOperation, ObjectRequests, classify_object_kind}; use prometheus_client::encoding::{EncodeLabelSet, text::encode}; @@ -178,6 +180,17 @@ pub struct NodeMetrics { pub search_projection_mode_skipped_partitions: Counter, /// Stopped partitions that kept their previous history offload mode at startup. pub history_offload_mode_skipped_partitions: Counter, + /// Sealed history segments waiting for upload, per partition, as of the uploader's last scan. + pub history_upload_backlog: HistoryUploadBacklog, + /// History segment uploads by outcome: each upload once its acknowledgment committed, and + /// each failed attempt. + pub history_uploads: HistoryUploadOutcomes, + /// Bytes of the history segment objects this node wrote. + pub history_upload_bytes: Counter, + /// Existing history segment objects whose content checksum differed from committed state. + pub history_upload_checksum_mismatches: Counter, + /// Runs or index rows that the history upload rescan could not read and skipped. + pub history_upload_rescan_errors: Counter, /// Hot standby state, with one series for each closed state. pub hot_standby_states: HotStandbyStateGauges, /// Log batches the hot standby has yet to apply, as of its last listing. @@ -554,6 +567,41 @@ impl NodeMetrics { "Stopped partitions that kept their previous history offload mode at startup.", metrics.history_offload_mode_skipped_partitions.clone(), ); + registry.register( + "dexzd_history_upload_backlog_segments", + "Sealed history segments waiting for upload.", + HistoryUploadBacklogSegments(metrics.history_upload_backlog.clone()), + ); + registry.register( + "dexzd_history_upload_backlog_bytes", + "Uncompressed bytes of the sealed history segments waiting for upload.", + HistoryUploadBacklogBytes(metrics.history_upload_backlog.clone()), + ); + registry.register( + "dexzd_history_upload_oldest_sealed_age_seconds", + "Age of the oldest sealed history segment waiting for upload.", + HistoryUploadOldestSealedAge(metrics.history_upload_backlog.clone()), + ); + registry.register( + "dexzd_history_uploads", + "History segment uploads by outcome: acknowledged uploads and failed attempts.", + metrics.history_uploads.clone(), + ); + registry.register( + "dexzd_history_upload_bytes", + "Bytes of the history segment objects this node wrote.", + metrics.history_upload_bytes.clone(), + ); + registry.register( + "dexzd_history_upload_checksum_mismatches", + "Existing history segment objects whose content differed from committed state.", + metrics.history_upload_checksum_mismatches.clone(), + ); + registry.register( + "dexzd_history_upload_rescan_errors", + "Runs or index rows that the history upload rescan could not read and skipped.", + metrics.history_upload_rescan_errors.clone(), + ); registry.register( "dexzd_hot_standby_state", "Hot standby state of this process.", @@ -598,6 +646,12 @@ impl NodeMetrics { Self::with_partitions(0) } + /// Creates unregistered handles whose per-partition gauges track `partition_count` + /// partitions, for tests and simulations that read those gauges. + pub fn unregistered_with_partitions(partition_count: usize) -> Self { + Self::with_partitions(partition_count) + } + fn with_partitions(partition_count: usize) -> Self { Self { behavior_version: Gauge::default(), @@ -656,6 +710,11 @@ impl NodeMetrics { search_initial_loads: Counter::default(), search_projection_mode_skipped_partitions: Counter::default(), history_offload_mode_skipped_partitions: Counter::default(), + history_upload_backlog: HistoryUploadBacklog::new(partition_count), + history_uploads: HistoryUploadOutcomes::default(), + history_upload_bytes: Counter::default(), + history_upload_checksum_mismatches: Counter::default(), + history_upload_rescan_errors: Counter::default(), hot_standby_states: HotStandbyStateGauges::default(), hot_standby_lag_batches: Gauge::default(), hot_standby_lag_seconds: Gauge::default(), diff --git a/server-zd/crates/dexzd-metrics/src/object_requests.rs b/server-zd/crates/dexzd-metrics/src/object_requests.rs index e7d851119b..5a4d7dc4b4 100644 --- a/server-zd/crates/dexzd-metrics/src/object_requests.rs +++ b/server-zd/crates/dexzd-metrics/src/object_requests.rs @@ -52,18 +52,21 @@ pub enum ObjectKind { Manifest, /// Node lease. Lease, + /// History segment object. + History, /// Cluster metadata or value object. Other, } -label_value!(ObjectKind { Log => "log", Snapshot => "snapshot", Manifest => "manifest", Lease => "lease", Other => "other" }); +label_value!(ObjectKind { Log => "log", Snapshot => "snapshot", Manifest => "manifest", Lease => "lease", History => "history", Other => "other" }); impl ObjectKind { - const ALL: [Self; 5] = [ + const ALL: [Self; 6] = [ Self::Log, Self::Snapshot, Self::Manifest, Self::Lease, + Self::History, Self::Other, ]; @@ -73,7 +76,8 @@ impl ObjectKind { Self::Snapshot => 1, Self::Manifest => 2, Self::Lease => 3, - Self::Other => 4, + Self::History => 4, + Self::Other => 5, } } } @@ -88,6 +92,8 @@ pub fn classify_object_kind(key: &str) -> ObjectKind { ObjectKind::Manifest } else if key.starts_with("lease/") || key == "lease" { ObjectKind::Lease + } else if key.starts_with("history/") || key == "history" { + ObjectKind::History } else { ObjectKind::Other } @@ -102,7 +108,7 @@ struct ObjectLabel { /// Fixed atomic object request counters, emitting only observed label pairs. #[derive(Clone, Debug)] pub struct ObjectRequests { - counts: Arc<[[Counter; 5]; 4]>, + counts: Arc<[[Counter; 6]; 4]>, } impl Default for ObjectRequests { @@ -171,6 +177,11 @@ mod tests { ), ("manifest", ObjectKind::Manifest), ("lease/node-a", ObjectKind::Lease), + ( + "history/0000000000000000/00000000000000ab/0192f3a4-5b6c-7d8e-9fa0-b1c2d3e4f5a6/\ + 00000000000000000001-00000000000000000009", + ObjectKind::History, + ), ("cluster", ObjectKind::Other), ("hard-delete/tenant/value", ObjectKind::Other), ] { diff --git a/server-zd/crates/dexzd-server/Cargo.toml b/server-zd/crates/dexzd-server/Cargo.toml index 969d54fd08..6b60516479 100644 --- a/server-zd/crates/dexzd-server/Cargo.toml +++ b/server-zd/crates/dexzd-server/Cargo.toml @@ -62,6 +62,7 @@ base64.workspace = true bytes.workspace = true # Unit tests wait on a local store directory through the engine's test-only waiter. dexzd-engine = { workspace = true, features = ["test-util"] } +dexzd-history-segment.workspace = true dexzd-visibility-query.workspace = true hyper-util.workspace = true libc.workspace = true diff --git a/server-zd/crates/dexzd-server/src/component_configs.rs b/server-zd/crates/dexzd-server/src/component_configs.rs index a9cad9c488..e6fef6e3c2 100644 --- a/server-zd/crates/dexzd-server/src/component_configs.rs +++ b/server-zd/crates/dexzd-server/src/component_configs.rs @@ -248,6 +248,10 @@ impl ComponentConfigs { ..EffectsConfig::default().child_run_settings }, heads_resync_interval: config.matching.heads_resync_interval, + history_upload_concurrency: to_usize( + "history.uploadConcurrency", + u64::from(config.history.upload_concurrency), + )?, ..EffectsConfig::default() }; let matching = MatchingConfig { diff --git a/server-zd/crates/dexzd-server/src/config/engine_config.rs b/server-zd/crates/dexzd-server/src/config/engine_config.rs index 815053a2a6..9fe9a629b8 100644 --- a/server-zd/crates/dexzd-server/src/config/engine_config.rs +++ b/server-zd/crates/dexzd-server/src/config/engine_config.rs @@ -272,6 +272,11 @@ pub struct HistoryConfig { /// superseded values reach this many bytes. Recorded in RunRecord at FlowStart; changes /// affect only later runs. pub segment_target_bytes: u64, + /// Defaults to 16; from 1 to 1024. Most sealed history segments the lease holder reads, + /// encodes, and uploads to object storage at once. Higher values drain an upload backlog + /// faster and hold more segments in memory; each in-flight segment holds about its encoded + /// size, usually up to `segmentTargetBytes` plus one command's history. + pub upload_concurrency: u32, /// Deprecated keys that the parsed YAML used; never serialized. #[serde(skip)] deprecated_keys: Vec, @@ -295,6 +300,7 @@ impl Default for HistoryConfig { max_retained_superseded_bytes: 256 * 1024 * 1024, is_offload_enabled: false, segment_target_bytes: 4 * 1024 * 1024, + upload_concurrency: 16, deprecated_keys: Vec::new(), } } @@ -344,6 +350,7 @@ struct HistoryConfigSource { #[serde(rename = "offloadEnabled")] is_offload_enabled: bool, segment_target_bytes: u64, + upload_concurrency: u32, } /// A key's YAML value: `None` when the key is absent, `Some(None)` when it is an explicit null. @@ -373,6 +380,7 @@ impl Default for HistoryConfigSource { max_retained_superseded_bytes: defaults.max_retained_superseded_bytes, is_offload_enabled: defaults.is_offload_enabled, segment_target_bytes: defaults.segment_target_bytes, + upload_concurrency: defaults.upload_concurrency, } } } @@ -413,6 +421,7 @@ impl TryFrom for HistoryConfig { max_retained_superseded_bytes: source.max_retained_superseded_bytes, is_offload_enabled: source.is_offload_enabled, segment_target_bytes: source.segment_target_bytes, + upload_concurrency: source.upload_concurrency, deprecated_keys, }) } diff --git a/server-zd/crates/dexzd-server/src/config/tests.rs b/server-zd/crates/dexzd-server/src/config/tests.rs index 0eb8fc3e34..c46e87f1b0 100644 --- a/server-zd/crates/dexzd-server/src/config/tests.rs +++ b/server-zd/crates/dexzd-server/src/config/tests.rs @@ -151,6 +151,28 @@ fn history_segment_target_bytes_reaches_run_and_child_settings() { ); } +#[test] +fn history_upload_concurrency_reaches_the_effect_runner() { + assert_eq!(Config::default().history.upload_concurrency, 16); + assert_eq!( + ComponentConfigs::from_config(&valid_config(), false) + .expect("component config") + .effects + .history_upload_concurrency, + 16 + ); + let parsed = Config::parse_yaml("history:\n uploadConcurrency: 3\n") + .expect("parse the history upload concurrency"); + let mut configuration = valid_config(); + configuration.history.upload_concurrency = parsed.history.upload_concurrency; + configuration + .validate() + .expect("a valid history upload concurrency"); + let components = + ComponentConfigs::from_config(&configuration, false).expect("component config"); + assert_eq!(components.effects.history_upload_concurrency, 3); +} + #[test] fn run_limits_and_the_child_result_limit_reach_run_settings() { assert_eq!( @@ -1265,6 +1287,12 @@ fn every_field_constraint_names_the_rejected_yaml_path() { ("history.segmentTargetBytes", |config| { config.history.segment_target_bytes = 256 * 1024 * 1024 + 1 }), + ("history.uploadConcurrency", |config| { + config.history.upload_concurrency = 0 + }), + ("history.uploadConcurrency", |config| { + config.history.upload_concurrency = 1025 + }), ("values.maxValueBytes", |config| { config.values.max_value_bytes = 0 }), diff --git a/server-zd/crates/dexzd-server/src/config/validation.rs b/server-zd/crates/dexzd-server/src/config/validation.rs index d1f2c17a9d..57969ee48c 100644 --- a/server-zd/crates/dexzd-server/src/config/validation.rs +++ b/server-zd/crates/dexzd-server/src/config/validation.rs @@ -18,6 +18,8 @@ use super::{Config, ConfigError, CredentialSource, SearchBackend}; const MAX_LOG_RECORD_BYTES: u64 = 32 * 1024 * 1024; /// Bounds the memory a reader needs to decompress one segment's records block. const MAX_HISTORY_SEGMENT_TARGET_BYTES: u64 = 256 * 1024 * 1024; +/// Bounds the memory that in-flight history segment uploads hold. +const MAX_HISTORY_UPLOAD_CONCURRENCY: u32 = 1024; impl Config { /// Checks cross-section limits and reports the first failing camelCase YAML field path. @@ -270,6 +272,11 @@ impl Config { "history.segmentTargetBytes", "must be positive and at most 256 MiB", )?; + ensure( + (1..=MAX_HISTORY_UPLOAD_CONCURRENCY).contains(&self.history.upload_concurrency), + "history.uploadConcurrency", + "must be from 1 to 1024", + )?; ensure( self.retention.closed_flow_duration > Duration::ZERO, "retention.closedFlowDuration", diff --git a/server-zd/crates/dexzd-server/src/server_node.rs b/server-zd/crates/dexzd-server/src/server_node.rs index 6c321fdc8f..14107d81f7 100644 --- a/server-zd/crates/dexzd-server/src/server_node.rs +++ b/server-zd/crates/dexzd-server/src/server_node.rs @@ -498,6 +498,7 @@ impl ServerNode { metrics: metrics.clone(), dispatcher: Arc::new(dispatch.clone()), hard_delete_objects: hard_delete_objects.clone(), + history_object_store: object_stores.primary.clone(), partition_hasher: partition_hasher.clone(), clock: clock.clone(), validation_config: components.validation().clone(), diff --git a/server-zd/crates/dexzd-server/tests/serve_process/harness.rs b/server-zd/crates/dexzd-server/tests/serve_process/harness.rs index 2fa2574b93..fb5f0bb650 100644 --- a/server-zd/crates/dexzd-server/tests/serve_process/harness.rs +++ b/server-zd/crates/dexzd-server/tests/serve_process/harness.rs @@ -155,6 +155,18 @@ impl S3Emulator { } } + /// Returns the whole object at `key` in `bucket`. + pub(crate) fn read_object(&self, bucket: &str, key: &str) -> Vec { + let body = self + .store + .read_object(bucket, key, &s3local::ObjectReadRequest::default()) + .expect("the emulator reads the object"); + let mut bytes = Vec::new(); + std::io::Read::read_to_end(&mut std::io::Read::take(body.file, body.length), &mut bytes) + .expect("the object body reads"); + bytes + } + /// Replaces the emulator's fault rules with `rules`, a JSON array of s3local fault rules. pub(crate) async fn set_fault_rules(&self, rules: &str) { let response = self diff --git a/server-zd/crates/dexzd-server/tests/serve_process/history_uploads.rs b/server-zd/crates/dexzd-server/tests/serve_process/history_uploads.rs new file mode 100644 index 0000000000..f8bc904e2a --- /dev/null +++ b/server-zd/crates/dexzd-server/tests/serve_process/history_uploads.rs @@ -0,0 +1,241 @@ +// Copyright (c) 2026 Super Durable, Inc. +// +// Licensed under the Sustainable Use License 1.0. +// You may not use this file except in compliance with the License. +// See the LICENSE file in the repository root. +// +// SPDX-License-Identifier: LicenseRef-Sustainable-Use-1.0 + +//! History offload across an object storage outage and a takeover: commands commit while every +//! history upload fails, and a waiting standby that restores a snapshot finishes the uploads. + +use std::fs; +use std::net::SocketAddr; +use std::time::Duration; + +use dexzd_history_segment::{decode_history_segment, history_segment_object_key}; +use dexzd_keyspace::{ + ColumnFamily, IndexKind, PartitionId, RunTable, TenantId, parse_run_key, partition_index_prefix, +}; +use dexzd_protocol::dex::flow_service_client::FlowServiceClient; +use dexzd_protocol::dexzd::v1::{HistorySegmentRecord, HistorySegmentState}; +use dexzd_storage::{LocalReadView, RocksDbLocalStore, RocksDbStoreOptions, ScanDirection}; +use prost::Message; +use s3local_server::SignatureVerification; +use tempfile::TempDir; + +use super::harness::{S3Emulator, ServeConfig, ServeProcess, free_loopback_address}; +use super::metrics::try_scrape; +use super::{ + HELD_LEASE_DURATION, PRIMARY_BUCKET, PRIMARY_PREFIX, SHUTDOWN_BUDGET, run_greeting_flow, + start_greeting_worker, +}; + +const FLOW_COUNT: usize = 4; +/// Matches `cluster.partitionCount` of [`ServeConfig`]. +const PARTITION_COUNT: u16 = 16; +const SNAPSHOT_BUDGET: Duration = Duration::from_secs(30); +const STANDBY_WAIT_BUDGET: Duration = Duration::from_secs(30); +const TAKEOVER_BUDGET: Duration = Duration::from_secs(15); +const UPLOAD_BUDGET: Duration = Duration::from_secs(60); +const LEASE_WAIT_LOG_LINE: &str = "another process holds the node lease"; +const UPLOAD_FAILURE_LOG_LINE: &str = "uploading a history segment failed; retrying"; +/// Upload outcomes counted once the acknowledgment committed. +const ACKNOWLEDGED_OUTCOMES: [&str; 3] = ["created", "already_present", "replaced"]; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_standby_takeover_finishes_uploads_that_failed_on_the_previous_owner() { + let primary = S3Emulator::start(PRIMARY_BUCKET, SignatureVerification::Strict).await; + let history_prefix = format!("{PRIMARY_PREFIX}/history/"); + primary + .set_fault_rules(&format!( + r#"[{{"operation":"PutObject","key_prefix":"{history_prefix}","action":{{"type":"error","status":503}}}}]"# + )) + .await; + let primary_endpoint = primary.endpoint(); + let directory = TempDir::new().expect("a temporary test directory can be created"); + let serve_config = |name: &str, listen, metrics_listen: SocketAddr| { + let path = directory.path().join(format!("{name}.yaml")); + ServeConfig { + primary_endpoint: &primary_endpoint, + hard_delete_endpoint: None, + listen, + local_directory: &directory.path().join(format!("{name}-store")), + lease_duration: HELD_LEASE_DURATION, + fail_if_held: false, + parade_db: None, + } + .write_with_snapshot_schedule(&path, "1s", "1h"); + let mut contents = fs::read_to_string(&path).expect("the serve configuration is readable"); + contents.push_str(&format!( + "history:\n offloadEnabled: true\nmetrics:\n listen: '{metrics_listen}'\n" + )); + fs::write(&path, contents).expect("the serve configuration is written"); + path + }; + + let serving_address = free_loopback_address(); + let mut serving = ServeProcess::spawn( + &serve_config("serving", serving_address, free_loopback_address()), + serving_address, + directory.path().join("serving.log"), + ); + let channel = serving.wait_until_serving().await; + let worker = start_greeting_worker(channel.clone()).await; + let mut flow_client = FlowServiceClient::new(channel); + for index in 0..FLOW_COUNT { + run_greeting_flow(&mut flow_client, &format!("offload-flow-{index}")).await; + } + assert!( + primary + .keys_under(PRIMARY_BUCKET, &history_prefix) + .is_empty(), + "every history upload failed, yet each Flow completed" + ); + wait_for_snapshot(&primary).await; + serving + .wait_for_log_line(UPLOAD_FAILURE_LOG_LINE, UPLOAD_BUDGET) + .await; + + let standby_address = free_loopback_address(); + let standby_metrics = free_loopback_address(); + let mut standby = ServeProcess::spawn( + &serve_config("standby", standby_address, standby_metrics), + standby_address, + directory.path().join("standby.log"), + ); + standby + .wait_for_log_line(LEASE_WAIT_LOG_LINE, STANDBY_WAIT_BUDGET) + .await; + let serving_status = serving.terminate(SHUTDOWN_BUDGET).await; + assert_eq!(serving_status.code(), Some(0), "{}", serving.log()); + worker.stop().await; + primary.clear_fault_rules().await; + tokio::time::timeout(TAKEOVER_BUDGET, standby.wait_until_serving()) + .await + .unwrap_or_else(|_| panic!("the standby did not take over:\n{}", standby.log())); + + let waited = tokio::time::timeout(UPLOAD_BUDGET, async { + while acknowledged_uploads(standby_metrics).await < FLOW_COUNT as u64 { + tokio::time::sleep(Duration::from_millis(50)).await; + } + }) + .await; + waited.unwrap_or_else(|_| { + panic!( + "the new owner did not acknowledge {FLOW_COUNT} uploads:\n{}", + standby.log() + ) + }); + let standby_status = standby.terminate(SHUTDOWN_BUDGET).await; + assert_eq!(standby_status.code(), Some(0), "{}", standby.log()); + + let local_store = RocksDbLocalStore::open( + directory.path().join("standby-store"), + RocksDbStoreOptions::default(), + ) + .expect("the stopped standby's store opens"); + let expected_keys = assert_every_segment_uploaded(&local_store, &primary); + assert_eq!( + expected_keys.len(), + FLOW_COUNT, + "one closed run, one segment" + ); + assert_eq!( + primary.keys_under(PRIMARY_BUCKET, &history_prefix), + expected_keys, + "one object per segment and no other object" + ); + primary.shutdown().await; +} + +/// Returns the uploads whose acknowledgment committed on the node, or 0 while its metrics +/// listener is down. +async fn acknowledged_uploads(metrics: SocketAddr) -> u64 { + let Ok(body) = try_scrape(metrics, "/metrics").await else { + return 0; + }; + ACKNOWLEDGED_OUTCOMES + .iter() + .filter_map(|outcome| { + let series = format!("dexzd_history_uploads_total{{outcome=\"{outcome}\"}} "); + body.lines() + .find_map(|line| line.strip_prefix(series.as_str())) + .and_then(|value| value.trim().parse::().ok()) + }) + .sum() +} + +/// Waits until the serving node published a snapshot, which the standby restores at takeover. +async fn wait_for_snapshot(primary: &S3Emulator) { + let snapshot_prefix = format!("{PRIMARY_PREFIX}/snapshot/"); + tokio::time::timeout(SNAPSHOT_BUDGET, async { + while !primary.has_object_under(PRIMARY_BUCKET, &snapshot_prefix) { + tokio::time::sleep(Duration::from_millis(50)).await; + } + }) + .await + .unwrap_or_else(|_| panic!("no snapshot was published within {SNAPSHOT_BUDGET:?}")); +} + +/// Checks that every history segment of the stopped node is UPLOADED with an object whose +/// content checksum it records, and that no run waits for an upload. Returns the object keys. +fn assert_every_segment_uploaded( + local_store: &RocksDbLocalStore, + primary: &S3Emulator, +) -> Vec { + let mut keys = Vec::new(); + for (key, value) in local_store + .scan( + ColumnFamily::Flow, + None, + None, + ScanDirection::Forward, + usize::MAX, + ) + .expect("the Flow rows scan") + { + let Ok(parsed) = parse_run_key(&key) else { + continue; + }; + if parsed.table != RunTable::HistorySegment { + continue; + } + let segment = + HistorySegmentRecord::decode(value.as_slice()).expect("the segment row decodes"); + assert_eq!( + segment.state, + HistorySegmentState::Uploaded as i32, + "{segment:?}" + ); + let object_key = format!( + "{PRIMARY_PREFIX}/{}", + history_segment_object_key( + TenantId::DEFAULT, + parsed.flow_hash, + parsed.run_id, + segment.first_seq, + segment.last_seq, + ) + ); + let object = primary.read_object(PRIMARY_BUCKET, &object_key); + let content = decode_history_segment(&object).expect("the uploaded object decodes"); + assert_eq!(content.identity.run_id, parsed.run_id); + assert_eq!(content.identity.last_seq, segment.last_seq); + assert_eq!(segment.object_size_bytes, object.len() as u64); + keys.push(object_key); + } + for partition in 0..PARTITION_COUNT { + let pending = local_store + .scan_prefix( + ColumnFamily::PartitionIndex, + &partition_index_prefix(PartitionId(partition), IndexKind::HistoryUploadPending), + ScanDirection::Forward, + 1, + ) + .expect("the index scans"); + assert!(pending.is_empty(), "partition {partition} still uploads"); + } + keys.sort(); + keys +} diff --git a/server-zd/crates/dexzd-server/tests/serve_process/main.rs b/server-zd/crates/dexzd-server/tests/serve_process/main.rs index 68e30bacc8..299339649b 100644 --- a/server-zd/crates/dexzd-server/tests/serve_process/main.rs +++ b/server-zd/crates/dexzd-server/tests/serve_process/main.rs @@ -16,6 +16,7 @@ mod dev_restart; mod dev_soak; mod harness; mod history_offload_mode; +mod history_uploads; mod lease_release; mod lease_validity; mod local_command; diff --git a/server-zd/crates/dexzd-server/tests/serve_process/metrics.rs b/server-zd/crates/dexzd-server/tests/serve_process/metrics.rs index b8eb8e710b..61c78690e8 100644 --- a/server-zd/crates/dexzd-server/tests/serve_process/metrics.rs +++ b/server-zd/crates/dexzd-server/tests/serve_process/metrics.rs @@ -324,7 +324,7 @@ async fn metrics_endpoint_tracks_one_completed_flow_and_worker_poll() { .lines() .filter(|line| !line.starts_with('#') && !line.is_empty()) .count(); - assert!(series_count <= 192, "scrape has {series_count} series"); + assert!(series_count <= 202, "scrape has {series_count} series"); eprintln!("metrics scenario series count: {series_count}"); let status = process.terminate(SHUTDOWN_BUDGET).await; assert_eq!(status.code(), Some(0), "{}", process.log()); diff --git a/server-zd/crates/dexzd-simulation/src/engine_simulation/simulated_node.rs b/server-zd/crates/dexzd-simulation/src/engine_simulation/simulated_node.rs index 2621379c6b..72487e92e6 100644 --- a/server-zd/crates/dexzd-simulation/src/engine_simulation/simulated_node.rs +++ b/server-zd/crates/dexzd-simulation/src/engine_simulation/simulated_node.rs @@ -284,6 +284,7 @@ impl NodeProcess { let factory_parts = Arc::clone(&startup_parts); let factory_clock = Arc::clone(&wall_clock); let factory_store = Arc::clone(&object_store_port); + let factory_history_store = Arc::clone(&object_store_port); let factory_applier: Arc = Arc::new(DryRunCountingApplier { applier: Arc::clone(&applier), applied_command_ids: Mutex::new(BTreeSet::new()), @@ -316,6 +317,7 @@ impl NodeProcess { metrics: Arc::new(dexzd_metrics::NodeMetrics::unregistered()), dispatcher: worker_port.clone(), hard_delete_objects: hard_delete_objects.clone(), + history_object_store: factory_history_store, partition_hasher: partition_hasher.clone(), clock: factory_clock, validation_config: ValidationConfig::default(), diff --git a/server-zd/testdata/v1/README.md b/server-zd/testdata/v1/README.md index 298593fef7..884a0849a1 100644 --- a/server-zd/testdata/v1/README.md +++ b/server-zd/testdata/v1/README.md @@ -59,8 +59,10 @@ Then the node took a snapshot. After the snapshot: - `travel-flow`: two Steps and a client Attribute write complete the run. Time travel to the second Step's Execute, with writes reapplied, reopens it in run attempt 2, which completes. - `idle-flow`: a timer wait is skipped, the Flow config is updated, and the Flow is canceled. -- `short-lived-flow`: a run with a 60-second retention completes; the retention timer fires - and the run is purged. +- `short-lived-flow`: a run with a 60-second retention completes in partition 0, which + offloads history, so its close seals a segment. A HistorySegmentUploaded acknowledgment + marks it uploaded, because retention postpones while a segment is SEALED. Then the retention + timer fires and the run is purged. - `handout-sync`: an Execute that locks an Attribute is started and returned three times, with reasons UNSUPPORTED, LEGACY_NO_CAPACITY, and NOT_DELIVERED, then fails before any start. Its retries fail as STARTED_ATTEMPT_INTERRUPTED after a heartbeat with a task ID, as a not-run diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000070 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000070 index 8dc7674765f4318ed83346b1e6724c0c7cc86292..088e2164164e87e05ea9e12fdf0a271e03c3c91c 100644 GIT binary patch delta 116 zcmdnOc#?5~xpKBflEJKJ504+-dS=IaMvsP9$M1exG?Q6ltNvex2tj6sM5*|RQJQWr zaVr7F#S*d73Nuffh;Z@0bb`|$A>qsM_imwtU(G?Q6ltNvexh)gC1A1SYiQJRJ@ waVr7F#S(Fx88t7u-8Pjob$C9X#LU9Qq{T4p^5vF((T~PP%x~(4FG?@}ijoD77#B09a`P08AQ6ky%3* ak^=+^0xAGGTftlxI0TNva)(Fbt=!xf4;?80 delta 64 zcmV-G0KfmU0j~j&HzHCfNigJv+xETxww^sm(4FG?@}ijoD77#B09Z)^07x1}ky%3% WiUj~De?x%oSZ=sD7t=NKAejBOSsZ}? diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000072 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000072 index 24121bc74b74a762b55a402a66202b5aa0bdc238..587cd6dd99259cdc36d64cd55901a3e93b625c31 100644 GIT binary patch delta 146 zcmcb{w4RaM#WfBk->fgMA-=eOfe= zS!1jIUxo-zCI$~FH!gwVjQpY!-JHy_)D+#cocwa2&czb3;udAPYcx0tg{(s*6&N)b r4PctBK+Bk-=#qw_3fZ0HZ{J!Z${ZRwjcB`~Tj0xAW5Ft&AQI_I>>I zY0*q(jjj5B86tAo8Mg2?Gf6ctaq(m%=B4D9mgp8&<|QjIYA_nWlv)WeE|!Snj;VL+ zGFaUu=QH165#xDAF0P~s<#vsPP5cRR3_=M)+#YT@`Q=*NL8-;5MP;e=!6m5$UP?+V z9E=Jq2255!@toAeVkagKMi0g;*5Z=<3-$nzEK9@$zXXtY29RLVVv=CA60pwMDk)z0*68`-n+#0v zq`929IG6?0Qxq6M(yR=t59CBk->2yw_3f30HcJz!Z${ZRwjeF)33g}|8~;1ERPTO8-9OU zG?Q6ltNvexh+KAtExgT4QVmR8JQ<03Dfy)(y2X`w$qI}bj0P~JRsxKRCE|F(N)ApF zUMh9r-_oo_jOQ7-xMS+wx(rr#$@$DT5K0i@_HfI|FW2G@N-a(;DoYKnFG(%%Qc_~! zU{qi+V6p;==cFbUJ281MdN5|O7MJ8FXGaMYmn0UI=t`M$Oz;zU!7kAN1dIv|2m0F2 z3TmVn0C^l14Gj(s6Z|}0um^x-St2I*C4j^;fCQ5klLVucfOXDRN%6wBM$Z@DWMFzH y&E>?!!7PxXroaf2W@T7?Ajj%$WCt(Dv{ykh0t%~lX0X@26cJIi-6G@AArAn`DQuSj delta 182 zcmcc0bcvDM#WfBk->f%t005jpj{pDw diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000074 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000074 index 481eb995e99b950e06a0ee75644a48144da52cbb..11d0a2b119100ce5c6b56941b75235bf82c01429 100644 GIT binary patch delta 112 zcmV-$0FVFF0n!1GH$rtNY%tJ?w}z#N-kvx}fPs+v@}ijoD77#B09auI0Bag)3Jhpr zZe(wDbS-mvZetP<0izgR3tWFjiZH8+9!#Gwk$E2j@&PB2n;RSqR&+y?R`+J92ADy2MtN%a#r<2><{9 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000075 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000075 index 58749cf8e5795d6c03ae4d093f0b38817c0f4220..3ad9cd163812e1a2b0a1b00ee2c0b20ffd6f01f0 100644 GIT binary patch delta 142 zcmcc2c$IO2d1|>vp23RV?LRJj+|)LM(PP4*`M*Ccn#ruORsSzTL?JUnu2eP`Pex*1 zN`7gHZgFK^vXB7dVu?8Jn0mJ^gVkMfKJyI}7&RCTU>dE!(tKVejx#?=Co;0DPME}` pIdP7(98Xw@^W?DaX)D=gxGFHNId^;u&$k)b`%3vgFdSIw1pp(pIL!b6 delta 140 zcmcc0c$sm6d1|Rfj=|c)SN}Cl*tp~_qsN3r^M8L@G?Q6ltNvexh(uE{X-Qvo;WFY~@#S(E`NfpZN8VQ^D6XX~a7&RCTU>dE!(tKXU=G!MFeT-kw^5PSt n`ouZXGQ0sLjx#?=Co;0DPFSORnD_avuPxsdK78kx;d2xK#C18m diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000076 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000076 index 9e6423e7f3089482efe152a59ad795815ca5c125..5beadc188bbd29ad56511311e92fde7811f28882 100644 GIT binary patch delta 132 zcmV-~0DJ$`0n-5sL`7IyL`+Xd0RTXe3nobu5CIrYAay8gFt>@=xa8-9mb(E+pu?{F z@}ijoD77#B09auI0Bag)3JhprZe(wDbS-mvZetP<0izgR3tWFjiZH8+9!#Gwk$E2j m@&PB2n;RSsN^>{2l49{+poHiPp5}Wn@u%OB?Y$u@O95(Z*)tCS delta 133 zcmV;00DAw^0n`BtL`7IyL`+Xd0RTUd3noYs5CIrXAa^KkF#4#Rz}>LDh+#;e!>;@C zqL~9IwJ-euSaAaYZ5nF|3}|6)WN&qJEpvHpV-gSnqZnQaVt6BmC}6S=U>*RGdLILl n0x6N48y*c%bU~T&8({$lC6FKjMSRg8rM;o(oT=n5+OS?PLrE~0 diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000077 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000077 index dde473ee3ca253f06c37fb283e4b1eabf692246d..4843562453b7f9a3cf0f277f685e045ef364b570 100644 GIT binary patch delta 181 zcmcb_c$Jac#WfBk-=diw_3fh0HcJTLb*nsL37i>h9AGzJZ@z4*zjcU?@xB91$z-mS}Eb(fsad;Bk->fBk-=dix0-5 s(wsP7R7DCbDPUK@wc^X+@Kx8piz+bR$zWR}8pa{Vc_Lgid)Y)G0O|cQasU7T delta 221 zcmV<303!d@0hIy^L`7IyL`+Xd0RTaf3noky5CIrZAhrQ0tpPB*RG zdLMa$0%i&k3Jp?ab7gXNWm9xza4j(q0uTWh1S(5JNmEckR6|H#R6$cqUqw<Usq33OGQ#&K~_OYOhH0SMLHmEZy;B1a%*LBAYo%;WpH$5WFT~CWgv86 Xb8A2W&BsatW$-XBk->2yw_3f30HcJz!Zt>YwTuSWPtACKa@oA!WgZWXzyAGc z(M)EIt@?i%A~IPSRx?ViVC3S-NX$#gFD=n6uFOjo5@1{`5yu@<@786ox=YSyzJUUx z2BQH?s})#USf@g8r{U}3tJX8ZnwZkK1i5&FQj1fI%2I<%QVVnqQw5lS4q(ypcJT}j za13$wh!1fL_KtTA3i1z%4{~)3_Vmigyfi^z?Cb@^Q6N$jes<&o9bO rEmBBKPEIW-Nlj5G$w*ZwNi5EGU`$H&6ThSobMha9-15!a&$s~q##mPP delta 171 zcmbQrbexgf#WfBk-=diw_3fh0HcJTLZ(K7!PV!tUM}0!w{ktB$Ab$`|9o0B zlUZY{{$GX&MrMY1saP(ajKsW@{L&KL;>x^aApyq45^-Ef70T@z37hy6Bk->f3a&s1 delta 165 zcmX@cc$tyg#WfBk-=diw_3Ka0HcJTLa9cM!QK}S&i0-Dw`w_~$A^z!{(M?A zlUZY{{$GZOL}rF;sZ1`OjKsW@{L&KL;>x^aApyq45^-Ef70T@z37hy6KBk->fP%V2evoX>oNiPiE}pBUA- z1i5&FQj1fI%2I<%QVVnq1(*bwq`<1Bzm>Bk->2yx0;uT0HcJzLI)d-gc&i(W7B<%im9n zW-@DR)&I*7@spXMPO65BCnGU0CBL*px41GdSxA6!u|ynKQiXE6M#3ik1UZI@mGZv7 z7_GPjxp;$8i&KlrQiDrU3v>+ym;{)l1f}?kAc83h#tQjq3Z`0M1El^GyG+`%?!||@ XY#KgtoL3HrDhAwoyx9D@W&TD0s@FZB diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000083 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000083 index dbbe1407b3849cc137c45f67d1bc87079c28cc76..0a96a2ec1ced67efb220fee905bbc8437c1bf544 100644 GIT binary patch delta 157 zcmV;O0Al~C0_Fh=L`7IyL`+Xd0RTXe3nobu5CIrYAcz4de=xd`Bk->2yx0;WL0HcJz!YM|LgNz2xCLf)D@7vXbLLL*gO#b_6 z(M)EIt@?i%A~@L?_A^TDVdUb;NX$#gFD=n6uFOjo5@1{`5yzEOq1>*Ku!%oGj$vYz ze0>vB6_+3vZ%}G+YEfBga7k)`uAu;v0FwllmVcrGSg=T;D7CmGv8Y5Lzepi1F*7H> zEVW1>Gp{7IsHn7{BsE2$BqLQJv7{t5x1dBJF%2lLkf;!zUz81G6s0DoW|jffWR`e= l9U%3m*k#h5buT{LWz#T7<-26&A^7k5@6)G@Z_mxw1OR2~SO)+A diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000084 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000084 index e67b568a6ca19e6058c659b236c4610009017650..72e711c42527fc06db30a9822e7f7de95e471272 100644 GIT binary patch delta 253 zcmX@cxQdC}#WfBk-=#qw_3ia0HZ{J!YM|LgNz1S-`zj9_CZhkOGb|cR~P;L zv}h)?##a5m3=y1c4Eq_S_AqksWF+RLG{7FXsa3kfhTmWbnysdwu#SluP(Gv7dg zQG?L{rqv28E%&6LcB+@r*WW&K9GaM_CN2~yW0K&~@=sJKPAw`+Em9~-EiOqcDpANU zQbV%YbS! mOT56ANq;MkTd}VH&TPfW3pI>iPBOo^*UD+xcIyD{q(lHShF|pn delta 168 zcmZ3*bc~VP#WfBk->f(mWwANF)t;*v_!YKGA|jZYq3NeS5k#?yGFt${scJ&1x5`<1DHlD zu(Z~kg4(HGMqhvX%yF2+EX^g!!6+rlCCJ4alvdE!(%RE19NO!D_HSDCBk->fPQslg?w1-ga;OajbOV3pd_ikwz&`pDru=Zoh4qrr2p2fdTO MBW7T|T6g1f0J@YvDgXcg delta 259 zcmcb_IG>5z#WfBk-=#qw_3fZ0HZ{J!hS}LJ&Xn)r)-<||NPo{PZ&KKmj3wn zY0*q(jjj5B86vi_GL&%bVwBp>$i zPgTgt%*`zE(wbA@&|d$uf77Zbmjf7C7%C_1I^Qj6UVUWgw`~)@*Z-QC+_fk&?uqGy Gx^Dpd=wea; diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000087 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000087 index 8a2faea17637ad9a3a4fff692fc4903f46d0f936..b4296359ca40d2996a3e8231973183fc275d36da 100644 GIT binary patch delta 241 zcmX@cIFE_j#WfBk->2yw_3J{0HcJz!hS}LJ&Xodn$KVBS+k^}#ACv-_Wz$2 z&1BZts{fZEVk;{{3D+)0sqKthJQ<03Dfy)(y2X`w$wC5*izVW?W9r?y3|4o^`OG&^ zVANnVfN8eU?J8B9mLt{7cfWT-6I1ELg(7*(5*%6{;rT_`sYMD!sfDGf#U%>)X$mF= z3Q3hEsl^JZ70IcoDa8sU8L0}!#-^qwCdNhzxv9m)iRq~dIhnbcC0^Roikwz&`pDru o=Zj_lBMU?2gk9&mCC#gkEd3ViHA7jLBk->f12~!krzBpBK$! z*4V25mmz|YnIT>(mWwANF)t;*v_!YKGA|jZYq3NeS5k#?yGFt${scJ&1x5`<1DHlD zu(YF4soJz0sb;?Wy&EPmOLIwbFiMGX33BlUr52|am8Axkq!x(j8VWE8upHmcFL%k# P*=(NU%&yzNUfux!4}dxI diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000088 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000088 index 75d789e9114405cb955398200e848fef40e89ec1..34acd6eb7170669fcf68a4b7f8e6b22b7f23647e 100644 GIT binary patch delta 151 zcmcb_c#Ls^d3~lvg2BvlkCt~|d(B91$z-mS}Eb(fsad;c}l59j=|gO%lA(|dhv^#$AiDe8$U0a$*i$e|1U#CA~Qp_R3;ZsMq*w{ zerbtrab;ezkO1Rii8!vL3gvcl35; p#D$_tQeX*3q1qeE!Y@jnvszJdCvADn-Sp@PS>K?ZhYLdf0su{7JlFsL diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000089 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000089 index dff3fa00de5fb2d01ef3b413a7887b6ab63d7751..589be962a16c360e0d1294d60a91e3965143c15f 100644 GIT binary patch delta 165 zcmZo>y2Qxs;u;a<;^Xhm$Y4K_TTMeqfKkF%p;RNs;LOI&>wjI}@;=n#!@TQ_pBK$! z*4V25mmwmNnIT&$b7F!>DtAo1TbIG=E;*n11`3QCj0P};RsxKRCE}dfs+u~CqnBO% zW?%4$QJqVWi#I5>IJKxOHMk_TK-W-!Nq|KPtkQ|6MsM$dZzs2y)!uq~$Z?<3>DYyg Mzc_ZC;Cej)0ESXObN~PV delta 231 zcmcb_*v!Q3;u;a<;^Xhm$lx@QTdiJHfKehqVKJk|N=Ad*doIts*L1r37o*3AdDk01 zFPh1$u~q*sL&Q}UhAAA&8Kssma`9v&=B4D9mgp8&<|PXWFfNve<4USfZr4cI#GfF? zpuniXXaLh}<;+&q)L|UG?CLlBf+nUUE0Tu~HC37y> z%)GMHypsH)N`=hiRE5&K#InT9oW!J@R3Q=1Pz^pWN1@sq%fc^8pR-y~!pe{#Fu6Fs dfvNF^vfftqyd-{KyRdEhcc@0a?^M_<3IK5{Q7Hfb diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000090 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000090 index ea368aa67516f426c9e4bfe0900870e5d6b4330c..af1853b563776f492efb256e9f7c90f621a94d59 100644 GIT binary patch delta 230 zcmcb_*u=!`;u;a<;^Xhm$ly4UTdiJ1fKkFO>=>uqKCz8jYpKl@O4#GZM}65spp1_0H(RILC2 delta 182 zcmZo-y2Qxs;u;a<;^Xhm$Y4K_TdiJ5fKkF%p;RNs;Pvdy)6dO1_Q%Pi;lQ)T&x>X< zYi!m3%Mg*s%#baW$;FeAn3s}YTB2KAnU@UIwOAsKE2%=cT_a%=e}Wu?0;2|_0ZgM6 zSlWf5^!2QD1;NK(?sWgesLmzG#T%5GlbTqZ8eEcEplc|=B*3U71y<_JR=i=;nRMYl YrGcMLL_dn{zj*h|YsrQOw%l<-0N<5D{Qv*} diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000091 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000091 index 1da542284daa7c34247e1be9e691147a6bc50bf8..38bec1924dbf214db330fbe86040858608f06535 100644 GIT binary patch delta 137 zcmey$c!`nQ#WfBk->f2FL%0sGBk-=#qx0Bk-=#qw_3fZ0HZ{J!c<0$9!7(QGgqHn+PtQ>o6%!IQ)|=b zMKhT-w(9?7h)85%=wg&=XXN6^NX$#gFD=n6uFOjo5@1{`5yu@<@786ox=YSyzJUUx z2BQH?s})$<^-CRBzr{jb+lLE0yO>Bk->Q)x0;g}2crO^M4-Y=MvWJY22I^NzCAg8yQ9EkK~rnf z=S4G_HMZ*iWr%2HV_@NV#>l}a^^lQ^udp<=G&M!HxFoTtBsFDXomjo(UNglxLLV0I zTEw`Hk;~;*9aq1_LS5U33p|DFg*XF2>fLhk%e6QH5|guoOHvEGm^>JxM0BOhIVSiC zykM7T00KsZh68==X9YD<41hcii-raVhY5ZjFW3V>vMdo3{1QOo89;(bi%EjfN36nVPmB75)myh1 JEh+J=0RYgvT<8D* diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000093 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000093 index 81f8bd80fad76de36dcac4b517572093f1bc029b..dc9847acb5d472d3739f093ac26d0f835f5174b8 100644 GIT binary patch delta 259 zcmX@ZxQmI~#WfBk->Q)x0;g}2crO^M4-Y=MvWJY1|8jvf6vWpIA86tVeyov z&x>XLDW+Utwu#X=;jYaYaINprn}+xh7LBk->2yw_3dj2crO^gugBk->2yw_3dj2crO^gug?# zn#ruORsSzTgd#ITG6$nnJQrVKX=-U|if(a9Vo^zIijV-~Vu?7{FLjc8%@pSdeOSCp zfl-6e0H)arEbacHLTpdfp37^uPdqV;Nt%m~iz^^8IXk!{wLsTUK!you`-a6+z_#@i T-+Wb3+2vjIS@ne8kLOhYps+)J delta 272 zcmX@ZxR;6B#WfBk-=pmx0* zH+^0-lUZY{{$GZOscZ}(JkJ<8n4}&ua`6?Grk19r=$0pDmSpCoPplWMubI7cPvP0m z3qEr#VqC|_<@T;ZY){pm%WJn!JRxK+#2EBk-=pmx0 z&FrOn3eSFC@R>`RxFof}i^+pAN~E|Xv8Y5>%A8|@pTG-t zi3T8GRA@NR*M3$|BgFv7Bk-=#qw_3d@2a^D!M1VrBMvB3)2YZ{Rc3(I%jnU)6(@jmE z7tLhW*sA}RAwrRvA(?|oDxQn4ur#$aHAS~PF|#ByFI`B0aj`_4+q(*}Jym-yuiZZJ zgaV@mqXA5_6{k-5a*Ti~Ji^Bb##T_fJGwc?f u#U#ze$Hf(pn4BG4l3Ji^C;+nS!qZJ)vlFymX#)Uk z2mu;s3J!5~Wp!m_Epv2Xa&%>65)c8S7+ysBehQB`p)Eb(ph=N`6+v^coB&j0B2@Yq z@&P0Y4hjlTVPk7kbY*ZYF%SX}0U8hiqZnRD;&H!;bBL+e0J|04F8dJIqmc>@4$2mu;u3J!5~Wp!m_Epv2Xa&%>65)c8S7+ysBehQB`p)Eb( uph=N`9|MvCCXt~S9!TPGzln2*sn-Cz6(9i>86Oai`KY9CZzt^>Uz}3c1vE$i diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000098 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000098 index 953f68a9290332c39ebd19e8eced23d82657f07c..992bf39a0cc54965d2e4d2769766b224eaf3ce56 100644 GIT binary patch delta 141 zcmV;80CNA$0oMTwL`7IyL`+Xd0RTaf3nnZS2mufQ7*8O0C~q*>?uLlJzQx%QNPx-U zg7c!811PmG{Qy{b0|0Ia0UBzNUKR{R^nMw=HX)o6@T0nsfFC)M0wxL$3JOqRV{21% vWpFJq5CRYZ8V~`a7+y)-bFrKNRAeGl`WPSq*{FTz*bl=sn4Tc%b$vwpPxCU& delta 149 zcmcc4c$Sgd#WfBk-=#qx0=5w2crO^M1VrBMv6iIqlc@u?Em=r1*1p9$p=lJ z7tLhW*sA}RAwrRvA(?|wDxQn4ur#$aHAT0$B(bO@HAP5(aj`_4%da}Fev5^=whtG0 zPOO*Lo5du}#mB`JkeHkuT#{O#Ybd}3v|>S1E7(NF%9N(_*J^W4WKLP|s-eCa02`Dx AfdBvi diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000099 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000099 index 4ae0788d7f05e229290f4547be7bad952072bc53..e048972c455114cbb89ba9047366fd91ad93dd4d 100644 GIT binary patch delta 137 zcmV;40CxY?0n7oAIYwZ%0izgRMf83dy*44767ZwCk$)99eDkZ%0izgRMEZUTk2s+%J>j59k$)9IcB;K9f(SsmV7u}GBnl1+ z3Q%EVYg2S(a4j(q0uTWj5CNkYUQ5t?^SEa<^kr+|X@jyLu}fmFPCvUoyGcS)W%I~6 AdjJ3c diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000100 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000100 index ec67c2da14f04b7aafe83e1095ffbcb5d133c003..89c592c5ea23b3aef2fa915cf40f6d9e01c564f6 100644 GIT binary patch delta 185 zcmX@ibd{0Y#WfBk->2yw_3dj2crO^gug&SX7dlA|$}LSR&5#OP%CiGsQVV9~SRY zVANnVfN8b@OZy%v+quJZu2<|CKI2b}DqMVATmgy6*})~L1-ga;s!RfmQebsHr^;6C d)oS8!*p;yR-rWoBk->Q)x0;g}2crO^M4-YWMvYI52Ip=~T>9+7&dF>Z3%dU| zeO@$^S!1jIUxtXYYz$L)J}`1HO1);};>$?POUW-S(Jd)TEKALqST9;%_jH;*+o^fE zea99t?qKBdIaRiEhv{6e*fV^_LY_k09&S1LBk-=pmx0z45$AmnExINr*^2@b2f-^D;f=f~hytq6Vv)Bp}i;`1w zqC|>I5{pW7rOY`d_zApVmuLV2Mumn0eeGujHBtg3-oGK>Um5oDcO6Y*ycvUDec)E#Sn(!7PxXzzC9NVR)o9U+l&LmXO)k YTbMe(DROUrw{|AWfvpY68%_QK0LC(Bx&QzG delta 185 zcmX@Ybd{0Y#WfBk->2yw_3dj2crO^gugi=bk$Yf^7;b4@?;Nr_j%uC5HEzvD0N-Rsw5fWfrED`5(s%+;D)45);XZVa2 z7&RCTV4AJK(ti6&PHc3Q+n9R&NyR5d6)rw5uHcN!g5Z+W0$oD^RVD#ODX==Bk->2yw_3dj2crO^gug9d`hm)mzt zfl-6e0H)arEbYIevS5vPa?To_8DgIpRk--LxPmh>3xZ2h3v>+yRG9=ArNHX^_LZF2 d=qk4{_4<>FCDQd<7;c^4sj>D}hwUHTFaSPg=#WfBk->Q)w_3dz2crO^M4&=1qsC-LgIj01KkV=N@=?g+z}dOY zpBK$!*4V25mmxxxg`tFfA|nT*R1YH;Uq)hHN`7gHZb?yMS!#}u0OMkbIG9sX) JuN~?72>=&2PQm~H diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000103 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000103 index fddb8b0b72576dd028deb5c233f44192e419a5e0..083aa4bcf00ff43fb32c1ed80b22c4e5b9f4600e 100644 GIT binary patch delta 169 zcmeyy_>OUcrDiXq#$-l=?=PB8-o4+up~BoA;p$nlwX>cqQqOATB56KC?!xD(etzWkVyY-+4{MfS2b5KGrUeoZW5fcFObW&Kn}jUGmW={u)fIQXJxLY>EjcWatIFPh1$u~q*sLqsAALl+|lqf{#+ z7hgtVUP^vxiEc?zVp(dAkO1Rii8!BAWjlA6&h?5t!)L6(sKICe({2Tp4%k$;YW>aF z7K^%R6S|ljx%jxaf-^D;f=f~hbPWZV1Q?a<1cms8csZD*IHlN&fl>vjDOw!m`9;~O oMY@SjU_*S5lr8Gm{FmjDYst14m8lA~zvaGii{Faca(Lln04uCXwEzGB diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000104 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000104 index d7db0fcda2da7582b32b6461a53897367cf5a0f6..db5ef2b43ef6ae5dabb3cb80350a4e8e88f475ae 100644 GIT binary patch delta 199 zcmcb^_>Ga<#WfBk->Q)x0-<%2crO^M4-Y{MvWdugSS^(9!}f(Z)b?dgZ@3u zpBK$!*4V25mmwmNg`ta)gHfuLabluaz3-8_r_=P=PR+~hJEp*>!Ds+eY$d?BSRyWP zO?CF(TvdaznNdf(m>jwIxVVBdG7ExBQVVnq1(*aFl>~+Og?Krbr8uS7i-9r)sVQ0< t<@rU~sYSYpPGB?q_LZF2=qk4{_4<>FRJ&s{W;f_}f98u{tG{`T5C9RZM$7;J delta 173 zcmeyyc!!bO#WfBk-=#qx0-<{2crO^M1Vq#Ml++qhLtm4&f5NKHHXK8{yoi~ z7tLhW*sA}RA!05wLke3XBL|~Y?ZiZ}RG(92J9n7Q^@=^iXRN@e!Ds+eZ56ntI(u)f zszKSzsH2aVExAl3n3TARGcpSl64Mhi^R$EwEet@w(Adzx0cbHZ!{Z|}SXk8bwz7ZU U9oSs>?T$$PvNJ7T_~xAh0Dmw(Qvd(} diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000105 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000105 index f48a5836b1e4c5d04534195864ec5e1377ab4133..025605ca829afb862a1434d4eeb4e868a7c51f34 100644 GIT binary patch delta 177 zcmcb{_<)hy#WfBk->Q)x0<0C2crO^M4&>wMl++qws+rt9G?Dh&2C1I4;xN5 ze_k|`S!1jIUxtXS%nVc58W}klrD`X}iDmd6se3w2pY7DV+`eN9j2esvFx6H;%POa> ze!lj>#gZMjA2C~UnMyDzaTRA|7APd9CuZhp2^(4%fPkT~p^>S9fdkNTW`->+=A1ix YVz{`%xMdp!KfGe*7UQ3AbXge-00d<}Qvd(} delta 186 zcmaFBc#VBk-=#qw_3d@2crO^M1Vq>My|o^xBU~2Z+$vvBcsQM4X2ww zFPh1$u~q*sLqsMsLkBk-=#qw_3d@2crO^M1Vq>My|o?N9U(@KYMxX6{APPrCZIP z7tLhW*sA}RAtIBRA%}xeDuatJBQY-}zqCZRq$sg0HAhH*aj`_4?~%Hv)AZR+&CBgO zrogDdXaLh}1(pt8Q1^B}pUKYAcW-iQWpXWHLJ$F? h7+z7Vd6ui^t>DpgxZ4aKtXB(~w`BC?FRw`PdeqPOKYIWG diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000107 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000107 index 954c5fd0131c3bfaa333bf55e1d15b03f2b16ca6..bd0dc358973aae7012c14bc5c30891a5c5a458d1 100644 GIT binary patch delta 196 zcmZ3*_=l0(#WfBk->Q)w_1=G2crO^M4-ZKMvX~~22Gdup1bzs{aP-M37=m! ze_k|`S!1jIUxtW@EDRGEIT)q78M*i}67y2>OG|W1iW19GbA$vK7fZzX9;tgeO`q-5 zyxhKH6U*eIx|rO#__(-&GcpT;OHvDT4F#A47?s3@M1_Pon5Fooc#DBD1*s_tiRp=% wd0HIh`9;~OMY@SjVAFz@RadWMUleHjP$b3djTkfI;x6q)9RCBg9#(G$022T~$^ZZW delta 141 zcmeyvxQdb6#WfBk->2yw_2qL2crO^gujBfhLgdO_orV?c-Zr!-ebb&m(8CS z&1BZts{fZELXe5Uk%LjnmP4 pu|!Fv1oUw!H77G&xBSX325C8IZt^A}F*SJLLH2~RmGqC^w diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000108 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000108 index 1834b86f4bcc0de7a6608fab32f6fc5f2f6b8ae6..9bcf91797502f6d5165c68602c02aa393904b1d0 100644 GIT binary patch delta 123 zcmV->0EGXz0;&NEL`7IyL`+Xd0RTaf3noPr2mufQ7*8NeC_*sl;JCYzyojJPNT8JY zgY%-911PmG{Qy`I0suh>0UA9D7i494EpTCSbZK;HZ*DDPZ*6U1Ze%Smk#H6si<9h| dvt2-EW!eM+2m#CXoDBk->Q)w_3dz2crO^M4-ZTMvaGz2FE&HUwCk8*4s#r1yg=D ze_k|`S!1jIUxtWMHij?U_Zc}DrEW2DaTlj17A0rsrsd?9D==y>8o<0#g2;@S21#h%qs1k{ATv%ScmjfA!{M7U^k#fEl$t8l+=p);F8n=FD4Jh zD8b^A#G(>iDRYhqegZGpB^rQ$QK8{LU;9}>jT8eQkHey&!NFmIpT`UK0FW$8#00+t zkaz}=VA5ieV6+lA@p`#Hwf|>r(~T1g@7wCWcjDq;7D!QG1j({6xNmhl`c&?r$TQQ0 W_k`r-c4sMbKA9z)?R7wD*((6m(r7gR diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000109 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000109 index 9e6f7e0703931cd46adb3f0b5159fcac4786d7a5..e4c19f20c573692f18fb92dc9b947a5a2bc42c98 100644 GIT binary patch delta 285 zcmaFGxPyt?#WfBk->Q)w_3dz2crO^M4-ZTMvaGz1}``Ft-8JE>+Lj;4O=@} zJ};Wdtg%)9FGEBr8^agw`-~inQnwhnxQkO0i;^>R({l356&N)b4Pa`m1Q-`f#D(>h z**C2>p0Rh|K95C=s~EXLr<4}0W}4i*{C=^ckhKt3up3aL7N=)kN@_)Ya7k)`7n28L zlwfg5Vo`~%lsU%)KYBfnL_ic6GJ8^L^3#2G8f@E13+_yR&eJb}*Bk-=pmw_3e82crO^M36!Uqed^I!Htd?dv^Vp@Z%$+$A+yP zEuR<7WY*ZK|Cb@+FEaxRdp9Eoqf{Fs7k6=LVo`F2Zdy)$xsU+kVu`qrS*6{R-^|_| z>yVzRz^K7!0Ml(1)>mfVwBC5e-hKN#b}_ng8A^$Av6f_(gZJ&Bw%lBk-=pmw_3e82crO^M36!Uqed^I!J)6yI_AHayJ`xf$AN3p zTRtzE$*i$e|1U$tUuFgt_HITFMyWPNF7D#g#G>R3-L#ziav=f6#S(F$Q%Z|gGfi$@ ze!tjJfl-6e0H)h2yrals@vKXRGCA=|yBJ-$45dW5SW7ZXa#DqOq_`5Tl5$d&faaCx z>KaN3$Rs`fc&M!+D0Cy|+c)9efh-I*tecGguBk->2yw_3dj2crO^gugw1tt|#WfBk-=#qw_3d@2crO^M1VqwhPT1xi~r{y+_iuDSw@cw-w$u y#>EnG5lw}EHdgOByYtt&Etgohf!1u;+5tAxi+l42pJl%$EQnpx#PYfPwg&)t$USNR delta 273 zcmdnOxR#0A#WfBk-=pmw_3e82crO^M3BO1MvYsH22U0(`}KL!q!$wyJuZA- z*7AALOlFO(`hOWBMA;ZjxNk6WFiKr! zfwfE6Qez&?D1EYM!FnYl!OCYA_nWlv@cfE|!Rktf^Al=BA^&?PcbdMT`p=xgwg% zQ|1KkW_WNRW`~fW5VwaLP-~P>aYBIXh!LiMGNK|-gZ0e#KplZ okfOi{l4W6VtGtvXF{L5v^Q?f_s)O4Wd%5rYoK~qhGtKH20Gm6$k+G4?H+f zZ_)wd1ti=U6{%jx_x@{HELhP$pUY@Z8rjR9E%aJqNpABJyumvUKROW14FJKz&0H|L z+^|=g1D63*j44ZjeY6)a$5d&xnAy*rsxffNy^m@v8O+~ zBZ%d@ue~9j>3SR-`&(&%^-$M4v0ln6ahWVCsNkSNxX76B0C<43Xs9|?vX}69MC65J zWrC4{8Er^V6g7mr4W|RUD$-!7>$P(uurjCXX)+%H1ON`nNK8Sma|viqTxt*3lZwMf O@G~HFLSKuMf6tR$ua_$T diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000113 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000113 index c1e5d9474163a59c002dcef375e6f3b5d5de2fe7..a8e81f9b5cb7d15e82cf0180c8580c9b6adb77b5 100644 GIT binary patch delta 316 zcmV-C0mJ_P0_*~iIe&`+D4qf^kIa_y_Rh_#a7ggE#f0;snFA=bFZ}>m6$k+G4?HJ?c7Nrr8q4?WEZ+R?Zlv1; zF=qRprIAUphAL%x?qdpj$(PpFxH~_-7vI3xMKt!M=gVpN+~ss#E5U6V=x delta 335 zcmV-V0kHn;0{;S#Ie(r4D5?T5g1+OR`M$E&J4o=k#f0;snFA=bFZ}>mtq1_N&_Fsc zB_|!!BXD4as+Knh_#z)QnEXi$ti~B4T@<ZmX{&^Zay^0~|Pa^IWXA6S$0RJ+8Y{_Kt?SGa=M#(y=l>-VC<9>yKUYKcRgcS3U9X+pf#q3Ykif$d)AGB^^>QCIy+^Pn z>8ZAfL;cwuODx}g?p*;@7(BV`Z>0kkY{4Ln1$7`yZ+@R{q^2oi&GtV_Bd25~Rm$|; z#~SvMFRjgWcYb_SJOyhn(b$)sF9T2@92bYt$i&KJU{Nz;qR<}Bk-=#qw_3d@2crO^M1Vq?My^5gt)s8k9NKW=1EWX7?JF&x z7tLhW*sA}RA>u6)0}E>o2cuL57hgtZaY=qrrEXeIez~r(kO1RiiMa5N!ap0U_nh7N zYuy$FMh!*-n0Bk^f|{udmp*pgBbhsC6{8B5l9WPmYGO`miUK1ugK^6Yv4j{^JyzBP UZ;=^N7hSuwnv#wpLVo1QUj delta 135 zcmX@cc$#s7rE0cDl0o~M(;uHKJ~i<%qsN3lk6S)3n#ruORsSzTgdj6RA_t>X?8F4I z@bHepKO3v}oZb0r-4+E#4Mqc)QY!(*#S(EbS#@Fy`J#DNMu@mhVwL1z6kw4Oh+k0f m$Lh?ZlSfRMneDC8UMkGzSj{Pv^zdN*g0<37_Y*{aOalP2C^;Me diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000116 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000116 index 2dfd30052bd931513911a5a9049dc3eb0741d9eb..4240617f3d53e39710e96994a5555ecf9e69e203 100644 GIT binary patch delta 175 zcmZ3=c$$&h#WfBk-=#qw_3d@2crO^M1VrJMv}p&uA4U=^>%;#!RWDI?&p@z zi)J!wY}NnE5FyCSkjTL(70bn!ky%`lUsS1^mXlwuYb+$dxL6`CqNzM(PVjDq2Nz;? zC@^X;8o)GLfu&

RXp&-WPN_lDc6Mt0d4q7Ab+4^jwXrC&F8#R~+PCd+oaD9)$%Q VMe0VM3q8I(nsx5=zj@2`SOJ8sL!06iKz3KwK$cr9>Ya&&2QX>V>VV{dJ3VQyqCHIaZH6XFB_a_+%+ P<75_4-P>oVuH{QuWz#Cq diff --git a/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000117 b/server-zd/testdata/v1/objects/log/node-a/0000000001/00000000000000000117 new file mode 100644 index 0000000000000000000000000000000000000000..8be935c078cf1c047c17542606df1b65f1d4ca6d GIT binary patch literal 164 zcmZ>9jfisb@porruxDUk;1Xb5ED;mHQ17?Oc51Mfz?ls~Yv zLIR8uz6$Odb_SoHEZ+C=#QjA%9vk*GwSHbSlUZY{{$GZOkBkhqQr2AJDXA5@1&Ku^ znI)O|dAiB@xw(mXDY~XWa~KU^I;=nz#>K@|UtB1{;MV>kE#MIg!?VXrE_tShd|F`R Ko^W($*B<~O|2%&H literal 0 HcmV?d00001 diff --git a/server-zd/testdata/v1/state.txt b/server-zd/testdata/v1/state.txt index 3c17e741af..f62c379312 100644 --- a/server-zd/testdata/v1/state.txt +++ b/server-zd/testdata/v1/state.txt @@ -13,37 +13,37 @@ flow 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f1900000000000000011600000 flow 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f1900000000000000021500000000 =08f70110011801200128b09799d8f2a29903300240025001 flow 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f1900000000000000031500000000 =087f20012880cfd9daf2a2990330025002 flow 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f1900000000000000041500000000 =08890120012890f0e8daf2a2990330025002 -flow 0808577b47782b9201000000000000000068616e646f75742d73796e63 =0a3a0a1001a3185e0a627823872860b20f601e00100218e0bef1faf2a299032090e1f581f3a299032a1273746172742d68616e646f75742d73796e63 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0001 #267:1121f4afe8c5e6da549ba88fca342483 -flow 0808577b47782b920201a3185e0a627823872860b20f601e000200000001 =0801200128e0bef1faf2a299033a0101 -flow 0808577b47782b920201a3185e0a627823872860b20f601e000652656c65617365537465702d31 #125:a34741264c64b5ffcea7aaf1166bb906 -flow 0808577b47782b920201a3185e0a627823872860b20f601e000652657365727665537465702d31 #95:e926de9967853b70492a434bd8d7b18c -flow 0808577b47782b920201a3185e0a627823872860b20f601e000752656c6561736553746570 =0801 -flow 0808577b47782b920201a3185e0a627823872860b20f601e00075265736572766553746570 =0801 -flow 0808577b47782b920201a3185e0a627823872860b20f601e000a2a00 =0a022a001090a18489c0b49903 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0017 #67:d06b71dc812a9607cfa5b3943a5783cd -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000011500000000 =08f1031001200128e0bef1faf2a2990330013a200a0d52657365727665537465702d31120b52657365727665537465701802200140015000 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000011600000001 =088402 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000021500000000 =089a01200128f0df80fbf2a2990330015001 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000031500000000 =08bc01200128808190fbf2a2990330013a1e0a0d52657365727665537465702d31120b526573657276655374657018025001 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000041500000000 =089a0120012890a29ffbf2a2990330015001 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000051500000000 =08bc01200128a0c3aefbf2a2990330013a1e0a0d52657365727665537465702d31120b526573657276655374657018025001 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000061500000000 =089a01200128b0e4bdfbf2a2990330015001 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000071500000000 =08bc01200128c085cdfbf2a2990330013a1e0a0d52657365727665537465702d31120b526573657276655374657018025001 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000081500000000 =08dc011001200128d0a6dcfbf2a2990330015001 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000091500000000 =0890011001200128e0d0e5fcf2a2990330015002 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0019000000000000000a1500000000 =089a01200128f0f1f4fcf2a2990330015003 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0019000000000000000b1500000000 =08ad01200128809384fdf2a2990330015003 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0019000000000000000c1500000000 =08f001100120012890b493fdf2a2990330015003 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0019000000000000000d1500000000 =0890011001200128e0e2d9fef2a2990330015004 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0019000000000000000e1500000000 =089a01200128f083e9fef2a2990330015005 -flow 0808577b47782b920201a3185e0a627823872860b20f601e0019000000000000000f1500000000 =08eb01100120012880a5f8fef2a2990330015005 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000101500000000 =0890011001200128d0dcb881f3a2990330015006 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000111500000000 =089a01200128e0fdc781f3a2990330015007 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000121500000000 =08fd041001200128f09ed781f3a2990330023a200a0d52656c65617365537465702d31120b52656c65617365537465701802200140025007 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000131500000000 =089a0120012880c0e681f3a2990330025008 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001900000000000000141500000000 =089b03100120012890e1f581f3a29903300440044a005008 -flow 0808577b47782b920201a3185e0a627823872860b20f601e001b0000000000000001 =080110141801200428013090e1f581f3a29903 +flow 0808577b47782b9201000000000000000068616e646f75742d73796e63 =0a3a0a1001a3185e0b5c7f468a30ab8a1e4c9f30100218f0df80fbf2a2990320a0828582f3a299032a1273746172742d68616e646f75742d73796e63 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3001 #267:77e78141f0e76bea745920c7d239c113 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f300200000001 =0801200128f0df80fbf2a299033a0101 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f300652656c65617365537465702d31 #125:eb4a349009ba6d118508090462f25fc7 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f300652657365727665537465702d31 #95:4f2a8ab6fd668a193b3648036a94f988 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f300752656c6561736553746570 =0801 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30075265736572766553746570 =0801 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f300a2a00 =0a022a0010a0c29389c0b49903 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3017 #67:0e1d5173bad3a09e086a156754a05839 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000011500000000 =08f1031001200128f0df80fbf2a2990330013a200a0d52657365727665537465702d31120b52657365727665537465701802200140015000 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000011600000001 =088402 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000021500000000 =089a01200128808190fbf2a2990330015001 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000031500000000 =08bc0120012890a29ffbf2a2990330013a1e0a0d52657365727665537465702d31120b526573657276655374657018025001 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000041500000000 =089a01200128a0c3aefbf2a2990330015001 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000051500000000 =08bc01200128b0e4bdfbf2a2990330013a1e0a0d52657365727665537465702d31120b526573657276655374657018025001 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000061500000000 =089a01200128c085cdfbf2a2990330015001 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000071500000000 =08bc01200128d0a6dcfbf2a2990330013a1e0a0d52657365727665537465702d31120b526573657276655374657018025001 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000081500000000 =08dc011001200128e0c7ebfbf2a2990330015001 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000091500000000 =0890011001200128f0f1f4fcf2a2990330015002 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3019000000000000000a1500000000 =089a01200128809384fdf2a2990330015003 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3019000000000000000b1500000000 =08ad0120012890b493fdf2a2990330015003 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3019000000000000000c1500000000 =08f0011001200128a0d5a2fdf2a2990330015003 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3019000000000000000d1500000000 =0890011001200128f083e9fef2a2990330015004 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3019000000000000000e1500000000 =089a0120012880a5f8fef2a2990330015005 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3019000000000000000f1500000000 =08eb01100120012890c687fff2a2990330015005 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000101500000000 =0890011001200128e0fdc781f3a2990330015006 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000111500000000 =089a01200128f09ed781f3a2990330015007 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000121500000000 =08f804100120012880c0e681f3a2990330023a200a0d52656c65617365537465702d31120b52656c65617365537465701802200140025007 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000131500000000 =089a0120012890e1f581f3a2990330025008 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301900000000000000141500000000 =089c031001200128a0828582f3a29903300440044a005008 +flow 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f301b0000000000000001 =0801101418012004280130a0828582f3a29903 flow 0f0b8625e5f9724701000000000000000066756c66696c6c6d656e742d706172656e74 =0a390a1001a3185ce75e7c9ba774c48afeb866c0100218c0afaed7f2a2990320b0a093d9f2a299032a1173746172742d706172656e742d666c6f77 flow 0f0b8625e5f972470201a3185ce75e7c9ba774c48afeb866c001 #263:badc593c1169517a4191d8d6ed8b5d0d flow 0f0b8625e5f972470201a3185ce75e7c9ba774c48afeb866c00200000001 =0801200128c0afaed7f2a299033a0101 @@ -122,72 +122,72 @@ flow 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b1900000000000000081500000 flow 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b1900000000000000091500000000 =08b7031001200228e09eafdcf2a29903300940094a005007 flow 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b1b0000000000000001 =0801100618012005280130b0bb81dcf2a29903 flow 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b1b0000000000000007 =0807100918062009280130e09eafdcf2a29903 -flow 55e03d097c903c0b01000000000000000068616e646f75742d74726176656c =0a3c0a1001a3185e4cca76b9b8359d4a5dcc0e33100218a08bff82f3a299032080d2da83f3a299032a1473746172742d68616e646f75742d74726176656c -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e3301 #270:03585a848d6ae68dedfeb7450725a97e -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e330200000001 =0801200128a08bff82f3a299033a0101 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e330200000002 =080210011803200528e08fbc83f3a29903320a7368697020616761696e3a0105 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e330363617272696572 =0a0a0a081a0667726f756e642003 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e330653686970537465702d31 #65:ece9e378915ef7c9e45707298d67471c -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33075368697053746570 =0801 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e330a2a00 =0a022a00108092e98ac0b49903 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e330d027365742d636172726965722d31 =0a1075588cf98bc2148fbb1d7f9d29d529a812022a001803 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e3317 #69:5e148b5f676e1132c39e169ea52f2c43 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33180000000000000003 =027365742d636172726965722d31 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000011500000000 =08bb031001200128a08bff82f3a2990330013a1a0a0a53686970537465702d31120853686970537465701802200140015000 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000011600000001 =08dd01 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000021500000000 =089901200128b0ac8e83f3a2990330015001 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000031500000000 =08ea0110011801200128c0cd9d83f3a29903300240025001 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000041500000000 =088e031001200128d0eeac83f3a29903300440044a005002 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000051500000000 =08f3011001200228e08fbc83f3a29903300540055004 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000051600000002 =08fa01 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000061500000000 =089901200228f0b0cb83f3a2990330055005 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331900000000000000071500000000 =089503100120022880d2da83f3a29903300740074a005005 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331b0000000000000001 =0801100418012004280130d0eeac83f3a29903 -flow 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e331b0000000000000005 =080510071805200728013080d2da83f3a29903 -flow 599e1f765cf66c9c0100000000000000007365617263682d666c6f77 =0a300a1001a3185e549a758b93ec9bb35d406765100118a094f983f3a299032a1173746172742d7365617263682d666c6f77 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d40676501 #188:a246baedcf3e744ab6f3f5522c4c12cd -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d4067650200000001 =0801200128a094f983f3a299033a0101 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d406765037469746c65 =0a0e0a0c1a0a73656172636861626c652002 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d40676506496e646578537465702d31 #91:4b36a470aa712a900b2f25c6342fe4c4 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d40676507496e64657853746570 =0801 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d4067650d027365742d7469746c652d31 =0a101c62e3f1c286785255b109edec578b5112022a001802 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d40676517 =08011a0b7365617263682d666c6f77221001a3185e549a758b93ec9bb35d4067652a0a536561726368466c6f77300138a094f983f3a29903 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d406765180000000000000002 =027365742d7469746c652d31 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d4067651900000000000000011500000000 =08a1031001200128a094f983f3a2990330013a1c0a0b496e646578537465702d311209496e646578537465701802200140015000 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d4067651900000000000000011600000001 =08af01 -flow 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d4067651900000000000000021500000000 =08ec0110011801200128b0b58884f3a29903300240025001 -flow 5fbe9f01fc12f3b90100000000000000007175657565642d73746172746564 =0a330a1001a3185e44fa7e0a8f38a12d3de1a049100118a0828582f3a299032a1473746172742d7175657565642d73746172746564 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a04901 #196:068b8156f8a9d607bcc80ac178eaa8d9 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a0490200000001 =0801200128a0828582f3a299033a0101 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a049065061636b537465702d31 #128:8538a07ae527dee4e2e3fdecfa3e96de -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a049075061636b53746570 =0801 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a04917 =08011a0e7175657565642d73746172746564221001a3185e44fa7e0a8f38a12d3de1a0492a095175657565466c6f77300138a0828582f3a29903 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a0491900000000000000011500000000 =08a7031001200128a0828582f3a2990330013a1a0a0a5061636b537465702d3112085061636b537465701802200140015000 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a0491900000000000000011600000001 =08ab01 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a0491900000000000000021500000000 =089901200128e086c282f3a2990330015001 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a0491900000000000000031500000000 =08b501200128f0a7d182f3a2990330013a180a0a5061636b537465702d3112085061636b5374657018025001 -flow 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a0491900000000000000041500000000 =08990120012890eaef82f3a2990330015001 -flow 7becb50ef3da384a010000000000000000686973746f72792d666c6f772d33 =0a3a0a1001a3185e578871fcb17bbccdb9faaeb4100618d0f7a684f3a299032080dbd484f3a299032a1273746172742d686973746f72792d666c6f77 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb401 #219:7c6fb80138ff4931aafc4e795102fbf5 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb40200000001 =0801200128d0f7a684f3a299033a0101 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4036e6f746573 =0a0b12090a020803109f0118012003 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb40a2a00 =0a022a0010809be38bc0b49903 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb40d027365742d6e6f7465732d31 =0a10336dd9cd2dbf97b13090fe9d0df4230812022a001802 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb40d027365742d6e6f7465732d32 =0a10721b9dc8ca590d12a746945834add93212022a001803 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb411000000000000000200000000 =08930110021803 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb411000000000000000300000000 =089f011003 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4120000000000000003000000000200056e6f746573 = -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4130000000000000003000000000000000200000000 =089301 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb417 #69:8719657ce4974af2f18e217782720c1a -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4180000000000000002 =027365742d6e6f7465732d31 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4180000000000000003 =027365742d6e6f7465732d32 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb41900000000000000011500000000 =08f0021001200128d0f7a684f3a29903300140015000 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb41900000000000000011600000001 =082e -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb41900000000000000021500000000 =089f0310011801200128e098b684f3a29903300240025001 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb41900000000000000031500000000 =08ab0310011801200128f0b9c584f3a29903300340035002 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb41900000000000000041500000000 =08ae01100120012880dbd484f3a29903300440044a005003 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb41b0000000000000001 =080110041801200428023080dbd484f3a29903380142105fa070fc3acce2c9c43503033f3a66e9489f08 -flow 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb41c0000000000000001000000000000000200000000 =0803109301 +flow 55e03d097c903c0b01000000000000000068616e646f75742d74726176656c =0a3c0a1001a3185e4dc47ee5962f06ca9e6d8ec6100218b0ac8e83f3a299032090f3e983f3a299032a1473746172742d68616e646f75742d74726176656c +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec601 #270:ad9da1dfc7e13eeb2a46388fba5b4dd2 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec60200000001 =0801200128b0ac8e83f3a299033a0101 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec60200000002 =080210011803200528f0b0cb83f3a29903320a7368697020616761696e3a0105 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec60363617272696572 =0a0a0a081a0667726f756e642003 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec60653686970537465702d31 #65:db586b4c4b474e6bea3f4ac715a17772 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6075368697053746570 =0801 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec60a2a00 =0a022a001090b3f88ac0b49903 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec60d027365742d636172726965722d31 =0a1075588cf98bc2148fbb1d7f9d29d529a812022a001803 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec617 #69:98fd7b5e8b2d84f550dcb8b4863a64e4 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6180000000000000003 =027365742d636172726965722d31 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000011500000000 =08bb031001200128b0ac8e83f3a2990330013a1a0a0a53686970537465702d31120853686970537465701802200140015000 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000011600000001 =08dd01 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000021500000000 =089901200128c0cd9d83f3a2990330015001 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000031500000000 =08ef0110011801200128d0eeac83f3a29903300240025001 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000041500000000 =088a031001200128e08fbc83f3a29903300440044a005002 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000051500000000 =08f3011001200228f0b0cb83f3a29903300540055004 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000051600000002 =08fa01 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000061500000000 =08990120022880d2da83f3a2990330055005 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61900000000000000071500000000 =089903100120022890f3e983f3a29903300740074a005005 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61b0000000000000001 =0801100418012004280130e08fbc83f3a29903 +flow 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec61b0000000000000005 =080510071805200728013090f3e983f3a29903 +flow 599e1f765cf66c9c0100000000000000007365617263682d666c6f77 =0a300a1001a3185e55947572ab029383a7df7341100118b0b58884f3a299032a1173746172742d7365617263682d666c6f77 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df734101 #188:71cdbbab96cfff473d09ba928d487cd8 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df73410200000001 =0801200128b0b58884f3a299033a0101 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df7341037469746c65 =0a0e0a0c1a0a73656172636861626c652002 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df734106496e646578537465702d31 #91:e009708d653dc7da8de54468b51a6a24 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df734107496e64657853746570 =0801 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df73410d027365742d7469746c652d31 =0a101c62e3f1c286785255b109edec578b5112022a001802 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df734117 =08011a0b7365617263682d666c6f77221001a3185e55947572ab029383a7df73412a0a536561726368466c6f77300138b0b58884f3a29903 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df7341180000000000000002 =027365742d7469746c652d31 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df73411900000000000000011500000000 =08a1031001200128b0b58884f3a2990330013a1c0a0b496e646578537465702d311209496e646578537465701802200140015000 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df73411900000000000000011600000001 =08af01 +flow 599e1f765cf66c9c0201a3185e55947572ab029383a7df73411900000000000000021500000000 =08e60110011801200128c0d69784f3a29903300240025001 +flow 5fbe9f01fc12f3b90100000000000000007175657565642d73746172746564 =0a330a1001a3185e45f47e19bd36219c12f0a3ba100118b0a39482f3a299032a1473746172742d7175657565642d73746172746564 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba01 #196:5c02c1d9d4fdca79a629e7c62930465d +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba0200000001 =0801200128b0a39482f3a299033a0101 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba065061636b537465702d31 #128:9842dc37d010d94754bd251c73d845c6 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba075061636b53746570 =0801 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba17 =08011a0e7175657565642d73746172746564221001a3185e45f47e19bd36219c12f0a3ba2a095175657565466c6f77300138b0a39482f3a29903 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba1900000000000000011500000000 =08a7031001200128b0a39482f3a2990330013a1a0a0a5061636b537465702d3112085061636b537465701802200140015000 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba1900000000000000011600000001 =08ab01 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba1900000000000000021500000000 =089901200128f0a7d182f3a2990330015001 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba1900000000000000031500000000 =08b50120012880c9e082f3a2990330013a180a0a5061636b537465702d3112085061636b5374657018025001 +flow 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba1900000000000000041500000000 =089901200128a08bff82f3a2990330015001 +flow 7becb50ef3da384a010000000000000000686973746f72792d666c6f772d33 =0a3a0a1001a3185e588277649c53bb00e0d05cb8100618e098b684f3a299032090fce384f3a299032a1273746172742d686973746f72792d666c6f77 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb801 #219:34df31e58acc948df33a5ce0e0d60882 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb80200000001 =0801200128e098b684f3a299033a0101 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8036e6f746573 =0a0b12090a020803109f0118012003 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb80a2a00 =0a022a001090bcf28bc0b49903 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb80d027365742d6e6f7465732d31 =0a10336dd9cd2dbf97b13090fe9d0df4230812022a001802 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb80d027365742d6e6f7465732d32 =0a10721b9dc8ca590d12a746945834add93212022a001803 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb811000000000000000200000000 =08930110021803 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb811000000000000000300000000 =089f011003 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8120000000000000003000000000200056e6f746573 = +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8130000000000000003000000000000000200000000 =089301 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb817 #69:431e280624464c023320cbe47eec7c87 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8180000000000000002 =027365742d6e6f7465732d31 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8180000000000000003 =027365742d6e6f7465732d32 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb81900000000000000011500000000 =08f1021001200128e098b684f3a29903300140015000 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb81900000000000000011600000001 =082e +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb81900000000000000021500000000 =089f0310011801200128f0b9c584f3a29903300240025001 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb81900000000000000031500000000 =08a5031001180120012880dbd484f3a29903300340035002 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb81900000000000000041500000000 =08b301100120012890fce384f3a29903300440044a005003 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb81b0000000000000001 =080110041801200428023090fce384f3a29903380142105c676d28d5c85784a8c10badd6d715bc48a008 +flow 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb81c0000000000000001000000000000000200000000 =0803109301 flow 957748ea772a492d01000000000000000070726f66696c652d726576696577 =0a3c0a1001a3185ce1827fd69291cf63846a12f3100218e0e8d2d6f2a2990320e08cbbdaf2a299032a1473746172742d70726f66696c652d726576696577 flow 957748ea772a492d0201a3185ce1827fd69291cf63846a12f301 #262:24f914ad660701729ddb7ab496a751a3 flow 957748ea772a492d0201a3185ce1827fd69291cf63846a12f30200000001 =0801200128e0e8d2d6f2a299033a0101 @@ -228,14 +228,14 @@ flow 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b681721900000000000000011600000 flow 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b681721900000000000000021500000000 =08b501200128e0fac6d8f2a2990330015001 flow 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b681721900000000000000031500000000 =08c5031001200128f09bd6d8f2a29903300340034a005001 flow 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b681721900000000000000041500000000 =08a10120012890def4d8f2a2990330035003 -flow bd8ac614675559d00100000000000000007175657565642d77616974696e67 =0a330a1001a3185e46ee7816bc7abcd3adb791c8100118c0c4a382f3a299032a1473746172742d7175657565642d77616974696e67 -flow bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c801 #177:80f11609a162e139f6e293da3423e371 -flow bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c80200000001 =0801200128c0c4a382f3a299033a0101 -flow bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c8065061636b537465702d31 #89:2528f53f6a7a12f47084bdcfd2c1528c -flow bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c8075061636b53746570 =0801 -flow bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c817 =08011a0e7175657565642d77616974696e67221001a3185e46ee7816bc7abcd3adb791c82a095175657565466c6f77300138c0c4a382f3a29903 -flow bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c81900000000000000011500000000 =08a1031001200128c0c4a382f3a2990330013a1a0a0a5061636b537465702d3112085061636b537465701802200140015000 -flow bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c81900000000000000011600000001 =08ab01 +flow bd8ac614675559d00100000000000000007175657565642d77616974696e67 =0a330a1001a3185e47e87c9ba5bc71cdf3a0f30a100118d0e5b282f3a299032a1473746172742d7175657565642d77616974696e67 +flow bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a01 #177:bd3f67b6ec434f0df3659debe7e5287e +flow bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a0200000001 =0801200128d0e5b282f3a299033a0101 +flow bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a065061636b537465702d31 #89:255402f20b64517d95a2366dbeb7795f +flow bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a075061636b53746570 =0801 +flow bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a17 =08011a0e7175657565642d77616974696e67221001a3185e47e87c9ba5bc71cdf3a0f30a2a095175657565466c6f77300138d0e5b282f3a29903 +flow bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a1900000000000000011500000000 =08a6031001200128d0e5b282f3a2990330013a1a0a0a5061636b537465702d3112085061636b537465701802200140015000 +flow bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a1900000000000000011600000001 =08ab01 flow cef57bddf594cff001000000000000000068656c642d636c6f7365 =0a380a1001a3185d04aa7a8486e4fc3d7b333df3100218a091f8daf2a2990320d0f4a5dbf2a299032a1073746172742d68656c642d636c6f7365 flow cef57bddf594cff00201a3185d04aa7a8486e4fc3d7b333df301 #248:7ae9c4d41898d5f9b52cc5590cee199d flow cef57bddf594cff00201a3185d04aa7a8486e4fc3d7b333df30200000001 =0801200128a091f8daf2a299033a0101 @@ -295,27 +295,27 @@ history 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f150000000000000002 #21 history 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f150000000000000003 #93:a823d7c055c3a44c410c4f7741b7c4a2 history 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f150000000000000004 #103:626cecdcf123127730e2057e75c5abb6 history 00c77b2df1ff0c810201a3185ced3a76f5860d3203b7c6fd9f16000000010000000000000001 #107:8cc6426e16bf9d6ecef4647e39e0310f -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000001 #463:23ea33e479c7fc22c9a5270ee1953590 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000002 #120:8767241857ca2afc92e4f9a9b88fecce -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000003 #154:168547a721f874c2e5ef22ffc99a7bd0 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000004 #120:364d02e467aecec343f3154d695197d8 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000005 #154:3822488612318813e7476108df85856f -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000006 #120:1df8f85d6d33b99df51b4ee1a95e5529 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000007 #154:bfee81509678954a9f8029b1b410a9a4 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000008 #186:1693cdd5999be531f45beeb1e9de7b93 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000009 #110:122f0a431dad54bd5f8d46174848aac7 -history 0808577b47782b920201a3185e0a627823872860b20f601e0015000000000000000a #120:49babf0095b09bf4473bbbd2346e3228 -history 0808577b47782b920201a3185e0a627823872860b20f601e0015000000000000000b #139:c39daa901fda20ca52a2b48c0c6bf8cd -history 0808577b47782b920201a3185e0a627823872860b20f601e0015000000000000000c #206:cbbfa3b93fc20a8f790a69b45f47b849 -history 0808577b47782b920201a3185e0a627823872860b20f601e0015000000000000000d #110:e0072f77d2bc148e1385986d08253592 -history 0808577b47782b920201a3185e0a627823872860b20f601e0015000000000000000e #120:e4f6e5428623b7fcbb3de6fc22c8f406 -history 0808577b47782b920201a3185e0a627823872860b20f601e0015000000000000000f #201:9e2d96e0ab65ea10eab5bbd0058a6d31 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000010 #110:ab89fea8f7723fc220284c3b2fe2ab05 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000011 #120:440340a1f7805630c998cc5305036c3f -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000012 #603:eb08fb237db4a14997a47315697659c7 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000013 #120:f0c145b348abff9fca887ad2642f1162 -history 0808577b47782b920201a3185e0a627823872860b20f601e00150000000000000014 #377:d12a3cdbf46d910cb46642b6fe3e5a43 -history 0808577b47782b920201a3185e0a627823872860b20f601e0016000000010000000000000001 #222:49d2639af4256d5461eaeeced9ed8638 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000001 #463:416d7250cd2e8dbc412f35f796e23f76 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000002 #120:9752d13a3997f3323c001347dcaa97ac +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000003 #154:917b85cde1639721c7a3a9c0dc901c69 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000004 #120:6a580f439f691af918a5294f4f07b110 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000005 #154:ab7f7b39e3fb72e966c8c86ee0a1cefd +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000006 #120:a279398a8ba7f4624d9ad43d6b61abfa +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000007 #154:6e0e75aacbc71125a986d35d38bb03d8 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000008 #186:ec7ff34356df30f45e0e8f6f32bb6e55 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000009 #110:cfb45d411dc454b2259a362192d19ca8 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3015000000000000000a #120:f10cb55a70b45b3ca1bff191d497e59c +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3015000000000000000b #139:33d1f26626f2104085fa05090fca4978 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3015000000000000000c #206:f166fe460a46a103a954bc69d0bf273d +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3015000000000000000d #110:0ca4fc244fa171248558a057ef542b7d +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3015000000000000000e #120:a8d8b7fa500fca6d3122830acb2741da +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3015000000000000000f #201:29912b662c7cd7aea398c86210b8382d +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000010 #110:52b1b77173584c83b86681a4f5aa9382 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000011 #120:0469bfc70a451ffa16948e5f55ab5227 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000012 #598:f45d3b29856f720a53ac14cf365b1a32 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000013 #120:092d24d0d31cc95f1b2a4d193b594364 +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f30150000000000000014 #378:ce01cdf6f014bd28cf85c4286a519a6b +history 0808577b47782b920201a3185e0b5c7f468a30ab8a1e4c9f3016000000010000000000000001 #222:7468bd14b241d32d95d78f6632cd452a history 0f0b8625e5f972470201a3185ce75e7c9ba774c48afeb866c0150000000000000001 #390:a7d2ba6aea3471ba4234b3c58678e3c0 history 0f0b8625e5f972470201a3185ce75e7c9ba774c48afeb866c0150000000000000002 #124:a2505e32abfd925d115ca5bad2ed8212 history 0f0b8625e5f972470201a3185ce75e7c9ba774c48afeb866c0150000000000000003 #441:b963254987de08f15a69b01eb9b301e5 @@ -349,28 +349,28 @@ history 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b150000000000000008 #11 history 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b150000000000000009 #405:2b93ea12a8c08ef09ed610b630bf2c0b history 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b16000000010000000000000001 #137:05e856ee6f8ae13bf63d05ad924b83c4 history 4a90b11d2e1acfc70201a3185d08927201ac6ac5c708d6402b16000000020000000000000007 #300:d0380b9412c4b1c638699dc6c4d3e3fa -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33150000000000000001 #409:0052d540a8b4c0524a743a9c09ec13f5 -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33150000000000000002 #119:7a4067953aee295b96fc1aeda28a593d -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33150000000000000003 #200:9e05b25a08724b6d5ff5175cc1c6d5f7 -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33150000000000000004 #364:a0d944888f0693d75edcbf57873f593d -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33150000000000000005 #209:d11479bf56138d29b1867b799da7e23a -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33150000000000000006 #119:02a13a55854230831ad1339ef22f0a5b -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e33150000000000000007 #371:cca904884bba28917220de26ed821bbd -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e3316000000010000000000000001 #183:df2f4ccfb9f9e620f13dd0a4c31a926c -history 55e03d097c903c0b0201a3185e4cca76b9b8359d4a5dcc0e3316000000020000000000000005 #212:040f300d62ac9bee1437ae15d76dade4 -history 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d406765150000000000000001 #383:22f769ac364e51b810ede53032720f66 -history 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d406765150000000000000002 #202:961759739431bdd646b65bcb15413cc6 -history 599e1f765cf66c9c0201a3185e549a758b93ec9bb35d40676516000000010000000000000001 #137:dea5f6561a892c4e3b30411f6ea0d8cc -history 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a049150000000000000001 #389:a557daac74a0d2c53b3d9f4c35f2937c -history 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a049150000000000000002 #119:7ee4af93e1ddda9f0e865383383cfb72 -history 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a049150000000000000003 #147:aceba48007e2e0311ce8e13ba0fef4b8 -history 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a049150000000000000004 #119:384476d9876db03107970e161b7c687b -history 5fbe9f01fc12f3b90201a3185e44fa7e0a8f38a12d3de1a04916000000010000000000000001 #133:123bd2a3572d3dcd06b99e55fcea36bf -history 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4150000000000000001 #334:536378d3eeec5f1b3609b37e93e946b4 -history 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4150000000000000002 #381:87150dcc7a0a6dc04da7e04f1f8b3fc8 -history 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4150000000000000003 #393:37ed21fb9b481fe92f0268b1da2deb10 -history 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb4150000000000000004 #140:ef6591c8aedcd62f2c89559aba0b0397 -history 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb416000000010000000000000001 =0801100122020801 +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6150000000000000001 #409:6e45a5f412125df4a4f5647355eed33f +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6150000000000000002 #119:97441310a09d8af3b194eaf98c28fbd2 +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6150000000000000003 #205:2e666d8724db18414fe1e346ab427804 +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6150000000000000004 #360:82496e99d12540662a4f452fcfdee7ae +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6150000000000000005 #209:6b8f8df75d7e2dbe6510ba640a5a4891 +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6150000000000000006 #119:1cfd7bfc1443e03fe9fd1e1ec6ae6854 +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec6150000000000000007 #375:e1e10a336c479a3d3a2f52bf4fbe6672 +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec616000000010000000000000001 #183:4625fd4165b9dbd073ab84be25d5bfa7 +history 55e03d097c903c0b0201a3185e4dc47ee5962f06ca9e6d8ec616000000020000000000000005 #212:0196871a50e865f6d8796b0d5f6878fb +history 599e1f765cf66c9c0201a3185e55947572ab029383a7df7341150000000000000001 #383:95571f912afbca573db48246da0f0420 +history 599e1f765cf66c9c0201a3185e55947572ab029383a7df7341150000000000000002 #196:01a66914051d6ee30318090a4940908c +history 599e1f765cf66c9c0201a3185e55947572ab029383a7df734116000000010000000000000001 #137:8a947e5f06d2ae0c429e5426a292cf76 +history 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba150000000000000001 #389:a4efe16c385211ddee69b324f56c7aab +history 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba150000000000000002 #119:f241e0dc14141fc4ea019760556f0462 +history 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba150000000000000003 #147:8537d4d2ae2d8e977da99d0b2ffe67f2 +history 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba150000000000000004 #119:4a41f44126f5278ed249f586fb77c44e +history 5fbe9f01fc12f3b90201a3185e45f47e19bd36219c12f0a3ba16000000010000000000000001 #133:40c2b8505d06685cea40629baa792187 +history 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8150000000000000001 #335:e61b93ee4ab29916f0ccaa0ad9d7f1d4 +history 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8150000000000000002 #381:9ce8414129a0f1e3cb1f6f5a78eb8565 +history 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8150000000000000003 #387:4e0a58cbc7129ea45f0f0ba7c35cda50 +history 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb8150000000000000004 #145:b8915db22cfcc52bcfc0e726e491d254 +history 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb816000000010000000000000001 =0801100122020801 history 957748ea772a492d0201a3185ce1827fd69291cf63846a12f3150000000000000001 #393:a5706d57a699118e0b45412f75b8d67d history 957748ea772a492d0201a3185ce1827fd69291cf63846a12f3150000000000000002 #121:7dc2aefc1ddf60488d6abb2e757e1653 history 957748ea772a492d0201a3185ce1827fd69291cf63846a12f3150000000000000003 #297:94fea4a1ea446cb26713e1f5ce2644d2 @@ -387,8 +387,8 @@ history 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b68172150000000000000002 #14 history 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b68172150000000000000003 #419:4c7379ece2c41eefe68ec6fd0871147a history 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b68172150000000000000004 #127:95e84b705457202aef3a5c57dc7a5188 history 9b749ea64ab099e80201a3185cea4c7d5982c3e7ad08b6817216000000010000000000000001 #224:4bf8ea34dcee365f7091af69238434c6 -history bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c8150000000000000001 #383:e4b777c59775bb15d1436d46c2496afc -history bd8ac614675559d00201a3185e46ee7816bc7abcd3adb791c816000000010000000000000001 #133:56501899281aa791c44ccff89c8f587a +history bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a150000000000000001 #388:28d5bc7bd34f6d2e492284de0724246f +history bd8ac614675559d00201a3185e47e87c9ba5bc71cdf3a0f30a16000000010000000000000001 #133:e918f86e9004d8053b88019b7fbed067 history cef57bddf594cff00201a3185d04aa7a8486e4fc3d7b333df3150000000000000001 #381:6405bffdc7fb8573dfa7192b4b8a3b9b history cef57bddf594cff00201a3185d04aa7a8486e4fc3d7b333df3150000000000000002 #115:b897fbab54e667cee99b4d3156598112 history cef57bddf594cff00201a3185d04aa7a8486e4fc3d7b333df3150000000000000003 #262:7ee45f442b6ef5a8714b5e0aa7af0417 @@ -414,102 +414,102 @@ history d879b97faa5ef0cf0201a3185c50007c4ab7515b188df65b7b1600000001000000000000 partition_index 0000018006652b48d23a000f0b8625e5f9724701a3185ce75e7c9ba774c48afeb866c032120a1001000000000000000300000000000000 = partition_index 000001800665a3fc0870300f0b8625e5f9724701a3185ce75e7c9ba774c48afeb866c02a00 = partition_index 000001800665a3fc2ac54030cdf0c23882674401a3185cf9ec7d97b2eddbc862722e0c2a00 = -partition_index 000001800665a4012110900808577b47782b9201a3185e0a627823872860b20f601e002a00 = +partition_index 000001800665a40124e1200808577b47782b9201a3185e0b5c7f468a30ab8a1e4c9f302a00 = partition_index 000002000000000000000000094f72646572466c6f770230cdf0c23882674401a3185cf9ec7d97b2eddbc862722e0c = partition_index 0000020000000000000000000a506172656e74466c6f77020f0b8625e5f9724701a3185ce75e7c9ba774c48afeb866c0 = partition_index 0000020000000000000000000b4163636f756e74466c6f770100c77b2df1ff0c8101a3185ced3a76f5860d3203b7c6fd9f = -partition_index 0000020000000000000000000b48616e646f7574466c6f77020808577b47782b9201a3185e0a627823872860b20f601e00 = +partition_index 0000020000000000000000000b48616e646f7574466c6f77020808577b47782b9201a3185e0b5c7f468a30ab8a1e4c9f30 = partition_index 000003000000000000000000c77b2df1ff0c8101a3185ced3a76f5860d3203b7c6fd9f =0000000000000001 -partition_index 00000300000000000000000808577b47782b9201a3185e0a627823872860b20f601e00 =0000000000000014 +partition_index 00000300000000000000000808577b47782b9201a3185e0b5c7f468a30ab8a1e4c9f30 =0000000000000014 partition_index 00000300000000000000000f0b8625e5f9724701a3185ce75e7c9ba774c48afeb866c0 =0000000000000008 partition_index 000003000000000000000030cdf0c23882674401a3185cf9ec7d97b2eddbc862722e0c =0000000000000005 partition_index 000004000000000000000000c77b2df1ff0c8101a3185ced3a76f5860d3203b7c6fd9f = -partition_index 00000400000000000000000808577b47782b9201a3185e0a627823872860b20f601e00 = +partition_index 00000400000000000000000808577b47782b9201a3185e0b5c7f468a30ab8a1e4c9f30 = partition_index 00000400000000000000000f0b8625e5f9724701a3185ce75e7c9ba774c48afeb866c0 = partition_index 000004000000000000000030cdf0c23882674401a3185cf9ec7d97b2eddbc862722e0c = partition_index 000007800665a3fc0870300f0b8625e5f9724701a3185ce75e7c9ba774c48afeb866c0 = partition_index 000007800665a3fc2ac54030cdf0c23882674401a3185cf9ec7d97b2eddbc862722e0c = -partition_index 000007800665a4012110900808577b47782b9201a3185e0a627823872860b20f601e00 = +partition_index 000007800665a40124e1200808577b47782b9201a3185e0b5c7f468a30ab8a1e4c9f30 = partition_index 00000800c77b2df1ff0c8101a3185ced3a76f5860d3203b7c6fd9f = partition_index 00000a000000000000000000000100000000000000007ff99ae8d4fd84df01a3185ced3a76f5860d3203b7c6fd9f00c77b2df1ff0c81 = -partition_index 00000a00000000000000000001007ff99ae8cfc28f6f7ff99ae8d0a3a09f01a3185e0a627823872860b20f601e000808577b47782b92 = +partition_index 00000a00000000000000000001007ff99ae8cfbebedf7ff99ae8d09fd00f01a3185e0b5c7f468a30ab8a1e4c9f300808577b47782b92 = partition_index 00000a00000000000000000001007ff99ae8d4b8dabf7ff99ae8d4cbed8f01a3185cf9ec7d97b2eddbc862722e0c30cdf0c238826744 = partition_index 00000a00000000000000000001007ff99ae8d4db2fcf7ff99ae8d514683f01a3185ce75e7c9ba774c48afeb866c00f0b8625e5f97247 = partition_index 00000a00000000000000000100094f72646572466c6f7701007ff99ae8d4b8dabf7ff99ae8d4cbed8f01a3185cf9ec7d97b2eddbc862722e0c30cdf0c238826744 = partition_index 00000a000000000000000001000a506172656e74466c6f7701007ff99ae8d4db2fcf7ff99ae8d514683f01a3185ce75e7c9ba774c48afeb866c00f0b8625e5f97247 = partition_index 00000a000000000000000001000b4163636f756e74466c6f77000100000000000000007ff99ae8d4fd84df01a3185ced3a76f5860d3203b7c6fd9f00c77b2df1ff0c81 = -partition_index 00000a000000000000000001000b48616e646f7574466c6f7701007ff99ae8cfc28f6f7ff99ae8d0a3a09f01a3185e0a627823872860b20f601e000808577b47782b92 = -partition_index 00000f0808577b47782b9201a3185e0a627823872860b20f601e00 = +partition_index 00000a000000000000000001000b48616e646f7574466c6f7701007ff99ae8cfbebedf7ff99ae8d09fd00f01a3185e0b5c7f468a30ab8a1e4c9f300808577b47782b92 = +partition_index 00000f0808577b47782b9201a3185e0b5c7f468a30ab8a1e4c9f30 = partition_index 00000f30cdf0c23882674401a3185cf9ec7d97b2eddbc862722e0c = partition_index 000101800665a3fc6f6f604a90b11d2e1acfc701a3185d08927201ac6ac5c708d6402b2a00 = partition_index 000101800665a3fc8652c04174ea676bdbb0f301a3185d115c786fbaaa7a89714731b52a00 = -partition_index 000101800665a4015a490055e03d097c903c0b01a3185e4cca76b9b8359d4a5dcc0e332a00 = -partition_index 000101800665a40178cd807becb50ef3da384a01a3185e578871fcb17bbccdb9faaeb42a00 = +partition_index 000101800665a4015e199055e03d097c903c0b01a3185e4dc47ee5962f06ca9e6d8ec62a00 = +partition_index 000101800665a4017c9e107becb50ef3da384a01a3185e588277649c53bb00e0d05cb82a00 = partition_index 0001020000000000000000000849646c65466c6f77064174ea676bdbb0f301a3185d115c786fbaaa7a89714731b5 = -partition_index 000102000000000000000000095175657565466c6f77015fbe9f01fc12f3b901a3185e44fa7e0a8f38a12d3de1a049 = -partition_index 0001020000000000000000000a536561726368466c6f7701599e1f765cf66c9c01a3185e549a758b93ec9bb35d406765 = +partition_index 000102000000000000000000095175657565466c6f77015fbe9f01fc12f3b901a3185e45f47e19bd36219c12f0a3ba = +partition_index 0001020000000000000000000a536561726368466c6f7701599e1f765cf66c9c01a3185e55947572ab029383a7df7341 = partition_index 0001020000000000000000000a54726176656c466c6f77024a90b11d2e1acfc701a3185d08927201ac6ac5c708d6402b = -partition_index 0001020000000000000000000b48616e646f7574466c6f770255e03d097c903c0b01a3185e4cca76b9b8359d4a5dcc0e33 = -partition_index 0001020000000000000000000b486973746f7279466c6f77067becb50ef3da384a01a3185e578871fcb17bbccdb9faaeb4 = +partition_index 0001020000000000000000000b48616e646f7574466c6f770255e03d097c903c0b01a3185e4dc47ee5962f06ca9e6d8ec6 = +partition_index 0001020000000000000000000b486973746f7279466c6f77067becb50ef3da384a01a3185e588277649c53bb00e0d05cb8 = partition_index 00010300000000000000004174ea676bdbb0f301a3185d115c786fbaaa7a89714731b5 =0000000000000006 partition_index 00010300000000000000004a90b11d2e1acfc701a3185d08927201ac6ac5c708d6402b =0000000000000009 -partition_index 000103000000000000000055e03d097c903c0b01a3185e4cca76b9b8359d4a5dcc0e33 =0000000000000007 -partition_index 0001030000000000000000599e1f765cf66c9c01a3185e549a758b93ec9bb35d406765 =0000000000000001 -partition_index 00010300000000000000005fbe9f01fc12f3b901a3185e44fa7e0a8f38a12d3de1a049 =0000000000000001 -partition_index 00010300000000000000007becb50ef3da384a01a3185e578871fcb17bbccdb9faaeb4 =0000000000000004 +partition_index 000103000000000000000055e03d097c903c0b01a3185e4dc47ee5962f06ca9e6d8ec6 =0000000000000007 +partition_index 0001030000000000000000599e1f765cf66c9c01a3185e55947572ab029383a7df7341 =0000000000000001 +partition_index 00010300000000000000005fbe9f01fc12f3b901a3185e45f47e19bd36219c12f0a3ba =0000000000000001 +partition_index 00010300000000000000007becb50ef3da384a01a3185e588277649c53bb00e0d05cb8 =0000000000000004 partition_index 00010400000000000000004174ea676bdbb0f301a3185d115c786fbaaa7a89714731b5 = partition_index 00010400000000000000004a90b11d2e1acfc701a3185d08927201ac6ac5c708d6402b = -partition_index 000104000000000000000055e03d097c903c0b01a3185e4cca76b9b8359d4a5dcc0e33 = -partition_index 0001040000000000000000599e1f765cf66c9c01a3185e549a758b93ec9bb35d406765 = -partition_index 00010400000000000000005fbe9f01fc12f3b901a3185e44fa7e0a8f38a12d3de1a049 = -partition_index 00010400000000000000007becb50ef3da384a01a3185e578871fcb17bbccdb9faaeb4 = +partition_index 000104000000000000000055e03d097c903c0b01a3185e4dc47ee5962f06ca9e6d8ec6 = +partition_index 0001040000000000000000599e1f765cf66c9c01a3185e55947572ab029383a7df7341 = +partition_index 00010400000000000000005fbe9f01fc12f3b901a3185e45f47e19bd36219c12f0a3ba = +partition_index 00010400000000000000007becb50ef3da384a01a3185e588277649c53bb00e0d05cb8 = partition_index 000107800665a3fc6f6f604a90b11d2e1acfc701a3185d08927201ac6ac5c708d6402b = partition_index 000107800665a3fc8652c04174ea676bdbb0f301a3185d115c786fbaaa7a89714731b5 = -partition_index 000107800665a4015a490055e03d097c903c0b01a3185e4cca76b9b8359d4a5dcc0e33 = -partition_index 000107800665a40178cd807becb50ef3da384a01a3185e578871fcb17bbccdb9faaeb4 = -partition_index 000108599e1f765cf66c9c01a3185e549a758b93ec9bb35d406765 = -partition_index 0001085fbe9f01fc12f3b901a3185e44fa7e0a8f38a12d3de1a049 = -partition_index 00010a000000000000000000000100000000000000007ff99ae8cf81b5df01a3185e549a758b93ec9bb35d406765599e1f765cf66c9c = -partition_index 00010a000000000000000000000100000000000000007ff99ae8cfbebedf01a3185e44fa7e0a8f38a12d3de1a0495fbe9f01fc12f3b9 = -partition_index 00010a00000000000000000001007ff99ae8cf6ad27f7ff99ae8cf76442f01a3185e578871fcb17bbccdb9faaeb47becb50ef3da384a = -partition_index 00010a00000000000000000001007ff99ae8cf8956ff7ff99ae8cfa03a5f01a3185e4cca76b9b8359d4a5dcc0e3355e03d097c903c0b = +partition_index 000107800665a4015e199055e03d097c903c0b01a3185e4dc47ee5962f06ca9e6d8ec6 = +partition_index 000107800665a4017c9e107becb50ef3da384a01a3185e588277649c53bb00e0d05cb8 = +partition_index 000108599e1f765cf66c9c01a3185e55947572ab029383a7df7341 = +partition_index 0001085fbe9f01fc12f3b901a3185e45f47e19bd36219c12f0a3ba = +partition_index 00010a000000000000000000000100000000000000007ff99ae8cf7de54f01a3185e55947572ab029383a7df7341599e1f765cf66c9c = +partition_index 00010a000000000000000000000100000000000000007ff99ae8cfbaee4f01a3185e45f47e19bd36219c12f0a3ba5fbe9f01fc12f3b9 = +partition_index 00010a00000000000000000001007ff99ae8cf6701ef7ff99ae8cf72739f01a3185e588277649c53bb00e0d05cb87becb50ef3da384a = +partition_index 00010a00000000000000000001007ff99ae8cf85866f7ff99ae8cf9c69cf01a3185e4dc47ee5962f06ca9e6d8ec655e03d097c903c0b = partition_index 00010a00000000000000000001007ff99ae8d45d4d3f7ff99ae8d470600f01a3185d115c786fbaaa7a89714731b54174ea676bdbb0f3 = partition_index 00010a00000000000000000001007ff99ae8d474309f7ff99ae8d492b51f01a3185d08927201ac6ac5c708d6402b4a90b11d2e1acfc7 = partition_index 00010a000000000000000001000849646c65466c6f7701007ff99ae8d45d4d3f7ff99ae8d470600f01a3185d115c786fbaaa7a89714731b54174ea676bdbb0f3 = -partition_index 00010a00000000000000000100095175657565466c6f77000100000000000000007ff99ae8cfbebedf01a3185e44fa7e0a8f38a12d3de1a0495fbe9f01fc12f3b9 = -partition_index 00010a000000000000000001000a536561726368466c6f77000100000000000000007ff99ae8cf81b5df01a3185e549a758b93ec9bb35d406765599e1f765cf66c9c = +partition_index 00010a00000000000000000100095175657565466c6f77000100000000000000007ff99ae8cfbaee4f01a3185e45f47e19bd36219c12f0a3ba5fbe9f01fc12f3b9 = +partition_index 00010a000000000000000001000a536561726368466c6f77000100000000000000007ff99ae8cf7de54f01a3185e55947572ab029383a7df7341599e1f765cf66c9c = partition_index 00010a000000000000000001000a54726176656c466c6f7701007ff99ae8d474309f7ff99ae8d492b51f01a3185d08927201ac6ac5c708d6402b4a90b11d2e1acfc7 = -partition_index 00010a000000000000000001000b48616e646f7574466c6f7701007ff99ae8cf8956ff7ff99ae8cfa03a5f01a3185e4cca76b9b8359d4a5dcc0e3355e03d097c903c0b = -partition_index 00010a000000000000000001000b486973746f7279466c6f7701007ff99ae8cf6ad27f7ff99ae8cf76442f01a3185e578871fcb17bbccdb9faaeb47becb50ef3da384a = -partition_index 00010b000665173089bbd07becb50ef3da384a01a3185e578871fcb17bbccdb9faaeb4 = -partition_index 00010b0006651730952d807becb50ef3da384a01a3185e578871fcb17bbccdb9faaeb4 = -partition_index 00010c =080110a094f983f3a299031890f3e983f3a29903 +partition_index 00010a000000000000000001000b48616e646f7574466c6f7701007ff99ae8cf85866f7ff99ae8cf9c69cf01a3185e4dc47ee5962f06ca9e6d8ec655e03d097c903c0b = +partition_index 00010a000000000000000001000b486973746f7279466c6f7701007ff99ae8cf6701ef7ff99ae8cf72739f01a3185e588277649c53bb00e0d05cb87becb50ef3da384a = +partition_index 00010b00066517308d8c607becb50ef3da384a01a3185e588277649c53bb00e0d05cb8 = +partition_index 00010b000665173098fe107becb50ef3da384a01a3185e588277649c53bb00e0d05cb8 = +partition_index 00010c =080110b0b58884f3a2990318a094f983f3a29903 partition_index 00010e =0801 partition_index 00010f4174ea676bdbb0f301a3185d115c786fbaaa7a89714731b5 = partition_index 00010f4a90b11d2e1acfc701a3185d08927201ac6ac5c708d6402b = -partition_index 00010f55e03d097c903c0b01a3185e4cca76b9b8359d4a5dcc0e33 = +partition_index 00010f55e03d097c903c0b01a3185e4dc47ee5962f06ca9e6d8ec6 = partition_index 0002018006652b48f48f109b749ea64ab099e801a3185cea4c7d5982c3e7ad08b6817232120a1002000000000000000300000000000000 = partition_index 000201800665a3fbf92df09b749ea64ab099e801a3185cea4c7d5982c3e7ad08b681722a00 = partition_index 000201800665a3fc326660957748ea772a492d01a3185ce1827fd69291cf63846a12f32a00 = partition_index 000202000000000000000000094368696c64466c6f77029b749ea64ab099e801a3185cea4c7d5982c3e7ad08b68172 = -partition_index 000202000000000000000000095175657565466c6f7701bd8ac614675559d001a3185e46ee7816bc7abcd3adb791c8 = +partition_index 000202000000000000000000095175657565466c6f7701bd8ac614675559d001a3185e47e87c9ba5bc71cdf3a0f30a = partition_index 0002020000000000000000000b50726f66696c65466c6f7702957748ea772a492d01a3185ce1827fd69291cf63846a12f3 = partition_index 0002030000000000000000957748ea772a492d01a3185ce1827fd69291cf63846a12f3 =0000000000000009 partition_index 00020300000000000000009b749ea64ab099e801a3185cea4c7d5982c3e7ad08b68172 =0000000000000003 -partition_index 0002030000000000000000bd8ac614675559d001a3185e46ee7816bc7abcd3adb791c8 =0000000000000001 +partition_index 0002030000000000000000bd8ac614675559d001a3185e47e87c9ba5bc71cdf3a0f30a =0000000000000001 partition_index 0002040000000000000000957748ea772a492d01a3185ce1827fd69291cf63846a12f3 = partition_index 00020400000000000000009b749ea64ab099e801a3185cea4c7d5982c3e7ad08b68172 = -partition_index 0002040000000000000000bd8ac614675559d001a3185e46ee7816bc7abcd3adb791c8 = +partition_index 0002040000000000000000bd8ac614675559d001a3185e47e87c9ba5bc71cdf3a0f30a = partition_index 000207800665a3fbf92df09b749ea64ab099e801a3185cea4c7d5982c3e7ad08b68172 = partition_index 000207800665a3fc326660957748ea772a492d01a3185ce1827fd69291cf63846a12f3 = -partition_index 000208bd8ac614675559d001a3185e46ee7816bc7abcd3adb791c8 = -partition_index 00020a000000000000000000000100000000000000007ff99ae8cfb71dbf01a3185e46ee7816bc7abcd3adb791c8bd8ac614675559d0 = +partition_index 000208bd8ac614675559d001a3185e47e87c9ba5bc71cdf3a0f30a = +partition_index 00020a000000000000000000000100000000000000007ff99ae8cfb34d2f01a3185e47e87c9ba5bc71cdf3a0f30abd8ac614675559d0 = partition_index 00020a00000000000000000001007ff99ae8d4b1399f7ff99ae8d52b4b9f01a3185ce1827fd69291cf63846a12f3957748ea772a492d = partition_index 00020a00000000000000000001007ff99ae8d4ea720f7ff99ae8d508f68f01a3185cea4c7d5982c3e7ad08b681729b749ea64ab099e8 = partition_index 00020a00000000000000000100094368696c64466c6f7701007ff99ae8d4ea720f7ff99ae8d508f68f01a3185cea4c7d5982c3e7ad08b681729b749ea64ab099e8 = -partition_index 00020a00000000000000000100095175657565466c6f77000100000000000000007ff99ae8cfb71dbf01a3185e46ee7816bc7abcd3adb791c8bd8ac614675559d0 = +partition_index 00020a00000000000000000100095175657565466c6f77000100000000000000007ff99ae8cfb34d2f01a3185e47e87c9ba5bc71cdf3a0f30abd8ac614675559d0 = partition_index 00020a000000000000000001000b50726f66696c65466c6f7701007ff99ae8d4b1399f7ff99ae8d52b4b9f01a3185ce1827fd69291cf63846a12f3957748ea772a492d = -partition_index 00020d000000000000000000095175657565466c6f7700085061636b5374657001800665173048e240bd8ac614675559d001a3185e46ee7816bc7abcd3adb791c802000000015061636b537465702d31 =08011001 +partition_index 00020d000000000000000000095175657565466c6f7700085061636b537465700180066517304cb2d0bd8ac614675559d001a3185e47e87c9ba5bc71cdf3a0f30a02000000015061636b537465702d31 =08011001 partition_index 000301800665a3fc13e1e0d879b97faa5ef0cf01a3185c50007c4ab7515b188df65b7b2a00 = partition_index 000301800665a3fc4d1a50cef57bddf594cff001a3185d04aa7a8486e4fc3d7b333df32a00 = partition_index 000302000000000000000000094f72646572466c6f7702d879b97faa5ef0cf01a3185c50007c4ab7515b188df65b7b = @@ -524,10 +524,10 @@ partition_index 00030a00000000000000000001007ff99ae8d49685af7ff99ae8d4a1f75f01a3 partition_index 00030a00000000000000000001007ff99ae8d4cfbe1f7ff99ae8d763af6f01a3185c50007c4ab7515b188df65b7bd879b97faa5ef0cf = partition_index 00030a00000000000000000100094f72646572466c6f7701007ff99ae8d4cfbe1f7ff99ae8d763af6f01a3185c50007c4ab7515b188df65b7bd879b97faa5ef0cf = partition_index 00030a000000000000000001000d48656c64436c6f7365466c6f7701007ff99ae8d49685af7ff99ae8d4a1f75f01a3185d04aa7a8486e4fc3d7b333df3cef57bddf594cff0 = -partition_meta 0000 =18a09df384f3a2990320013001 -partition_meta 0001 =08011890fce384f3a2990320013001 -partition_meta 0002 =080218d0e5b282f3a2990320013001 +partition_meta 0000 =18b0be8285f3a2990320013001 +partition_meta 0001 =080118a09df384f3a2990320013001 +partition_meta 0002 =080218e086c282f3a2990320013001 partition_meta 0003 =080318d0f4a5dbf2a2990320013001 -value 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb410000000000000000200000000 #150:582bacc585ea0c8aa8ac6c9592ba64a3 -value 7becb50ef3da384a0201a3185e578871fcb17bbccdb9faaeb410000000000000000300000000 #162:f5da54fe0d1772299e0c1fe67b606a81 +value 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb810000000000000000200000000 #150:582bacc585ea0c8aa8ac6c9592ba64a3 +value 7becb50ef3da384a0201a3185e588277649c53bb00e0d05cb810000000000000000300000000 #162:f5da54fe0d1772299e0c1fe67b606a81 value d879b97faa5ef0cf0201a3185c50007c4ab7515b188df65b7b10000000000000000100000000 #150:582bacc585ea0c8aa8ac6c9592ba64a3