[longquanzheng]

Summary

• Upload every committed SEALED history segment from the server-zd engine to the primary object store under history/, and propose HistorySegmentUploaded acknowledgments in seq order. One effect task per run is started by the committed HistorySegmentSealed effect, by partition rebuild from the HistoryUploadPending index (restart, takeover, snapshot restore), and by a 60-second rescan.
• Write with CreateOnly. An existing object that fully decodes (on tracked blocking work) with the same content checksum is acknowledged with its stored size. Any other existing object is replaced with If-Match, logged as an error, and counted. A read or encode failure never writes.
• The upload permit owns the encoded body; after the store only key, checksum, and size are kept, so memory is bounded by history.uploadConcurrency while acknowledgments wait.
• Retryable object storage failures back off exponentially and never delay commands. Permanent failures (corrupt input, rejected store, rejected ack) are logged and counted, and the run is retried only by the next rescan, whatever triggers it. A run the rescan cannot read is logged, counted, and skipped without stopping the partition's scan. Uploads stop for runs that are gone or purging, and continue after the offload mode is turned off, so every sealed segment is eventually acknowledged.
• Apply: retention postpones a closed run while it has a SEALED segment, as it does for an unacknowledged outbox message.
• Purge of a run that ever had segment state waits 60 s for abandoned requests to end (all history I/O stops with the node), deletes its history objects by catalog key, then sweeps the run's history/ prefix, before the hard-delete prefix and RunPurge. A write the object store had already accepted may still land after the purge; it leaves an unreadable object of a purged run, which a later periodic orphan sweep removes.
• Metrics: dexzd_history_upload_backlog_segments, dexzd_history_upload_backlog_bytes, dexzd_history_upload_oldest_sealed_age_seconds, dexzd_history_uploads_total{outcome}, dexzd_history_upload_bytes_total, dexzd_history_upload_checksum_mismatches_total, dexzd_history_upload_rescan_errors_total, and object_kind="history".
• New setting history.uploadConcurrency (default 16, 1–1024).

Why

Refs #1325. Stacked on #1356, which seals history segments in apply and accepts upload acknowledgments, but nothing uploads them yet. While history offload is enabled, window pruning keeps every row after a run's uploaded seq, so without an uploader local history grows without bound. This PR adds the uploader, so sealed segments reach object storage and pruning can advance. Retention must wait for uploads: purging stops uploads, so a segment sealed before the purge would never be stored. A segment's key and content depend only on committed state, so a new owner that repeats an upload finds the same content.

Compatibility

• No persistent format change: no new proto fields, key bytes, or commands.
• One apply-result change: retention firing on a closed run with a SEALED segment now postpones by one hour instead of purging. It is reachable only with the experimental history offload mode on (default off). Before server-zd 1.0 this needs no behavior version; stop both nodes and upgrade them together.
• Purge of a run with segment state waits one minute before deleting and adds one prefix LIST; runs without segment state are unaffected.
• New additive config key history.uploadConcurrency; new metrics; object_kind gains the value history.
• With offload off (the default), no segments exist. Retention adds one point read and purge adds one empty range scan, and the rescan reads one empty index per partition each minute.
• The format compatibility workload acknowledges the short-lived run's segment before its retention fires; testdata/v1 is regenerated with its generator and its pin updated.

Testing

• Effect harness, real apply and in-memory object store with fault injection: seq-ordered upload and acknowledgment; owner restart between upload and acknowledgment; unknown write outcome; replaced decodable and undecodable objects; outage backlog gauges growing then draining; retention postponement and purge of the run's history/ prefix; purging runs; snapshot restore before acknowledgment; a stopping owner abandons its pending write before the new owner purges; concurrency 2 with 20 runs whose acks are blocked; content kept through mode-off pruning (all already_present); permanent failures retried only by the rescan, including 10 rapid seals; a corrupt run does not stop the partition's rescan.
• dev server e2e on embedded s3local: a closed run's segment uploads, decodes, and is UPLOADED in committed state.
• serve_process with s3local: 503 on history/ PUTs while Flows keep completing, then a waiting standby restores a snapshot, takes over, and uploads every segment.
• Changed existing tests: the B5 purge test marks the run purging directly, with a new test for postpone, acknowledge, then purge; retention tests poll for the purge; the serve_process metrics series ceiling is raised by the 10 new series (195 measured, ceiling 202).
• cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, tests of apply, engine (lib, engine, format_compat), metrics, history-segment, server (lib, serve_process), simulation, e2e; make rust-rules-check; make copyright-check; cd docs && npm run check.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!