Skip to content

server-zd: default the log deletion delay to 1 hour in single-node modes - #1353

Merged
longquanzheng merged 2 commits into
mainfrom
server-zd/single-machine-log-deletion-delay
Nov 7, 2026
Merged

longquanzheng merged 2 commits into
mainfrom
server-zd/single-machine-log-deletion-delay

Conversation

@longquanzheng

Copy link
Copy Markdown
Contributor

Summary

dex-server-zd dev and dex-server-zd local now default snapshot.logDeletionDelay to 1 hour. serve keeps the 24-hour default. An explicitly configured value still wins in every mode, and validation is unchanged (must be greater than zero). The single-node default follows the existing pattern in Config::load_single_node (applied only when the overrides YAML does not set the key), like the lease duration and partition count defaults. The configuration reference gains a "Disk use" section for single-node mode and a comparison-table row.

Why

A single-machine disk soak test showed that with the 24-hour default, the embedded s3local object store, which shares the disk with RocksDB, grew without bound for a full day. Superseded log batches and old snapshots stayed for 24 hours, about 2–3 times the day's write volume. With a short delay, disk use levelled off.

The delay protects three things:

  • falling back to an older snapshot plus its log;
  • a standby or a long-stopped node catching up from the log instead of restoring a full snapshot;
  • orphan files of an in-progress snapshot upload. The delay must exceed the longest upload, and multi-GB snapshots take minutes.

Single-machine modes have no hot standby, so the second reason does not apply there. One hour still covers the first and third. Clustered serve deployments keep 24 hours, where the extra object storage is cheap and a lagging standby benefits from log catch-up.

Compatibility

This changes a default for dev and local only. Configurations that set snapshot.logDeletionDelay explicitly are unaffected in every mode, and serve defaults are unchanged. No persistent format or API changes.

Testing

  • New config test single_node_log_deletion_delay_defaults_to_one_hour_unless_the_overrides_set_it: dev and local default to 1 h, including when another snapshot key is set; an explicit 24h overrides in both; serve stays 24 h.
  • cargo test -p dexzd-server --lib config::, cargo fmt --all --check, cargo clippy -p dexzd-server --all-targets -- -D warnings, make rust-rules-check, make copyright-check, cd docs && npm run check: all pass locally.
  • Changed docs page is in the Developer reference (English only).

Single-node modes have no standby and keep the embedded object store on
the same disk as RocksDB. A 24-hour delay kept superseded log batches and
snapshots for a day, so disk use grew to about 2-3 times a day's writes.
serve keeps 24 hours, and an explicit value still wins in every mode.
…e disk use includes

Log objects wait for the covering snapshot to reach the delay, counted from
its creation time, and displaced snapshots wait for the newer one. The disk
estimate now names the retained snapshots in the embedded object store.
@codecov

codecov Bot commented Nov 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@longquanzheng
longquanzheng merged commit 5ce4913 into main Nov 7, 2026
48 checks passed
@longquanzheng
longquanzheng deleted the server-zd/single-machine-log-deletion-delay branch November 7, 2026 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant