diff --git a/CHANGELOG.md b/CHANGELOG.md index 1cdecf45b..0e2f0e538 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,6 +77,81 @@ supported under per-shard AOF layout`. Tracked for v0.2.0. `CDC.SUBSCRIBE` push channel (C3b), and the multi-shard master PSYNC deferred from v0.1.10. Tracked in `.planning/rfcs/v02-enterprise-architecture.md`. +### Fixed — Multishard idle-RAM blowup + tokio-Linux serving hang + graph parser DoS (PR #136) + +Closes the "multishard RAM zombie": a fresh multishard instance with no +`maxmemory` could commit multiple GB of RSS while idle, and the tokio +runtime could hang its accept loop on Linux. Three independent root +causes, all fixed and verified in the OrbStack VM under a cgroup memory +cap (idle RSS **3791 MB → 29 MB** for a 4-shard no-`maxmemory` instance; +~45 MB under load): + +- **PageCache eager pre-allocation.** Each shard committed + `num_frames × PAGE` zeroed bytes at construction, sized to 25% of the + **whole-instance** `maxmemory` (or a large default when unset). Now the + page buffers allocate lazily and the frame budget is divided by shard + count (`per_shard_pagecache_budget`); a startup `WARN` reports the + resolved per-shard budget. +- **tokio listener bind-race.** The central accept listener bound the port + *without* `SO_REUSEPORT` while per-shard listeners bound *with* it, + producing a bind-order race that could leave the port served by a shard + that never received connections. The central listener now also binds + `SO_REUSEPORT`; `--per-shard-accept` defaults to `false` under tokio. +- **io_uring-under-tokio default-off.** The tokio→io_uring bridge floods + errors under load. It is now **opt-in via `MOON_URING=1`**; tokio shards + run plain epoll/kqueue by default. `MOON_NO_URING=1` still force-disables + io_uring everywhere. The monoio runtime is unaffected (always io_uring + unless `MOON_NO_URING`). + +Two durability defects found during review were fixed in-PR with +red/green TDD: + +- **Manifest compact-reopen failure no longer silently loses commits.** + `manifest.rs compact()` reopened `self.file` *after* `rename(tmp, path)`; + if that reopen failed, later commits silently wrote to an orphaned inode. + A `needs_reopen` flag now reattaches to `self.path` before any subsequent + commit (or fails loudly). +- **tokio per-shard AOF writer now latches after a torn write.** The tokio + per-shard writer lacked the `write_error` latch present on the single-file + and monoio writers; a torn write (header OK, data fails) corrupted the + frame stream. It now latches on any write failure and acks `WriteFailed`. + +- **Cypher parser stack-overflow DoS bounded.** The precedence-climbing + expression grammar pushes ~9 stack frames per source nesting level but + `check_depth()` counts only one level per recursion, so the previous + limit of 64 overflowed a 2 MiB async-worker stack (SIGABRT) **before** + the guard could fire. `DEFAULT_MAX_NESTING_DEPTH` is now `32` + (~288 worst-case debug frames, ~50% margin); deep nesting returns + `NestingDepthExceeded` gracefully. + +Low-severity durability edge cases filed as follow-ups: #137 +(`apply_spill_completions` failed-commit cold-key window), #138 +(`do_rewrite_per_shard` panic wedges `--experimental-per-shard-rewrite`), +#139 (multi-DB `SELECT >0` cold recovery restores db0 only). + +### Fixed — `maxmemory` is now a whole-instance cap across shards (G2) + +**Behavior change for multishard deployments.** Previously each shard +enforced eviction against the *full* `maxmemory`, so an N-shard server +tolerated ~N× the configured cap before evicting (a 4-shard server at +`--maxmemory 100mb` retained ~307 MB / 768K keys vs a 1-shard server's +124 MB / 322K — the "RAM keeps growing in multishard mode" report). + +`maxmemory` is now a true **whole-instance** cap. Each shard enforces +eviction against `maxmemory / num_shards`, so aggregate RSS converges on +the configured value regardless of shard count. + +- **`CONFIG GET maxmemory` / INFO are unchanged** — they report the + whole-instance value verbatim (Redis-compatible). Division happens only + at enforcement. +- **Operators running explicit `--maxmemory N --shards M (M>1)`** now get + an effective ceiling of `N` (not `N×M`). A startup log line states the + resolved per-shard budget so the change is visible: + `maxmemory bytes is a whole-instance cap; each of shards enforces + eviction against a per-shard budget of bytes`. +- Single-shard servers are byte-for-byte unaffected (`num_shards == 1` ⇒ + no division). + ### Docs — Hash-field TTL three-way benchmark suite (PR #127) - `scripts/bench-hash-ttl.sh` (2-way harness) + `scripts/bench-hash-ttl-3way.sh` diff --git a/CLAUDE.md b/CLAUDE.md index 05293565e..1db9a9217 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -85,7 +85,8 @@ orb run -m moon-dev bash -c 'sudo apt-get update -qq && sudo apt-get install -y ## Environment Variables - `RUST_LOG=moon=debug` — enable tracing output (uses `tracing-subscriber` with `env-filter`) -- `MOON_NO_URING=1` — disable io_uring at runtime; used in CI/containers/WSL where io_uring is unavailable +- `MOON_NO_URING=1` — force-disable io_uring everywhere (monoio runtime + tokio bridge); used in CI/containers/WSL where io_uring is unavailable +- `MOON_URING=1` — opt **into** the tokio→io_uring bridge. The bridge is **default-off under the tokio runtime** (it floods errors under load and can hang the accept loop); tokio shards run plain epoll/kqueue unless this is set. No effect on the monoio runtime, which always uses io_uring unless `MOON_NO_URING` is set. - `RUSTFLAGS="-C target-cpu=native"` — enable CPU-specific optimizations for benchmarking ## Key Design Decisions diff --git a/README.md b/README.md index 1754f43fb..f9232e242 100644 --- a/README.md +++ b/README.md @@ -233,6 +233,14 @@ cargo build --release --maxmemory 8g --maxmemory-policy allkeys-lfu ``` +> **`--maxmemory` is a whole-instance cap.** With `--shards M`, each +> shard enforces eviction against `maxmemory / M`, so total RSS converges +> on the value you set regardless of shard count. `CONFIG GET maxmemory` +> and `INFO` report the whole-instance value (Redis-compatible); a startup +> log line shows the resolved per-shard budget. If `--maxmemory` is omitted, +> Moon auto-caps at ~80% of detected RAM with `allkeys-lru` (pass +> `--maxmemory 0` for unlimited). + ### Connect with any Redis client ```bash diff --git a/src/command/persistence.rs b/src/command/persistence.rs index a1976785c..6a80f0496 100644 --- a/src/command/persistence.rs +++ b/src/command/persistence.rs @@ -300,6 +300,34 @@ pub fn bgrewriteaof_start_sharded( b"ERR Background AOF rewrite already in progress", )); } + + // [F6] PerShard pools use the per-shard fan-out (synchronized seq bump + + // single manifest commit across all writers). TopLevel multi-DB pools keep + // the legacy single-writer RewriteSharded path. + if pool.layout() == crate::persistence::aof_manifest::AofLayout::PerShard { + // Per-shard fan-out for BOTH runtimes. The fold is synchronous std::fs + // IO; on monoio the writer runs it inline, on tokio the writer converts + // its `tokio::fs` handle to `std::fs` for the fold's duration (both run + // on a dedicated block_on_local thread, so blocking is safe). The fold + // body — and therefore the exactly-once invariant — is identical. + // + // try_send_rewrite_per_shard loads the manifest, builds the shared + // coordinator, and reliably fans out to every writer. The in-progress + // flag is cleared by the coordinator's final commit + // (PerShardRewriteCoord::shard_done), not here. + match pool.try_send_rewrite_per_shard(shard_databases) { + Ok(()) => { + return Frame::SimpleString(Bytes::from_static( + b"Background append only file rewriting started", + )); + } + Err(e) => { + AOF_REWRITE_IN_PROGRESS.store(false, Ordering::SeqCst); + return rewrite_pool_error_frame(e); + } + } + } + match pool.try_send_rewrite(AofMessage::RewriteSharded(shard_databases)) { Ok(()) => Frame::SimpleString(Bytes::from_static( b"Background append only file rewriting started", diff --git a/src/config.rs b/src/config.rs index 13b4b475c..eb5e7eb29 100644 --- a/src/config.rs +++ b/src/config.rs @@ -29,7 +29,7 @@ pub enum CrossShardFastPath { } /// Server configuration parsed from command-line arguments. -#[derive(Parser, Debug, Clone)] +#[derive(Parser, Debug, Clone, Default)] #[command(name = "moon", about = "A Redis-compatible server")] pub struct ServerConfig { /// Bind address @@ -117,10 +117,34 @@ pub struct ServerConfig { #[arg(long, default_value_t = false)] pub unsafe_multishard_aof: bool, + /// [EXPERIMENTAL] Enable per-shard BGREWRITEAOF (compaction) for the + /// `--shards >= 2 + --appendonly yes` PerShard layout. + /// + /// Default `false`: BGREWRITEAOF stays gated in PerShard mode (the + /// shipped, crash-safe "append-only, no in-place compaction" behavior). + /// When `true`, BGREWRITEAOF fans the rewrite out to every per-shard + /// writer (synchronized seq bump + single manifest commit). This path is + /// validated by `tests/crash_matrix_per_shard_bgrewriteaof.rs` and is + /// opt-in until the both-runtime crash matrix is green by default. + /// + /// The flag only takes effect alongside `per_shard_aof_active`; it is a + /// no-op for `--shards 1` (TopLevel rewrite already works) and for + /// `--appendonly no`. + #[arg(long, default_value_t = false)] + pub experimental_per_shard_rewrite: bool, + /// AOF fsync policy (always/everysec/no) #[arg(long, default_value = "everysec")] pub appendfsync: String, + /// Max time (ms) a write may block awaiting the `appendfsync=always` + /// fsync ack before the write is failed instead of parking the + /// connection forever. Design-for-failure bound: a stalled disk must + /// not turn write connections into zombies holding their buffers. + /// 0 disables the bound (legacy unbounded await). Default 2000ms. + #[arg(long = "aof-fsync-timeout-ms", default_value_t = 2000)] + pub aof_fsync_timeout_ms: u64, + /// RDB auto-save rules (e.g., "3600 1 300 100") #[arg(long)] pub save: Option, @@ -137,9 +161,21 @@ pub struct ServerConfig { #[arg(long, default_value = "appendonly.aof")] pub appendfilename: String, - /// Maximum memory in bytes (0 = unlimited) - #[arg(long, default_value_t = 0)] - pub maxmemory: usize, + /// Maximum memory in bytes. + /// + /// G1 memory guardrail (design-for-failure against unbounded keyspace + /// growth → OOM kill): + /// - **flag omitted** (`None`) → Moon auto-caps at ~80% of the detected + /// memory limit (cgroup-aware on Linux, host RAM otherwise) and switches + /// a `noeviction` policy to `allkeys-lru`, logging a startup notice. + /// - `--maxmemory 0` → explicitly UNLIMITED (Redis-compatible escape hatch). + /// - `--maxmemory N` → exact cap in bytes (honored verbatim). + /// + /// Resolved once at startup by [`ServerConfig::apply_memory_guardrail`]; + /// downstream code reads the resolved `usize` from `RuntimeConfig` + /// (`0 = unlimited`). + #[arg(long)] + pub maxmemory: Option, /// Eviction policy when maxmemory is reached #[arg(long, default_value = "noeviction")] @@ -334,8 +370,15 @@ pub struct ServerConfig { /// the on-disk DataFile, and tombstones the manifest entry. /// /// Set to 0 to disable the sweeper entirely. - /// Default: 300 (5 minutes). Recommended range: 60–3600. - #[arg(long = "cold-orphan-sweep-interval-secs", default_value_t = 300)] + /// Default: 60 (1 minute). Recommended range: 60–3600. + /// + /// Lowered from 300 → 60: at 300s the sweep never fired within a typical + /// benchmark window, which both let cold orphans accumulate on disk for up + /// to 5 minutes AND masked a batch-file shared-deletion data-loss bug (fixed + /// by the per-file-liveness refcount in ColdIndex). 60s reclaims promptly; + /// the sweep's per-file unlinks run off the hot path so a shorter interval + /// keeps each batch small rather than churning under the shard lock. + #[arg(long = "cold-orphan-sweep-interval-secs", default_value_t = 60)] pub cold_orphan_sweep_interval_secs: u64, // ── MoonStore v2: Point-in-time recovery (PITR) ──────────────── @@ -652,10 +695,49 @@ impl ServerConfig { .unwrap_or(maxmemory / 4) } + /// Resolve the G1 memory guardrail, mutating `self` in place, and return + /// the [`GuardrailOutcome`] for the caller to log as a startup notice. + /// + /// MUST be called once at startup, after parsing and before + /// [`Self::to_runtime_config`]. Idempotent in effect: once `maxmemory` is + /// `Some`, re-calling returns `Explicit` and changes nothing. + pub fn apply_memory_guardrail(&mut self) -> GuardrailOutcome { + let detected = detect_memory_limit_bytes(); + let outcome = resolve_memory_guardrail( + self.maxmemory, + &self.maxmemory_policy, + detected, + MAXMEMORY_GUARDRAIL_PERCENT, + ); + match &outcome { + GuardrailOutcome::Applied { + cap_bytes, + policy_changed_to, + .. + } => { + self.maxmemory = Some(*cap_bytes); + if let Some(p) = policy_changed_to { + self.maxmemory_policy = p.clone(); + } + } + GuardrailOutcome::Explicit(_) => { /* operator set it; honor verbatim */ } + GuardrailOutcome::Skipped => { + // Omitted but no limit detectable → leave UNLIMITED but make it + // concrete so downstream sees the `0` sentinel, not `None`. + self.maxmemory = Some(0); + } + } + outcome + } + /// Create a RuntimeConfig from this server config, copying mutable parameters. + /// + /// `maxmemory` resolves `None`/`Some(0)` → `0` (the downstream "unlimited" + /// sentinel). Call [`Self::apply_memory_guardrail`] BEFORE this if the G1 + /// auto-guardrail should populate an unset `--maxmemory`. pub fn to_runtime_config(&self) -> RuntimeConfig { RuntimeConfig { - maxmemory: self.maxmemory, + maxmemory: self.maxmemory.unwrap_or(0), maxmemory_policy: self.maxmemory_policy.clone(), maxmemory_samples: self.maxmemory_samples, lfu_log_factor: 10, @@ -674,8 +756,188 @@ impl ServerConfig { maxclients: self.maxclients, timeout: self.timeout, tcp_keepalive: self.tcp_keepalive, + // Default to single-shard (no division). The server overwrites this + // on the shared RuntimeConfig with the resolved shard count once it + // is known (main.rs / embedded.rs), so the per-shard eviction budget + // bounds aggregate RSS. Tests that call this directly run 1 shard. + num_shards: 1, + } + } +} + +/// Fraction (percent) of the detected memory limit used as the G1 auto +/// guardrail cap when `--maxmemory` is omitted. 80% leaves headroom for +/// allocator fragmentation, page cache, and non-keyspace overhead. +pub const MAXMEMORY_GUARDRAIL_PERCENT: u64 = 80; + +/// Result of resolving the G1 memory guardrail — drives the startup notice. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum GuardrailOutcome { + /// Operator set `--maxmemory` explicitly (including `0` = unlimited). + /// The value is honored verbatim; no guardrail applied. + Explicit(usize), + /// `--maxmemory` was omitted and a memory limit was detected: Moon + /// auto-capped at `cap_bytes` (`~PERCENT%` of `detected_limit_bytes`). + /// `policy_changed_to` is `Some` when a `noeviction` policy was switched + /// to an evicting one so the cap actually sheds memory instead of OOM-ing. + Applied { + cap_bytes: usize, + detected_limit_bytes: usize, + policy_changed_to: Option, + }, + /// `--maxmemory` was omitted but no memory limit could be detected (e.g. + /// non-Linux dev host, or `/proc`/`/sys` unreadable). Left UNLIMITED — the + /// caller warns the operator to set `--maxmemory` explicitly. + Skipped, +} + +/// Parse the `MemTotal:` line of `/proc/meminfo` contents into bytes. +/// Format: `MemTotal: 16384256 kB`. Pure for testability. +/// +/// Only compiled where used: the Linux detector and the unit tests (avoids a +/// dead-code warning on non-Linux non-test builds). +#[cfg(any(target_os = "linux", test))] +fn parse_meminfo_memtotal(contents: &str) -> Option { + for line in contents.lines() { + if let Some(rest) = line.strip_prefix("MemTotal:") { + let kb: usize = rest.split_whitespace().next()?.parse().ok()?; + return kb.checked_mul(1024); } } + None +} + +/// Parse a cgroup memory-limit file's contents into a byte cap. +/// +/// Handles cgroup v2 (`memory.max`: a number or the literal `max`) and v1 +/// (`memory.limit_in_bytes`: a number, with a near-`i64::MAX` "no limit" +/// sentinel). Returns `None` for "unlimited". Pure for testability. +#[cfg(any(target_os = "linux", test))] +fn parse_cgroup_mem_max(contents: &str) -> Option { + let t = contents.trim(); + if t.is_empty() || t == "max" { + return None; + } + let v: usize = t.parse().ok()?; + // cgroup v1 uses a huge page-rounded value (~i64::MAX) to mean "no limit". + if v >= (1usize << 62) { + return None; + } + Some(v) +} + +/// Detect the effective memory limit in bytes: the minimum of the cgroup limit +/// (v2 then v1) and host RAM. Linux-only; returns `None` elsewhere or when +/// nothing is readable (the guardrail then fails open with a warning). +#[cfg(target_os = "linux")] +fn detect_memory_limit_bytes() -> Option { + let host = std::fs::read_to_string("/proc/meminfo") + .ok() + .and_then(|c| parse_meminfo_memtotal(&c)); + let cgroup = std::fs::read_to_string("/sys/fs/cgroup/memory.max") + .ok() + .and_then(|c| parse_cgroup_mem_max(&c)) + .or_else(|| { + std::fs::read_to_string("/sys/fs/cgroup/memory/memory.limit_in_bytes") + .ok() + .and_then(|c| parse_cgroup_mem_max(&c)) + }); + match (host, cgroup) { + (Some(h), Some(c)) => Some(h.min(c)), + (Some(h), None) => Some(h), + (None, Some(c)) => Some(c), + (None, None) => None, + } +} + +/// Non-Linux: no portable, dependency-free memory-limit probe. The guardrail +/// is skipped (operator sets `--maxmemory` explicitly on dev hosts). Production +/// targets Linux per the platform policy. +#[cfg(not(target_os = "linux"))] +fn detect_memory_limit_bytes() -> Option { + None +} + +/// Pure resolution of the guardrail decision (no I/O) for unit testing. +fn resolve_memory_guardrail( + maxmemory: Option, + policy: &str, + detected_limit: Option, + percent: u64, +) -> GuardrailOutcome { + if let Some(explicit) = maxmemory { + return GuardrailOutcome::Explicit(explicit); + } + match detected_limit { + Some(limit) if limit > 0 => { + // u128 intermediate avoids overflow on large-RAM hosts. + let cap = ((limit as u128 * percent as u128) / 100) as usize; + let policy_changed_to = (policy == "noeviction").then(|| "allkeys-lru".to_string()); + GuardrailOutcome::Applied { + cap_bytes: cap, + detected_limit_bytes: limit, + policy_changed_to, + } + } + _ => GuardrailOutcome::Skipped, + } +} + +/// Emit the G1 startup notice for a resolved guardrail outcome. Shared by the +/// binary entry (`main`) and the embedded entry so the message is identical. +pub fn log_memory_guardrail(outcome: GuardrailOutcome) { + match outcome { + GuardrailOutcome::Applied { + cap_bytes, + detected_limit_bytes, + policy_changed_to, + } => { + let policy_note = policy_changed_to + .map(|p| format!("; eviction policy set to '{p}'")) + .unwrap_or_default(); + tracing::warn!( + "Memory guardrail: --maxmemory not set; auto-capping at {} bytes \ + (~{}% of detected {} bytes){}. Override with --maxmemory , \ + or --maxmemory 0 for unlimited.", + cap_bytes, + MAXMEMORY_GUARDRAIL_PERCENT, + detected_limit_bytes, + policy_note + ); + } + GuardrailOutcome::Skipped => { + tracing::warn!( + "Memory guardrail: --maxmemory not set and no memory limit could be \ + detected on this platform; running UNLIMITED. Set --maxmemory \ + to bound keyspace growth and avoid OOM termination." + ); + } + GuardrailOutcome::Explicit(_) => { /* operator chose; stay silent */ } + } +} + +/// Emit a one-line startup notice when `maxmemory` is split across multiple shards. +/// +/// `maxmemory` is a whole-instance cap, but each shard enforces eviction +/// independently against `maxmemory / num_shards` (see +/// [`RuntimeConfig::maxmemory_per_shard`]). Without this notice an operator +/// running e.g. `--maxmemory 8gb --shards 4` would silently get an 8 GB +/// (not 32 GB) effective ceiling and see "surprise" evictions with nothing +/// in the logs explaining why. Only fires when the division actually changes +/// the effective budget (`num_shards > 1` and a finite cap). +pub fn log_maxmemory_sharding(maxmemory: usize, num_shards: usize) { + if maxmemory == 0 || num_shards <= 1 { + return; + } + let per_shard = maxmemory.div_ceil(num_shards); + tracing::info!( + "maxmemory {} bytes is a whole-instance cap; each of {} shards enforces \ + eviction against a per-shard budget of {} bytes (maxmemory / shards). \ + CONFIG GET / INFO continue to report the whole-instance value.", + maxmemory, + num_shards, + per_shard + ); } /// Runtime-mutable configuration parameters. @@ -722,6 +984,38 @@ pub struct RuntimeConfig { pub timeout: u64, /// TCP keepalive interval in seconds (0 = disabled). pub tcp_keepalive: u64, + /// Resolved shard count — used only to derive the per-shard eviction budget. + /// + /// `maxmemory` is a whole-instance cap (Redis-compatible: `CONFIG GET` / + /// INFO report it verbatim). But each shard is shared-nothing and enforces + /// eviction independently, so without dividing, an N-shard server would + /// tolerate ~N×`maxmemory` before evicting. The per-shard threshold is + /// therefore `maxmemory / num_shards` (see [`RuntimeConfig::maxmemory_per_shard`]). + /// Defaults to `1` (single shard ⇒ no division, preserving prior behavior); + /// the server overwrites it on the shared instance at startup with the + /// resolved shard count. + pub num_shards: usize, +} + +impl RuntimeConfig { + /// Per-shard eviction budget in bytes. + /// + /// `maxmemory` is a whole-instance cap. Because each shard enforces eviction + /// independently (shared-nothing), the effective per-shard threshold is + /// `maxmemory / num_shards`, so the aggregate across all shards converges on + /// the configured whole-instance cap instead of overshooting it ~N×. + /// + /// Uses `div_ceil` so the summed per-shard budgets never undershoot the cap, + /// and `max(1)` on the divisor guards against a mis-set `num_shards`. Returns + /// `0` (unlimited) iff `maxmemory == 0`. + #[inline] + #[must_use] + pub fn maxmemory_per_shard(&self) -> usize { + if self.maxmemory == 0 { + return 0; + } + self.maxmemory.div_ceil(self.num_shards.max(1)) + } } impl Default for RuntimeConfig { @@ -746,6 +1040,7 @@ impl Default for RuntimeConfig { maxclients: 10000, timeout: 0, tcp_keepalive: 300, + num_shards: 1, } } } @@ -826,7 +1121,9 @@ mod tests { #[test] fn test_maxmemory_defaults() { let config = ServerConfig::parse_from::<[&str; 0], &str>([]); - assert_eq!(config.maxmemory, 0); + // G1: omitted flag parses to None (the auto-guardrail sentinel), + // distinct from an explicit `--maxmemory 0` (= unlimited). + assert_eq!(config.maxmemory, None); assert_eq!(config.maxmemory_policy, "noeviction"); assert_eq!(config.maxmemory_samples, 5); } @@ -842,11 +1139,104 @@ mod tests { "--maxmemory-samples", "10", ]); - assert_eq!(config.maxmemory, 1048576); + assert_eq!(config.maxmemory, Some(1048576)); assert_eq!(config.maxmemory_policy, "allkeys-lru"); assert_eq!(config.maxmemory_samples, 10); } + #[test] + fn test_maxmemory_explicit_zero_is_unlimited() { + // The Redis escape hatch: explicit `--maxmemory 0` must stay unlimited + // and NOT trigger the guardrail. + let mut config = ServerConfig::parse_from(["moon", "--maxmemory", "0"]); + assert_eq!(config.maxmemory, Some(0)); + let outcome = config.apply_memory_guardrail(); + assert_eq!(outcome, GuardrailOutcome::Explicit(0)); + assert_eq!(config.maxmemory, Some(0)); + assert_eq!(config.to_runtime_config().maxmemory, 0); + } + + // ── G1 pure resolution + parsing ── + + #[test] + fn guardrail_applies_percent_and_flips_noeviction() { + let out = resolve_memory_guardrail(None, "noeviction", Some(1000), 80); + assert_eq!( + out, + GuardrailOutcome::Applied { + cap_bytes: 800, + detected_limit_bytes: 1000, + policy_changed_to: Some("allkeys-lru".to_string()), + } + ); + } + + #[test] + fn guardrail_keeps_existing_evicting_policy() { + // Operator already chose an evicting policy → don't override it. + let out = resolve_memory_guardrail(None, "allkeys-lfu", Some(2000), 80); + assert_eq!( + out, + GuardrailOutcome::Applied { + cap_bytes: 1600, + detected_limit_bytes: 2000, + policy_changed_to: None, + } + ); + } + + #[test] + fn guardrail_explicit_value_is_honored() { + assert_eq!( + resolve_memory_guardrail(Some(4096), "noeviction", Some(1 << 30), 80), + GuardrailOutcome::Explicit(4096) + ); + } + + #[test] + fn guardrail_skipped_when_no_limit_detected() { + assert_eq!( + resolve_memory_guardrail(None, "noeviction", None, 80), + GuardrailOutcome::Skipped + ); + // Zero/invalid detection also skips (fail open, never cap at 0-by-math). + assert_eq!( + resolve_memory_guardrail(None, "noeviction", Some(0), 80), + GuardrailOutcome::Skipped + ); + } + + #[test] + fn guardrail_skipped_outcome_leaves_unlimited() { + let mut config = ServerConfig::parse_from::<[&str; 0], &str>([]); + // Force the Skipped branch deterministically by resolving against a + // None detection (mirrors non-Linux / unreadable /proc). + let out = resolve_memory_guardrail(config.maxmemory, &config.maxmemory_policy, None, 80); + assert_eq!(out, GuardrailOutcome::Skipped); + // apply_* on Skipped must concretize to Some(0) = unlimited. + if let GuardrailOutcome::Skipped = out { + config.maxmemory = Some(0); + } + assert_eq!(config.to_runtime_config().maxmemory, 0); + } + + #[test] + fn parse_meminfo_extracts_memtotal_bytes() { + let sample = "MemTotal: 16384256 kB\nMemFree: 1000 kB\n"; + assert_eq!(parse_meminfo_memtotal(sample), Some(16_384_256 * 1024)); + assert_eq!(parse_meminfo_memtotal("MemFree: 10 kB\n"), None); + assert_eq!(parse_meminfo_memtotal(""), None); + } + + #[test] + fn parse_cgroup_max_handles_v1_v2_sentinels() { + assert_eq!(parse_cgroup_mem_max("2147483648\n"), Some(2_147_483_648)); + assert_eq!(parse_cgroup_mem_max("max\n"), None); // v2 unlimited + assert_eq!(parse_cgroup_mem_max(""), None); + // v1 near-i64::MAX "no limit" sentinel. + assert_eq!(parse_cgroup_mem_max("9223372036854771712"), None); + } + #[test] fn test_to_runtime_config() { let config = ServerConfig::parse_from([ @@ -862,6 +1252,59 @@ mod tests { assert_eq!(rt.maxmemory_samples, 5); assert_eq!(rt.lfu_log_factor, 10); assert_eq!(rt.lfu_decay_time, 1); + // to_runtime_config defaults num_shards to 1 (no per-shard division until + // the server sets the resolved count on the shared instance). + assert_eq!(rt.num_shards, 1); + assert_eq!(rt.maxmemory_per_shard(), 1024); + } + + #[test] + fn maxmemory_per_shard_unlimited_stays_zero() { + let mut rt = RuntimeConfig::default(); + rt.maxmemory = 0; + for n in [1, 2, 4, 16] { + rt.num_shards = n; + assert_eq!( + rt.maxmemory_per_shard(), + 0, + "unlimited (0) must stay 0 regardless of shard count" + ); + } + } + + #[test] + fn maxmemory_per_shard_single_shard_is_whole_instance() { + let mut rt = RuntimeConfig::default(); + rt.maxmemory = 1_000; + rt.num_shards = 1; + assert_eq!(rt.maxmemory_per_shard(), 1_000); + } + + #[test] + fn maxmemory_per_shard_divides_by_shard_count() { + let mut rt = RuntimeConfig::default(); + rt.maxmemory = 400; + rt.num_shards = 4; + assert_eq!(rt.maxmemory_per_shard(), 100); + } + + #[test] + fn maxmemory_per_shard_div_ceil_never_undershoots() { + // 10 / 3 = 3.33 -> ceil 4 so the summed per-shard budgets (12) >= cap (10). + let mut rt = RuntimeConfig::default(); + rt.maxmemory = 10; + rt.num_shards = 3; + assert_eq!(rt.maxmemory_per_shard(), 4); + assert!(rt.maxmemory_per_shard() * rt.num_shards >= rt.maxmemory); + } + + #[test] + fn maxmemory_per_shard_guards_zero_shard_count() { + // A mis-set num_shards == 0 must not divide-by-zero; treat as 1 shard. + let mut rt = RuntimeConfig::default(); + rt.maxmemory = 500; + rt.num_shards = 0; + assert_eq!(rt.maxmemory_per_shard(), 500); } #[test] diff --git a/src/graph/cypher/mod.rs b/src/graph/cypher/mod.rs index 13a5985ea..f710076fa 100644 --- a/src/graph/cypher/mod.rs +++ b/src/graph/cypher/mod.rs @@ -5,7 +5,8 @@ //! DELETE, SET, MERGE, WITH, UNWIND, CALL ... YIELD. //! //! Parameterized queries (`$param`) prevent Cypher injection. -//! Nesting depth is limited (default 64) to prevent stack overflow DoS. +//! Nesting depth is limited (`DEFAULT_MAX_NESTING_DEPTH`) to prevent a +//! stack-overflow DoS — deep nesting returns `NestingDepthExceeded`, never aborts. pub mod ast; pub mod executor; @@ -15,14 +16,15 @@ pub mod planner; pub use ast::{Clause, CypherQuery, Expr}; pub use executor::{ExecResult, OpProfile, ProfileResult, Value}; -pub use parser::{CypherError, Parser}; +pub use parser::{CypherError, DEFAULT_MAX_NESTING_DEPTH, Parser}; pub use planner::{CostEstimate, PhysicalPlan, PlanCache, Strategy}; /// Parse a Cypher query from a byte slice. /// -/// Uses the default nesting depth limit of 64. +/// Uses `DEFAULT_MAX_NESTING_DEPTH` so pathologically nested input returns +/// `NestingDepthExceeded` instead of overflowing the (2 MiB worker-thread) stack. pub fn parse_cypher(input: &[u8]) -> Result { - let mut parser = Parser::new(input, 64); + let mut parser = Parser::new(input, DEFAULT_MAX_NESTING_DEPTH); parser.parse() } diff --git a/src/graph/cypher/parser/mod.rs b/src/graph/cypher/parser/mod.rs index ab63f3a89..d24a65ffa 100644 --- a/src/graph/cypher/parser/mod.rs +++ b/src/graph/cypher/parser/mod.rs @@ -2,7 +2,7 @@ //! //! Operates on tokens from the `logos`-based lexer. Each clause type has its //! own parsing method. Nesting depth is tracked to prevent stack overflow -//! from pathologically nested expressions (default limit: 64). +//! from pathologically nested expressions (`DEFAULT_MAX_NESTING_DEPTH`). //! //! Parameters (`$name`) are stored as `Expr::Parameter` -- never interpolated //! into the query string, preventing Cypher injection (CVE-2024-8309). @@ -73,6 +73,20 @@ impl core::fmt::Display for CypherError { } } +/// Default nesting-depth limit for Cypher expressions/patterns. +/// +/// The recursive-descent expression grammar pushes ~9 stack frames per source +/// nesting level (`parse_expression` → or → and → not → comparison → addition → +/// multiplication → unary → primary → recurse). `check_depth` only counts source +/// levels, so the limit must be low enough that `limit × ~9` debug frames fit the +/// SMALLEST stack the parser can run on. Cypher is parsed on async worker threads +/// (tokio/monoio default to 2 MiB stacks, not the 8 MiB main thread), so a limit +/// that overflows 2 MiB is a real DoS: a pathologically nested query aborts the +/// process (SIGABRT) instead of returning `NestingDepthExceeded`. 32 levels keeps +/// the worst-case descent to ~288 frames (well under 2 MiB even in debug) while +/// still allowing far deeper nesting than any real query needs. +pub const DEFAULT_MAX_NESTING_DEPTH: u32 = 32; + /// Recursive descent Cypher parser. pub struct Parser<'a> { lexer: Lexer<'a>, @@ -562,7 +576,7 @@ mod tests { use super::*; fn parse(input: &str) -> Result { - let mut parser = Parser::new(input.as_bytes(), 64); + let mut parser = Parser::new(input.as_bytes(), DEFAULT_MAX_NESTING_DEPTH); parser.parse() } @@ -754,22 +768,30 @@ mod tests { #[test] fn test_nesting_depth_exceeded() { - // Create deeply nested expression: (((((...))))) > 64 levels + // Nesting well beyond DEFAULT_MAX_NESTING_DEPTH must return a graceful + // error — NOT overflow the stack (SIGABRT). The guard has to fire before + // the recursive-descent frames exhaust a 2 MiB worker-thread stack, which + // is why the limit is bounded; this test runs on a default (small) test + // thread stack and historically aborted the whole process here. + let depth = (DEFAULT_MAX_NESTING_DEPTH as usize) + 8; let mut query = String::from("MATCH (n) WHERE "); - for _ in 0..70 { + for _ in 0..depth { query.push('('); } query.push('1'); - for _ in 0..70 { + for _ in 0..depth { query.push(')'); } query.push_str(" = 1 RETURN n"); let result = parse(&query); assert!( - matches!(result, Err(CypherError::NestingDepthExceeded { limit: 64 })), - "expected NestingDepthExceeded, got {:?}", - result + matches!( + result, + Err(CypherError::NestingDepthExceeded { limit }) + if limit == DEFAULT_MAX_NESTING_DEPTH + ), + "expected NestingDepthExceeded(limit={DEFAULT_MAX_NESTING_DEPTH}), got {result:?}" ); } diff --git a/src/main.rs b/src/main.rs index 0e0cf082e..1d55d2099 100644 --- a/src/main.rs +++ b/src/main.rs @@ -68,7 +68,7 @@ fn main() -> anyhow::Result<()> { ) .init(); - let config = ServerConfig::parse(); + let mut config = ServerConfig::parse(); // ── AOF v1→v2 migration (FIX-W3-2): early-exit before normal boot ── // When `--migrate-aof-from` is set, run the migration tool and exit. @@ -128,6 +128,11 @@ fn main() -> anyhow::Result<()> { ); } + // G1 memory guardrail: resolve --maxmemory before any RuntimeConfig is + // built so an unset cap is auto-populated (cgroup-aware) and the startup + // notice prints exactly once. + moon::config::log_memory_guardrail(config.apply_memory_guardrail()); + // Build TLS configuration if tls_port is set. // Uses ArcSwap for SIGHUP-based certificate hot-reload. let tls_config: Option = if config.tls_port > 0 { @@ -482,7 +487,15 @@ fn main() -> anyhow::Result<()> { "AOF enabled (PerShard, {} writers, fsync: {:?})", num_shards, fsync ); - Some(AofWriterPool::per_shard_with_policy(senders, fsync)) + // [F6] per_shard_with_base_dir records the persistence base dir so a + // per-shard BGREWRITEAOF can load the authoritative manifest fresh + // at rewrite time (try_send_rewrite_per_shard). + Some(AofWriterPool::per_shard_with_base_dir( + senders, + fsync, + std::time::Duration::from_millis(config.aof_fsync_timeout_ms), + base_dir.clone(), + )) } else { let (tx, rx) = channel::mpsc_bounded::(10_000); let aof_token = cancel_token.child_token(); @@ -499,7 +512,11 @@ fn main() -> anyhow::Result<()> { }) .expect("failed to spawn AOF writer thread"); info!("AOF enabled (TopLevel, fsync: {:?})", fsync); - Some(AofWriterPool::top_level_with_policy(tx, fsync)) + Some(AofWriterPool::top_level_with_policy( + tx, + fsync, + std::time::Duration::from_millis(config.aof_fsync_timeout_ms), + )) } } else { None @@ -518,14 +535,30 @@ fn main() -> anyhow::Result<()> { // Verified 2026-05-26: multi-shard BGREWRITEAOF loses ~38% of keys on // restart. Gate lifted only when multi-part AOF replay ships (v2.0+). // See docs/runbooks/multi-shard-aof-rewrite.md. + // [F6] When `--experimental-per-shard-rewrite` is set, leave the gate OPEN + // so BGREWRITEAOF routes to the per-shard fan-out coordinator + // (try_send_rewrite_per_shard): synchronized seq bump + single manifest + // commit across all per-shard writers, validated by + // tests/crash_matrix_per_shard_bgrewriteaof.rs. Default (flag off) keeps + // the gate closed — the shipped, crash-safe "no in-place compaction" + // behavior that avoided the historical ~38%-key-loss-on-restart. if config.per_shard_aof_active(num_shards) { - moon::command::persistence::MULTI_SHARD_AOF_REWRITE_UNSAFE - .store(true, std::sync::atomic::Ordering::Relaxed); - tracing::warn!( - shards = num_shards, - appendonly = %config.appendonly, - "BGREWRITEAOF gated: per-shard AOF layout active (see docs/runbooks/multi-shard-aof-rewrite.md). Use --shards 1 to re-enable rewrite." - ); + if config.experimental_per_shard_rewrite { + tracing::warn!( + shards = num_shards, + appendonly = %config.appendonly, + "BGREWRITEAOF per-shard rewrite ENABLED (--experimental-per-shard-rewrite). \ + Per-shard fan-out compaction is active; this path is experimental." + ); + } else { + moon::command::persistence::MULTI_SHARD_AOF_REWRITE_UNSAFE + .store(true, std::sync::atomic::Ordering::Relaxed); + tracing::warn!( + shards = num_shards, + appendonly = %config.appendonly, + "BGREWRITEAOF gated: per-shard AOF layout active (see docs/runbooks/multi-shard-aof-rewrite.md). Use --shards 1, or --experimental-per-shard-rewrite to enable per-shard compaction." + ); + } } // Create watch channel for snapshot triggers (auto-save and BGSAVE) @@ -578,6 +611,11 @@ fn main() -> anyhow::Result<()> { // Build shared runtime config for sharded handlers let runtime_config_shared: std::sync::Arc> = { std::sync::Arc::new(parking_lot::RwLock::new(config.to_runtime_config())) }; + // Publish the resolved shard count so eviction enforces maxmemory as a + // whole-instance cap (per-shard budget = maxmemory / num_shards). Without + // this, each shard would tolerate the full maxmemory → ~N× aggregate RSS. + runtime_config_shared.write().num_shards = num_shards; + moon::config::log_maxmemory_sharding(runtime_config_shared.read().maxmemory, num_shards); let server_config_shared: std::sync::Arc = { std::sync::Arc::new(config.clone()) }; @@ -652,8 +690,18 @@ fn main() -> anyhow::Result<()> { config.initial_keyspace_hint, config.to_runtime_config(), ); - if let Some(ref dir) = persistence_dir { - shard.restore_from_persistence(dir, disk_offload_base.as_deref()); + // Recover whenever there is something to recover. Disk-offload cold + // recovery (v3: heap reload + rebuild_from_manifest) is INDEPENDENT of + // AOF, but `persistence_dir` is intentionally None under appendonly=no + // (to avoid per-tick WAL fsync writers). Gating recovery on it alone + // silently dropped ALL cold data on restart under --appendonly no + + // disk-offload (cold read-through 0/200; "v3 recovery complete" never + // logged). Fire recovery when an offload base exists too; the v3 path + // reads the offload manifest, and the dir arg is used only by the v2 + // fallback (a no-op when no appendonly.aof/snapshot exists). + if persistence_dir.is_some() || disk_offload_base.is_some() { + let recover_dir = persistence_dir.as_deref().unwrap_or(config.dir.as_str()); + shard.restore_from_persistence(recover_dir, disk_offload_base.as_deref()); } // Initialize cold_index + cold_shard_dir for disk offload if let Some(ref offload_base) = disk_offload_base { @@ -669,20 +717,62 @@ fn main() -> anyhow::Result<()> { }) .collect(); - // Multi-part AOF replay layered on top of v2/v3 recovery. + // Multi-part AOF replay/init layered on top of v2/v3 recovery. // Priority: if appendonlydir/ manifest exists → load multi-part (skip legacy v2 fallback). // Otherwise v2 already handled legacy appendonly.aof during restore_from_persistence. // - // A corrupt manifest is FATAL: overwriting it silently destroys the reference - // to the real base RDB and loses all persisted data. + // A corrupt manifest is FATAL on BOTH runtimes (previously warn+continue + // under tokio): overwriting it silently destroys the reference to the real + // base RDB and loses all persisted data. // - // Gated to runtime-monoio: BGREWRITEAOF under runtime-tokio writes a - // single-file appendonly.aof with RDB preamble (legacy v2 format) and - // never advances the manifest. Engaging this block under tokio creates an - // empty manifest at first boot, then on the next boot wipes v2-loaded - // state because the multi-part replay finds no base RDB. This caused the - // tokio TXN replay regression that surfaced via test_txn_commit_wal_crash_recovery. - #[cfg(feature = "runtime-monoio")] + // Runtime split (the regression boundary): the multi-shard PerShard paths + // (initialize_multi + replay_per_shard) run under BOTH runtimes — the tokio + // per_shard_aof_writer_task writes the identical framed incr format + // (`[u64 lsn LE][u32 len LE][RESP]`) and replay_per_shard is + // runtime-agnostic. The SINGLE-shard multi-part paths stay monoio-only: + // tokio --shards 1 uses legacy single-file appendonly.aof (v2) + in-place + // BGREWRITEAOF, so engaging multi-part replay there finds an empty manifest, + // no rewritten base RDB, and wipes the v2-loaded state — the + // test_txn_commit_wal_crash_recovery regression. Each single-shard branch + // below is therefore #[cfg(runtime-monoio)] with a tokio warn fallback. + // + // ── B-2: preserve cold-tier wiring across AOF/RDB recovery ────────────── + // The multi-part AOF replay below invokes `rdb::load`, which loads the base + // RDB into fresh `Database::new()` temporaries and swaps them wholesale into + // the live databases (`*live = temp`). That swap silently drops + // `cold_shard_dir` and the rebuilt `cold_index`: both are live-tier topology, + // NOT part of the RDB hot snapshot, so a 0-key base RDB still wipes them. + // Capture them here — after `restore_from_persistence` rebuilt the index and + // the map wired the dir, before replay clobbers — and re-attach after the + // replay block so disk-offload read-through survives a restart. + // + // The fix lives here (recovery path), NOT in the generic `rdb::load`, on + // purpose: `rdb::load` also serves replica full-sync and DEBUG RELOAD, which + // load a *foreign* dataset whose values do NOT live in this node's cold + // files — preserving the local index there would surface stale reads. Here + // the loaded base + replayed incrs are this node's own data, so the rebuilt + // index is authoritative. Pairs with the spill file_id seed + // (eviction.rs::next_spill_file_id_seed): the seed keeps recovered cold + // files immutable so these preserved entries stay valid until the + // steady-state cascade refreshes them. + let preserved_cold_wiring: Vec< + Vec<( + Option, + Option, + )>, + > = if disk_offload_base.is_some() { + shards + .iter_mut() + .map(|s| { + s.databases + .iter_mut() + .map(|db| (db.cold_shard_dir.take(), db.cold_index.take())) + .collect() + }) + .collect() + } else { + Vec::new() + }; if config.appendonly == "yes" && let Some(ref dir) = persistence_dir { @@ -698,42 +788,67 @@ fn main() -> anyhow::Result<()> { })?; if let Some(ref manifest) = manifest_opt { if num_shards == 1 { - // Multi-part AOF is authoritative. Wipe any state that earlier - // recovery phases (per-shard WAL replay, legacy appendonly.aof - // fallback inside restore_from_persistence) may have loaded — - // otherwise non-idempotent commands from the incr log would - // double-apply on top of that pre-existing state. - for db in shards[0].databases.iter_mut() { - db.clear(); - } - let loaded = moon::persistence::aof_manifest::replay_multi_part( - &mut shards[0].databases, - manifest, - &DispatchReplayEngine::new(), - ) - .with_context(|| "multi-part AOF replay failed")?; - info!( - "AOF multi-part loaded (seq {}): {} entries", - manifest.seq, loaded - ); + // Single-shard multi-part replay is monoio-only (the regression + // boundary). Under tokio, --shards 1 uses legacy v2 single-file + // recovery; engaging multi-part replay here wipes v2 state. + #[cfg(feature = "runtime-monoio")] + { + // Multi-part AOF is authoritative. Wipe any state that earlier + // recovery phases (per-shard WAL replay, legacy appendonly.aof + // fallback inside restore_from_persistence) may have loaded — + // otherwise non-idempotent commands from the incr log would + // double-apply on top of that pre-existing state. + for db in shards[0].databases.iter_mut() { + db.clear(); + } + let loaded = moon::persistence::aof_manifest::replay_multi_part( + &mut shards[0].databases, + manifest, + &DispatchReplayEngine::new(), + ) + .with_context(|| "multi-part AOF replay failed")?; + info!( + "AOF multi-part loaded (seq {}): {} entries", + manifest.seq, loaded + ); - // Retire legacy appendonly.aof so future boots don't double- - // replay it via restore_from_persistence's fallback path. - // Rename (not delete) so an operator can recover if something - // went wrong. - let legacy = base_dir.join("appendonly.aof"); - if legacy.exists() { - let retired = base_dir.join("appendonly.aof.legacy"); - if let Err(e) = std::fs::rename(&legacy, &retired) { - tracing::warn!("Failed to retire legacy AOF {}: {}", legacy.display(), e); - } else { - info!( - "Retired legacy AOF {} → {}", - legacy.display(), - retired.display() - ); + // Retire legacy appendonly.aof so future boots don't double- + // replay it via restore_from_persistence's fallback path. + // Rename (not delete) so an operator can recover if something + // went wrong. + let legacy = base_dir.join("appendonly.aof"); + if legacy.exists() { + let retired = base_dir.join("appendonly.aof.legacy"); + if let Err(e) = std::fs::rename(&legacy, &retired) { + tracing::warn!( + "Failed to retire legacy AOF {}: {}", + legacy.display(), + e + ); + } else { + info!( + "Retired legacy AOF {} → {}", + legacy.display(), + retired.display() + ); + } } } + #[cfg(not(feature = "runtime-monoio"))] + { + // tokio + --shards 1: single-shard multi-part replay is + // monoio-only. Legacy v2 (appendonly.aof) recovery already + // ran in restore_from_persistence; warn so an operator who + // switched from monoio knows multi-part data isn't loaded by + // this build. + tracing::warn!( + "multi-part AOF manifest at {}/appendonlydir/ found but runtime is \ + tokio with --shards 1; single-shard multi-part replay is monoio-only. \ + Legacy v2 (appendonly.aof) recovery active. Switch to monoio to load \ + multi-part single-shard data.", + base_dir.display() + ); + } } else if manifest.layout == moon::persistence::aof_manifest::AofLayout::PerShard { // Per-shard AOF replay (RFC § 2 rules 1-3, Option B step 4). // @@ -877,22 +992,33 @@ fn main() -> anyhow::Result<()> { // mode the multi-part path currently supports). let has_state = num_shards == 1 && shards[0].databases.iter().any(|db| db.len() > 0); if has_state { - let rdb_bytes = moon::persistence::rdb::save_to_bytes(&shards[0].databases) - .with_context(|| "failed to serialize legacy state for AOF base")?; - AofManifest::initialize_with_base(&base_dir, &rdb_bytes) - .with_context(|| "failed to initialize AOF manifest with base")?; - info!( - "First-upgrade: captured legacy state as AOF base seq 1 ({} bytes)", - rdb_bytes.len() - ); - // Retire legacy appendonly.aof — its contents are now in - // the base RDB, and leaving it would cause v2 recovery on - // the next boot to double-replay it. - let legacy = base_dir.join("appendonly.aof"); - if legacy.exists() { - let retired = base_dir.join("appendonly.aof.legacy"); - if let Err(e) = std::fs::rename(&legacy, &retired) { - tracing::warn!("Failed to retire legacy AOF {}: {}", legacy.display(), e); + // Single-shard legacy-upgrade capture is monoio-only. Under + // tokio, --shards 1 keeps v2 single-file recovery (no manifest); + // creating a seq-1 base here would trigger the empty-manifest + // replay regression on the next boot. + #[cfg(feature = "runtime-monoio")] + { + let rdb_bytes = moon::persistence::rdb::save_to_bytes(&shards[0].databases) + .with_context(|| "failed to serialize legacy state for AOF base")?; + AofManifest::initialize_with_base(&base_dir, &rdb_bytes) + .with_context(|| "failed to initialize AOF manifest with base")?; + info!( + "First-upgrade: captured legacy state as AOF base seq 1 ({} bytes)", + rdb_bytes.len() + ); + // Retire legacy appendonly.aof — its contents are now in + // the base RDB, and leaving it would cause v2 recovery on + // the next boot to double-replay it. + let legacy = base_dir.join("appendonly.aof"); + if legacy.exists() { + let retired = base_dir.join("appendonly.aof.legacy"); + if let Err(e) = std::fs::rename(&legacy, &retired) { + tracing::warn!( + "Failed to retire legacy AOF {}: {}", + legacy.display(), + e + ); + } } } } else if num_shards >= 2 { @@ -910,28 +1036,32 @@ fn main() -> anyhow::Result<()> { base_dir.display() ); } else { + // Single-shard fresh boot. + #[cfg(feature = "runtime-monoio")] AofManifest::initialize(&base_dir) .with_context(|| "failed to initialize AOF manifest")?; + // tokio --shards 1 fresh: no manifest (v2 single-file recovery + // owns single-shard durability). Creating one here would trigger + // the empty-manifest replay regression. } } } - // Under tokio, multi-part AOF is not supported. If a manifest exists, the - // operator likely switched from monoio — warn so they don't think their - // data is silently corrupted. v2 recovery (single-file appendonly.aof) - // remains active. - #[cfg(not(feature = "runtime-monoio"))] - if config.appendonly == "yes" - && let Some(ref dir) = persistence_dir - { - let manifest_path = std::path::PathBuf::from(dir).join("appendonlydir/moon.aof.manifest"); - if manifest_path.exists() { - tracing::warn!( - "multi-part AOF manifest found at {} but runtime is tokio; ignoring. \ - Switch to monoio (cargo run --no-default-features --features runtime-monoio,jemalloc) \ - to load multi-part AOF data.", - manifest_path.display() - ); + // (The former standalone tokio "multi-part AOF ignored" warn block was + // removed: multi-shard PerShard AOF is now loaded on tokio too, and the + // single-shard tokio warn is emitted inline above.) + + // ── B-2: re-attach cold-tier wiring dropped by the rdb::load swap ─────── + // Restore the `cold_shard_dir` + rebuilt `cold_index` captured before the + // AOF replay block. Without this, the handler reads a database with + // `cold_shard_dir = None`, so every read-through of a re-evicted key misses + // and disk-offload silently loses data across a restart. + if !preserved_cold_wiring.is_empty() { + for (shard, dbs) in shards.iter_mut().zip(preserved_cold_wiring) { + for (db, (cold_shard_dir, cold_index)) in shard.databases.iter_mut().zip(dbs) { + db.cold_shard_dir = cold_shard_dir; + db.cold_index = cold_index; + } } } @@ -1146,7 +1276,20 @@ fn main() -> anyhow::Result<()> { info!("Cluster bus and gossip ticker started"); } - let per_shard_accept = cfg!(target_os = "linux"); + // The central tokio listener plain-binds the port (no SO_REUSEPORT, + // see listener::run_sharded), which makes EVERY per-shard SO_REUSEPORT + // bind fail with EADDRINUSE — both the io_uring multishot path and the + // non-uring per_shard_listener fall back to `conn_rx`. `conn_rx`'s only + // feeder is THIS central accept loop, so it MUST run. Gating it off on + // Linux (the old `cfg!(target_os = "linux")`) left nobody accepting: the + // server bound the port and kernel-accepted TCP but never dispatched a + // command — it hung (the "zombie eating RAM" signature). Keep the central + // accept loop always on for tokio: identical to the already-working macOS + // tokio path and to monoio's central-accept model. Per-shard SO_REUSEPORT + // accept on tokio/Linux additionally rides the io_uring-accept path that + // is known-fragile under load; central-accept + conn_rx is the proven one. + // Guarded by tests/multishard_serve_smoke.rs (non-ignored, both runtimes). + let per_shard_accept = false; if let Err(e) = server::listener::run_sharded( config, conn_txs, diff --git a/src/persistence/aof.rs b/src/persistence/aof.rs index d4d3bd5a7..77b8d09b5 100644 --- a/src/persistence/aof.rs +++ b/src/persistence/aof.rs @@ -15,7 +15,7 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; -use std::time::Instant; +use std::time::{Duration, Instant}; use crate::runtime::cancel::CancellationToken; use crate::runtime::channel; @@ -92,6 +92,25 @@ pub enum AofAck { pub static AOF_BACKPRESSURE_DROPPED: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); +/// Result of awaiting an `AppendSync` ack under a bounded timeout (F2). +/// +/// Distinguishes the three terminal states the `Always` durability path +/// can reach so the caller can map each to the correct client-facing +/// outcome: +/// - `Ack(_)` — the writer reported back; inspect the `AofAck`. +/// - `Disconnected` — the writer task is gone / channel dropped (no ack +/// will ever arrive). Treated as `WriteFailed`. +/// - `TimedOut` — the fsync did not confirm within the configured +/// bound. Durability is unconfirmed; treated as `FsyncFailed`. The +/// entry may still reach disk later, so the caller must NOT report +/// success but also must not assume the write was rejected. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum AckOutcome { + Ack(AofAck), + Disconnected, + TimedOut, +} + /// AOF fsync policy controlling when data is flushed to disk. #[derive(Debug, Clone, Copy, PartialEq)] pub enum FsyncPolicy { @@ -160,10 +179,154 @@ pub enum AofMessage { Rewrite(SharedDatabases), /// Trigger AOF rewrite in sharded mode (all shards' databases). RewriteSharded(Arc), + /// [F6] Trigger a per-shard AOF rewrite (compaction) in the PerShard + /// layout. Sent to EVERY per-shard writer at once. Each writer folds its + /// own shard (drain → lock → snapshot → write new base+incr at the + /// coordinator's `new_seq` → reopen), then decrements the shared + /// `PerShardRewriteCoord`; the last writer commits the manifest once + /// (single seq flip) and prunes the old generation. The synchronized seq + /// + single commit are what make multi-shard BGREWRITEAOF crash-safe. + RewritePerShard { + shard_dbs: Arc, + coord: Arc, + }, /// Shut down the AOF writer task gracefully. Shutdown, } +/// Coordinator shared by all per-shard writers participating in one +/// BGREWRITEAOF fan-out (F6). +/// +/// Crash-safety contract (mirrors `AofManifest::advance` ordering, but +/// distributed across N writer threads): +/// +/// 1. Each writer writes its new base+incr at `new_seq` via +/// `manifest.advance_shard(shard_id, new_seq, rdb)` — which does NOT bump +/// `manifest.seq` or rewrite the manifest. So until the final commit, the +/// on-disk manifest still resolves to `old_seq`; a crash here recovers the +/// intact old generation (no loss, no double-apply). +/// 2. The LAST writer to finish (countdown reaches zero) performs the single +/// durable commit: `manifest.seq = new_seq; write_manifest()`. This is the +/// atomic point at which recovery flips to the new generation. +/// 3. Only AFTER the commit are the old-generation files pruned. +/// +/// The manifest is shared via `Arc>` and locked ONLY for the brief, +/// await-free `advance_shard` and final-commit critical sections — never held +/// across a blocking disk write of the base RDB (that happens before the lock) +/// nor across `.await`. +pub struct PerShardRewriteCoord { + /// Writers still to finish. Starts at the shard count; the writer that + /// decrements it to zero performs the commit + prune. + remaining: std::sync::atomic::AtomicUsize, + /// Shared manifest, loaded fresh from disk by the BGREWRITEAOF handler at + /// rewrite time (normal appends never touch the manifest, and BGREWRITEAOF + /// is CAS-serialized, so a fresh load is the authoritative current state). + manifest: Arc>, + /// The generation every writer advances to. Computed once = old_seq + 1. + new_seq: u64, + /// The generation being retired; pruned only after the commit. + old_seq: u64, + /// Number of shards participating (= initial `remaining`). + n_shards: usize, + /// Set by any shard whose fold fails. The final writer checks this and + /// ABORTS the commit if set — committing `new_seq` while a shard never + /// wrote its new base would make recovery look for a missing base and + /// refuse to start. On abort the old generation (`old_seq`) stays the + /// committed state for all shards (crash-safe). + failed: std::sync::atomic::AtomicBool, +} + +impl PerShardRewriteCoord { + /// Construct a coordinator for an `n_shards`-way rewrite advancing the + /// shared `manifest` from its current seq to `current_seq + 1`. + pub fn new( + manifest: Arc>, + current_seq: u64, + n_shards: usize, + ) -> Arc { + Arc::new(Self { + remaining: std::sync::atomic::AtomicUsize::new(n_shards), + manifest, + new_seq: current_seq + 1, + old_seq: current_seq, + n_shards, + failed: std::sync::atomic::AtomicBool::new(false), + }) + } + + /// The generation writers advance to. + #[inline] + pub fn new_seq(&self) -> u64 { + self.new_seq + } + + /// Mark the whole rewrite as failed (called by a shard whose fold errored). + /// The final writer will abort the commit, leaving `old_seq` authoritative. + #[inline] + pub fn mark_failed(&self) { + self.failed + .store(true, std::sync::atomic::Ordering::Release); + } + + /// Called by each writer AFTER it has durably written its new base+incr at + /// `new_seq` and reopened its append file. Decrements the countdown; the + /// final caller commits the manifest (single seq flip) and prunes the old + /// generation, then clears the global in-progress flag. + /// + /// Crash-safety: the commit (`write_manifest`) is the atomic flip point; + /// pruning runs strictly after it, so a crash mid-prune only orphans + /// already-superseded files (recovery uses `new_seq`). + pub fn shard_done(&self) { + use std::sync::atomic::Ordering; + // AcqRel: the decrement-to-zero must observe all prior writers' + // advance_shard manifest mutations before committing. + if self.remaining.fetch_sub(1, Ordering::AcqRel) == 1 { + // Abort if any shard failed to fold: committing new_seq while a + // shard lacks its new base would break recovery. Keep old_seq. + if self.failed.load(Ordering::Acquire) { + let m = self.manifest.lock(); + // Best-effort: prune the orphaned new-seq files written by the + // shards that DID fold, so they don't linger. + for sid in 0..self.n_shards { + m.prune_shard_files(sid as u16, self.new_seq); + } + drop(m); + error!( + "F6 per-shard rewrite ABORTED: a shard failed to fold; seq stays {}. \ + Old generation remains authoritative (crash-safe). A RESTART is \ + recommended so successful shards' writers stop appending to the \ + discarded new generation.", + self.old_seq + ); + crate::command::persistence::AOF_REWRITE_IN_PROGRESS.store(false, Ordering::SeqCst); + return; + } + let mut m = self.manifest.lock(); + m.seq = self.new_seq; + if let Err(e) = m.write_manifest() { + error!( + "F6 per-shard rewrite: final manifest commit (seq {}) failed: {}. \ + Old generation remains authoritative; rewrite did not take effect.", + self.new_seq, e + ); + // Do NOT prune — old generation is still the committed state. + drop(m); + crate::command::persistence::AOF_REWRITE_IN_PROGRESS.store(false, Ordering::SeqCst); + return; + } + for sid in 0..self.n_shards { + m.prune_shard_files(sid as u16, self.old_seq); + } + drop(m); + info!( + "F6 per-shard rewrite complete: committed seq {} across {} shards, pruned seq {}", + self.new_seq, self.n_shards, self.old_seq + ); + crate::command::persistence::AOF_REWRITE_IN_PROGRESS.store(false, Ordering::SeqCst); + } + } +} + /// Reasons a pool send may be refused without queueing. #[derive(Debug, Clone, PartialEq, Eq)] pub enum AofPoolSendError { @@ -187,6 +350,11 @@ pub enum AofPoolSendError { /// Step 2a is additive — this type is defined here but no call site is wired /// to it yet. Step 2c performs the type plumbing in `conn_state` and /// `conn/core`; steps 2d/2e/2f update the call sites and spawn paths. +/// Default bound for the `appendfsync=always` fsync-ack await. Mirrors the +/// `--aof-fsync-timeout-ms` config default; used by constructors that don't +/// take an explicit timeout (non-production / test helpers). +pub const DEFAULT_AOF_FSYNC_TIMEOUT: Duration = Duration::from_millis(2000); + #[derive(Clone)] pub struct AofWriterPool { senders: Vec>, @@ -196,6 +364,16 @@ pub struct AofWriterPool { /// fsync-before-ack durability (H1 fix); everything else stays on /// the fire-and-forget `Append` path. fsync_policy: FsyncPolicy, + /// F2: max time `try_send_append_durable` waits for the `Always` fsync + /// ack before failing the write. `Duration::ZERO` means unbounded + /// (legacy behavior). Prevents a stalled disk from parking write + /// connections forever (design-for-failure). + fsync_timeout: Duration, + /// F6: persistence base dir (the parent of `appendonlydir/`), set only for + /// PerShard pools that may service a per-shard BGREWRITEAOF. Needed to load + /// the authoritative manifest fresh at rewrite time. `None` for TopLevel + /// pools and test pools that never rewrite. + base_dir: Option, } impl AofWriterPool { @@ -203,20 +381,25 @@ impl AofWriterPool { /// legacy v1 deployments and `--shards 1` v2 deployments where one writer /// thread services every shard. pub fn top_level(sender: channel::MpscSender) -> Arc { - Self::top_level_with_policy(sender, FsyncPolicy::EverySec) + Self::top_level_with_policy(sender, FsyncPolicy::EverySec, DEFAULT_AOF_FSYNC_TIMEOUT) } /// Same as [`Self::top_level`] but with an explicit fsync policy. The /// policy controls whether [`Self::try_send_append_durable`] takes the /// fast (fire-and-forget) or rendezvous (`AppendSync`) path. + /// `fsync_timeout` bounds the `Always` ack await (F2); `Duration::ZERO` + /// = unbounded. pub fn top_level_with_policy( sender: channel::MpscSender, fsync_policy: FsyncPolicy, + fsync_timeout: Duration, ) -> Arc { Arc::new(Self { senders: vec![sender], layout: crate::persistence::aof_manifest::AofLayout::TopLevel, fsync_policy, + fsync_timeout, + base_dir: None, }) } @@ -225,13 +408,39 @@ impl AofWriterPool { /// shard count; passing a length-1 vector here is a bug — use /// [`AofWriterPool::top_level`] instead. pub fn per_shard(senders: Vec>) -> Arc { - Self::per_shard_with_policy(senders, FsyncPolicy::EverySec) + Self::per_shard_with_policy(senders, FsyncPolicy::EverySec, DEFAULT_AOF_FSYNC_TIMEOUT) } /// Same as [`Self::per_shard`] but with an explicit fsync policy. + /// `fsync_timeout` bounds the `Always` ack await (F2); `Duration::ZERO` + /// = unbounded. pub fn per_shard_with_policy( senders: Vec>, fsync_policy: FsyncPolicy, + fsync_timeout: Duration, + ) -> Arc { + debug_assert!( + senders.len() >= 2, + "per_shard pool needs >=2 writers; use top_level for single-writer" + ); + Arc::new(Self { + senders, + layout: crate::persistence::aof_manifest::AofLayout::PerShard, + fsync_policy, + fsync_timeout, + base_dir: None, + }) + } + + /// F6: same as [`Self::per_shard_with_policy`] but records the persistence + /// `base_dir` so a per-shard BGREWRITEAOF can load the authoritative + /// manifest fresh at rewrite time. This is the production constructor used + /// by `main.rs` for the PerShard layout. + pub fn per_shard_with_base_dir( + senders: Vec>, + fsync_policy: FsyncPolicy, + fsync_timeout: Duration, + base_dir: PathBuf, ) -> Arc { debug_assert!( senders.len() >= 2, @@ -241,6 +450,8 @@ impl AofWriterPool { senders, layout: crate::persistence::aof_manifest::AofLayout::PerShard, fsync_policy, + fsync_timeout, + base_dir: Some(base_dir), }) } @@ -278,12 +489,18 @@ impl AofWriterPool { match self.fsync_policy { FsyncPolicy::Always => { let rx = self.try_send_append_sync(shard_id, lsn, bytes); - match rx.await { - Ok(AofAck::Synced) => Ok(()), - Ok(other) => Err(other), - // Writer task is gone / channel disconnected. Caller - // treats this as a hard failure. - Err(_) => Err(AofAck::WriteFailed), + // F2 (design-for-failure): bound the wait so a stalled disk + // can't park this connection forever. On elapse the write is + // failed — the entry may still land on disk later, but + // durability is NOT confirmed, so the caller must not report + // success. `Duration::ZERO` keeps the legacy unbounded await. + match Self::await_ack(rx, self.fsync_timeout).await { + AckOutcome::Ack(AofAck::Synced) => Ok(()), + AckOutcome::Ack(other) => Err(other), + // Writer task gone / channel disconnected. + AckOutcome::Disconnected => Err(AofAck::WriteFailed), + // Fsync did not confirm within the bound. + AckOutcome::TimedOut => Err(AofAck::FsyncFailed), } } FsyncPolicy::EverySec | FsyncPolicy::No => { @@ -293,6 +510,49 @@ impl AofWriterPool { } } + /// Await an `AppendSync` ack receiver under a bounded timeout (F2). + /// + /// `timeout == Duration::ZERO` preserves the legacy unbounded await + /// (used when the operator explicitly opts out via + /// `--aof-fsync-timeout-ms 0`). Otherwise the await is capped by the + /// runtime-appropriate timer; on elapse the in-flight fsync is + /// abandoned (the receiver is dropped) and `TimedOut` is returned. + /// + /// Runtime-agnostic: monoio uses `select! { rx, sleep }` (matching the + /// established `cluster::failover` pattern — monoio 0.2 has no + /// `time::timeout`); tokio uses `tokio::time::timeout`. Both resolve the + /// ack first if it arrives within the bound, otherwise `TimedOut`. + async fn await_ack( + rx: crate::runtime::channel::OneshotReceiver, + timeout: Duration, + ) -> AckOutcome { + if timeout.is_zero() { + return match rx.await { + Ok(ack) => AckOutcome::Ack(ack), + Err(_) => AckOutcome::Disconnected, + }; + } + + #[cfg(feature = "runtime-monoio")] + { + monoio::select! { + res = rx => match res { + Ok(ack) => AckOutcome::Ack(ack), + Err(_) => AckOutcome::Disconnected, + }, + _ = monoio::time::sleep(timeout) => AckOutcome::TimedOut, + } + } + #[cfg(all(feature = "runtime-tokio", not(feature = "runtime-monoio")))] + { + match tokio::time::timeout(timeout, rx).await { + Ok(Ok(ack)) => AckOutcome::Ack(ack), + Ok(Err(_)) => AckOutcome::Disconnected, + Err(_) => AckOutcome::TimedOut, + } + } + } + /// Return the writer sender that owns the given shard's AOF file. /// /// For TopLevel pools, `shard_id` is ignored — all shards multiplex onto @@ -475,6 +735,87 @@ impl AofWriterPool { .map_err(|_| AofPoolSendError::SendFailed) } + /// [F6] Initiate a per-shard BGREWRITEAOF across every writer in a + /// PerShard pool. + /// + /// Loads the authoritative manifest fresh from `base_dir` (normal appends + /// never mutate the manifest, and BGREWRITEAOF is CAS-serialized by + /// `AOF_REWRITE_IN_PROGRESS`, so a fresh load is the current committed + /// state), builds a shared [`PerShardRewriteCoord`] that advances the + /// generation by one, and hands every writer the same `coord` + a cheap + /// `Arc` clone of `shard_dbs`. + /// + /// **Reliable delivery (design-for-failure):** the fan-out uses the + /// *blocking* `send` rather than `try_send`. A dropped rewrite message + /// would leave the countdown unable to reach zero — folded writers would + /// have reopened to new-seq files that the manifest never commits, silently + /// losing their post-rewrite appends. The writers run on dedicated threads + /// draining continuously, so `send` blocks only until a channel slot frees + /// (sub-millisecond), which is acceptable for a rare admin command. + /// + /// Returns `SendFailed` if `base_dir` is unset, the manifest can't be + /// loaded, or a writer thread is gone (disconnected channel). On the last + /// case the rewrite aborts WITHOUT committing — the old generation stays + /// authoritative (crash-safe), but a dead writer already means that shard's + /// persistence was compromised before this call. + pub fn try_send_rewrite_per_shard( + &self, + shard_dbs: Arc, + ) -> Result<(), AofPoolSendError> { + use crate::persistence::aof_manifest::{AofLayout, AofManifest}; + if self.layout != AofLayout::PerShard { + // A TopLevel pool rewrites via try_send_rewrite; this entry point + // is PerShard-only. + return Err(AofPoolSendError::RewriteUnsupportedInPerShard); + } + let base_dir = self.base_dir.as_ref().ok_or(AofPoolSendError::SendFailed)?; + let manifest = match AofManifest::load(base_dir) { + Ok(Some(m)) if m.layout == AofLayout::PerShard => m, + Ok(_) => { + error!( + "F6 per-shard rewrite: manifest at {} missing or not PerShard; aborting", + base_dir.display() + ); + return Err(AofPoolSendError::SendFailed); + } + Err(e) => { + error!( + "F6 per-shard rewrite: failed to load manifest at {}: {}", + base_dir.display(), + e + ); + return Err(AofPoolSendError::SendFailed); + } + }; + let current_seq = manifest.seq; + let n_shards = self.senders.len(); + let shared_manifest = Arc::new(parking_lot::Mutex::new(manifest)); + let coord = PerShardRewriteCoord::new(shared_manifest, current_seq, n_shards); + for s in &self.senders { + // Blocking send for guaranteed delivery — see the doc comment. + if s.send(AofMessage::RewritePerShard { + shard_dbs: shard_dbs.clone(), + coord: coord.clone(), + }) + .is_err() + { + error!( + "F6 per-shard rewrite: a writer channel is disconnected; \ + rewrite aborted (no manifest commit, old generation remains \ + authoritative). Inspect AOF writer threads." + ); + return Err(AofPoolSendError::SendFailed); + } + } + info!( + "F6 per-shard rewrite dispatched: seq {} -> {} across {} shards", + current_seq, + current_seq + 1, + n_shards + ); + Ok(()) + } + /// Broadcast `Shutdown` to every writer. Used by orchestrated shutdown /// paths in `main.rs`/`embedded.rs`. Each writer drains its channel and /// fsyncs before exiting. @@ -704,6 +1045,176 @@ mod pool_tests { assert_eq!(result, AofAck::FsyncFailed); } + // F2 (design-for-failure): `appendfsync=always` must bound its fsync-ack + // await. A stalled writer must surface a hard error within the budget, + // never park the connection forever. Tokio-gated because it drives the + // runtime timer; the monoio path shares the proven `select! + sleep` + // shape from `cluster::failover`, exercised end-to-end by the crash tests. + #[cfg(feature = "runtime-tokio")] + #[tokio::test] + async fn always_fsync_times_out_when_writer_never_acks() { + // Writer channel is held (kept open) but never drained → the + // AppendSync sits buffered with its ack sender alive, so the receiver + // never resolves. The bounded await MUST elapse and report failure. + let (tx0, _rx0) = channel::mpsc_bounded::(4); + let (tx1, _rx1) = channel::mpsc_bounded::(4); + let pool = AofWriterPool::per_shard_with_policy( + vec![tx0, tx1], + FsyncPolicy::Always, + Duration::from_millis(50), + ); + + let start = Instant::now(); + let res = pool + .try_send_append_durable(0, 1, Bytes::from_static(b"x")) + .await; + let elapsed = start.elapsed(); + + assert_eq!( + res, + Err(AofAck::FsyncFailed), + "timed-out fsync must map to FsyncFailed (durability unconfirmed)" + ); + assert!( + elapsed < Duration::from_secs(2), + "must fail within the bound, not hang (took {:?})", + elapsed + ); + // Keep the receivers alive until here so the message stays buffered. + drop((_rx0, _rx1)); + } + + #[cfg(feature = "runtime-tokio")] + #[tokio::test] + async fn always_fsync_succeeds_when_writer_acks_in_time() { + // Happy path: a writer drains the AppendSync and acks `Synced` well + // within the bound → the durable append returns Ok(()). + let (tx0, rx0) = channel::mpsc_bounded::(4); + let (tx1, _rx1) = channel::mpsc_bounded::(4); + let pool = AofWriterPool::per_shard_with_policy( + vec![tx0, tx1], + FsyncPolicy::Always, + Duration::from_millis(500), + ); + + tokio::spawn(async move { + if let Ok(AofMessage::AppendSync { ack, .. }) = rx0.recv_async().await { + let _ = ack.send(AofAck::Synced); + } + }); + + let res = pool + .try_send_append_durable(0, 1, Bytes::from_static(b"x")) + .await; + assert_eq!(res, Ok(()), "ack within the bound must succeed"); + drop(_rx1); + } + + /// Parse the PerShard incr framing `[u64 lsn LE][u32 len LE][len bytes]`, + /// stopping at a truncated tail (the crash/torn boundary) — exactly what + /// `replay_incr_framed` does. Returns the cleanly-replayable prefix. + #[cfg(feature = "runtime-tokio")] + fn parse_framed(buf: &[u8]) -> Vec<(u64, Vec)> { + let mut out = Vec::new(); + let mut i = 0usize; + while i + 12 <= buf.len() { + let lsn = u64::from_le_bytes(buf[i..i + 8].try_into().unwrap()); + let len = u32::from_le_bytes(buf[i + 8..i + 12].try_into().unwrap()) as usize; + if i + 12 + len > buf.len() { + break; // truncated tail → crash boundary, stop + } + out.push((lsn, buf[i + 12..i + 12 + len].to_vec())); + i += 12 + len; + } + out + } + + // Regression (PR #136 review, BUG #2): the tokio per-shard writer must carry + // a `write_error` latch like the single-file (~:1467) and monoio (~:2125) + // writers. A torn write (header lands, payload fails) must NOT be followed by + // more records — a lone orphaned header makes the framed replay misread the + // next record's bytes as the orphan's payload, corrupting everything after. + // The latch suppresses all writes after the tear and reports WriteFailed to + // AppendSync callers (so they error instead of ack'ing a corrupt write). + #[cfg(feature = "runtime-tokio")] + #[tokio::test] + async fn tokio_per_shard_writer_latches_after_torn_write() { + use crate::persistence::aof_manifest::AofManifest; + use std::sync::atomic::Ordering; + + let tmp = tempfile::tempdir().unwrap(); + let base_dir = tmp.path().to_path_buf(); + // PerShard layout, 2 shards (the per-shard pool needs >=2); drive shard 0. + let manifest = AofManifest::initialize_multi(&base_dir, 2).unwrap(); + let incr = manifest.shard_incr_path(0); + + // Inject: the 2nd Append tears (header written, payload "fails"). + TEST_FAIL_WRITE_AT.store(2, Ordering::SeqCst); + + let (tx, rx) = channel::mpsc_bounded::(16); + let cancel = CancellationToken::new(); + let writer = tokio::spawn(per_shard_aof_writer_task( + rx, + base_dir.clone(), + 0, + FsyncPolicy::Always, + cancel.clone(), + )); + + // 1: clean. 2: torn (header only). 3: must be suppressed by the latch. + tx.try_send(AofMessage::Append { + lsn: 1, + bytes: Bytes::from_static(b"AAAA"), + }) + .unwrap(); + tx.try_send(AofMessage::Append { + lsn: 2, + bytes: Bytes::from_static(b"BBBB"), + }) + .unwrap(); + tx.try_send(AofMessage::Append { + lsn: 3, + bytes: Bytes::from_static(b"CCCC"), + }) + .unwrap(); + + // Barrier + assertion: an AppendSync after the tear MUST come back + // WriteFailed (latched), never Synced. + let (ack_tx, ack_rx) = crate::runtime::channel::oneshot::(); + tx.try_send(AofMessage::AppendSync { + lsn: 4, + bytes: Bytes::from_static(b"DDDD"), + ack: ack_tx, + }) + .unwrap(); + + let ack = tokio::time::timeout(std::time::Duration::from_secs(5), ack_rx) + .await + .expect("writer must answer the AppendSync within 5s") + .expect("ack channel must not drop"); + assert_eq!( + ack, + AofAck::WriteFailed, + "after a torn write the latch must reject further writes (got {ack:?})" + ); + + cancel.cancel(); + TEST_FAIL_WRITE_AT.store(0, Ordering::SeqCst); + let _ = tokio::time::timeout(std::time::Duration::from_secs(5), writer).await; + + // On disk: exactly one replayable frame (lsn=1, "AAAA"). The orphaned + // lsn=2 header is a truncated tail (crash boundary); lsn 3 and 4 were + // never written (latch held) — no corruption. + let raw = std::fs::read(&incr).unwrap(); + let frames = parse_framed(&raw); + assert_eq!( + frames, + vec![(1u64, b"AAAA".to_vec())], + "only the pre-tear record may replay; orphaned headers / suppressed \ + records must not corrupt the stream" + ); + } + #[test] fn broadcast_shutdown_reaches_every_writer() { let (tx0, rx0) = channel::mpsc_bounded::(2); @@ -736,6 +1247,7 @@ mod pool_tests { let pool = std::sync::Arc::new(AofWriterPool::per_shard_with_policy( vec![tx0, tx1], FsyncPolicy::Always, + Duration::ZERO, // legacy unbounded await — disconnect/ack resolves it )); // Spawn a mock writer that drains AppendSync and responds with FsyncFailed. @@ -870,7 +1382,11 @@ mod pool_tests { // ack sender, simulating a dead writer. let (tx0, rx0) = channel::mpsc_bounded::(4); let (tx1, _rx1) = channel::mpsc_bounded::(4); - let pool = AofWriterPool::per_shard_with_policy(vec![tx0, tx1], FsyncPolicy::Always); + let pool = AofWriterPool::per_shard_with_policy( + vec![tx0, tx1], + FsyncPolicy::Always, + Duration::ZERO, // legacy unbounded await — disconnect resolves it + ); // Spawn a thread that pulls the AppendSync off the channel but drops // the ack without sending — simulating a writer crash mid-fsync. @@ -910,7 +1426,11 @@ mod pool_tests { // use try_send_append_durable so the policy is respected. let (tx0, _rx0) = channel::mpsc_bounded::(4); let (tx1, _rx1) = channel::mpsc_bounded::(4); - let pool = AofWriterPool::per_shard_with_policy(vec![tx0, tx1], FsyncPolicy::EverySec); + let pool = AofWriterPool::per_shard_with_policy( + vec![tx0, tx1], + FsyncPolicy::EverySec, + Duration::ZERO, + ); let result = futures::executor::block_on(pool.try_send_append_durable( 0, @@ -1185,6 +1705,14 @@ pub async fn aof_writer_task( crate::command::persistence::AOF_REWRITE_IN_PROGRESS .store(false, std::sync::atomic::Ordering::SeqCst); } + // [F6] A TopLevel writer never owns per-shard files; receiving + // RewritePerShard means a routing bug. Self-abort so the + // coordinator's countdown completes and the flag clears. + Ok(AofMessage::RewritePerShard { coord, .. }) => { + warn!("AOF TopLevel writer received RewritePerShard — routing bug; aborting"); + coord.mark_failed(); + coord.shard_done(); + } } } return; @@ -1272,6 +1800,13 @@ pub async fn aof_writer_task( Err(e) => { error!("Failed to reopen AOF after rewrite: {}", e); return; } } } + // [F6] TopLevel writer never owns per-shard files — routing + // bug. Self-abort so the countdown completes + flag clears. + Ok(AofMessage::RewritePerShard { coord, .. }) => { + warn!("AOF TopLevel writer received RewritePerShard — routing bug; aborting"); + coord.mark_failed(); + coord.shard_done(); + } Ok(AofMessage::Shutdown) | Err(_) => { let _ = writer.flush().await; let _ = writer.get_ref().sync_data().await; @@ -1314,6 +1849,15 @@ pub async fn aof_writer_task( /// /// Wait/timeout/corruption semantics for manifest loading match the existing /// `aof_writer_task` (60s bounded wait, hard fail on corrupt manifest). +/// Test-only torn-write injection for `per_shard_aof_writer_task`: when set to a +/// nonzero `N`, the `N`-th `Append` received by a tokio per-shard writer writes +/// its header and then simulates a payload write failure, exercising the +/// `write_error` latch. `0` disables. Atomic (not an env var) because +/// `std::env::set_var` is `unsafe` under edition 2024. Compiled out of release. +#[cfg(all(test, feature = "runtime-tokio"))] +pub(crate) static TEST_FAIL_WRITE_AT: std::sync::atomic::AtomicUsize = + std::sync::atomic::AtomicUsize::new(0); + pub async fn per_shard_aof_writer_task( rx: channel::MpscReceiver, base_dir: PathBuf, @@ -1422,8 +1966,11 @@ pub async fn per_shard_aof_writer_task( let mut writer = tokio::io::BufWriter::new(file); let mut last_fsync = Instant::now(); - let mut interval = tokio::time::interval(std::time::Duration::from_secs(1)); - interval.tick().await; + // (No `interval` here: the EverySec flush deadline is enforced by the + // timeout-bounded recv in the loop below, which wakes at least every + // 200ms regardless of message traffic. A long-lived `interval.tick()` + // select arm is fairness-starvable under sustained writes and proved + // unreliable when idle on this dedicated current-thread writer runtime.) // Test-only fault injection: if MOON_TEST_AOF_FSYNC_FAIL=1 is set in // the environment at writer task startup, every AppendSync ack resolves @@ -1433,24 +1980,81 @@ pub async fn per_shard_aof_writer_task( // hot path in production deployments where the var is absent. let fail_fsync_for_test = std::env::var("MOON_TEST_AOF_FSYNC_FAIL").as_deref() == Ok("1"); + // Torn-write latch: once any write to this incr file fails partway + // (e.g. the header landed but the payload did not), we must NEVER write + // another record — a lone orphaned header makes the framed reader + // misinterpret the next record's bytes as the orphan's payload, + // corrupting every record after it on replay. Stay latched for the + // writer's lifetime; recovery replays the clean prefix and a rewrite + // starts a fresh file. This mirrors the single-file (line ~1467) and + // monoio per-shard (line ~2125) writers, which already carry the latch. + let mut write_error = false; + // Test-only fault injection (no env var: edition-2024 set_var is unsafe). + // When `TEST_FAIL_WRITE_AT` is the ordinal of an incoming Append, that + // append writes its header then simulates a payload failure, exercising + // the latch. Compiled out of production builds. + #[cfg(test)] + let mut test_append_ordinal: usize = 0; + loop { tokio::select! { - msg = rx.recv_async() => { + // Bounded recv (EverySec durability): wake at least every 200ms + // even when idle so the flush deadline after this select! is + // honored within its 1s bound. flume's recv future is drop-safe + // on the Elapsed branch (no message consumed on timeout); the + // Ok(Ok(msg)) path below captures the message with no loss. + r = tokio::time::timeout( + std::time::Duration::from_millis(200), + rx.recv_async(), + ) => { + // On Elapsed (timeout) `r` is Err: skip the match and fall + // through to the EverySec deadline check after this select!. + if let Ok(msg) = r { match msg { // PerShard writer (tokio): per RFC § 2 Rule 1 the on-disk // format is `[u64 lsn LE][u32 len LE][RESP bytes]`. Header // is written sequentially with the body — both calls land // in the same BufWriter so this is one syscall under load. Ok(AofMessage::Append { lsn, bytes: data }) => { + // Latch: stream already torn — drop silently (Append + // is fire-and-forget; no ack channel to notify). + if write_error { + continue; + } + #[cfg(test)] + { + test_append_ordinal += 1; + let fail_at = TEST_FAIL_WRITE_AT + .load(std::sync::atomic::Ordering::Relaxed); + if fail_at != 0 && fail_at == test_append_ordinal { + // Reproduce a torn write: header lands, payload + // "fails". The orphaned header is flushed so the + // on-disk effect matches the real I/O-error case. + let mut header = [0u8; 12]; + header[..8].copy_from_slice(&lsn.to_le_bytes()); + header[8..] + .copy_from_slice(&(data.len() as u32).to_le_bytes()); + let _ = writer.write_all(&header).await; + let _ = writer.flush().await; + error!( + "AOF shard {}: injected torn write after header (test)", + shard_id + ); + write_error = true; + continue; + } + } let mut header = [0u8; 12]; header[..8].copy_from_slice(&lsn.to_le_bytes()); header[8..].copy_from_slice(&(data.len() as u32).to_le_bytes()); if let Err(e) = writer.write_all(&header).await { error!("AOF header write error shard {}: {}", shard_id, e); + write_error = true; continue; } if let Err(e) = writer.write_all(&data).await { error!("AOF write error shard {}: {}", shard_id, e); + write_error = true; continue; } if matches!(fsync, FsyncPolicy::Always) { @@ -1460,6 +2064,13 @@ pub async fn per_shard_aof_writer_task( } // AppendSync (tokio + PerShard): framed write + fsync + ack. Ok(AofMessage::AppendSync { lsn, bytes: data, ack }) => { + // Latch: stream already torn — refuse to write more and + // report failure so the caller does not hang to the F2 + // timeout and does not ack a write into a corrupt stream. + if write_error { + let _ = ack.send(AofAck::WriteFailed); + continue; + } let mut header = [0u8; 12]; header[..8].copy_from_slice(&lsn.to_le_bytes()); header[8..].copy_from_slice(&(data.len() as u32).to_le_bytes()); @@ -1468,6 +2079,7 @@ pub async fn per_shard_aof_writer_task( "AOF AppendSync header write error shard {}: {}", shard_id, e ); + write_error = true; let _ = ack.send(AofAck::WriteFailed); continue; } @@ -1476,6 +2088,7 @@ pub async fn per_shard_aof_writer_task( "AOF AppendSync write error shard {}: {}", shard_id, e ); + write_error = true; let _ = ack.send(AofAck::WriteFailed); continue; } @@ -1506,11 +2119,53 @@ pub async fn per_shard_aof_writer_task( Ok(AofMessage::Rewrite(_)) | Ok(AofMessage::RewriteSharded(_)) => { warn!( "AOF writer shard {}: received Rewrite/RewriteSharded — \ - not supported in PerShard layout, dropped. \ - Per-shard BGREWRITEAOF lands in RFC step 6.", + not applicable in PerShard layout, dropped.", shard_id ); } + // [F6] Per-shard rewrite (tokio): reuse the proven + // synchronous fold (`do_rewrite_per_shard`) verbatim, so + // the exactly-once invariant carries over unchanged. This + // writer runs on a DEDICATED std::thread (block_on_local, + // main.rs) — not a shared tokio worker — so executing the + // blocking fold here cannot starve the runtime. We flush + // the BufWriter (its `into_inner` does NOT flush) so any + // buffered appends are durable in the OLD incr, convert + // `tokio::fs::File` -> `std::fs::File` for the sync fold, + // then wrap the (reopened) file back into the BufWriter. + Ok(AofMessage::RewritePerShard { shard_dbs, coord }) => { + if let Err(e) = writer.flush().await { + error!( + "F6 tokio per-shard rewrite: shard {} pre-fold flush \ + failed: {}. Aborting; old generation stays authoritative.", + shard_id, e + ); + coord.mark_failed(); + coord.shard_done(); + } else { + // `into_std().await` waits for in-flight ops and is + // infallible; the buffer is already flushed above. + let mut sf = writer.into_inner().into_std().await; + let res = do_rewrite_per_shard( + shard_id, &shard_dbs, &mut sf, &rx, &coord, + ); + // On success `sf` points at the NEW incr (the fold + // reopened it + already called `shard_done()`); on + // error it is still the OLD incr (pre-reopen). Wrap + // it back either way so the writer stays valid. + writer = + tokio::io::BufWriter::new(tokio::fs::File::from_std(sf)); + if let Err(e) = res { + error!( + "F6 tokio per-shard rewrite: shard {} fold failed: {}. \ + Aborting commit; old generation stays authoritative.", + shard_id, e + ); + coord.mark_failed(); + coord.shard_done(); + } + } + } Ok(AofMessage::Shutdown) | Err(_) => { let _ = writer.flush().await; let _ = writer.get_ref().sync_data().await; @@ -1518,12 +2173,6 @@ pub async fn per_shard_aof_writer_task( break; } } - } - _ = interval.tick(), if fsync == FsyncPolicy::EverySec => { - if last_fsync.elapsed() >= std::time::Duration::from_secs(1) { - let _ = writer.flush().await; - let _ = writer.get_ref().sync_data().await; - last_fsync = Instant::now(); } } _ = cancel.cancelled() => { @@ -1533,6 +2182,19 @@ pub async fn per_shard_aof_writer_task( break; } } + // EverySec deadline — checked after EVERY wake (message OR timeout), + // so it is NOT subject to select! fairness and holds the 1s bound + // under sustained writes as well as when idle. (The old long-lived + // `interval.tick()` arm could be starved by the always-ready recv + // arm under load, leaving >1s of writes buffered in the BufWriter + // and lost on SIGKILL — the COMPOSE crash-matrix failure.) + if fsync == FsyncPolicy::EverySec + && last_fsync.elapsed() >= std::time::Duration::from_secs(1) + { + let _ = writer.flush().await; + let _ = writer.get_ref().sync_data().await; + last_fsync = Instant::now(); + } } } @@ -1752,11 +2414,32 @@ pub async fn per_shard_aof_writer_task( Ok(AofMessage::Rewrite(_)) | Ok(AofMessage::RewriteSharded(_)) => { warn!( "AOF writer shard {}: received Rewrite/RewriteSharded — \ - not supported in PerShard layout, dropped. \ - Per-shard BGREWRITEAOF lands in RFC step 6.", + not applicable in PerShard layout (use per-shard \ + BGREWRITEAOF), dropped.", shard_id ); } + // [F6] Per-shard rewrite fan-out (monoio). Fold THIS shard, + // then signal the coordinator; the last shard commits the + // manifest. On error the old generation stays authoritative + // (advance_shard did not commit the seq). + Ok(AofMessage::RewritePerShard { shard_dbs, coord }) => { + if let Err(e) = + do_rewrite_per_shard(shard_id, &shard_dbs, &mut file, &rx, &coord) + { + error!( + "F6 per-shard rewrite: shard {} fold failed: {}. \ + Aborting rewrite; old generation stays authoritative.", + shard_id, e + ); + // Mark the whole rewrite failed so the final writer + // aborts the commit (committing new_seq with a shard + // missing its new base would break recovery), then + // decrement so the countdown can still complete. + coord.mark_failed(); + coord.shard_done(); + } + } Ok(AofMessage::Shutdown) | Err(_) => { if !write_error { if let Err(e) = file.flush().and_then(|_| file.sync_data()) { @@ -2181,7 +2864,7 @@ fn snapshot_and_generate(db: &SharedDatabases) -> BytesMut { /// are dropped silently (duplicate rewrites while a rewrite is in progress) or /// returned via the flag for Shutdown (caller is responsible for honoring it /// after the rewrite completes). -#[cfg(feature = "runtime-monoio")] +#[cfg(any(feature = "runtime-monoio", feature = "runtime-tokio"))] #[derive(Default)] struct DrainOutcome { drained: usize, @@ -2229,7 +2912,9 @@ fn drain_pending_appends( AofMessage::Shutdown => { outcome.shutdown_requested = true; } - AofMessage::Rewrite(_) | AofMessage::RewriteSharded(_) => { + AofMessage::Rewrite(_) + | AofMessage::RewriteSharded(_) + | AofMessage::RewritePerShard { .. } => { // Already rewriting — drop redundant request. } } @@ -2237,6 +2922,225 @@ fn drain_pending_appends( Ok(outcome) } +/// [F6] Drain a per-shard writer's queued appends into its OLD incr file using +/// the framed `[u64 lsn LE][u32 len LE][RESP bytes]` on-disk format that +/// per-shard recovery expects. +/// +/// This is the per-shard twin of [`drain_pending_appends`] (which writes the +/// legacy TopLevel raw-RESP format). Correctness depends on the framing +/// matching `replay_per_shard`'s reader — an unframed write here would make the +/// drained appends unparseable on restart. +#[cfg(any(feature = "runtime-monoio", feature = "runtime-tokio"))] +fn drain_pending_appends_framed( + rx: &channel::MpscReceiver, + file: &mut std::fs::File, +) -> Result { + use std::io::Write; + let mut outcome = DrainOutcome::default(); + let write_framed = |file: &mut std::fs::File, lsn: u64, data: &[u8]| -> std::io::Result<()> { + let mut header = [0u8; 12]; + header[..8].copy_from_slice(&lsn.to_le_bytes()); + header[8..].copy_from_slice(&(data.len() as u32).to_le_bytes()); + file.write_all(&header)?; + file.write_all(data) + }; + while let Ok(msg) = rx.try_recv() { + match msg { + AofMessage::Append { lsn, bytes: data } => { + write_framed(file, lsn, &data).map_err(|e| AofError::Io { + path: PathBuf::from(""), + source: e, + })?; + outcome.drained += 1; + } + AofMessage::AppendSync { + lsn, + bytes: data, + ack, + } => { + write_framed(file, lsn, &data).map_err(|e| AofError::Io { + path: PathBuf::from(""), + source: e, + })?; + outcome.drained += 1; + // Durability for these is covered by the post-drain fsync at + // the rewrite boundary (mirrors drain_pending_appends). + let _ = ack.send(AofAck::Synced); + } + AofMessage::Shutdown => { + outcome.shutdown_requested = true; + } + AofMessage::Rewrite(_) + | AofMessage::RewriteSharded(_) + | AofMessage::RewritePerShard { .. } => { + // Already rewriting this shard — drop redundant request. + } + } + } + Ok(outcome) +} + +/// [F6] Per-shard rewrite fold (monoio). Run by a single per-shard writer for +/// ITS shard only; the manifest commit is coordinated across all shards by the +/// shared [`PerShardRewriteCoord`]. +/// +/// Correctness ordering (prevents double-apply of non-idempotent commands like +/// INCR after the rewrite) — identical discipline to [`do_rewrite_sharded`], +/// scoped to one shard: +/// +/// 1. Drain queued appends into the OLD incr (framed) and fsync. +/// 2. Acquire write locks on this shard's databases. +/// 3. Re-drain appends that arrived between phase 1 and the lock, into OLD +/// incr, and fsync. +/// 4. Snapshot this shard's databases under the locks. +/// 5. Release the locks before the expensive base-RDB write. +/// 6. Write the new base + new (empty) incr at `coord.new_seq` via +/// `advance_shard` (which does NOT bump `manifest.seq`), then reopen +/// `file` to the new incr. Subsequent appends land in the new generation. +/// 7. Signal completion to the coordinator; the last shard commits the +/// manifest (single seq flip) and prunes the old generation. +/// +/// Until step 7's commit, the on-disk manifest still resolves to the old seq, +/// so a crash anywhere in steps 1-6 recovers the intact old generation. +/// +/// # Cross-thread exactly-once invariant (load-bearing) +/// +/// This fold runs on the per-shard *writer* thread, which is distinct from the +/// shard event-loop thread that applies commands. Exactly-once across the +/// rewrite boundary depends on a single ordering fact: the live write path +/// enqueues each command's AOF append **inside** the same `RwLock` +/// write guard under which it mutated the db (see `spsc_handler.rs`: +/// `wal_append_and_fanout` is called before `drop(guard)`). Phase 2 here +/// acquires those *same* locks (`all_shard_dbs()[sidx]` is +/// `ShardDatabases::shards[sidx]`, the exact `RwLock`s `write_db` locks), so +/// RwLock mutual exclusion forces the order +/// `enqueue → guard-release → fold-acquire → mid-drain(phase 3)`. Hence every +/// INCR whose mutation lands in the phase-4 snapshot had its append drained +/// into the OLD incr (then pruned at commit) — never replayed on top of the +/// new base. Were the append enqueued *after* the guard drop, a snapshot would +/// capture the mutation while its append still raced toward the NEW incr → +/// double-apply. The in-guard append is therefore the invariant; do not move it. +/// +/// This also assumes the `RwLock`-backed `ShardDatabases` is the *live* store. +/// It is, because the thread-local `ShardSlice` fast path is dead code until +/// Phase 4 wires `init_shard` (`is_initialized()` is always false today). A +/// future Phase 4 that makes ShardSlice live MUST revisit this fold: the writer +/// thread cannot lock another thread's `!Send` `Rc>`, so the +/// per-shard rewrite would need a different snapshot-coordination mechanism. +/// +/// # Known limitation — channel saturation during the fold +/// +/// Exactly-once holds *absent append-channel saturation during the fold*. While +/// this function runs (phases 2-6, including the base-RDB serialize + write + +/// fsync of phase 6, which is hundreds of ms on a large shard) the writer is +/// NOT in its recv loop, so it is not draining the bounded +/// `mpsc_bounded::(10_000)` append channel. Post-snapshot appends +/// queue there for the new incr; the event loop enqueues them with +/// `try_send_append` (drop-on-full, return ignored — `spsc_handler.rs`). Under +/// *sustained concurrent* writes on a large dataset, > 10_000 appends can pile +/// up during the window and the overflow is silently dropped — lost even on a +/// clean restart (worse than the everysec contract, which only loses on crash). +/// The single-client crash matrix cannot surface this (serialized `redis-cli` +/// never pressures the channel). This window is *pre-existing*: the shipped +/// `do_rewrite_sharded` has the identical non-draining gap. Tracked as a +/// known limitation (F6 is behind `--experimental-per-shard-rewrite`); the fix +/// (keep draining during phase 6, or block-on-full for the rewrite's duration) +/// is a separate scoped task. See `tmp/F6-known-limitations.md`. +#[cfg(any(feature = "runtime-monoio", feature = "runtime-tokio"))] +fn do_rewrite_per_shard( + shard_id: u16, + shard_dbs: &crate::shard::shared_databases::ShardDatabases, + file: &mut std::fs::File, + rx: &channel::MpscReceiver, + coord: &PerShardRewriteCoord, +) -> Result<(), MoonError> { + let sidx = shard_id as usize; + let all_shards = shard_dbs.all_shard_dbs(); + if sidx >= all_shards.len() { + return Err(AofError::RewriteFailed { + detail: format!( + "do_rewrite_per_shard: shard {} out of range ({} shards)", + sidx, + all_shards.len() + ), + } + .into()); + } + + // Phase 1: drain pre-rewrite queued appends into old incr (framed). + let pre_drain = drain_pending_appends_framed(rx, file)?; + file.sync_data().map_err(|e| AofError::Io { + path: PathBuf::from(""), + source: e, + })?; + + // Phase 2: acquire write locks on this shard's db(s) (db_idx ascending). + let shard_locks = &all_shards[sidx]; + let guards: Vec<_> = shard_locks.iter().map(|lock| lock.write()).collect(); + + // Phase 3: drain appends that completed between phase 1 and phase 2. + let mid_drain = drain_pending_appends_framed(rx, file)?; + file.sync_data().map_err(|e| AofError::Io { + path: PathBuf::from(""), + source: e, + })?; + + // Phase 4: snapshot this shard's databases under the locks. + let now_ms = current_time_ms(); + let mut snapshot: Vec<( + Vec<( + crate::storage::compact_key::CompactKey, + crate::storage::entry::Entry, + )>, + u32, + )> = Vec::with_capacity(guards.len()); + for guard in &guards { + let base_ts = guard.base_timestamp(); + let mut entries = Vec::new(); + for (key, entry) in guard.data().iter() { + if !entry.is_expired_at(base_ts, now_ms) { + entries.push((key.clone(), entry.clone())); + } + } + snapshot.push((entries, base_ts)); + } + + // Phase 5: release locks before the expensive disk write. + drop(guards); + + // Phase 6: write new base, advance THIS shard's manifest entry (no seq + // commit), reopen to the new incr. The manifest lock is held only for the + // brief, await-free advance_shard call. + let rdb_bytes = crate::persistence::rdb::save_snapshot_to_bytes(&snapshot)?; + let new_incr = { + let mut m = coord.manifest.lock(); + m.advance_shard(shard_id, coord.new_seq, &rdb_bytes)? + }; + *file = std::fs::OpenOptions::new() + .create(true) + .append(true) + .open(&new_incr) + .map_err(|e| AofError::Io { + path: new_incr, + source: e, + })?; + + info!( + "F6 per-shard rewrite: shard {} folded (drained {}+{} appends), new seq {}", + shard_id, pre_drain.drained, mid_drain.drained, coord.new_seq + ); + if pre_drain.shutdown_requested || mid_drain.shutdown_requested { + warn!( + "F6 per-shard rewrite: shard {} saw shutdown during rewrite (honored after commit)", + shard_id + ); + } + + // Phase 7: signal completion; the last writer commits + prunes. + coord.shard_done(); + Ok(()) +} + /// Multi-part rewrite: snapshot single-shard databases → RDB base → advance manifest. /// /// Correctness ordering (prevents double-apply of non-idempotent commands like diff --git a/src/persistence/aof_manifest.rs b/src/persistence/aof_manifest.rs index 91b16087f..572e16e85 100644 --- a/src/persistence/aof_manifest.rs +++ b/src/persistence/aof_manifest.rs @@ -1152,17 +1152,27 @@ impl AofManifest { } /// Advance a single shard to a new sequence: write the shard's new base RDB, - /// create a new empty incr file, delete old shard files, then update the - /// shard's `max_lsn` in the in-memory manifest. + /// create a new empty incr file, then update the shard's `max_lsn` in the + /// in-memory manifest. + /// + /// **Does NOT delete the old generation's files.** Deletion is deferred to + /// the coordinator via [`prune_shard_files`](Self::prune_shard_files), + /// called only AFTER `write_manifest()` durably commits the new seq. + /// Deleting before the commit would leave a crash window where the + /// persisted (old) seq points at files that are already gone — recovery + /// resolves base/incr by `self.seq`, so a crash mid-fan-out would lose data + /// for any shard that had already advanced. This matches the post-commit + /// deletion ordering in [`advance`](Self::advance) (TopLevel layout). /// /// **Caller MUST call `write_manifest()` after all shards have been advanced** - /// to persist the updated manifest atomically. Advancing shards one at a time - /// and writing the manifest per-shard would leave the manifest in an - /// inconsistent state between calls. + /// (and set `self.seq` to the new seq) to persist the updated manifest + /// atomically — this is the single durable commit point for the rewrite. + /// Advancing shards one at a time and writing the manifest per-shard would + /// leave the manifest in an inconsistent state between calls. /// /// For `TopLevel` layout, `shard_id` must be 0 and this delegates to - /// `advance()`. For `PerShard` layout, files are written to - /// `shard_dir(shard_id)/`. + /// `advance()` (which deletes post-commit internally). For `PerShard` + /// layout, files are written to `shard_dir(shard_id)/`. /// /// Returns the path to the new incremental file for this shard. pub fn advance_shard( @@ -1189,7 +1199,6 @@ impl AofManifest { ), })?; - let old_seq = self.seq; let shard_dir = self.shard_dir(shard_id); std::fs::create_dir_all(&shard_dir).map_err(|e| crate::error::AofError::Io { path: shard_dir.clone(), @@ -1234,42 +1243,56 @@ impl AofManifest { source: e, })?; - // 3. Delete old shard files (best-effort). - let old_base = self.shard_base_path_seq(shard_id, old_seq); - let old_incr = self.shard_incr_path_seq(shard_id, old_seq); - if old_base.exists() { - if let Err(e) = std::fs::remove_file(&old_base) { + // 3. Update per-shard LSN in-memory (manifest write is the caller's job). + // Old-generation files are intentionally NOT deleted here — the + // coordinator prunes them via `prune_shard_files` only after + // `write_manifest()` durably commits the new seq (see the fn doc; + // delete-before-commit would lose data on a mid-fan-out crash). + self.shards[shard_idx].max_lsn = self.shards[shard_idx].max_lsn.max(new_seq); + + info!( + "AOF shard {} advanced to seq {}: base={} bytes, incr={}", + shard_id, + new_seq, + rdb_bytes.len(), + new_incr.display() + ); + + Ok(new_incr) + } + + /// Delete a shard's base + incr files for a specific `seq`. Best-effort. + /// + /// **Crash-safety contract:** the rewrite coordinator MUST call this only + /// AFTER `write_manifest()` has durably committed the new seq. Deleting an + /// old generation's files before the manifest flips would orphan the + /// persisted (old) seq whose files are already gone — recovery resolves + /// base/incr by `self.seq`, so it would read a missing base and lose data + /// for any shard that completed before the crash. This mirrors the + /// post-commit deletion ordering in `advance()` (TopLevel layout). + pub fn prune_shard_files(&self, shard_id: u16, seq: u64) { + let base = self.shard_base_path_seq(shard_id, seq); + let incr = self.shard_incr_path_seq(shard_id, seq); + if base.exists() { + if let Err(e) = std::fs::remove_file(&base) { warn!( - "advance_shard {}: failed to delete old base {}: {}", + "prune_shard_files {}: failed to delete old base {}: {}", shard_id, - old_base.display(), + base.display(), e ); } } - if old_incr.exists() { - if let Err(e) = std::fs::remove_file(&old_incr) { + if incr.exists() { + if let Err(e) = std::fs::remove_file(&incr) { warn!( - "advance_shard {}: failed to delete old incr {}: {}", + "prune_shard_files {}: failed to delete old incr {}: {}", shard_id, - old_incr.display(), + incr.display(), e ); } } - - // 4. Update per-shard LSN in-memory (manifest write is the caller's job). - self.shards[shard_idx].max_lsn = self.shards[shard_idx].max_lsn.max(new_seq); - - info!( - "AOF shard {} advanced to seq {}: base={} bytes, incr={}", - shard_id, - new_seq, - rdb_bytes.len(), - new_incr.display() - ); - - Ok(new_incr) } } @@ -1456,9 +1479,13 @@ pub struct OrderedEntry { /// in global LSN order across all shards. /// /// Returns `(commands_replayed, max_lsn)` — the count covers only inline -/// (non-ordered) replays, and `max_lsn` covers both inline AND ordered -/// entries (the high bit is masked out before max comparison, so it reflects -/// the true issued LSN). +/// (non-ordered) replays. `max_lsn` is the NEXT-FREE replication offset: +/// `max(entry.lsn + entry.len)` across both inline AND ordered entries (the +/// high bit is masked out before the computation). It is the offset AFTER the +/// last byte on disk, NOT the start LSN of the last entry — because +/// `ReplicationState::issue_lsn` returns the offset BEFORE adding the entry +/// length, seeding `master_repl_offset` with a start LSN would reissue the +/// last on-disk LSN and break the lsn->entry uniqueness invariant (F5). /// /// **Truncated entries:** a header partly written at crash time is treated as /// EOF (parity with `replay_incr_resp` semantics). A whole header followed by @@ -1530,8 +1557,12 @@ fn replay_incr_framed( lsn, bytes, }); - if lsn > max_lsn { - max_lsn = lsn; + // F5: track the next-free replication offset (entry end), not the + // start LSN — `issue_lsn` returns the offset BEFORE adding the + // entry length, so the seed must clear every byte already on disk. + let entry_end = lsn + len as u64; + if entry_end > max_lsn { + max_lsn = entry_end; } offset = payload_end; continue; @@ -1578,8 +1609,10 @@ fn replay_incr_framed( }; engine.replay_command(databases, cmd, cmd_args, &mut selected_db); count += 1; - if lsn > max_lsn { - max_lsn = lsn; + // F5: next-free offset = entry start LSN + RESP byte length. + let entry_end = lsn + len as u64; + if entry_end > max_lsn { + max_lsn = entry_end; } } Ok(None) => { @@ -2316,13 +2349,43 @@ mod tests_v2 { assert!(ordered.is_empty(), "no ordered entries in this stream"); assert_eq!(count, 2); - assert_eq!(max_lsn, 11); + // F5: max_lsn is the NEXT-FREE offset = max(lsn + len) = max(7+14, 11+16) = 27. + assert_eq!(max_lsn, 27); let calls = engine.calls.borrow(); assert_eq!(calls.len(), 2); assert_eq!(calls[0], "PING"); assert_eq!(calls[1], "DBSIZE"); } + #[test] + fn replay_incr_framed_max_lsn_is_next_free_offset() { + // F5: replay must return the next-free replication offset (entry end = + // start LSN + RESP byte length), not the START LSN of the last entry. + // `issue_lsn` hands out the offset BEFORE adding the entry's length, so + // seeding `master_repl_offset` with a start LSN reissues the last + // pre-crash entry's LSN — breaking lsn->entry uniqueness (RFC § 2 Rule 3). + let ping = b"*1\r\n$4\r\nPING\r\n"; // 14 bytes + let dbsize = b"*1\r\n$6\r\nDBSIZE\r\n"; // 16 bytes + // Cumulative LSNs as the writer issues them: each entry starts at the + // previous entry's end. + let mut bytes = frame_entry(100, ping); + bytes.extend_from_slice(&frame_entry(100 + ping.len() as u64, dbsize)); + + let mut dbs: Vec = vec![crate::storage::Database::new()]; + let engine = RecordingEngine::new(); + let mut ordered: Vec = Vec::new(); + let (_count, max_lsn) = + replay_incr_framed(0, &mut dbs, &bytes, &engine, &mut ordered).expect("framed replay"); + + // Last entry: start 114 + len 16 = 130. The next write MUST get >= 130. + let expected_next_free = 100 + ping.len() as u64 + dbsize.len() as u64; + assert_eq!(expected_next_free, 130); + assert_eq!( + max_lsn, expected_next_free, + "max_lsn must be the next-free offset (entry end), not the last start LSN" + ); + } + #[test] fn replay_incr_framed_truncated_header_is_crash_eof() { // One valid entry, then a partial 5-byte header (crash mid-write). @@ -2336,7 +2399,8 @@ mod tests_v2 { .expect("truncated-header is EOF"); assert_eq!(count, 1); - assert_eq!(max_lsn, 3); + // F5: next-free offset = PING entry start 3 + RESP len 14 = 17. + assert_eq!(max_lsn, 17); } #[test] @@ -2408,7 +2472,12 @@ mod tests_v2 { }; assert_eq!(total, 2, "two SETs replayed"); - assert_eq!(global_max_lsn, 20, "global max lsn = max(shard maxes)"); + // F5: global_max_lsn = max next-free offset across shards. shard-1 SET + // is 29 RESP bytes at lsn 20 → next-free 49 (> shard-0's 10+29=39). + assert_eq!( + global_max_lsn, 49, + "global max lsn = max(shard next-free offsets)" + ); assert!(ordered.is_empty(), "no ordered entries in this stream"); // Each shard's DB now holds its key (and only its key). @@ -2487,10 +2556,13 @@ mod tests_v2 { }; assert_eq!(total, n_shards as usize, "one SET per shard = N total"); + // F5: global_max_lsn = highest shard's NEXT-FREE offset. The highest + // shard (sid=N-1) SETs at lsn N*10 with a 29-byte RESP → next-free + // N*10 + 29 (here 40 + 29 = 69), not the bare start LSN. assert_eq!( global_max_lsn, - n_shards as u64 * 10, - "global max lsn = highest shard lsn" + n_shards as u64 * 10 + 29, + "global max lsn = highest shard next-free offset" ); assert!(ordered.is_empty(), "no ordered entries"); @@ -2542,7 +2614,13 @@ mod tests_v2 { .expect("framed replay with ordered"); assert_eq!(count, 2, "two inline entries dispatched (PING, DBSIZE)"); - assert_eq!(max_lsn, 12, "max LSN tracks both inline and ordered"); + // F5: max_lsn is the next-free offset across inline AND ordered. The + // ordered SET (27 RESP bytes) at lsn 8 → next-free 35, exceeding the + // PING (5+14=19) and DBSIZE (12+16=28) ends. + assert_eq!( + max_lsn, 35, + "max LSN = next-free offset across inline and ordered" + ); assert_eq!(ordered.len(), 1, "one entry buffered as ordered"); let buffered = &ordered[0]; assert_eq!(buffered.shard_id, 3, "shard_id forwarded"); @@ -2820,10 +2898,16 @@ mod tests_v2 { } // ----------------------------------------------------------------------- - // FIX-W2-3 (partial): advance_shard writes new base+incr, deletes old + // F6 crash-safety ordering: advance_shard writes new base+incr but MUST + // NOT delete old files. Deleting before the manifest durably commits the + // new seq leaves a window where a crash orphans the persisted (old) seq + // whose files are already gone → recovery reads a missing base → data + // loss for completed shards. Deletion is the coordinator's job, AFTER + // write_manifest(), via prune_shard_files(). This mirrors the proven + // ordering in advance() (TopLevel), which deletes only post-commit. // ----------------------------------------------------------------------- #[test] - fn advance_shard_writes_new_seq_and_deletes_old() { + fn advance_shard_defers_delete_until_after_commit() { let dir = temp_dir(); // Initialize 2-shard manifest at seq=1. @@ -2833,41 +2917,81 @@ mod tests_v2 { let empty_rdb = crate::persistence::rdb::save_to_bytes(&[] as &[crate::storage::Database]) .expect("empty rdb"); - // Old shard-0 files at seq=1 must exist before advance. - let old_base_s0 = manifest.shard_base_path(0); - let old_incr_s0 = manifest.shard_incr_path(0); + // Old shard files at seq=1 must exist before advance. + let old_base_s0 = manifest.shard_base_path_seq(0, 1); + let old_incr_s0 = manifest.shard_incr_path_seq(0, 1); + let old_base_s1 = manifest.shard_base_path_seq(1, 1); + let old_incr_s1 = manifest.shard_incr_path_seq(1, 1); assert!(old_base_s0.exists(), "seq=1 base must exist for shard 0"); assert!(old_incr_s0.exists(), "seq=1 incr must exist for shard 0"); - // Advance shard-0 to seq=2. - let new_incr = manifest + // Fan out: coordinator picks new_seq=2 once and advances every shard + // to it. No manifest write, no deletion, happens inside the fan-out. + let new_incr_s0 = manifest .advance_shard(0, 2, &empty_rdb) .expect("advance_shard 0 → seq=2"); - assert!(new_incr.exists(), "new incr file must be created"); + let new_incr_s1 = manifest + .advance_shard(1, 2, &empty_rdb) + .expect("advance_shard 1 → seq=2"); + assert!(new_incr_s0.exists(), "new incr file must be created (s0)"); + assert!(new_incr_s1.exists(), "new incr file must be created (s1)"); + + // PRE-COMMIT INVARIANT: new files written, OLD files NOT yet deleted. + // This is the regression guard against delete-before-commit. assert!( manifest.shard_base_path_seq(0, 2).exists(), "new seq=2 base must exist for shard 0" ); assert!( - !old_base_s0.exists(), - "old seq=1 base must be deleted for shard 0" + manifest.shard_base_path_seq(1, 2).exists(), + "new seq=2 base must exist for shard 1" ); assert!( - !old_incr_s0.exists(), - "old seq=1 incr must be deleted for shard 0" + old_base_s0.exists(), + "old seq=1 base (s0) MUST survive until the manifest commits" ); - - // Shard-1 must be unaffected. assert!( - manifest.shard_base_path(1).exists(), - "shard-1 seq=1 base must survive advance of shard-0" + old_incr_s0.exists(), + "old seq=1 incr (s0) MUST survive until the manifest commits" + ); + assert!( + old_base_s1.exists(), + "old seq=1 base (s1) MUST survive until the manifest commits" ); - // Caller must write_manifest after all shards advanced. + // COMMIT: coordinator bumps seq and persists the manifest atomically. + // This is the single durable commit point for the whole rewrite. manifest.seq = 2; manifest .write_manifest() .expect("write manifest after advance"); + + // POST-COMMIT: coordinator prunes old files — safe now that recovery + // resolves base/incr by the durably-committed new seq. + manifest.prune_shard_files(0, 1); + manifest.prune_shard_files(1, 1); + assert!( + !old_base_s0.exists(), + "old seq=1 base (s0) pruned post-commit" + ); + assert!( + !old_incr_s0.exists(), + "old seq=1 incr (s0) pruned post-commit" + ); + assert!( + !old_base_s1.exists(), + "old seq=1 base (s1) pruned post-commit" + ); + assert!( + !old_incr_s1.exists(), + "old seq=1 incr (s1) pruned post-commit" + ); + assert!( + manifest.shard_base_path_seq(0, 2).exists(), + "new seq=2 base (s0) must remain after prune" + ); + + // Recovery reads base by manifest.seq — must resolve to the new seq. let reloaded = AofManifest::load(&dir).expect("load").expect("present"); assert_eq!(reloaded.seq, 2); diff --git a/src/persistence/manifest.rs b/src/persistence/manifest.rs index a90d5525c..1f19cb504 100644 --- a/src/persistence/manifest.rs +++ b/src/persistence/manifest.rs @@ -235,6 +235,11 @@ pub const MANIFEST_FORMAT_V2: u8 = 2; pub const MAX_INLINE_ENTRIES: usize = (PAGE_4K - MOONPAGE_HEADER_SIZE - ROOT_META_SIZE) / FileEntry::SIZE; +/// FileEntry records per overflow (`ManifestEntry`) page: a full 4 KB page +/// minus the 64-byte MoonPageHeader, no per-page meta. (4096 - 64) / 56 = 72. +/// Entries beyond `MAX_INLINE_ENTRIES` spill into append-only overflow pages. +pub const ENTRIES_PER_OVERFLOW_PAGE: usize = (PAGE_4K - MOONPAGE_HEADER_SIZE) / FileEntry::SIZE; + /// In-memory representation of one manifest root page. /// /// Fields match MOONSTORE-V2-COMPREHENSIVE-DESIGN.md §4.2. @@ -293,6 +298,18 @@ pub struct ShardManifest { /// after the commit flip — we record the pre-commit value so age = current - tombstone_epoch). /// `tombstoned_at` is a monotonic `Instant` for wall-clock retention. tombstone_registry: HashMap, + /// Set when `compact()` rewrote+renamed the manifest durably but then failed + /// to reopen `self.file` against the new inode. The old handle now refers to + /// the pre-rename (orphaned) inode, so writing through it would silently + /// discard every subsequent commit. While set, `commit()` reattaches to + /// `self.path` BEFORE writing (or fails loudly if that reopen also fails). + needs_reopen: bool, + /// Test-only fault injection: when true, `compact()` simulates the + /// "rename succeeded, reopen failed" race by pointing `self.file` at a + /// throwaway inode and returning an error, exercising the `needs_reopen` + /// recovery path without needing real fd/permission failures. + #[cfg(test)] + fail_compact_reopen: bool, } impl ShardManifest { @@ -314,7 +331,7 @@ impl ShardManifest { shard_uuid: [0u8; 16], entries: Vec::new(), }; - Self::serialize_root(&root, &mut buf[..PAGE_4K]); + Self::serialize_root(&root, 0, &mut buf[..PAGE_4K]); // Write file std::fs::write(path, &buf)?; @@ -337,6 +354,9 @@ impl ShardManifest { active_root: root, active_slot: 0, tombstone_registry: HashMap::new(), + needs_reopen: false, + #[cfg(test)] + fail_compact_reopen: false, }) } @@ -357,8 +377,11 @@ impl ShardManifest { )); } - let root_a = Self::try_parse_root(&buf[..PAGE_4K]); - let root_b = Self::try_parse_root(&buf[PAGE_4K..2 * PAGE_4K]); + // Load each slot fully (inline root + its append-only overflow run). A + // slot whose overflow is torn/short/corrupt returns None so selection + // falls back to the other (last-good) slot — never surfacing garbage. + let root_a = Self::load_root(&buf, 0); + let root_b = Self::load_root(&buf, PAGE_4K); let (active_root, active_slot) = match (root_a, root_b) { (Some(a), Some(b)) => { @@ -402,6 +425,9 @@ impl ShardManifest { active_root, active_slot, tombstone_registry, + needs_reopen: false, + #[cfg(test)] + fail_compact_reopen: false, }) } @@ -412,22 +438,56 @@ impl ShardManifest { /// 3. `sync_data()` — this is the atomic commit point /// 4. Flip active_slot pub fn commit(&mut self) -> std::io::Result<()> { - if self.active_root.entries.len() > MAX_INLINE_ENTRIES { - return Err(std::io::Error::new( - std::io::ErrorKind::InvalidInput, - format!( - "too many entries for inline root page: {} > {}", - self.active_root.entries.len(), - MAX_INLINE_ENTRIES, - ), - )); + // A prior compaction renamed a fresh manifest into place durably but then + // failed to reopen our handle, so `self.file` still points at the + // pre-rename (orphaned) inode. Reattach to the live file BEFORE any write; + // writing through the stale handle would silently discard this and every + // later commit. If the reopen still fails, surface the error (this commit + // genuinely cannot be persisted) rather than losing data silently. + if self.needs_reopen { + self.file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(&self.path)?; + self.active_slot = 0; + self.needs_reopen = false; } - self.active_root.epoch += 1; - self.active_root.file_count = self.active_root.entries.len() as u32; + let total = self.active_root.entries.len(); + self.active_root.file_count = total as u32; + + // Entries beyond the inline cap go into append-only overflow pages. + // ORDER IS THE CRASH-SAFETY INVARIANT: overflow pages are appended at + // EOF and `sync_data`'d BEFORE the root that references them. Because + // the run is appended (never overwriting the currently-active slot's + // older overflow), a crash before the root's atomic commit leaves the + // previously-committed state — its root AND its overflow run — fully + // intact, so `open()` falls back to it. Partial loss, never corruption. + let inline_count = total.min(MAX_INLINE_ENTRIES); + let overflow = &self.active_root.entries[inline_count..]; + let npages = overflow.len().div_ceil(ENTRIES_PER_OVERFLOW_PAGE); + + let overflow_start_page: u32 = if npages > 0 { + let eof = self.file.seek(SeekFrom::End(0))?; + // The manifest is always a whole number of 4 KB pages. + debug_assert_eq!(eof % PAGE_4K as u64, 0); + let start_page = (eof / PAGE_4K as u64) as u32; + let mut buf = vec![0u8; npages * PAGE_4K]; + for (pi, chunk) in overflow.chunks(ENTRIES_PER_OVERFLOW_PAGE).enumerate() { + Self::serialize_overflow_page(chunk, &mut buf[pi * PAGE_4K..(pi + 1) * PAGE_4K]); + } + self.file.seek(SeekFrom::Start(eof))?; + self.file.write_all(&buf)?; + self.file.sync_data()?; // overflow durable BEFORE the root points at it + start_page + } else { + 0 + }; + + self.active_root.entry_page_count = npages as u32; let mut page = [0u8; PAGE_4K]; - Self::serialize_root(&self.active_root, &mut page); + Self::serialize_root(&self.active_root, overflow_start_page, &mut page); // Write to the inactive slot let write_offset = if self.active_slot == 0 { @@ -443,6 +503,21 @@ impl ShardManifest { // Flip active slot self.active_slot = if self.active_slot == 0 { 1 } else { 0 }; + // Bound append-only growth: when the file is mostly dead overflow from + // superseded commits, rewrite it compactly. Best-effort — the commit is + // already durable, so a compaction failure must not fail the commit. + if npages > 0 { + if let Ok(file_len) = self.file.seek(SeekFrom::End(0)) { + let live_pages = 2 + npages as u64; + let live_bytes = live_pages * PAGE_4K as u64; + if file_len > live_bytes.saturating_mul(4) && file_len > 16 * PAGE_4K as u64 { + if let Err(e) = self.compact() { + tracing::warn!(error = %e, "manifest compaction failed (commit already durable)"); + } + } + } + } + Ok(()) } @@ -597,25 +672,37 @@ impl ShardManifest { &self.path } - /// Serialize a ManifestRoot into a 4KB page buffer (always v2 format). + /// Serialize a ManifestRoot's INLINE portion into a 4KB root page (v2). + /// + /// Only the first `min(entries.len(), MAX_INLINE_ENTRIES)` records live in + /// the root page; any surplus lives in append-only overflow pages written + /// by `commit`. `overflow_start_page` is the file-relative page index where + /// this root's overflow run begins (0 when none) and is stamped into the + /// header `next_page` field; `entry_page_count` (root meta) records the run + /// length. `file_count` in the meta is the TOTAL across inline + overflow. /// - /// Layout per §4.2: epoch(8) + redo_lsn(8) + wal_flush_lsn(8) + file_count(4) + - /// entry_page_count(4) + snapshot_lsn(8) + created_at(8) + shard_uuid(16) = 64 bytes, - /// then file_count * 56-byte FileEntry records. - fn serialize_root(root: &ManifestRoot, page: &mut [u8]) { + /// Layout: epoch(8) + redo_lsn(8) + wal_flush_lsn(8) + file_count(4) + + /// entry_page_count(4) + snapshot_lsn(8) + created_at(8) + shard_uuid(16) = + /// 64 bytes, then inline_count * 56-byte FileEntry records. + fn serialize_root(root: &ManifestRoot, overflow_start_page: u32, page: &mut [u8]) { assert!(page.len() >= PAGE_4K); // Zero the page page[..PAGE_4K].fill(0); - // Payload: 64 bytes meta + file_count * 56 bytes entries (v2) - let payload_bytes = ROOT_META_SIZE + root.entries.len() * FileEntry::SIZE; + let total = root.entries.len(); + let inline_count = total.min(MAX_INLINE_ENTRIES); + + // Payload framing covers ONLY the inline entries — the region this + // page's CRC32C authenticates. Overflow pages carry their own CRC. + let payload_bytes = ROOT_META_SIZE + inline_count * FileEntry::SIZE; // Header — stamp v2 format so readers know to expect 56-byte entries. let mut hdr = MoonPageHeader::new(PageType::ManifestRoot, 0, 0); hdr.format_version = MANIFEST_FORMAT_V2; hdr.payload_bytes = payload_bytes as u32; - hdr.entry_count = root.entries.len() as u32; + hdr.entry_count = inline_count as u32; + hdr.next_page = overflow_start_page; // 0 when no overflow run hdr.write_to(page); // Manifest-specific metadata after header (64 bytes) @@ -623,15 +710,17 @@ impl ShardManifest { page[p..p + 8].copy_from_slice(&root.epoch.to_le_bytes()); page[p + 8..p + 16].copy_from_slice(&root.redo_lsn.to_le_bytes()); page[p + 16..p + 24].copy_from_slice(&root.wal_flush_lsn.to_le_bytes()); - page[p + 24..p + 28].copy_from_slice(&root.file_count.to_le_bytes()); + // file_count = TOTAL entries (inline + overflow), the source of truth + // used on read to compute how many overflow entries to expect. + page[p + 24..p + 28].copy_from_slice(&(total as u32).to_le_bytes()); page[p + 28..p + 32].copy_from_slice(&root.entry_page_count.to_le_bytes()); page[p + 32..p + 40].copy_from_slice(&root.snapshot_lsn.to_le_bytes()); page[p + 40..p + 48].copy_from_slice(&root.created_at.to_le_bytes()); page[p + 48..p + 64].copy_from_slice(&root.shard_uuid); - // FileEntry records + // Inline FileEntry records (first `inline_count`) let entries_start = p + ROOT_META_SIZE; - for (i, entry) in root.entries.iter().enumerate() { + for (i, entry) in root.entries.iter().take(inline_count).enumerate() { let offset = entries_start + i * FileEntry::SIZE; entry.write_to(&mut page[offset..offset + FileEntry::SIZE]); } @@ -640,6 +729,175 @@ impl ShardManifest { MoonPageHeader::compute_checksum(page); } + /// Serialize up to `ENTRIES_PER_OVERFLOW_PAGE` FileEntry records into a 4KB + /// `ManifestEntry` overflow page (header + entries + CRC32C, no per-page + /// meta). `entries.len()` must be ≤ `ENTRIES_PER_OVERFLOW_PAGE`. + fn serialize_overflow_page(entries: &[FileEntry], page: &mut [u8]) { + assert!(page.len() >= PAGE_4K); + assert!(entries.len() <= ENTRIES_PER_OVERFLOW_PAGE); + page[..PAGE_4K].fill(0); + + let mut hdr = MoonPageHeader::new(PageType::ManifestEntry, 0, 0); + hdr.format_version = MANIFEST_FORMAT_V2; + hdr.payload_bytes = (entries.len() * FileEntry::SIZE) as u32; + hdr.entry_count = entries.len() as u32; + hdr.write_to(page); + + let start = MOONPAGE_HEADER_SIZE; + for (i, e) in entries.iter().enumerate() { + let off = start + i * FileEntry::SIZE; + e.write_to(&mut page[off..off + FileEntry::SIZE]); + } + MoonPageHeader::compute_checksum(page); + } + + /// Parse a `ManifestEntry` overflow page. Returns `None` on ANY header, + /// type, CRC, or framing failure so a torn/corrupt overflow page makes the + /// whole root fall back to the last-good slot (never surfaces garbage). + fn parse_overflow_page(page: &[u8]) -> Option> { + if page.len() < PAGE_4K { + return None; + } + let hdr = MoonPageHeader::read_from(page)?; + if hdr.page_type != PageType::ManifestEntry { + return None; + } + if !MoonPageHeader::verify_checksum(page) { + return None; + } + let n = hdr.entry_count as usize; + if n > ENTRIES_PER_OVERFLOW_PAGE { + return None; + } + if hdr.payload_bytes as usize != n * FileEntry::SIZE { + return None; + } + let start = MOONPAGE_HEADER_SIZE; + let mut out = Vec::with_capacity(n); + for i in 0..n { + let off = start + i * FileEntry::SIZE; + out.push(FileEntry::read_from(&page[off..])?); + } + Some(out) + } + + /// Load one slot fully: parse its inline root, then read its append-only + /// overflow run. Returns `None` (→ dual-root fallback) if the root is + /// invalid OR its overflow is torn/short/corrupt/inconsistent — so a crash + /// mid-overflow-write can never surface a torn mix or garbage entries. + fn load_root(buf: &[u8], slot_offset: usize) -> Option { + let end = slot_offset.checked_add(PAGE_4K)?; + if end > buf.len() { + return None; + } + let slice = &buf[slot_offset..end]; + let mut root = Self::try_parse_root(slice)?; + + if root.entry_page_count > 0 { + // Overflow run start lives in the root header's `next_page`. + let hdr = MoonPageHeader::read_from(slice)?; + let start = hdr.next_page as usize; + let npages = root.entry_page_count as usize; + for pi in 0..npages { + let off = start.checked_add(pi)?.checked_mul(PAGE_4K)?; + let pend = off.checked_add(PAGE_4K)?; + if pend > buf.len() { + return None; // torn/short overflow → fall back to last-good + } + let entries = Self::parse_overflow_page(&buf[off..pend])?; + root.entries.extend(entries); + } + } + + // The reconstructed total must reconcile with the root's file_count, + // else treat the slot as corrupt and fall back. + if root.entries.len() != root.file_count as usize { + return None; + } + Some(root) + } + + /// Rewrite the manifest compactly, reclaiming dead overflow pages from + /// superseded commits. Layout becomes `[Root A][Root B][fresh overflow]` + /// with the active root written to BOTH slots (same epoch) so either is a + /// valid recovery target. Crash-safe via temp-file + atomic rename: the + /// live manifest stays valid until the rename completes. + fn compact(&mut self) -> std::io::Result<()> { + let total = self.active_root.entries.len(); + let inline_count = total.min(MAX_INLINE_ENTRIES); + let overflow = &self.active_root.entries[inline_count..]; + let npages = overflow.len().div_ceil(ENTRIES_PER_OVERFLOW_PAGE); + + // Overflow starts immediately after the two root pages. + let overflow_start_page: u32 = if npages > 0 { 2 } else { 0 }; + self.active_root.entry_page_count = npages as u32; + + let mut buf = vec![0u8; (2 + npages) * PAGE_4K]; + for (pi, chunk) in overflow.chunks(ENTRIES_PER_OVERFLOW_PAGE).enumerate() { + let o = (2 + pi) * PAGE_4K; + Self::serialize_overflow_page(chunk, &mut buf[o..o + PAGE_4K]); + } + Self::serialize_root(&self.active_root, overflow_start_page, &mut buf[0..PAGE_4K]); + Self::serialize_root( + &self.active_root, + overflow_start_page, + &mut buf[PAGE_4K..2 * PAGE_4K], + ); + + let tmp = self.path.with_extension("manifest.compact.tmp"); + std::fs::write(&tmp, &buf)?; + { + let tf = std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(&tmp)?; + tf.sync_data()?; + } + std::fs::rename(&tmp, &self.path)?; + if let Some(parent) = self.path.parent() { + crate::persistence::fsync::fsync_directory(parent)?; + } + // Repoint the file handle at the freshly-rewritten manifest. Both slots + // carry the active root at the same epoch; treat slot 0 as active so the + // next commit writes the incremented epoch to slot 1 (the newest). + // + // The rename above is the durability point — the compacted manifest is + // already safe on disk. If the reopen below fails, our `self.file` still + // refers to the pre-rename (orphaned) inode; do NOT keep using it (that + // silently discards every later commit). Mark `needs_reopen` so the next + // `commit()` reattaches to `self.path` first, and propagate the error. + #[cfg(test)] + if self.fail_compact_reopen { + // Simulate the lost-handle race: point `self.file` at a throwaway + // inode (the real "orphaned pre-rename fd"), mark for reopen, return. + let orphan = self.path.with_extension("compact.orphan.test"); + self.file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(true) + .open(&orphan)?; + self.needs_reopen = true; + return Err(std::io::Error::other("simulated compact reopen failure")); + } + match std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(&self.path) + { + Ok(f) => { + self.file = f; + self.active_slot = 0; + self.needs_reopen = false; + Ok(()) + } + Err(e) => { + self.needs_reopen = true; + Err(e) + } + } + } + /// Try to parse a root page from a 4KB buffer. /// /// Returns `None` if magic/type mismatch or CRC32C fails. Recognizes both @@ -677,18 +935,26 @@ impl ShardManifest { let epoch = u64::from_le_bytes(page[p..p + 8].try_into().ok()?); let redo_lsn = u64::from_le_bytes(page[p + 8..p + 16].try_into().ok()?); let wal_flush_lsn = u64::from_le_bytes(page[p + 16..p + 24].try_into().ok()?); + // `file_count` is the TOTAL across inline + overflow. Only the first + // `inline_count` records live in this root page; the surplus lives in + // overflow pages loaded later by `load_root`. v1 never overflows. let file_count = u32::from_le_bytes(page[p + 24..p + 28].try_into().ok()?); let entry_page_count = u32::from_le_bytes(page[p + 28..p + 32].try_into().ok()?); + let inline_count = if entry_size == FileEntry::SIZE_V1 { + file_count as usize + } else { + (file_count as usize).min(MAX_INLINE_ENTRIES) + }; - // Validate payload framing: root metadata + declared entries must match - // the authenticated payload_bytes and entry_count in the header. This - // prevents reading unchecked trailing bytes on a corrupted root page. - let expected_payload = - ROOT_META_SIZE.checked_add((file_count as usize).checked_mul(entry_size)?)?; + // Validate payload framing against the INLINE count — the region this + // page's CRC32C authenticates. Overflow pages carry their own CRC and + // are validated when loaded. This prevents reading unchecked trailing + // bytes on a corrupted root page. + let expected_payload = ROOT_META_SIZE.checked_add(inline_count.checked_mul(entry_size)?)?; if hdr.payload_bytes as usize != expected_payload { return None; } - if hdr.entry_count != file_count { + if hdr.entry_count as usize != inline_count { return None; } let snapshot_lsn = u64::from_le_bytes(page[p + 32..p + 40].try_into().ok()?); @@ -696,12 +962,12 @@ impl ShardManifest { let mut shard_uuid = [0u8; 16]; shard_uuid.copy_from_slice(&page[p + 48..p + 64]); - // Parse entries. Use the size dictated by format_version so v1 - // manifests remain readable; FileEntry::read_v1 synthesizes + // Parse the inline entries. Use the size dictated by format_version so + // v1 manifests remain readable; FileEntry::read_v1 synthesizes // `last_modified_lsn = created_lsn` for the upgraded in-memory view. let entries_start = p + ROOT_META_SIZE; let mut entries = Vec::with_capacity(file_count as usize); - for i in 0..file_count as usize { + for i in 0..inline_count { let offset = entries_start + i * entry_size; let entry = if entry_size == FileEntry::SIZE_V1 { FileEntry::read_v1(&page[offset..])? @@ -957,6 +1223,55 @@ mod tests { assert_eq!(m2.files().len(), 2); } + // Regression (PR #136 review, BUG #1): compact() renames a fresh manifest + // into place durably, then reopens self.file against the new inode. If that + // reopen fails, the old handle refers to the pre-rename (orphaned) inode — + // continuing to write through it silently discards every later commit, and + // recovery sees only the compaction snapshot. The fix flags `needs_reopen` + // and makes the next commit reattach to self.path before writing. + #[test] + fn compact_reopen_failure_does_not_silently_lose_later_commits() { + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("shard-0.manifest"); + let mut m = ShardManifest::create(&path).unwrap(); + + // Seed committed state large enough to exercise inline + overflow. + for id in 1..=80 { + m.add_file(make_entry(id)); + } + m.commit().unwrap(); + + // Simulate the race: compaction rewrote + renamed durably (data safe on + // disk) but then failed to reopen the handle. + m.fail_compact_reopen = true; + let err = m.compact().unwrap_err(); + assert_eq!(err.kind(), std::io::ErrorKind::Other); + assert!( + m.needs_reopen, + "compact() must flag needs_reopen when it loses the file handle" + ); + m.fail_compact_reopen = false; + + // The next commit MUST reattach to the real manifest first. Without the + // guard this write lands in the orphaned inode and is lost on recovery. + m.add_file(make_entry(999)); + m.commit().unwrap(); + assert!( + !m.needs_reopen, + "commit() must clear needs_reopen after reattaching to self.path" + ); + + // Recover from disk: the post-failure entry MUST be durable. + let recovered = ShardManifest::open(&path).unwrap(); + let ids: Vec = recovered.files().iter().map(|e| e.file_id).collect(); + assert!( + ids.contains(&999), + "an entry committed after a compact-reopen failure must survive \ + recovery (stale-handle silent data-loss regression)" + ); + assert_eq!(recovered.files().len(), 81); + } + #[test] fn test_manifest_recovery_picks_higher_epoch() { let tmp = tempfile::tempdir().unwrap(); @@ -1062,12 +1377,174 @@ mod tests { let m2 = ShardManifest::open(&path).unwrap(); assert_eq!(m2.files().len(), 70); - // Adding one more should fail on commit + // Beyond the inline cap, entries now persist via overflow pages + // (was: commit rejected the 71st entry). drop(m2); let mut m3 = ShardManifest::open(&path).unwrap(); m3.add_file(make_entry(71)); - let result = m3.commit(); - assert!(result.is_err()); + m3.commit() + .expect("71st entry must persist via an overflow page"); + drop(m3); + assert_eq!( + ShardManifest::open(&path).unwrap().files().len(), + 71, + "overflow entry must survive reopen", + ); + } + + /// #15 RED — durability contract: a manifest holding FAR more than the + /// 70-entry inline cap must persist every entry and recover them all. + /// 70 entries ≈ 9 MB cold/shard, so every real disk-offload deployment + /// blows past the cap immediately (Phase C hit "495 > 70"). RED today: + /// `commit()` returns Err at the 71st entry. + #[test] + fn test_overflow_persists_beyond_inline_cap() { + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("shard-0.manifest"); + + let n = 200u64; // ~3 overflow pages worth + let mut m = ShardManifest::create(&path).unwrap(); + for i in 0..n { + m.add_file(make_entry(i + 1)); + } + m.commit() + .expect("manifest must persist >70 entries via overflow pages"); + drop(m); + + // Reopen from disk only — proves on-disk durability, not in-memory state. + let m2 = ShardManifest::open(&path).unwrap(); + assert_eq!( + m2.files().len(), + n as usize, + "all {n} entries must recover from overflow pages", + ); + let ids: std::collections::HashSet = m2.files().iter().map(|e| e.file_id).collect(); + for i in 0..n { + assert!(ids.contains(&(i + 1)), "file_id {} lost on recovery", i + 1); + } + } + + /// #15 RED — crash-atomicity: a torn overflow write during a later commit + /// must NEVER corrupt the previously-committed state. The advisor's key + /// risk: a naïve overflow chain converts today's PARTIAL loss (last-good + /// inline root intact) into TOTAL loss (root points at a half-written + /// overflow page → garbage → lose everything). + /// + /// Model: commit state1 (N1=150 > cap), then commit state2 (N2=300), then + /// truncate the file by one 4 KB page to simulate a crash mid-overflow-write + /// of state2. Reopen MUST yield a consistent state — the last-good state1 + /// (150) via dual-root fallback — never a panic, never a torn mix. + #[test] + fn test_overflow_commit_crash_atomicity() { + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("shard-0.manifest"); + + let n1 = 150u64; + let n2 = 300u64; + + let mut m = ShardManifest::create(&path).unwrap(); + for i in 0..n1 { + m.add_file(make_entry(i + 1)); + } + m.commit().expect("state1 (>cap) must commit via overflow"); + // state2: extend to n2 and commit again (flips active slot; state1's + // root remains in the now-inactive slot as the last-good fallback). + for i in n1..n2 { + m.add_file(make_entry(i + 1)); + } + m.commit().expect("state2 (>cap) must commit via overflow"); + drop(m); + + // Simulate a crash that tore the tail of state2's overflow write: + // lop off the final 4 KB page. state2's root now references an + // incomplete overflow region; state1's root + overflow are untouched. + let full = std::fs::metadata(&path).unwrap().len(); + assert!(full > (2 * PAGE_4K) as u64, "overflow must extend the file"); + let f = std::fs::OpenOptions::new().write(true).open(&path).unwrap(); + f.set_len(full - PAGE_4K as u64).unwrap(); + drop(f); + + // Recovery must be CONSISTENT, never corrupt. Newest root (state2) has a + // torn overflow → open() must fall back to the last-good state1 root. + let recovered = ShardManifest::open(&path) + .expect("torn overflow tail must not make the manifest unopenable"); + let len = recovered.files().len(); + assert_eq!( + len, n1 as usize, + "torn state2 overflow must fall back to last-good state1 ({n1}), got {len}", + ); + // And every recovered entry must be a real state1 entry (no garbage ids). + for e in recovered.files() { + assert!( + e.file_id >= 1 && e.file_id <= n1, + "garbage file_id {} after torn-overflow recovery", + e.file_id, + ); + } + } + + /// #16 — append-only overflow must not grow the manifest without bound: + /// repeatedly committing a >cap set (each commit appends a fresh overflow + /// run) must trigger compaction so the file stays a small multiple of the + /// live set, and the data must still fully recover. + #[test] + fn test_overflow_compaction_bounds_growth() { + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("shard-0.manifest"); + + let mut m = ShardManifest::create(&path).unwrap(); + for i in 0..100u64 { + m.add_file(make_entry(i + 1)); + } + // Each commit re-appends the overflow run; without compaction the file + // would grow ~1 page per commit (60+ dead runs). Compaction bounds it. + for _ in 0..60 { + m.commit().unwrap(); + } + drop(m); + + let len = std::fs::metadata(&path).unwrap().len(); + // Live set = 2 roots + ceil(30/72)=1 overflow page = 3 pages (12 KB). + // Compaction keeps the file within a small multiple of that. + assert!( + len <= 20 * PAGE_4K as u64, + "manifest grew unbounded despite compaction: {len} bytes", + ); + + let m2 = ShardManifest::open(&path).unwrap(); + assert_eq!(m2.files().len(), 100, "data lost after compaction"); + let ids: std::collections::HashSet = m2.files().iter().map(|e| e.file_id).collect(); + for i in 0..100u64 { + assert!(ids.contains(&(i + 1)), "file_id {} lost", i + 1); + } + } + + /// #16 — tombstoning + GC of an entry that lives in the OVERFLOW region + /// (id > inline cap) must remove exactly that entry and survive reopen. + #[test] + fn test_overflow_tombstone_and_gc() { + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("shard-0.manifest"); + + let mut m = ShardManifest::create(&path).unwrap(); + for i in 0..200u64 { + m.add_file(make_entry(i + 1)); + } + m.commit().unwrap(); + + // id 150 lives in the overflow region (inline cap is 70). + m.remove_file(150); + let pruned = m.gc_tombstones(0, 0, std::time::Instant::now()); + assert_eq!(pruned, 1, "overflow-region tombstone must be prunable"); + m.commit().unwrap(); + drop(m); + + let m2 = ShardManifest::open(&path).unwrap(); + assert_eq!(m2.files().len(), 199); + assert!( + m2.files().iter().all(|e| e.file_id != 150), + "tombstoned overflow entry survived recovery", + ); } /// P1 — manifest written today must always stamp format_version = 2. diff --git a/src/persistence/migrate_aof.rs b/src/persistence/migrate_aof.rs index 625a91007..db6153421 100644 --- a/src/persistence/migrate_aof.rs +++ b/src/persistence/migrate_aof.rs @@ -45,7 +45,7 @@ //! //! # Usage //! -//! ``` +//! ```text //! moon --migrate-aof-from /old/dir --migrate-aof-to /new/dir --migrate-aof-shards 4 //! ``` //! diff --git a/src/persistence/page_cache/mod.rs b/src/persistence/page_cache/mod.rs index 872c6e3e4..fae15762a 100644 --- a/src/persistence/page_cache/mod.rs +++ b/src/persistence/page_cache/mod.rs @@ -60,6 +60,28 @@ pub struct PageCache { sweep_64k: ClockSweep, } +/// Split a per-shard PageCache budget (bytes) into (4KB, 64KB) frame counts. +/// +/// 75% of the budget backs 4KB frames, 25% backs 64KB frames, with minimum +/// floors of 64 / 8 frames so a tiny budget still yields a usable cache. With +/// lazy buffers these are only *capacities* — actual memory grows on demand. +pub fn pagecache_frame_counts(budget_bytes: u64) -> (usize, usize) { + let num_4k = ((budget_bytes * 3 / 4) / PAGE_4K as u64) as usize; + let num_64k = ((budget_bytes / 4) / PAGE_64K as u64) as usize; + (num_4k.max(64), num_64k.max(8)) +} + +/// Divide a whole-instance PageCache budget across shards. +/// +/// `--pagecache-size` (and the 25%-of-maxmemory default) express a +/// whole-instance intent, but each shard builds its own PageCache. Sizing every +/// shard to the whole budget over-committed by `num_shards`× — the multishard +/// "zombie eating RAM". Dividing here bounds total pre-allocation to the budget +/// regardless of shard count. `num_shards == 0` is treated as 1 (never panics). +pub fn per_shard_pagecache_budget(whole_budget_bytes: u64, num_shards: usize) -> u64 { + whole_budget_bytes / (num_shards.max(1) as u64) +} + impl PageCache { /// Create a new PageCache with pre-allocated frame pools. /// @@ -68,15 +90,20 @@ impl PageCache { pub fn new(num_frames_4k: usize, num_frames_64k: usize) -> Self { let frames_4k: Vec = (0..num_frames_4k).map(|_| FrameDescriptor::new()).collect(); + // Lazy buffers: start EMPTY (zero heap), grown to a full page on first + // use in `fetch_page`. Eagerly committing `num_frames * PAGE` zeroed + // bytes here was the multishard "zombie eating RAM" — a 4-shard server + // with the auto memory guardrail pre-committed ≈80% of host RAM at + // startup before serving a command. RSS now tracks the working set. let buffers_4k: Vec>> = (0..num_frames_4k) - .map(|_| RwLock::new(vec![0u8; PAGE_4K])) + .map(|_| RwLock::new(Vec::new())) .collect(); let frames_64k: Vec = (0..num_frames_64k) .map(|_| FrameDescriptor::new()) .collect(); let buffers_64k: Vec>> = (0..num_frames_64k) - .map(|_| RwLock::new(vec![0u8; PAGE_64K])) + .map(|_| RwLock::new(Vec::new())) .collect(); Self { @@ -90,6 +117,17 @@ impl PageCache { } } + /// Total bytes currently committed across all page buffers. + /// + /// With lazy buffers this reflects the actual resident working set, not the + /// configured budget — a freshly constructed cache returns 0. Used by tests + /// and memory reporting; not a hot path (locks each buffer). + pub fn resident_buffer_bytes(&self) -> usize { + let small: usize = self.buffers_4k.iter().map(|b| b.read().len()).sum(); + let large: usize = self.buffers_64k.iter().map(|b| b.read().len()).sum(); + small + large + } + /// Fetch a page into the cache and return a pinned handle. /// /// On cache hit: pins the frame, touches usage count, returns handle. @@ -154,9 +192,15 @@ impl PageCache { // Reset frame for new page victim.reset(file_id, page_offset); - // Read page data from disk + // Read page data from disk. Lazily commit this frame's buffer to a full + // page on first use (it starts empty from `new`); a reused frame already + // has the right length so this is a no-op after the first miss. + let page_size = if is_large { PAGE_64K } else { PAGE_4K }; { let mut buf = buffers[victim_idx].write(); + if buf.len() != page_size { + buf.resize(page_size, 0); + } read_fn(&mut buf)?; } @@ -942,4 +986,89 @@ mod tests { "FPI should not be called when FPI_PENDING is not set" ); } + + // ── Zombie-RAM regression: lazy buffers + per-shard budget ────────────── + // + // Root cause of "moon eats RAM in multishard": disk-offload is on by + // default, so every shard EAGERLY committed a PageCache frame pool sized to + // 25% of the WHOLE maxmemory at startup. N shards => N x 25% x maxmemory + // pre-committed before serving one command (≈80% of host RAM with the auto + // guardrail). These tests lock in the two fixes: (1) buffers allocate + // lazily on first use, (2) the budget is divided across shards. + + #[test] + fn buffers_are_lazily_allocated_not_eagerly() { + // A pool big enough to be obvious if eager: 1000x4K + 100x64K ≈ 10.4 MB. + let cache = PageCache::new(1000, 100); + assert_eq!( + cache.resident_buffer_bytes(), + 0, + "a freshly constructed PageCache must NOT pre-commit page buffers \ + (the eager-alloc zombie); buffers grow on demand" + ); + + // First miss on a single 4K page commits exactly one frame's buffer. + let h = cache + .fetch_page(1, 0, false, |buf| { + // The miss path must size the buffer to a full page before the + // fill closure runs — callers index buf[..PAGE_4K]. + assert_eq!(buf.len(), PAGE_4K); + buf[0] = 7; + buf[PAGE_4K - 1] = 9; + Ok(()) + }) + .expect("fetch_page"); + assert_eq!( + cache.resident_buffer_bytes(), + PAGE_4K, + "exactly one 4K buffer resident after one 4K miss" + ); + { + let d = cache.page_data(&h); + assert_eq!(d.len(), PAGE_4K); + assert_eq!(d[0], 7); + assert_eq!(d[PAGE_4K - 1], 9); + } + cache.unpin_page(h); + + // A 64K miss commits exactly one 64K buffer on top. + let h2 = cache + .fetch_page(2, 0, true, |buf| { + assert_eq!(buf.len(), PAGE_64K); + Ok(()) + }) + .expect("fetch_page large"); + assert_eq!( + cache.resident_buffer_bytes(), + PAGE_4K + PAGE_64K, + "one 4K + one 64K buffer resident; the other 1098 frames stay unallocated" + ); + cache.unpin_page(h2); + } + + #[test] + fn pagecache_budget_divides_across_shards() { + let whole = 1024u64 * 1024 * 1024; // 1 GiB whole-instance budget + // The bug: each shard used the WHOLE budget. The fix: split by shards. + assert_eq!(per_shard_pagecache_budget(whole, 4), whole / 4); + assert_eq!(per_shard_pagecache_budget(whole, 1), whole); + // div-by-zero guard: 0 shards treated as 1 (never panics). + assert_eq!(per_shard_pagecache_budget(whole, 0), whole); + + // Frame counts scale down with the per-shard budget. + let (f4_whole, f64_whole) = pagecache_frame_counts(whole); + let (f4_quarter, f64_quarter) = pagecache_frame_counts(whole / 4); + assert!( + f4_quarter < f4_whole && f64_quarter < f64_whole, + "per-shard frame counts must shrink with the divided budget" + ); + assert!( + f4_quarter >= 64 && f64_quarter >= 8, + "minimum frame floors hold" + ); + + // Tiny budget still respects the minimum floors (never zero frames). + let (f4_min, f64_min) = pagecache_frame_counts(1); + assert_eq!((f4_min, f64_min), (64, 8)); + } } diff --git a/src/server/conn/handler_monoio/mod.rs b/src/server/conn/handler_monoio/mod.rs index a04173bed..bf96491ca 100644 --- a/src/server/conn/handler_monoio/mod.rs +++ b/src/server/conn/handler_monoio/mod.rs @@ -41,6 +41,20 @@ use crate::shard::dispatch::ShardMessage; // ResponseSlotPool NOT used on monoio — its AtomicWaker doesn't cross // monoio's single-threaded (!Send) executor boundary. Use oneshot channels. +// ── F3: cross-shard dispatch backpressure / response-wait bounds ── +// Design-for-failure: a wedged or saturated target shard must surface a +// bounded error, never park the connection (holding its buffers) forever. +// Push-retry bounds live in `crate::shard::dispatch` (shared with the tokio +// handler); the response-wait bound below is monoio-specific (the tokio path +// awaits via `ResponseSlotPool`, not this busy-wait relay). + +/// Max ~1ms event-loop wake cycles to wait for a cross-shard reply before +/// declaring the response lost. The batch was already dispatched, so this is +/// an *uncertain write* backstop (the command may have applied on the target) +/// — set generously (~30s) and primarily guarded by the shutdown check. +#[cfg(feature = "runtime-monoio")] +const CROSS_SHARD_RESPONSE_MAX_WAITS: u32 = 30_000; + /// Result of `handle_connection_sharded_monoio` execution. /// /// Same purpose as the Tokio handler's `HandlerResult`: the generic handler cannot @@ -1919,28 +1933,58 @@ pub(crate) async fn handle_connection_sharded_monoio< reply_tx, }; let target_idx = ChannelMesh::target_index(ctx.shard_id, target); - { - let mut pending = msg; - loop { - let push_result = { + // F3: bounded backpressure retry. The closure retains the + // message on a full ring; the helper checks `shutdown` before + // every backoff so a graceful shutdown can't hang on a wedged + // peer. Borrow of `ctx.dispatch_tx` is taken+released inside + // each attempt — never held across the await. + let mut pending = Some(msg); + let outcome = crate::shard::dispatch::push_with_backpressure( + &shutdown, + crate::shard::dispatch::CROSS_SHARD_PUSH_MAX_RETRIES, + crate::shard::dispatch::CROSS_SHARD_PUSH_BACKOFF, + || match pending.take() { + None => true, + Some(m) => { let mut producers = ctx.dispatch_tx.borrow_mut(); - producers[target_idx].try_push(pending) - }; - match push_result { - Ok(()) => { - tracing::trace!( - "Shard {}: pushed PipelineBatch to shard {}, notifying", - ctx.shard_id, - target - ); - ctx.spsc_notifiers[target].notify_one(); - break; - } - Err(val) => { - pending = val; - monoio::time::sleep(std::time::Duration::from_micros(10)).await; + match producers[target_idx].try_push(m) { + Ok(()) => true, + Err(back) => { + pending = Some(back); + false + } } } + }, + ) + .await; + match outcome { + crate::shard::dispatch::PushOutcome::Pushed => { + tracing::trace!( + "Shard {}: pushed PipelineBatch to shard {}, notifying", + ctx.shard_id, + target + ); + ctx.spsc_notifiers[target].notify_one(); + } + crate::shard::dispatch::PushOutcome::Backpressure + | crate::shard::dispatch::PushOutcome::Cancelled => { + // Target shard not draining (saturated/wedged) or + // shutting down. The PipelineBatch was NEVER accepted, + // so this is a clean reject — fail this batch's entries + // instead of parking the connection forever. + tracing::warn!( + "Shard {}: cross-shard push to shard {} gave up ({:?}); rejecting batch", + ctx.shard_id, + target, + outcome + ); + for (resp_idx, _, _) in &meta { + responses[*resp_idx] = Frame::Error(Bytes::from_static( + b"ERR cross-shard dispatch backpressure", + )); + } + continue; } } oneshot_futures.push((target, meta, reply_rx)); @@ -1958,11 +2002,33 @@ pub(crate) async fn handle_connection_sharded_monoio< ); let shard_responses = { let pw = pending_wakers.clone(); + // F3: bound the response wait. The batch was already + // dispatched, so a missing reply is an *uncertain write* + // (the target shard may have applied it) — the error must + // NOT imply rejection. Break on disconnect (writer gone), + // shutdown, or the generous wait cap (~30s of ~1ms wakes). + let mut waits: u32 = 0; loop { match reply_rx.try_recv() { Ok(value) => break Ok(value), - Err(flume::TryRecvError::Disconnected) => break Err(()), + Err(flume::TryRecvError::Disconnected) => { + break Err("ERR cross-shard dispatch failed"); + } Err(flume::TryRecvError::Empty) => { + if shutdown.is_cancelled() { + break Err("ERR cross-shard response aborted (shutdown)"); + } + waits += 1; + if waits > CROSS_SHARD_RESPONSE_MAX_WAITS { + tracing::warn!( + "Shard {}: cross-shard response wait exhausted; \ + target may have applied the write", + ctx.shard_id + ); + break Err( + "ERR cross-shard response timeout (write may have applied)", + ); + } // Yield once: register waker, return Pending, then Ready on wake. let mut yielded = false; std::future::poll_fn(|cx| { @@ -1982,11 +2048,10 @@ pub(crate) async fn handle_connection_sharded_monoio< }; let shard_responses = match shard_responses { Ok(r) => r, - Err(()) => { + Err(err_msg) => { for (resp_idx, _, _) in &meta { - responses[*resp_idx] = Frame::Error(Bytes::from_static( - b"ERR cross-shard dispatch failed", - )); + responses[*resp_idx] = + Frame::Error(Bytes::from_static(err_msg.as_bytes())); } continue; } diff --git a/src/server/conn/handler_sharded/mod.rs b/src/server/conn/handler_sharded/mod.rs index 8e75b2088..7de67190a 100644 --- a/src/server/conn/handler_sharded/mod.rs +++ b/src/server/conn/handler_sharded/mod.rs @@ -19,7 +19,7 @@ use crate::persistence::aof; use crate::protocol::Frame; use crate::shard::dispatch::{ShardMessage, key_to_shard}; use crate::shard::mesh::ChannelMesh; -use crate::storage::eviction::try_evict_if_needed; +use crate::storage::eviction::{try_evict_if_needed, try_evict_if_needed_async_spill}; use crate::workspace::{strip_workspace_prefix_from_response, workspace_rewrite_args}; use super::affinity::MigratedConnectionState; @@ -1272,7 +1272,34 @@ pub(crate) async fn handle_connection_sharded_inner< conn: &mut super::core::ConnectionState| -> WriteOutcome { let rt = ctx.runtime_config.read(); - if let Err(oom_frame) = try_evict_if_needed(db, &rt) { + // Disk-offload: when a per-shard spill sender is wired + // (ConnCtx populated by spawn_tokio_connection), evicted + // KVs are spilled to the cold tier on the background spill + // thread instead of being deleted — mirrors handler_monoio. + // Without the sender (disk-offload disabled) fall back to + // delete-only eviction. Both evictors run under this shard's + // db write lock, so they cannot race the persistence-tick + // cascade on the same key. + let evict_result = if let Some(ref sender) = ctx.spill_sender { + let mut fid = ctx.spill_file_id.get(); + let dir = ctx + .disk_offload_dir + .as_deref() + .unwrap_or(std::path::Path::new(".")); + let res = try_evict_if_needed_async_spill( + db, + &rt, + sender, + dir, + &mut fid, + conn.selected_db, + ); + ctx.spill_file_id.set(fid); + res + } else { + try_evict_if_needed(db, &rt) + }; + if let Err(oom_frame) = evict_result { drop(rt); return Err(oom_frame); } @@ -1674,17 +1701,51 @@ pub(crate) async fn handle_connection_sharded_inner< entries.into_iter().map(|(idx, arc_frame, aof, cmd, _db)| ((idx, aof, cmd), arc_frame)).unzip(); let msg = ShardMessage::PipelineBatchSlotted { db_index: batch_db, commands, response_slot: crate::shard::dispatch::ResponseSlotPtr(slot_ptr) }; let target_idx = ChannelMesh::target_index(ctx.shard_id, target); - { - let mut pending = msg; - loop { - let push_result = { let mut producers = ctx.dispatch_tx.borrow_mut(); producers[target_idx].try_push(pending) }; - match push_result { - Ok(()) => { ctx.spsc_notifiers[target].notify_one(); break; } - Err(val) => { pending = val; tokio::task::yield_now().await; } + // F3: bounded backpressure retry (shared helper). The + // closure retains the message on a full ring; the helper + // checks `shutdown` before each backoff so a wedged peer + // can't park this connection forever. On give-up the + // batch was NEVER sent — `slot_ptr` has no side effect, so + // simply skip `future_for` (no accounting leak) and error + // the entries directly. + let mut pending = Some(msg); + let outcome = crate::shard::dispatch::push_with_backpressure( + &shutdown, + crate::shard::dispatch::CROSS_SHARD_PUSH_MAX_RETRIES, + crate::shard::dispatch::CROSS_SHARD_PUSH_BACKOFF, + || match pending.take() { + None => true, + Some(m) => { + let mut producers = ctx.dispatch_tx.borrow_mut(); + match producers[target_idx].try_push(m) { + Ok(()) => true, + Err(back) => { + pending = Some(back); + false + } + } + } + }, + ) + .await; + match outcome { + crate::shard::dispatch::PushOutcome::Pushed => { + ctx.spsc_notifiers[target].notify_one(); + reply_futures.push((meta, target)); + } + crate::shard::dispatch::PushOutcome::Backpressure + | crate::shard::dispatch::PushOutcome::Cancelled => { + tracing::warn!( + "Shard {}: cross-shard push to shard {} gave up ({:?}); rejecting batch", + ctx.shard_id, target, outcome + ); + for (resp_idx, _, _) in &meta { + responses[*resp_idx] = Frame::Error(Bytes::from_static( + b"ERR cross-shard dispatch backpressure", + )); } } } - reply_futures.push((meta, target)); } let proto_ver = conn.protocol_version; for (meta, target) in reply_futures { diff --git a/src/server/conn/handler_single.rs b/src/server/conn/handler_single.rs index d7a938fae..d1375a49e 100644 --- a/src/server/conn/handler_single.rs +++ b/src/server/conn/handler_single.rs @@ -2508,7 +2508,11 @@ mod tests { async fn flush_with_aof_ack_ack_precedes_response() { // Build an Always-policy pool backed by a real bounded channel. let (tx, rx) = channel::mpsc_bounded::(4); - let pool = AofWriterPool::top_level_with_policy(tx, FsyncPolicy::Always); + let pool = AofWriterPool::top_level_with_policy( + tx, + FsyncPolicy::Always, + std::time::Duration::ZERO, + ); // Mock writer: receives one AppendSync, sleeps 60ms to simulate fsync, // then sends Synced. Runs on a blocking thread because flume's diff --git a/src/server/embedded.rs b/src/server/embedded.rs index 3d7e3cad0..01ca43bea 100644 --- a/src/server/embedded.rs +++ b/src/server/embedded.rs @@ -84,6 +84,10 @@ pub async fn run_embedded( ) })?; + // G1 memory guardrail: resolve --maxmemory before RuntimeConfig is built + // (matches the binary entry in main.rs). + crate::config::log_memory_guardrail(config.apply_memory_guardrail()); + // Resolve shard count (`0` => auto-detect core count, matches main.rs). if config.shards == 0 { config.shards = std::thread::available_parallelism() @@ -136,7 +140,11 @@ pub async fn run_embedded( .context("embedded moon: failed to spawn AOF writer thread")?; info!("embedded moon: AOF enabled (fsync: {:?})", fsync); ( - Some(AofWriterPool::top_level_with_policy(tx, fsync)), + Some(AofWriterPool::top_level_with_policy( + tx, + fsync, + std::time::Duration::from_millis(config.aof_fsync_timeout_ms), + )), Some(handle), ) } else { @@ -172,6 +180,10 @@ pub async fn run_embedded( // Shared runtime + server configs. let runtime_config_shared: Arc> = Arc::new(RwLock::new(config.to_runtime_config())); + // Publish the resolved shard count so maxmemory is enforced as a + // whole-instance cap (per-shard budget = maxmemory / num_shards). + runtime_config_shared.write().num_shards = num_shards; + crate::config::log_maxmemory_sharding(runtime_config_shared.read().maxmemory, num_shards); let server_config_shared: Arc = Arc::new(config.clone()); // Per-shard pubsub + remote-subscriber registries. diff --git a/src/server/listener.rs b/src/server/listener.rs index a93f26b7d..b4fb5ae38 100644 --- a/src/server/listener.rs +++ b/src/server/listener.rs @@ -125,7 +125,11 @@ pub async fn run_with_shutdown( let aof_file_path = PathBuf::from(&config.dir).join(&config.appendfilename); tokio::spawn(aof::aof_writer_task(rx, aof_file_path, fsync, aof_token)); info!("AOF enabled with fsync policy: {:?}", fsync); - Some(AofWriterPool::top_level_with_policy(tx, fsync)) + Some(AofWriterPool::top_level_with_policy( + tx, + fsync, + std::time::Duration::from_millis(config.aof_fsync_timeout_ms), + )) } else { None }; @@ -293,6 +297,36 @@ pub async fn run_sharded( affinity_tracker: Arc>, ) -> anyhow::Result<()> { let addr = format!("{}:{}", config.bind, config.port); + // Bind the central listener with SO_REUSEPORT so it COEXISTS with the + // per-shard SO_REUSEPORT listeners (shard/event_loop creates one per shard + // via the same `create_reuseport_socket` helper). A plain `TcpListener::bind` + // here is NOT REUSEPORT-compatible: it races the shards' REUSEPORT binds and, + // whenever the shards win that race, fails with EADDRINUSE — and a failure + // here returns Err from run_sharded, tearing down the ENTIRE server (the + // tokio multishard "zombie": port bound, shards parked, nothing serving). + // With every socket on the port set REUSEPORT, all of them coexist + // regardless of bind order, the kernel load-balances accepts across them, + // the shards accept their share directly, and this central listener accepts + // its share into conn_rx (also the fallback feeder for any shard that could + // not bind). `per_shard_accept` MUST stay false for that to hold: a + // bound-but-idle REUSEPORT socket black-holes the connections the kernel + // hashes to it at SYN time. Guarded by tests/multishard_serve_smoke.rs. + // On non-unix (no socket2 REUSEPORT here) fall back to the plain bind. + #[cfg(unix)] + let listener = match crate::shard::conn_accept::create_reuseport_socket(&addr) { + Ok(std_listener) => TcpListener::from_std(std_listener)?, + Err(e) => { + // e.g. `addr` is a hostname, not a parseable SocketAddr — keep the + // plain bind working (loses REUSEPORT coexistence in that case). + tracing::warn!( + "central SO_REUSEPORT bind failed ({}); falling back to plain bind on {}", + e, + addr + ); + TcpListener::bind(&addr).await? + } + }; + #[cfg(not(unix))] let listener = TcpListener::bind(&addr).await?; let num_shards = conn_txs.len(); info!("Listening on {} ({} shards)", addr, num_shards); diff --git a/src/shard/conn_accept.rs b/src/shard/conn_accept.rs index de77e1833..f3352a595 100644 --- a/src/shard/conn_accept.rs +++ b/src/shard/conn_accept.rs @@ -123,6 +123,14 @@ pub(crate) fn spawn_tokio_connection( shard_id: usize, num_shards: usize, config_port: u16, + // Disk-offload spill context (mirrors spawn_monoio_connection). The + // per-shard event loop owns the SpillThread; threading its sender + the + // shared file-id counter + the /shard-{id} dir into the ConnCtx + // lets handler_sharded spill evicted KVs to the cold tier instead of + // deleting them. All None/empty when disk-offload is disabled. + spill_sender: &Option>, + spill_file_id: &Rc>, + disk_offload_dir: &Option, ) { use crate::server::connection::handle_connection_sharded; use crate::server::connection::handle_connection_sharded_inner; @@ -198,9 +206,9 @@ pub(crate) fn spawn_tokio_connection( all_regs, all_rsm, aff, - None, // spill_sender (tokio handler doesn't use tiered storage) - Rc::new(std::cell::Cell::new(0)), // spill_file_id placeholder - None, // disk_offload_dir + spill_sender.clone(), + spill_file_id.clone(), + disk_offload_dir.clone(), ); if let (true, Some(tls_swap)) = (is_tls, tls_config.as_ref()) { diff --git a/src/shard/dispatch.rs b/src/shard/dispatch.rs index 3e2966576..1e72e6a75 100644 --- a/src/shard/dispatch.rs +++ b/src/shard/dispatch.rs @@ -592,6 +592,77 @@ const _: () = { ); }; +/// Backoff between cross-shard SPSC push retries when the target ring is full +/// (F3). Small so genuine transient backpressure adds minimal latency; runtime +/// timers may round it up to their tick granularity. +pub(crate) const CROSS_SHARD_PUSH_BACKOFF: std::time::Duration = + std::time::Duration::from_micros(100); + +/// Max push retries before giving up on a non-draining target ring (F3). With +/// the 100µs backoff the real budget is ~0.5s (timer honoured) to a few +/// seconds (coarse granularity) — wedged-shard detection scale, generous +/// enough not to false-reject a merely saturated shard under heavy load. +pub(crate) const CROSS_SHARD_PUSH_MAX_RETRIES: u32 = 5_000; + +/// Outcome of a bounded cross-shard SPSC push ([`push_with_backpressure`], F3). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum PushOutcome { + /// The target ring accepted the message. + Pushed, + /// The ring stayed full for the whole retry budget — the target shard is + /// not draining (wedged or saturated). The command was **never executed**, + /// so the caller returns a clean reject error to the client. + Backpressure, + /// Shutdown was requested mid-retry. The command was **never executed**. + Cancelled, +} + +/// Push a message onto a cross-shard SPSC ring with bounded backpressure +/// retry (F3 — design-for-failure). +/// +/// `try_push` returns `true` when the ring accepted the message and `false` +/// when the ring is full. On a `false`, the closure MUST retain ownership of +/// the message (stash it) so the next attempt can re-send it. We back off +/// `backoff` and retry up to `max_retries` times, checking `shutdown` before +/// every sleep so a graceful shutdown can never be blocked by a wedged peer. +/// +/// This replaces the previous unbounded `loop { try_push; sleep(10µs) }` in +/// the monoio cross-shard dispatch path. That loop parked a connection task +/// forever — holding its read/write buffers and piling wakers into the +/// `pending_wakers` relay — whenever a target shard stopped draining: the +/// multi-shard "zombie connection" RAM-growth vector from the investigation. +/// +/// **Hot path:** the happy case is a single `try_push()` returning `true` — +/// no sleep, no allocation. `TimerImpl::sleep` boxes a future, but only on +/// the contended retry path (ring already full), never on the fast path. +/// +/// **Wall-clock budget:** the total bound is roughly +/// `max_retries * max(backoff, timer_granularity)`. Callers pick +/// `max_retries` for the *intended* budget, not a literal microsecond count, +/// because runtime timers round small sleeps up to their tick granularity. +pub(crate) async fn push_with_backpressure( + shutdown: &crate::runtime::cancel::CancellationToken, + max_retries: u32, + backoff: std::time::Duration, + mut try_push: impl FnMut() -> bool, +) -> PushOutcome { + use crate::runtime::{TimerImpl, traits::RuntimeTimer}; + + if try_push() { + return PushOutcome::Pushed; + } + for _ in 0..max_retries { + if shutdown.is_cancelled() { + return PushOutcome::Cancelled; + } + TimerImpl::sleep(backoff).await; + if try_push() { + return PushOutcome::Pushed; + } + } + PushOutcome::Backpressure +} + #[cfg(test)] mod tests { use super::*; @@ -599,6 +670,93 @@ mod tests { #[cfg(feature = "runtime-tokio")] use std::sync::Arc; + // ── F3: bounded cross-shard push (push_with_backpressure) ── + // Tokio-gated: these drive the runtime timer via TimerImpl::sleep. + + #[cfg(feature = "runtime-tokio")] + #[tokio::test] + async fn push_backpressure_pushes_on_first_try_without_sleeping() { + let token = crate::runtime::cancel::CancellationToken::new(); + let mut calls = 0u32; + let start = std::time::Instant::now(); + let outcome = + push_with_backpressure(&token, 1000, std::time::Duration::from_millis(10), || { + calls += 1; + true // ring accepts immediately + }) + .await; + assert_eq!(outcome, PushOutcome::Pushed); + assert_eq!(calls, 1, "happy path must not retry"); + assert!( + start.elapsed() < std::time::Duration::from_millis(5), + "happy path must not sleep" + ); + } + + #[cfg(feature = "runtime-tokio")] + #[tokio::test] + async fn push_backpressure_succeeds_after_transient_full() { + let token = crate::runtime::cancel::CancellationToken::new(); + let mut calls = 0u32; + let outcome = + push_with_backpressure(&token, 1000, std::time::Duration::from_millis(1), || { + calls += 1; + calls >= 3 // full twice, then drains + }) + .await; + assert_eq!(outcome, PushOutcome::Pushed); + assert_eq!(calls, 3, "should retry until the ring drains"); + } + + #[cfg(feature = "runtime-tokio")] + #[tokio::test] + async fn push_backpressure_gives_up_when_ring_never_drains() { + // A wedged target shard: ring always full. The bounded retry MUST + // give up (Backpressure) rather than spin forever — the command was + // never accepted, so the caller returns a clean reject. + let token = crate::runtime::cancel::CancellationToken::new(); + let mut calls = 0u32; + let start = std::time::Instant::now(); + let outcome = + push_with_backpressure(&token, 5, std::time::Duration::from_millis(1), || { + calls += 1; + false // never accepts + }) + .await; + let elapsed = start.elapsed(); + assert_eq!(outcome, PushOutcome::Backpressure); + // initial attempt + 5 retries = 6 try_push calls. + assert_eq!(calls, 6, "initial try + max_retries attempts"); + assert!( + elapsed < std::time::Duration::from_secs(2), + "must give up within the budget, not hang (took {:?})", + elapsed + ); + } + + #[cfg(feature = "runtime-tokio")] + #[tokio::test] + async fn push_backpressure_aborts_on_shutdown() { + // Shutdown signalled before the call: a full ring must surface + // Cancelled immediately (checked before the first backoff sleep), + // never blocking a graceful shutdown on a wedged peer. + let token = crate::runtime::cancel::CancellationToken::new(); + token.cancel(); + let mut calls = 0u32; + let outcome = push_with_backpressure( + &token, + 1_000_000, // huge budget — only the shutdown check can break us out + std::time::Duration::from_millis(10), + || { + calls += 1; + false + }, + ) + .await; + assert_eq!(outcome, PushOutcome::Cancelled); + assert_eq!(calls, 1, "one attempt, then the shutdown check aborts"); + } + /// Diagnostic: print ShardMessage total size + representative payload sizes /// so we can see which variants cap the enum and target them for hot/cold split. #[test] diff --git a/src/shard/event_loop.rs b/src/shard/event_loop.rs index e2227897a..f2bffa41b 100644 --- a/src/shard/event_loop.rs +++ b/src/shard/event_loop.rs @@ -82,11 +82,22 @@ impl super::Shard { std::sync::atomic::Ordering::Relaxed, ); - // On Linux with tokio runtime, attempt to initialize io_uring for high-performance I/O. + // io_uring under the tokio runtime is an EXPERIMENTAL bridge (io_uring CQEs + // relayed into tokio via an eventfd). It is broken under sustained load: the + // driver floods `Unknown io_uring event type: 0` and then drops connections + // (BrokenPipe), taking multishard + disk-offload down with it. tokio is the + // PORTABILITY runtime — production io_uring lives in monoio (a separate path, + // unaffected by this gate). So default tokio to pure-tokio (epoll) I/O, which + // is stable under load. Opt back into the bridge with `MOON_URING=1` (for + // benchmarking / fixing it). `MOON_NO_URING` still force-disables and remains + // the CI default. Guarded by tests/multishard_serve_smoke + crash_recovery_*. #[cfg(all(target_os = "linux", feature = "runtime-tokio"))] let mut uring_state: Option = { - if std::env::var("MOON_NO_URING").is_ok() { - info!("Shard {} io_uring disabled via MOON_NO_URING", self.id); + if std::env::var("MOON_NO_URING").is_ok() || std::env::var("MOON_URING").is_err() { + info!( + "Shard {} io_uring disabled (tokio default; set MOON_URING=1 to opt in)", + self.id + ); None } else { match UringDriver::new(UringConfig { @@ -443,16 +454,37 @@ impl super::Shard { // Per-shard PageCache (None when disk-offload is disabled). // Manages 4KB + 64KB page frames with clock-sweep eviction. let page_cache: Option = if server_config.disk_offload_enabled() { - // Default: pagecache_size_bytes returns configured size or maxmemory/4. - // Split: 75% for 4KB frames, 25% for 64KB frames. - let budget = server_config.pagecache_size_bytes(server_config.maxmemory as u64); - let num_4k = ((budget * 3 / 4) / 4096) as usize; - let num_64k = ((budget / 4) / 65536) as usize; - let num_4k = num_4k.max(64); // minimum 64 frames - let num_64k = num_64k.max(8); // minimum 8 frames + use crate::persistence::page_cache::{ + pagecache_frame_counts, per_shard_pagecache_budget, + }; + // `pagecache_size_bytes` (explicit --pagecache-size, else 25% of + // maxmemory) is a WHOLE-INSTANCE intent, but each shard builds its + // own PageCache. Sizing every shard to the whole budget over-committed + // by num_shards× — the multishard "zombie eating RAM". Divide across + // shards so total pre-allocation is bounded by the budget. Buffers are + // also lazy now (grown on first use), so this is a ceiling, not RSS. + let whole_budget = + server_config.pagecache_size_bytes(server_config.maxmemory.unwrap_or(0) as u64); + let budget = per_shard_pagecache_budget(whole_budget, num_shards); + let (num_4k, num_64k) = pagecache_frame_counts(budget); + // Design-for-failure: an oversized explicit --pagecache-size lets the + // page cache crowd out the keyspace under pressure. Warn once. + if shard_id == 0 + && let Some(maxmem) = server_config.maxmemory + && maxmem > 0 + && whole_budget.saturating_mul(2) > maxmem as u64 + { + tracing::warn!( + "PageCache budget {} B is >50% of maxmemory {} B — under cache \ + pressure it competes with the keyspace; lower --pagecache-size", + whole_budget, + maxmem + ); + } info!( - "Shard {}: PageCache initialized ({} x 4KB + {} x 64KB frames, budget={})", - shard_id, num_4k, num_64k, budget + "Shard {}: PageCache initialized ({} x 4KB + {} x 64KB frames, \ + per-shard budget={} B of whole {} B across {} shard(s), lazy buffers)", + shard_id, num_4k, num_64k, budget, whole_budget, num_shards ); Some(PageCache::new(num_4k, num_64k)) } else { @@ -581,8 +613,30 @@ impl super::Shard { > = spill_thread.as_ref().map(|st| st.sender()); let spill_file_id: std::rc::Rc> = std::rc::Rc::new(std::cell::Cell::new(1)); - let mut next_file_id: u64 = 1; - let disk_offload_dir: Option = disk_offload_base.clone(); + // Per-shard spill directory for the write-path eviction (handler_monoio). + // MUST match the reader's `cold_shard_dir` (main.rs / shard::mod) and the + // persistence-tick cascade, which both use `/shard-{id}`. Using the + // bare base here wrote cold files to `/data` while reads looked in + // `/shard-{id}/data`, so spilled values were never read back. + let disk_offload_dir: Option = disk_offload_base + .clone() + .map(|base| base.join(format!("shard-{}", shard_id))); + + // B-2: resume the spill file_id counter ABOVE every recovered + // `heap-*.mpf`. Without this the counter restarts at 1 each boot and + // post-restart re-eviction overwrites cold files the rebuilt cold_index + // still points at, silently corrupting post-crash cold read-through. + // Fresh server / disk-offload off → seed 1 (unchanged from before). + let spill_seed = + crate::storage::eviction::next_spill_file_id_seed(disk_offload_dir.as_deref()); + spill_file_id.set(spill_seed); + let mut next_file_id: u64 = spill_seed; + if spill_seed > 1 { + info!( + "Shard {}: spill file_id counter seeded at {} from recovered cold files", + shard_id, spill_seed + ); + } // Per-shard warm-segment mmap budget enforcer. // Owned exclusively by this event-loop task; no locking needed. @@ -1053,6 +1107,7 @@ impl super::Shard { &cached_clock, &remote_sub_map_arc, &all_pubsub_registries, &all_remote_sub_maps, &affinity_tracker, shard_id, num_shards, config_port, + &spill_sender, &spill_file_id, &disk_offload_dir, ); } Err(e) => { @@ -1104,6 +1159,7 @@ impl super::Shard { &all_remote_sub_maps, &affinity_tracker, shard_id, num_shards, config_port, + &spill_sender, &spill_file_id, &disk_offload_dir, ); } Err(_) => { diff --git a/src/shard/persistence_tick.rs b/src/shard/persistence_tick.rs index f5c02de81..00441de61 100644 --- a/src/shard/persistence_tick.rs +++ b/src/shard/persistence_tick.rs @@ -374,8 +374,12 @@ pub(crate) fn drain_and_shutdown_spill( } } -/// For each successful completion: update manifest and ColdIndex. -/// Called on each eviction tick from the event loop. +/// For each successful completion: update manifest (ONE add_file+commit per +/// file) and ColdIndex (one insert per entry within that file). +/// +/// Under the batching model each `SpillCompletion` covers ONE DataFile that +/// may contain many KV entries. This makes manifest entries == #files, not +/// #keys, removing the ~70-entry inline-root cap. pub(crate) fn apply_spill_completions( spill_thread: &crate::storage::tiered::spill_thread::SpillThread, shard_manifest: &mut Option, @@ -390,45 +394,46 @@ pub(crate) fn apply_spill_completions( for c in completions { if !c.success { tracing::warn!( - key = %String::from_utf8_lossy(&c.key), - file_id = c.file_id, + file_id = c.file_entry.file_id, "Spill pwrite failed on background thread" ); continue; } - // Update manifest + let file_id = c.file_entry.file_id; + + // ONE manifest add_file + commit per flushed file. if let Some(ref mut manifest) = *shard_manifest { manifest.add_file(c.file_entry); if let Err(e) = manifest.commit() { - tracing::warn!(file_id = c.file_id, error = %e, "Manifest commit failed for spill completion"); + tracing::warn!( + file_id, + error = %e, + "Manifest commit failed for spill completion" + ); } } - // Update ColdIndex in the originating logical DB. - // Phase 2d: gate on is_initialized(); new path uses ShardSlice directly. - if crate::shard::slice::is_initialized() { - crate::shard::slice::with_shard_db(c.db_index, |db| { - if let Some(ref mut ci) = db.cold_index { - ci.insert( - c.key, - crate::storage::tiered::cold_index::ColdLocation { - file_id: c.file_id, - slot_idx: c.slot_idx, - }, - ); + // Insert one ColdIndex entry per KV within this file. + for entry in c.entries { + let location = crate::storage::tiered::cold_index::ColdLocation { + file_id, + page_idx: entry.page_idx, + slot_idx: entry.slot_idx, + }; + + // Phase 2d: gate on is_initialized(); new path uses ShardSlice directly. + if crate::shard::slice::is_initialized() { + crate::shard::slice::with_shard_db(entry.db_index, |db| { + if let Some(ref mut ci) = db.cold_index { + ci.insert(entry.key.clone(), location); + } + }); + } else { + let mut guard = shard_databases.write_db(shard_id, entry.db_index); + if let Some(ref mut ci) = guard.cold_index { + ci.insert(entry.key, location); } - }); - } else { - let mut guard = shard_databases.write_db(shard_id, c.db_index); - if let Some(ref mut ci) = guard.cold_index { - ci.insert( - c.key, - crate::storage::tiered::cold_index::ColdLocation { - file_id: c.file_id, - slot_idx: c.slot_idx, - }, - ); } } } @@ -452,7 +457,11 @@ pub(crate) fn should_run_pressure_cascade( if rt.maxmemory == 0 { return false; // No memory limit set -- no pressure possible } - let threshold = (rt.maxmemory as f64 * server_config.disk_offload_threshold) as usize; + // `used` is this shard's aggregate (across its DBs); compare against the + // PER-SHARD budget so the cascade fires at maxmemory/num_shards per shard, + // bounding aggregate RSS instead of the whole-instance cap per shard. + let threshold = + (rt.maxmemory_per_shard() as f64 * server_config.disk_offload_threshold) as usize; let used = shard_databases.aggregate_memory(shard_id); used > threshold } @@ -532,7 +541,9 @@ pub(crate) fn handle_memory_pressure( // Step 3: KV eviction -- run existing LRU/LFU eviction, with spill-to-disk // when disk-offload is enabled (evicted entries written to KvLeaf DataFiles). - // Use aggregate memory (server-wide) to match Redis maxmemory semantics. + // Compare this shard's aggregate (across its DBs) against the PER-SHARD + // budget (maxmemory/num_shards) so the summed eviction across shards bounds + // aggregate RSS at the whole-instance maxmemory. // // When a SpillThread is available, use the async path: entries are removed // from DashTable immediately (freeing RAM) and pwrite is deferred to the @@ -542,7 +553,7 @@ pub(crate) fn handle_memory_pressure( if rt.maxmemory > 0 { // Compute aggregate BEFORE acquiring write locks (same pattern as handler_sharded). let total_mem = shard_databases.aggregate_memory(shard_id); - if total_mem > rt.maxmemory { + if total_mem > rt.maxmemory_per_shard() { let db_count = shard_databases.db_count(); let shard_dir = server_config .effective_disk_offload_dir() diff --git a/src/shard/spsc_handler.rs b/src/shard/spsc_handler.rs index 57e2514ca..fec87d3be 100644 --- a/src/shard/spsc_handler.rs +++ b/src/shard/spsc_handler.rs @@ -3184,7 +3184,11 @@ mod wal_append_tests { // Build a pool backed by a real channel so we can observe what arrives. let (tx, rx) = mpsc_bounded::(16); - let pool = AofWriterPool::top_level_with_policy(tx, FsyncPolicy::EverySec); + let pool = AofWriterPool::top_level_with_policy( + tx, + FsyncPolicy::EverySec, + std::time::Duration::ZERO, + ); wal_append_and_fanout( b"world", @@ -3212,6 +3216,7 @@ mod wal_append_tests { AofMessage::AppendSync { .. } => panic!("expected Append, got AppendSync"), AofMessage::Rewrite(_) => panic!("expected Append, got Rewrite"), AofMessage::RewriteSharded(_) => panic!("expected Append, got RewriteSharded"), + AofMessage::RewritePerShard { .. } => panic!("expected Append, got RewritePerShard"), AofMessage::Shutdown => panic!("expected Append, got Shutdown"), } } @@ -3240,7 +3245,11 @@ mod wal_append_tests { // Build a 2-shard pool so per_shard_with_policy's debug_assert passes. let (tx0, rx0) = mpsc_bounded::(16); let (tx1, rx1) = mpsc_bounded::(16); - let pool = AofWriterPool::per_shard_with_policy(vec![tx0, tx1], FsyncPolicy::EverySec); + let pool = AofWriterPool::per_shard_with_policy( + vec![tx0, tx1], + FsyncPolicy::EverySec, + std::time::Duration::ZERO, + ); // ── PipelineBatch path: caller passes None ── // Pre-fix this was `aof_pool` (Some), which caused the double-write. diff --git a/src/shard/timers.rs b/src/shard/timers.rs index 01d7cf585..23c080a16 100644 --- a/src/shard/timers.rs +++ b/src/shard/timers.rs @@ -190,7 +190,16 @@ pub(crate) fn run_cold_orphan_sweep( }) .unwrap_or_default(); - if orphan_keys.is_empty() { + // Skip only when there is nothing to do: no hot-shadowed orphan keys AND + // no zero-ref files queued for unlink. Files orphaned by re-eviction + // (insert overwrite) or promotion (remove) carry no hot∩cold key, so the + // drain must still run for them even when `orphan_keys` is empty. + let has_pending_unlink = guard + .cold_index + .as_ref() + .map(|ci| ci.has_pending_unlink()) + .unwrap_or(false); + if orphan_keys.is_empty() && !has_pending_unlink { continue; } diff --git a/src/storage/eviction.rs b/src/storage/eviction.rs index c4b656f59..b5f1404e5 100644 --- a/src/storage/eviction.rs +++ b/src/storage/eviction.rs @@ -206,10 +206,13 @@ pub fn try_evict_if_needed_with_spill_and_total( let policy = EvictionPolicy::from_str(&config.maxmemory_policy); - // Check aggregate memory (server-wide maxmemory limit per Redis semantics). - // Evict from this DB until total memory drops below limit. + // Compare against the PER-SHARD budget, not the whole-instance maxmemory. + // `maxmemory` is a whole-instance cap; each shard enforces independently, so + // the threshold is `maxmemory / num_shards` (single shard ⇒ unchanged). This + // is what bounds aggregate RSS in multishard mode. + let budget = config.maxmemory_per_shard(); let mut current_total = total_memory; - while current_total > config.maxmemory { + while current_total > budget { if policy == EvictionPolicy::NoEviction { return Err(oom_error()); } @@ -234,6 +237,65 @@ pub fn try_evict_if_needed_with_spill_and_total( /// /// Callers must poll `SpillThread::drain_completions()` to apply manifest /// and ColdIndex updates from completed spills. +/// Seed value for a shard's spill `file_id` counter after recovery. +/// +/// On restart, AOF/RDB replay re-populates the hot tier and the persistence +/// cascade re-evicts the excess. If the counter restarted at 1 it would mint +/// `heap-000001.mpf`, `heap-000257.mpf`, … — the *same names* recovery just +/// loaded — and atomically overwrite cold files the rebuilt `cold_index` still +/// references, silently corrupting cold read-through (the B-2 bug). +/// +/// Scanning the on-disk `/data/heap-NNNNNN.mpf` files (rather than +/// the manifest) is deliberate: a spill that wrote its `.mpf` but crashed +/// before the manifest commit leaves an orphan the manifest doesn't know about, +/// yet it still occupies a filename that must not be clobbered. The physical +/// files are the authority on what can be overwritten. +/// +/// Returns `max(existing file_id) + 1`. Because each batch file is named after +/// its first request's `file_id` (gaps are harmless — recovery iterates +/// manifest entries, see `spill_thread`), any seed strictly above the current +/// max guarantees every future filename is new. Returns `1` when disk-offload +/// is off, the directory is absent, or it holds no heap files — identical to +/// the historical default, so a fresh server and the live hot path are +/// unchanged by this seeding. +#[must_use] +pub fn next_spill_file_id_seed(shard_dir: Option<&Path>) -> u64 { + let Some(dir) = shard_dir else { return 1 }; + let data_dir = dir.join("data"); + let entries = match std::fs::read_dir(&data_dir) { + Ok(e) => e, + Err(e) => { + // NotFound is the normal fresh-server case (data/ not created yet). + // Anything else (permissions, I/O) is anomalous: fail safe to the + // legacy default but surface it so a misseed can't hide silently. + if e.kind() != std::io::ErrorKind::NotFound { + tracing::warn!( + dir = %data_dir.display(), + error = %e, + "spill file_id seed: could not scan cold dir; defaulting to 1" + ); + } + return 1; + } + }; + let mut max_id: Option = None; + for entry in entries.flatten() { + let name = entry.file_name(); + if let Some(id) = name + .to_str() + .and_then(|n| n.strip_prefix("heap-")) + .and_then(|r| r.strip_suffix(".mpf")) + .and_then(|s| s.parse::().ok()) + { + max_id = Some(max_id.map_or(id, |m| m.max(id))); + } + } + match max_id { + Some(m) => m + 1, + None => 1, + } +} + pub fn try_evict_if_needed_async_spill( db: &mut Database, config: &RuntimeConfig, @@ -269,8 +331,10 @@ pub fn try_evict_if_needed_async_spill_with_total( let policy = EvictionPolicy::from_str(&config.maxmemory_policy); + // Per-shard budget (see `try_evict_if_needed_with_spill_and_total`). + let budget = config.maxmemory_per_shard(); let mut current_total = total_memory; - while current_total > config.maxmemory { + while current_total > budget { if policy == EvictionPolicy::NoEviction { return Err(oom_error()); } @@ -307,8 +371,10 @@ pub fn try_evict_deferred( return Ok(smallvec::SmallVec::new()); } + // Per-shard budget (see `try_evict_if_needed_with_spill_and_total`). + let budget = config.maxmemory_per_shard(); let total_memory = db.estimated_memory(); - if total_memory <= config.maxmemory { + if total_memory <= budget { return Ok(smallvec::SmallVec::new()); } @@ -316,7 +382,7 @@ pub fn try_evict_deferred( let mut evicted = smallvec::SmallVec::new(); let mut current_total = total_memory; - while current_total > config.maxmemory { + while current_total > budget { if policy == EvictionPolicy::NoEviction { return Err(oom_error()); } @@ -459,19 +525,13 @@ fn evict_one_async_spill( // produce a SpillCompletion that updates cold_index for this db_index. db.remove(key.as_bytes()); - // Insert a tentative cold_index entry so subsequent GETs in this DB - // can resolve the key while the bg pwrite is in flight. The completion - // handler in persistence_tick::apply_spill_completions will overwrite - // this with the authoritative ColdLocation once pwrite finishes. - if let Some(ref mut ci) = db.cold_index { - ci.insert( - Bytes::copy_from_slice(key.as_bytes()), - crate::storage::tiered::cold_index::ColdLocation { - file_id, - slot_idx: 0, - }, - ); - } + // NOTE: we do NOT insert a tentative cold_index entry here. + // Under batching the (page_idx, slot_idx) are unknown at evict time; + // slot 0 would return a *different* key's value in the pre-flush window. + // Accept a brief read-miss until the completion applies — the key is + // safe: it is in the SpillRequest and will be registered once the bg + // thread writes and the event loop processes the SpillCompletion. + // AOF incr log is the durability backstop for the pre-flush window. } else { // Entry disappeared (race with expiry), just remove db.remove(key.as_bytes()); @@ -695,9 +755,51 @@ mod tests { maxclients: 10000, timeout: 0, tcp_keepalive: 300, + num_shards: 1, } } + #[test] + fn per_shard_budget_divides_maxmemory_at_enforcement() { + // Regression for the multishard "zombie RAM" bug: maxmemory is a + // whole-instance cap, but each shard enforces eviction independently. + // Without per-shard division an N-shard server tolerates ~N×maxmemory. + let mut db = Database::new(); + db.set_string(Bytes::from_static(b"k1"), Bytes::from_static(b"v1")); + db.set_string(Bytes::from_static(b"k2"), Bytes::from_static(b"v2")); + db.set_string(Bytes::from_static(b"k3"), Bytes::from_static(b"v3")); + db.set_string(Bytes::from_static(b"k4"), Bytes::from_static(b"v4")); + let total = db.estimated_memory(); + assert!(total > 0); + + // Whole-instance cap == current memory, single shard: per-shard budget + // equals `total`, so nothing is evicted. + let mut cfg1 = make_config(total, "allkeys-lru"); + cfg1.num_shards = 1; + assert!(try_evict_if_needed(&mut db, &cfg1).is_ok()); + assert_eq!( + db.len(), + 4, + "single shard: nothing evicted at budget == total" + ); + + // SAME whole-instance cap, 4 shards: per-shard budget = ceil(total/4) << + // total, so this shard MUST shed keys until it is under the per-shard + // budget. Pre-fix (comparison vs raw maxmemory) this evicted nothing. + let mut cfg4 = make_config(total, "allkeys-lru"); + cfg4.num_shards = 4; + assert!(try_evict_if_needed(&mut db, &cfg4).is_ok()); + assert!( + db.len() < 4, + "4 shards: per-shard budget must force eviction (got len {})", + db.len() + ); + assert!( + db.estimated_memory() <= cfg4.maxmemory_per_shard(), + "evicted below the per-shard budget" + ); + } + #[test] fn test_eviction_policy_from_str() { assert_eq!( @@ -957,4 +1059,45 @@ mod tests { assert!(result.is_ok()); assert_eq!(db.len(), 0); } + + // ── B-2: post-crash cold-spill file_id seeding ────────────────────────── + // The spill file_id counter must resume ABOVE every recovered `heap-*.mpf` + // so post-restart re-eviction never overwrites a cold file the rebuilt + // cold_index still points at (which silently corrupts cold read-through). + + /// `None` dir (disk-offload off) → seed 1 (historical default, no change). + #[test] + fn test_spill_seed_none_dir_is_one() { + assert_eq!(next_spill_file_id_seed(None), 1); + } + + /// Fresh server (dir absent / empty) → seed 1 == legacy behaviour, so the + /// live read-through hot path cannot regress from this change. + #[test] + fn test_spill_seed_fresh_server_is_one() { + let tmp = tempfile::tempdir().unwrap(); + // data/ does not exist yet + assert_eq!(next_spill_file_id_seed(Some(tmp.path())), 1); + // data/ exists but empty + std::fs::create_dir_all(tmp.path().join("data")).unwrap(); + assert_eq!(next_spill_file_id_seed(Some(tmp.path())), 1); + } + + /// With recovered heap files, seed = max(file_id) + 1 so the next batch's + /// filename is strictly greater than any existing one (filenames are the + /// first request id of each batch; gaps are harmless). + #[test] + fn test_spill_seed_resumes_above_max_recovered() { + let tmp = tempfile::tempdir().unwrap(); + let data = tmp.path().join("data"); + std::fs::create_dir_all(&data).unwrap(); + for id in [1u64, 257, 513] { + std::fs::write(data.join(format!("heap-{id:06}.mpf")), b"x").unwrap(); + } + // non-heap files and bad names must be ignored + std::fs::write(data.join("manifest.bin"), b"x").unwrap(); + std::fs::write(data.join("heap-notanum.mpf"), b"x").unwrap(); + std::fs::write(data.join("base-000999.rdb"), b"x").unwrap(); + assert_eq!(next_spill_file_id_seed(Some(tmp.path())), 514); + } } diff --git a/src/storage/tiered/cold_index.rs b/src/storage/tiered/cold_index.rs index aff8e7621..d4ad71e8e 100644 --- a/src/storage/tiered/cold_index.rs +++ b/src/storage/tiered/cold_index.rs @@ -21,11 +21,18 @@ pub struct SweepStats { } /// Location of a cold KV entry on disk. +/// +/// Multi-page spill files store many KV entries across several KvLeafPages. +/// `page_idx` is the FILE-ABSOLUTE 4 KB chunk index (0-based); `slot_idx` +/// is the slot within that page. For the legacy single-page path, both are +/// always 0. #[derive(Debug, Clone, Copy)] pub struct ColdLocation { /// Manifest file_id of the heap DataFile. pub file_id: u64, - /// Slot index within the KvLeafPage (currently single-page files). + /// File-absolute 4KB page index within the DataFile (0 = first page). + pub page_idx: u32, + /// Slot index within the KvLeafPage at `page_idx`. pub slot_idx: u16, } @@ -36,23 +43,83 @@ pub struct ColdLocation { #[derive(Debug)] pub struct ColdIndex { map: HashMap, + /// Reverse liveness: `file_id` -> count of live `map` entries pointing at it. + /// + /// A batched spill file (`heap-NNNNNN.mpf`) holds up to `FLUSH_ENTRY_CAP` + /// (256) KVs across its pages, so it is safe to unlink ONLY when this count + /// reaches 0 (no co-located live key still references it). Deleting a file + /// on a single orphan key silently orphans its co-located keys — observed + /// empirically as cold read-through collapsing 200/200 -> 88/200 once the + /// orphan sweep runs. + file_refs: HashMap, + /// `file_id`s that dropped to zero live refs (via an `insert` overwrite or a + /// `remove`) and are awaiting unlink. Drained off the hot path by the orphan + /// sweep ([`Self::drain_pending_unlink`]). Pushed only on a zero-ref + /// transition (rare), so it does not allocate on the common insert path. + pending_unlink: Vec, } impl ColdIndex { pub fn new() -> Self { Self { map: HashMap::new(), + file_refs: HashMap::new(), + pending_unlink: Vec::new(), + } + } + + /// Increment a file's live-ref count. + #[inline] + fn ref_inc(&mut self, file_id: u64) { + *self.file_refs.entry(file_id).or_insert(0) += 1; + } + + /// Decrement a file's live-ref count. Returns `true` when it reaches zero — + /// the file no longer backs any live cold entry and may be unlinked. + #[inline] + fn ref_dec(&mut self, file_id: u64) -> bool { + if let Some(c) = self.file_refs.get_mut(&file_id) { + *c = c.saturating_sub(1); + if *c == 0 { + self.file_refs.remove(&file_id); + return true; + } } + false } /// Record a spilled key's disk location. + /// + /// Maintains the reverse [`Self::file_refs`] liveness index. When this call + /// overwrites an existing entry whose key moves to a *different* file (the + /// re-eviction case), the old file loses its referrer; if that was its last + /// referrer the old `file_id` is queued for unlink — the hot∩cold sweep can + /// never see such a file because no key references it anymore. pub fn insert(&mut self, key: Bytes, location: ColdLocation) { - self.map.insert(key, location); + let new_file = location.file_id; + if let Some(old) = self.map.insert(key, location) { + if old.file_id != new_file { + if self.ref_dec(old.file_id) { + self.pending_unlink.push(old.file_id); + } + self.ref_inc(new_file); + } + // Same file_id (different slot/page): live-ref count is unchanged. + } else { + self.ref_inc(new_file); + } } /// Remove a key from the cold index (e.g., when promoted back to RAM). + /// + /// Decrements the backing file's live-ref count; if this removes the file's + /// last referrer, the `file_id` is queued for unlink by the next sweep. pub fn remove(&mut self, key: &[u8]) { - self.map.remove(key); + if let Some(old) = self.map.remove(key) { + if self.ref_dec(old.file_id) { + self.pending_unlink.push(old.file_id); + } + } } /// Look up a key's cold location. @@ -61,8 +128,14 @@ impl ColdIndex { } /// Merge another ColdIndex into this one (used during recovery). + /// + /// Routes through [`Self::insert`] so the reverse [`Self::file_refs`] + /// liveness index is rebuilt for the merged entries (a raw `map.extend` + /// would leave the ref counts inconsistent and break safe reclamation). pub fn merge(&mut self, other: ColdIndex) { - self.map.extend(other.map); + for (key, location) in other.map { + self.insert(key, location); + } } /// Number of entries tracked. @@ -70,6 +143,16 @@ impl ColdIndex { self.map.len() } + /// Whether any zero-ref files are queued for unlink by the next sweep. + /// + /// Files orphaned by `insert` overwrite (re-eviction) or `remove` + /// (promotion) carry no hot∩cold key, so the sweep trigger must consult + /// this in addition to the orphan-key set — otherwise those files are never + /// reclaimed on ticks where no key is hot-shadowed. + pub fn has_pending_unlink(&self) -> bool { + !self.pending_unlink.is_empty() + } + /// Iterate over all cold entries as `(key, location)` pairs. /// /// Used by the orphan sweeper to walk all entries without taking ownership. @@ -184,20 +267,69 @@ impl ColdIndex { &mut self, orphan_keys: Vec, shard_dir: &Path, - mut manifest: Option<&mut crate::persistence::manifest::ShardManifest>, + manifest: Option<&mut crate::persistence::manifest::ShardManifest>, ) -> std::io::Result { let mut stats = SweepStats::default(); - let data_dir = shard_dir.join("data"); + // Phase 1: remove orphan ENTRIES and decrement their files' ref counts. + // A file becomes an unlink candidate only when its LAST live ref is + // removed — NEVER on a single orphan key while co-located keys in the + // same batched `.mpf` still reference it (the data-loss bug this fixes). for key in &orphan_keys { - let loc = match self.map.get(key.as_ref()) { - Some(l) => *l, - None => continue, // already removed by a concurrent path - }; + if let Some(old) = self.map.remove(key.as_ref()) { + stats.entries_reclaimed += 1; + if self.ref_dec(old.file_id) { + self.pending_unlink.push(old.file_id); + } + } + } - let file_path = data_dir.join(format!("heap-{:06}.mpf", loc.file_id)); + // Phase 2: unlink files that now have zero live refs (off the hot path). + stats.bytes_reclaimed = self.drain_pending_unlink(shard_dir, manifest)?; - // Determine file size before deletion for byte accounting. + if stats.entries_reclaimed > 0 || stats.bytes_reclaimed > 0 { + tracing::info!( + entries = stats.entries_reclaimed, + bytes = stats.bytes_reclaimed, + "orphan_sweep: completed", + ); + } + + Ok(stats) + } + + /// Unlink every `pending_unlink` file that still has zero live refs, + /// tombstone its manifest entry, and commit once. Returns bytes reclaimed. + /// + /// A file is queued only on a zero-ref transition, but a later `insert` + /// could re-reference the same `file_id` before the drain runs (file ids are + /// minted monotonically per process, so this is defensive); such files are + /// skipped. The `.mpf` is unlinked BEFORE the manifest commit, so disk is + /// freed even when the commit fails (e.g. manifest root overflow at >70 + /// entries) — the commit error is surfaced only after best-effort + /// reclamation, and a file whose unlink itself errors is re-queued for a + /// later sweep rather than leaked. + fn drain_pending_unlink( + &mut self, + shard_dir: &Path, + mut manifest: Option<&mut crate::persistence::manifest::ShardManifest>, + ) -> std::io::Result { + if self.pending_unlink.is_empty() { + return Ok(0); + } + let data_dir = shard_dir.join("data"); + let mut queued = std::mem::take(&mut self.pending_unlink); + queued.sort_unstable(); + queued.dedup(); + + let mut bytes_reclaimed: u64 = 0; + let mut manifest_dirty = false; + for file_id in queued { + // A live ref re-appeared after queueing -> keep the file. + if self.file_refs.contains_key(&file_id) { + continue; + } + let file_path = data_dir.join(format!("heap-{:06}.mpf", file_id)); let file_bytes = std::fs::metadata(&file_path).map(|m| m.len()).unwrap_or(0); // Delete the DataFile (idempotent — missing = already gone). @@ -206,52 +338,43 @@ impl ColdIndex { Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} Err(e) => { tracing::warn!( - key = %String::from_utf8_lossy(key), file = %file_path.display(), err = %e, - "orphan_sweep: failed to delete cold DataFile — skipping", + "orphan_sweep: failed to delete cold DataFile — will retry", ); + // Re-queue rather than leak; a later sweep retries. + self.pending_unlink.push(file_id); continue; } } // Tombstone manifest entry so GC / recovery doesn't re-index it. if let Some(ref mut m) = manifest.as_deref_mut() { - m.remove_file(loc.file_id); + m.remove_file(file_id); + manifest_dirty = true; } - // Remove from in-memory cold index. - self.map.remove(key.as_ref()); - - stats.entries_reclaimed += 1; - stats.bytes_reclaimed = stats.bytes_reclaimed.saturating_add(file_bytes); - + bytes_reclaimed = bytes_reclaimed.saturating_add(file_bytes); crate::command::info_reclamation::record_cold_orphan_reclaim(file_bytes); tracing::debug!( - key = %String::from_utf8_lossy(key), - file_id = loc.file_id, + file_id, bytes = file_bytes, - "orphan_sweep: reclaimed cold-tier orphan", + "orphan_sweep: reclaimed zero-ref cold DataFile", ); } - // Single manifest commit for all tombstones in this batch. - if stats.entries_reclaimed > 0 { + // Single manifest commit for all tombstones in this drain. + if manifest_dirty { if let Some(m) = manifest { if let Err(e) = m.commit() { tracing::error!(err = %e, "orphan_sweep: manifest commit failed"); return Err(e); } } - tracing::info!( - entries = stats.entries_reclaimed, - bytes = stats.bytes_reclaimed, - "orphan_sweep: completed", - ); } - Ok(stats) + Ok(bytes_reclaimed) } /// Rebuild the cold index from all heap DataFiles in a shard directory. @@ -263,7 +386,7 @@ impl ColdIndex { manifest: &crate::persistence::manifest::ShardManifest, ) -> Self { use crate::persistence::manifest::FileStatus; - use crate::persistence::page::PageType; + use crate::persistence::page::{PAGE_4K, PageType}; let mut index = Self::new(); let data_dir = shard_dir.join("data"); @@ -271,14 +394,25 @@ impl ColdIndex { for entry in manifest.files() { if entry.status == FileStatus::Active && entry.file_type == PageType::KvLeaf as u8 { let heap_path = data_dir.join(format!("heap-{:06}.mpf", entry.file_id)); - if let Ok(pages) = crate::persistence::kv_page::read_datafile(&heap_path) { - for page in &pages { + // Read raw bytes and iterate by absolute chunk index. + // `read_datafile` skips overflow pages (returns only KvLeaf pages), + // so its enumerate index ≠ file-absolute page index in multi-page files. + // We must use the raw chunk index to produce a correct `page_idx`. + let raw = match std::fs::read(&heap_path) { + Ok(b) => b, + Err(_) => continue, + }; + for (page_idx, chunk) in raw.chunks_exact(PAGE_4K).enumerate() { + let mut buf = [0u8; PAGE_4K]; + buf.copy_from_slice(chunk); + if let Some(page) = crate::persistence::kv_page::KvLeafPage::from_bytes(buf) { for slot_idx in 0..page.slot_count() { if let Some(kv) = page.get(slot_idx) { index.insert( Bytes::from(kv.key), ColdLocation { file_id: entry.file_id, + page_idx: page_idx as u32, slot_idx, }, ); @@ -301,14 +435,164 @@ mod tests { let mut idx = ColdIndex::new(); let loc = ColdLocation { file_id: 1, + page_idx: 0, slot_idx: 0, }; idx.insert(Bytes::from_static(b"key1"), loc); assert_eq!(idx.len(), 1); let found = idx.lookup(b"key1").unwrap(); assert_eq!(found.file_id, 1); + assert_eq!(found.page_idx, 0); assert_eq!(found.slot_idx, 0); idx.remove(b"key1"); assert!(idx.lookup(b"key1").is_none()); } + + /// Create a shard dir with a `data/` subdir and a dummy heap-NNNNNN.mpf + /// file standing in for a batched multi-KV spill file. + fn make_shard_with_heap(file_ids: &[u64]) -> tempfile::TempDir { + let tmp = tempfile::tempdir().unwrap(); + let data_dir = tmp.path().join("data"); + std::fs::create_dir_all(&data_dir).unwrap(); + for &fid in file_ids { + let p = data_dir.join(format!("heap-{:06}.mpf", fid)); + std::fs::write(&p, vec![0xABu8; 4096]).unwrap(); + } + tmp + } + + fn heap_path(shard_dir: &Path, file_id: u64) -> std::path::PathBuf { + shard_dir + .join("data") + .join(format!("heap-{:06}.mpf", file_id)) + } + + /// REGRESSION (batch-file shared-deletion data loss): sweeping ONE orphan + /// key must NOT delete its `.mpf` while a co-located live key still + /// references the same file. Under spill batching a single file holds up + /// to 256 KVs; deleting it on one orphan key silently orphans the rest + /// (observed empirically as cold read-through 200/200 -> 88/200). + #[test] + fn test_sweep_retains_file_with_colocated_live_key() { + let tmp = make_shard_with_heap(&[5]); + let shard_dir = tmp.path(); + + let mut ci = ColdIndex::new(); + // Two keys co-located in the SAME batched file (file_id = 5). + ci.insert( + Bytes::from_static(b"k_orphan"), + ColdLocation { + file_id: 5, + page_idx: 0, + slot_idx: 0, + }, + ); + ci.insert( + Bytes::from_static(b"k_live"), + ColdLocation { + file_id: 5, + page_idx: 0, + slot_idx: 1, + }, + ); + + // Sweep ONLY the orphan key. + ci.sweep_known_orphans(vec![Bytes::from_static(b"k_orphan")], shard_dir, None) + .unwrap(); + + // The co-located live key must remain resolvable AND its file present. + assert!( + ci.lookup(b"k_live").is_some(), + "co-located live key dropped from cold index", + ); + assert!( + heap_path(shard_dir, 5).exists(), + "DATA LOSS: file holding a live co-located key was deleted", + ); + // The orphan entry itself is gone. + assert!(ci.lookup(b"k_orphan").is_none(), "orphan entry not removed"); + } + + /// A batched file is unlinked only once its LAST live ref is removed. + #[test] + fn test_sweep_deletes_file_only_when_last_ref_removed() { + let tmp = make_shard_with_heap(&[7]); + let shard_dir = tmp.path(); + + let mut ci = ColdIndex::new(); + ci.insert( + Bytes::from_static(b"k1"), + ColdLocation { + file_id: 7, + page_idx: 0, + slot_idx: 0, + }, + ); + ci.insert( + Bytes::from_static(b"k2"), + ColdLocation { + file_id: 7, + page_idx: 0, + slot_idx: 1, + }, + ); + + // Sweep k1: one ref remains (k2) -> file MUST survive. + ci.sweep_known_orphans(vec![Bytes::from_static(b"k1")], shard_dir, None) + .unwrap(); + assert!( + heap_path(shard_dir, 7).exists(), + "file deleted while k2 still references it", + ); + + // Sweep k2: last ref removed -> file now reclaimed. + ci.sweep_known_orphans(vec![Bytes::from_static(b"k2")], shard_dir, None) + .unwrap(); + assert!( + !heap_path(shard_dir, 7).exists(), + "file not reclaimed after its last ref was swept", + ); + } + + /// Re-eviction churn: `insert` overwriting a key's location (old file_id -> + /// new file_id) drops the old file to zero refs. The hot∩cold sweep can + /// never see this file (no key references it anymore), so the index must + /// enqueue it for reclamation and a subsequent sweep must unlink it. + #[test] + fn test_overwrite_reclaims_orphaned_old_file() { + let tmp = make_shard_with_heap(&[10, 11]); + let shard_dir = tmp.path(); + + let mut ci = ColdIndex::new(); + ci.insert( + Bytes::from_static(b"k"), + ColdLocation { + file_id: 10, + page_idx: 0, + slot_idx: 0, + }, + ); + // Key re-spilled to a NEW file (re-eviction) -> file 10 orphaned. + ci.insert( + Bytes::from_static(b"k"), + ColdLocation { + file_id: 11, + page_idx: 0, + slot_idx: 0, + }, + ); + + // A sweep with NO orphan keys must still drain the pending unlink. + ci.sweep_known_orphans(vec![], shard_dir, None).unwrap(); + + assert!( + !heap_path(shard_dir, 10).exists(), + "orphaned old file (10) not reclaimed after overwrite", + ); + assert!( + heap_path(shard_dir, 11).exists(), + "live file (11) wrongly deleted", + ); + assert_eq!(ci.lookup(b"k").map(|l| l.file_id), Some(11)); + } } diff --git a/src/storage/tiered/cold_read.rs b/src/storage/tiered/cold_read.rs index 9ad5a621d..dae993415 100644 --- a/src/storage/tiered/cold_read.rs +++ b/src/storage/tiered/cold_read.rs @@ -45,19 +45,19 @@ fn read_cold_entry( location: ColdLocation, now_ms: u64, ) -> Option<(RedisValue, Option)> { + use std::os::unix::fs::FileExt as _; + let file_path = shard_dir .join("data") .join(format!("heap-{:06}.mpf", location.file_id)); - // Read the full file (needed for potential overflow chain reads) - let file_data = std::fs::read(&file_path).ok()?; - if file_data.len() < PAGE_4K { - return None; - } + let file = std::fs::File::open(&file_path).ok()?; - // Parse the KvLeaf page (page 0) + // Read only the specific 4KB page identified by page_idx (pread, no whole-file read). + let page_offset = (location.page_idx as u64) * (PAGE_4K as u64); let mut leaf_buf = [0u8; PAGE_4K]; - leaf_buf.copy_from_slice(&file_data[..PAGE_4K]); + file.read_exact_at(&mut leaf_buf, page_offset).ok()?; + let page = crate::persistence::kv_page::KvLeafPage::from_bytes(leaf_buf)?; let entry = page.get(location.slot_idx)?; @@ -68,13 +68,16 @@ fn read_cold_entry( } } - // Resolve value bytes: handle overflow chain if flagged + // Resolve value bytes: handle overflow chain if flagged. + // For overflow we need the full file to traverse the chain. let value_bytes = if entry.flags & entry_flags::OVERFLOW != 0 { // Overflow pointer: start_page_idx as u32 LE if entry.value.len() < 4 { return None; } let start_page_idx = u32::from_le_bytes(entry.value[..4].try_into().ok()?) as usize; + // Only read the full file when following an overflow chain. + let file_data = std::fs::read(&file_path).ok()?; read_overflow_chain(&file_data, start_page_idx)? } else { entry.value diff --git a/src/storage/tiered/kv_spill.rs b/src/storage/tiered/kv_spill.rs index 954a3462a..e7093044b 100644 --- a/src/storage/tiered/kv_spill.rs +++ b/src/storage/tiered/kv_spill.rs @@ -193,6 +193,7 @@ pub fn spill_to_datafile( Bytes::copy_from_slice(key), super::cold_index::ColdLocation { file_id, + page_idx: 0, slot_idx: 0, }, ); @@ -201,6 +202,163 @@ pub fn spill_to_datafile( Ok(()) } +// ── Multi-page batch spill ─────────────────────────────────────────────────── + +/// Maximum raw value size (in bytes) for an entry to be eligible for inline +/// batching. Entries whose serialized value exceeds this threshold are spilled +/// via the existing single-file path (`build_kv_spill_pages`) which handles +/// overflow chains correctly. +/// +/// The 4KB leaf page has ~3916B of usable payload after all headers + one slot +/// (PAGE_4K=4096 − MoonPage header 64B − KV header 16B − slot 4B). After LZ4 +/// compression (minimum 256B value → may not shrink) plus key overhead, the +/// safe inline threshold is 3500B. Using the raw value length is conservative +/// but correct: the caller pre-screens before building the entry, so the batch +/// builder never sees truly oversized values. +pub const INLINE_MAX_VALUE_BYTES: usize = 3500; + +/// One entry to include in a spill batch. +pub struct SpillEntry { + pub key: bytes::Bytes, + pub value_bytes: bytes::Bytes, + pub value_type: ValueType, + pub flags: u8, + pub ttl_ms: Option, +} + +/// Result of building a multi-page inline spill batch. +/// +/// `leaves` contains all KvLeafPages (in file order). `locations[i]` is the +/// `(page_idx, slot_idx)` for `entries[i]` — page_idx is the FILE-ABSOLUTE +/// 4KB chunk index. +/// +/// Inline-only MVP: the caller MUST pre-screen entries so that no entry's +/// `value_bytes.len()` exceeds `INLINE_MAX_VALUE_BYTES`. Entries larger than +/// that threshold are routed to `build_kv_spill_pages` (single-file path) by +/// the caller (`flush_buffer` in `spill_thread.rs`). Mixing overflow and +/// inline entries in a single file is deferred to a future phase. +pub struct BatchPages { + pub leaves: Vec, + pub overflow: Vec, + /// Parallel to the *accepted* entries slice: (file-absolute page_idx, slot_idx). + pub locations: Vec<(u32, u16)>, +} + +/// Build a multi-page inline spill batch from a slice of entries. +/// +/// Greedily packs entries into KvLeafPages using `KvLeafPage::insert`. When +/// a page fills up (`Err(PageFull)`) the current leaf is finalized and a fresh +/// leaf is started. +/// +/// **Inline-only** — the caller (`flush_buffer`) MUST pre-screen so that no +/// entry exceeds `INLINE_MAX_VALUE_BYTES`. If an entry still does not fit on a +/// fresh leaf (e.g. because post-LZ4 compression it is still too large), the +/// function returns `Err(io::ErrorKind::InvalidData)`. The caller must catch +/// that and fall back to the single-file path for that entry. +/// +/// The `overflow` field of `BatchPages` is always empty from this function. +/// It exists on the struct for forward-compatibility. +pub fn build_kv_spill_batch(entries: &[SpillEntry], file_id: u64) -> io::Result { + let mut leaves: Vec = Vec::new(); + let mut locations: Vec<(u32, u16)> = Vec::with_capacity(entries.len()); + + // Start with page 0. + let mut current_leaf = KvLeafPage::new(0, file_id); + let mut current_page_idx: u32 = 0; + + for entry in entries { + // Try inserting directly into the current leaf. + match current_leaf.insert( + &entry.key, + &entry.value_bytes, + entry.value_type, + entry.flags, + entry.ttl_ms, + ) { + Ok(slot_idx) => { + locations.push((current_page_idx, slot_idx)); + } + Err(PageFull) => { + // Current leaf is full. Finalize it and start a new one, + // then retry the insert on the fresh leaf. + current_leaf.finalize(); + leaves.push(current_leaf); + current_page_idx += 1; + current_leaf = KvLeafPage::new(current_page_idx as u64, file_id); + + match current_leaf.insert( + &entry.key, + &entry.value_bytes, + entry.value_type, + entry.flags, + entry.ttl_ms, + ) { + Ok(slot_idx) => { + locations.push((current_page_idx, slot_idx)); + } + Err(PageFull) => { + // Value is too large for a fresh leaf even after LZ4. + // The caller should have pre-screened using + // INLINE_MAX_VALUE_BYTES; this is a defensive fallback. + warn!( + key_len = entry.key.len(), + value_len = entry.value_bytes.len(), + "kv_spill batch: entry too large for inline leaf, skipping" + ); + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "entry too large for inline leaf page", + )); + } + } + } + } + } + + // Finalize and push the last (possibly only) leaf. + current_leaf.finalize(); + leaves.push(current_leaf); + + Ok(BatchPages { + leaves, + overflow: Vec::new(), + locations, + }) +} + +/// Write a `BatchPages` to `{shard_dir}/data/heap-{file_id:06}.mpf` atomically. +/// +/// Layout: all leaf pages first (file offsets 0..L*4KB), then overflow pages +/// (offsets L*4KB..). Writes to a `.tmp` file, fsyncs, then renames — so a +/// crash during write leaves no partial file visible. +/// +/// Returns the total byte size written. +pub fn write_kv_spill_batch(shard_dir: &Path, file_id: u64, batch: &BatchPages) -> io::Result { + use std::io::Write as _; + + let data_dir = shard_dir.join("data"); + std::fs::create_dir_all(&data_dir)?; + + let final_path = data_dir.join(format!("heap-{file_id:06}.mpf")); + let tmp_path = data_dir.join(format!("heap-{file_id:06}.tmp")); + + { + let mut file = std::fs::File::create(&tmp_path)?; + for leaf in &batch.leaves { + file.write_all(leaf.as_bytes())?; + } + for ov in &batch.overflow { + file.write_all(ov.as_bytes())?; + } + file.sync_all()?; + } + + std::fs::rename(&tmp_path, &final_path)?; + + let total_pages = (batch.leaves.len() + batch.overflow.len()) as u64; + Ok(total_pages * PAGE_4K as u64) +} + #[cfg(test)] mod tests { use super::*; @@ -449,4 +607,252 @@ mod tests { _ => panic!("expected String"), } } + + // ── Multi-page batch tests (TDD: written before implementation) ────────── + + /// Helper: build N entries with distinct keys/values small enough to fit + /// inline (≤200 bytes each), forcing page overflow by sheer count. + fn make_inline_entries(n: usize) -> Vec { + (0..n) + .map(|i| SpillEntry { + key: bytes::Bytes::from(format!("batch_key_{i:04}")), + value_bytes: bytes::Bytes::from(format!( + "batch_value_{i:04}_padding_to_200_bytes_{:0>150}", + i + )), + value_type: ValueType::String, + flags: 0, + ttl_ms: None, + }) + .collect() + } + + /// A KvLeafPage holds roughly 7-10 entries of ~200 B each. Generating 50 + /// entries guarantees ≥ 2 leaf pages. + #[test] + fn test_build_kv_spill_batch_multi_page() { + const N: usize = 50; + let entries = make_inline_entries(N); + let file_id = 42u64; + + let batch = + build_kv_spill_batch(&entries, file_id).expect("build_kv_spill_batch should succeed"); + + // Must have spanned at least 2 leaf pages. + assert!( + batch.leaves.len() >= 2, + "expected ≥2 leaf pages, got {}", + batch.leaves.len() + ); + // One location per entry. + assert_eq!(batch.locations.len(), N); + + // page_idx values must be monotonically non-decreasing and within range. + let max_page = batch.leaves.len() as u32 - 1; + for (i, &(page_idx, slot_idx)) in batch.locations.iter().enumerate() { + assert!( + page_idx <= max_page, + "entry {i}: page_idx {page_idx} out of range (max {max_page})" + ); + let _ = slot_idx; // just assert no panic + } + } + + /// write_kv_spill_batch must produce an atomic file and + /// read_cold_entry_at must recover every entry by (page_idx, slot_idx). + #[test] + fn test_write_and_read_batch_multi_page() { + use crate::storage::tiered::cold_index::ColdLocation; + use crate::storage::tiered::cold_read::read_cold_entry_at; + + const N: usize = 50; + let entries = make_inline_entries(N); + let file_id = 77u64; + + let tmp = tempfile::tempdir().unwrap(); + let shard_dir = tmp.path(); + + let batch = build_kv_spill_batch(&entries, file_id).unwrap(); + assert!(batch.leaves.len() >= 2, "test requires ≥2 leaf pages"); + + let byte_size = write_kv_spill_batch(shard_dir, file_id, &batch).unwrap(); + assert!(byte_size > 0); + + // The file must exist at the canonical path (not the .tmp). + let file_path = shard_dir + .join("data") + .join(format!("heap-{file_id:06}.mpf")); + assert!( + file_path.exists(), + "batch file should exist at canonical path" + ); + let tmp_path = shard_dir + .join("data") + .join(format!("heap-{file_id:06}.tmp")); + assert!(!tmp_path.exists(), ".tmp file should be renamed away"); + + // Round-trip: every entry must be readable by its location. + for (i, (&(page_idx, slot_idx), entry)) in + batch.locations.iter().zip(entries.iter()).enumerate() + { + let loc = ColdLocation { + file_id, + page_idx, + slot_idx, + }; + let result = read_cold_entry_at(shard_dir, loc, 0); + assert!( + result.is_some(), + "entry {i} (key={}) not readable at page_idx={page_idx} slot_idx={slot_idx}", + String::from_utf8_lossy(&entry.key) + ); + let (value, _ttl) = result.unwrap(); + match value { + crate::storage::entry::RedisValue::String(data) => { + assert_eq!( + data.as_ref(), + entry.value_bytes.as_ref(), + "entry {i}: value mismatch" + ); + } + _ => panic!("entry {i}: expected String"), + } + } + } + + /// An entry at page_idx=3 slot=2 (deep in the batch) resolves correctly. + #[test] + fn test_batch_deep_page_slot_resolves() { + use crate::storage::tiered::cold_index::ColdLocation; + use crate::storage::tiered::cold_read::read_cold_entry_at; + + // Generate enough entries to reach page_idx ≥ 3. + const N: usize = 100; + let entries = make_inline_entries(N); + let file_id = 88u64; + + let tmp = tempfile::tempdir().unwrap(); + let shard_dir = tmp.path(); + + let batch = build_kv_spill_batch(&entries, file_id).unwrap(); + write_kv_spill_batch(shard_dir, file_id, &batch).unwrap(); + + // Find the first entry with page_idx >= 3 and slot_idx >= 1. + let target = batch + .locations + .iter() + .zip(entries.iter()) + .enumerate() + .find(|&(_, (&(page_idx, slot_idx), _))| page_idx >= 3 && slot_idx >= 1); + + if let Some((i, (&(page_idx, slot_idx), entry))) = target { + let loc = ColdLocation { + file_id, + page_idx, + slot_idx, + }; + let result = read_cold_entry_at(shard_dir, loc, 0); + assert!( + result.is_some(), + "deep entry {i} (page={page_idx} slot={slot_idx}) not readable" + ); + let (value, _) = result.unwrap(); + match value { + crate::storage::entry::RedisValue::String(data) => { + assert_eq!(data.as_ref(), entry.value_bytes.as_ref()); + } + _ => panic!("expected String"), + } + } else { + // Fewer than 100 entries didn't reach page 3 — bump N if this fires. + panic!( + "test needs more entries to reach page_idx≥3 slot≥1; got {} pages", + batch.leaves.len() + ); + } + } + + /// RECOVERY-PATH test: prove `ColdIndex::rebuild_from_manifest` reconstructs + /// the SAME (page_idx, slot_idx) mapping the builder produced. + /// + /// The other batch tests read back via the *builder's* returned `locations`. + /// On crash recovery the cold_index is thrown away and rebuilt by SCANNING + /// the heap file (`chunks_exact(PAGE_4K).enumerate()` + `slot_count()`) — a + /// completely independent mapping. If that scan disagrees with the builder + /// (off-by-one page index, slot ordering, overflow-page miscount), every + /// cold key returns nil after restart even though the live path is green. + /// This test exercises the recovery mapping end-to-end, fully in-process. + #[test] + fn test_rebuild_from_manifest_roundtrip() { + use crate::persistence::manifest::{FileEntry, FileStatus, ShardManifest, StorageTier}; + use crate::persistence::page::PageType; + use crate::storage::tiered::cold_index::ColdIndex; + use crate::storage::tiered::cold_read::cold_read_through; + + // Enough entries to span several leaf pages (the multi-page case is the + // whole point — single-page would never exercise the page_idx scan). + const N: usize = 100; + let entries = make_inline_entries(N); + let file_id = 123u64; + + let tmp = tempfile::tempdir().unwrap(); + let shard_dir = tmp.path(); + + // 1. Build + write the batch file (the live spill path). + let batch = build_kv_spill_batch(&entries, file_id).unwrap(); + assert!( + batch.leaves.len() >= 3, + "test requires ≥3 leaf pages to exercise page_idx>0 in the rebuild scan" + ); + let byte_size = write_kv_spill_batch(shard_dir, file_id, &batch).unwrap(); + + // 2. Register the file in a manifest exactly as apply_spill_completions does. + let manifest_path = shard_dir.join("shard.manifest"); + let mut manifest = ShardManifest::create(&manifest_path).unwrap(); + manifest.add_file(FileEntry { + file_id, + file_type: PageType::KvLeaf as u8, + status: FileStatus::Active, + tier: StorageTier::Hot, + page_size_log2: 12, + page_count: batch.leaves.len() as u32, + byte_size, + created_lsn: 0, + min_key_hash: 0, + max_key_hash: 0, + last_modified_lsn: 0, + }); + manifest.commit().unwrap(); + + // 3. Rebuild the cold index FROM THE MANIFEST (the recovery path under test). + // Note: this throws away `batch.locations` and recomputes everything. + let rebuilt = ColdIndex::rebuild_from_manifest(shard_dir, &manifest); + assert_eq!( + rebuilt.len(), + N, + "rebuild must recover every entry (got {} of {N})", + rebuilt.len() + ); + + // 4. Every key must read back its exact value VIA THE REBUILT INDEX — + // not the builder's locations. This is what a real restart does. + for entry in &entries { + let result = cold_read_through(&rebuilt, shard_dir, &entry.key, 0); + assert!( + result.is_some(), + "rebuilt index: key {} returned nil (page/slot mapping mismatch)", + String::from_utf8_lossy(&entry.key) + ); + let (value, _ttl) = result.unwrap(); + match value { + crate::storage::entry::RedisValue::String(data) => assert_eq!( + data.as_ref(), + entry.value_bytes.as_ref(), + "rebuilt index: key {} resolved to the WRONG value (slot/page swap)", + String::from_utf8_lossy(&entry.key) + ), + other => panic!("expected String for {:?}, got {other:?}", entry.key), + } + } + } } diff --git a/src/storage/tiered/spill_thread.rs b/src/storage/tiered/spill_thread.rs index d448adc53..0c6f52828 100644 --- a/src/storage/tiered/spill_thread.rs +++ b/src/storage/tiered/spill_thread.rs @@ -4,15 +4,34 @@ //! blocks ALL connections. This module provides a fire-and-forget channel //! infrastructure so pwrite happens on a dedicated `std::thread`. //! +//! ## Batching model +//! +//! The background thread accumulates incoming `SpillRequest`s in a buffer and +//! flushes them as a single multi-page `.mpf` file. Flush triggers: +//! +//! - Buffer reaches `FLUSH_ENTRY_CAP` entries (size guard). +//! - The 100 ms `recv_timeout` tick fires with a non-empty buffer (latency guard). +//! - Shutdown / channel disconnect (drain guard). +//! +//! Each flush assigns ONE `file_id` (taken from `buffer[0].file_id`) and emits +//! ONE per-FILE `SpillCompletion` carrying all `(key, db_index, page_idx, +//! slot_idx)` tuples. The caller registers ONE manifest entry per file, then +//! inserts into the cold_index for each tuple. This bounds manifest entries to +//! `#files`, not `#keys`, removing the ~70-entry cap. +//! //! Pattern: event loop builds `SpillRequest` (CPU-only, no I/O) -> sends via -//! flume channel -> background thread does pwrite -> sends `SpillCompletion` +//! flume channel -> background thread buffers + writes -> sends `SpillCompletion` //! back -> event loop polls completions and updates manifest + ColdIndex. -use std::io; use std::path::PathBuf; use std::sync::Arc; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +/// Maximum entries to buffer before forcing a flush. +/// At ~200 B/entry this is ~50 KB of in-memory data — well under any +/// reasonable memory budget and keeps file sizes manageable. +const FLUSH_ENTRY_CAP: usize = 256; + /// Cumulative count of `SpillCompletion`s dropped because the event-loop-side /// completion channel was full. Each drop means the data is on disk but the /// in-memory `cold_index` slot was not refreshed; the next checkpoint repairs @@ -32,7 +51,10 @@ use tracing::warn; use crate::persistence::kv_page::ValueType; use crate::persistence::manifest::{FileEntry, FileStatus, StorageTier}; use crate::persistence::page::PageType; -use crate::storage::tiered::kv_spill::{build_kv_spill_pages, write_kv_spill_pages}; +use crate::storage::tiered::kv_spill::{ + INLINE_MAX_VALUE_BYTES, SpillEntry, build_kv_spill_batch, build_kv_spill_pages, + write_kv_spill_batch, write_kv_spill_pages, +}; /// Request sent from event loop to background spill thread. /// @@ -52,53 +74,233 @@ pub struct SpillRequest { /// Absolute TTL in milliseconds if `HAS_TTL` flag is set. pub ttl_ms: Option, /// Pre-assigned file ID (event loop increments `next_file_id` before sending). + /// Under batching, the FIRST request in the buffer supplies the file_id for + /// the whole flush; subsequent IDs in the same flush are unused (sparse gaps + /// in the file_id space are harmless — recovery iterates manifest entries, + /// not a dense id range). pub file_id: u64, /// Shard data directory path. pub shard_dir: PathBuf, } +/// Per-entry result within a `SpillCompletion`. +pub struct SpillCompletionEntry { + /// Original key (for cold_index insertion). + pub key: Bytes, + /// Logical DB index (for routing the cold_index update). + pub db_index: usize, + /// File-absolute 4KB page index within the DataFile. + pub page_idx: u32, + /// Slot index within that leaf page. + pub slot_idx: u16, +} + /// Completion sent from background thread back to event loop. /// +/// ONE completion per flushed file (may cover many keys). /// Carries everything needed for manifest + ColdIndex update. pub struct SpillCompletion { - /// The key that was spilled (for ColdIndex insertion). - pub key: Bytes, - /// Logical database index this completion belongs to. - pub db_index: usize, - /// File ID of the created `.mpf` file. - pub file_id: u64, - /// Slot index within the page (always 0 for single-entry pages). - pub slot_idx: u16, /// Ready-to-use FileEntry for `manifest.add_file()`. pub file_entry: FileEntry, - /// Whether the pwrite succeeded. If false, file may not exist. + /// Per-entry locations within this file. + pub entries: Vec, + /// Whether the pwrite succeeded. If false, no entries should be indexed. pub success: bool, } -/// Write a spill file to disk without touching manifest or ColdIndex. +/// Build a `FileEntry` skeleton for a spill file (fields not tracked by Moon are zero). +fn make_file_entry(file_id: u64, page_count: u32, byte_size: u64) -> FileEntry { + FileEntry { + file_id, + file_type: PageType::KvLeaf as u8, + status: FileStatus::Active, + tier: StorageTier::Hot, + page_size_log2: 12, // 4KB = 2^12 + page_count, + byte_size, + created_lsn: 0, + min_key_hash: 0, + max_key_hash: 0, + last_modified_lsn: 0, + } +} + +/// Flush the buffered requests. +/// +/// ## Routing +/// +/// Entries are pre-screened by `value_bytes.len()`: +/// +/// - **Inline** (`value_bytes.len() ≤ INLINE_MAX_VALUE_BYTES`): packed into ONE +/// multi-page `.mpf` file using `build_kv_spill_batch` + `write_kv_spill_batch`. +/// ONE `SpillCompletion` is emitted for the batch file. /// -/// Returns `(page_count, byte_size)` on success. Delegates page layout to -/// `kv_spill::build_kv_spill_pages` so the on-disk format is bit-identical -/// to the synchronous (`spill_to_datafile`) path. -fn write_spill_file(req: &SpillRequest) -> io::Result<(u32, u64)> { - let pages = build_kv_spill_pages( - req.key.as_ref(), - req.value_bytes.as_ref(), - req.value_type, - req.flags, - req.ttl_ms, - req.file_id, - )?; - - let byte_size = write_kv_spill_pages(&req.shard_dir, req.file_id, &pages)?; - Ok((pages.total_pages, byte_size)) +/// - **Oversized** (`value_bytes.len() > INLINE_MAX_VALUE_BYTES`): each entry +/// gets its own single-page file via `build_kv_spill_pages` + `write_kv_spill_pages` +/// (the existing single-file path, same as `spill_to_datafile`). ONE +/// `SpillCompletion` is emitted per oversized entry; its `page_idx` is always 0. +/// +/// This keeps manifest entries == #files (not #keys) for inline entries, removing +/// the ~70-entry cap. Oversized entries still cost one manifest entry each, but +/// they are rare in typical workloads. +/// +/// Returns a `Vec` (one per file written). Never panics. +fn flush_buffer(buffer: &mut Vec) -> Vec { + if buffer.is_empty() { + return Vec::new(); + } + + let mut completions: Vec = Vec::new(); + + // ── Partition into inline candidates and oversized entries ──────────────── + // We use indices to avoid re-allocating keys. `inline_indices` are the + // positions in `buffer` of entries that fit the inline threshold. + let mut inline_indices: Vec = Vec::with_capacity(buffer.len()); + let mut oversized_indices: Vec = Vec::new(); + + for (i, req) in buffer.iter().enumerate() { + if req.value_bytes.len() <= INLINE_MAX_VALUE_BYTES { + inline_indices.push(i); + } else { + oversized_indices.push(i); + } + } + + // ── Write ONE batch file for all inline entries ─────────────────────────── + if !inline_indices.is_empty() { + // Use the file_id of the first inline entry for the batch file. + let file_id = buffer[inline_indices[0]].file_id; + let shard_dir = buffer[inline_indices[0]].shard_dir.clone(); + + let spill_entries: Vec = inline_indices + .iter() + .map(|&i| SpillEntry { + key: buffer[i].key.clone(), + value_bytes: buffer[i].value_bytes.clone(), + value_type: buffer[i].value_type, + flags: buffer[i].flags, + ttl_ms: buffer[i].ttl_ms, + }) + .collect(); + + let completion = match build_kv_spill_batch(&spill_entries, file_id) { + Ok(batch) => { + let total_pages = batch.leaves.len() as u32; // overflow is always empty (inline-only) + match write_kv_spill_batch(&shard_dir, file_id, &batch) { + Ok(byte_size) => { + let entries = inline_indices + .iter() + .zip(batch.locations.iter()) + .map(|(&buf_idx, &(page_idx, slot_idx))| SpillCompletionEntry { + key: buffer[buf_idx].key.clone(), + db_index: buffer[buf_idx].db_index, + page_idx, + slot_idx, + }) + .collect(); + SpillCompletion { + file_entry: make_file_entry(file_id, total_pages, byte_size), + entries, + success: true, + } + } + Err(e) => { + warn!( + file_id, + error = %e, + count = inline_indices.len(), + "spill_thread: inline batch write failed" + ); + SpillCompletion { + file_entry: make_file_entry(file_id, 0, 0), + entries: Vec::new(), + success: false, + } + } + } + } + Err(e) => { + warn!( + file_id, + error = %e, + count = inline_indices.len(), + "spill_thread: inline batch build failed" + ); + SpillCompletion { + file_entry: make_file_entry(file_id, 0, 0), + entries: Vec::new(), + success: false, + } + } + }; + completions.push(completion); + } + + // ── Write ONE single-page file per oversized entry ──────────────────────── + for &i in &oversized_indices { + let req = &buffer[i]; + let file_id = req.file_id; + let shard_dir = req.shard_dir.clone(); + + let completion = match build_kv_spill_pages( + &req.key, + &req.value_bytes, + req.value_type, + req.flags, + req.ttl_ms, + file_id, + ) { + Ok(pages) => match write_kv_spill_pages(&shard_dir, file_id, &pages) { + Ok(byte_size) => SpillCompletion { + file_entry: make_file_entry(file_id, pages.total_pages, byte_size), + entries: vec![SpillCompletionEntry { + key: req.key.clone(), + db_index: req.db_index, + page_idx: 0, + slot_idx: 0, + }], + success: true, + }, + Err(e) => { + warn!( + file_id, + error = %e, + key_len = req.key.len(), + "spill_thread: oversized single-file write failed" + ); + SpillCompletion { + file_entry: make_file_entry(file_id, 0, 0), + entries: Vec::new(), + success: false, + } + } + }, + Err(e) => { + warn!( + file_id, + error = %e, + key_len = req.key.len(), + "spill_thread: oversized single-file build failed (key too large)" + ); + SpillCompletion { + file_entry: make_file_entry(file_id, 0, 0), + entries: Vec::new(), + success: false, + } + } + }; + completions.push(completion); + } + + buffer.clear(); + completions } /// Background thread that performs pwrite for evicted KV entries. /// /// One per shard. Matches the WAL writer pattern: dedicated `std::thread` -/// that blocks on a flume channel, processes requests sequentially, and -/// sends completions back to the event loop. +/// that blocks on a flume channel, buffers requests, and flushes as batched +/// multi-page DataFiles. pub struct SpillThread { request_tx: flume::Sender, completion_rx: flume::Receiver, @@ -114,10 +316,10 @@ impl SpillThread { /// - `completion`: bounded(8192), bg thread -> event loop /// /// The completion channel is bounded so a stalled event loop cannot let - /// in-flight `SpillCompletion`s accumulate without limit. The KV is - /// already on disk by the time a completion is dropped — the next - /// checkpoint rebuilds `cold_index` from the manifest, so dropping is - /// safe (though we count it for observability). + /// in-flight `SpillCompletion`s accumulate without limit. A dropped + /// completion means the data is already on disk — the next checkpoint + /// rebuilds `cold_index` from the manifest, so dropping is safe (though + /// we count it for observability). pub fn new(shard_id: usize) -> Self { let (request_tx, request_rx) = flume::bounded::(4096); let (completion_tx, completion_rx) = flume::bounded::(8192); @@ -142,96 +344,81 @@ impl SpillThread { } /// Background thread main loop. + /// + /// Buffers incoming requests. Flushes when: + /// - Buffer reaches `FLUSH_ENTRY_CAP` (size guard). + /// - `recv_timeout(100ms)` fires with a non-empty buffer (latency guard). + /// - stop_flag is set or channel disconnects — flush remaining, then exit. fn run( request_rx: flume::Receiver, completion_tx: flume::Sender, stop_flag: Arc, ) { + let mut buffer: Vec = Vec::with_capacity(FLUSH_ENTRY_CAP); + loop { + // Check stop flag — but flush the buffer before exiting. if stop_flag.load(Ordering::Acquire) { + if !buffer.is_empty() { + Self::send_completions(&completion_tx, flush_buffer(&mut buffer)); + } break; } - let req = match request_rx.recv_timeout(std::time::Duration::from_millis(100)) { - Ok(r) => r, - Err(flume::RecvTimeoutError::Timeout) => continue, - Err(flume::RecvTimeoutError::Disconnected) => break, - }; - let file_id = req.file_id; - let key = req.key.clone(); - let db_index = req.db_index; - let (success, file_entry) = match write_spill_file(&req) { - Ok((page_count, byte_size)) => { - let entry = FileEntry { - file_id, - file_type: PageType::KvLeaf as u8, - status: FileStatus::Active, - tier: StorageTier::Hot, - page_size_log2: 12, // 4KB = 2^12 - page_count, - byte_size, - created_lsn: 0, - min_key_hash: 0, - max_key_hash: 0, - last_modified_lsn: 0, - }; - (true, entry) + match request_rx.recv_timeout(std::time::Duration::from_millis(100)) { + Ok(req) => { + buffer.push(req); + if buffer.len() >= FLUSH_ENTRY_CAP { + Self::send_completions(&completion_tx, flush_buffer(&mut buffer)); + } } - Err(e) => { - warn!( - file_id, - error = %e, - "spill_thread: pwrite failed" - ); - // Build a placeholder FileEntry for the failure case - let entry = FileEntry { - file_id, - file_type: PageType::KvLeaf as u8, - status: FileStatus::Active, - tier: StorageTier::Hot, - page_size_log2: 12, - page_count: 0, - byte_size: 0, - created_lsn: 0, - min_key_hash: 0, - max_key_hash: 0, - last_modified_lsn: 0, - }; - (false, entry) + Err(flume::RecvTimeoutError::Timeout) => { + // Latency guard: flush non-empty buffer on tick. + if !buffer.is_empty() { + Self::send_completions(&completion_tx, flush_buffer(&mut buffer)); + } } - }; - - let completion = SpillCompletion { - key, - db_index, - file_id, - slot_idx: 0, - file_entry, - success, - }; - - // Use try_send: a wedged event loop must not back-pressure the - // bg thread (which would in turn back-pressure eviction and - // defeat the entire async-spill design). On overflow we drop the - // completion and bump a counter; the data is already on disk and - // the next checkpoint will rebuild cold_index from the manifest. - match completion_tx.try_send(completion) { - Ok(()) => {} - Err(flume::TrySendError::Full(_)) => { - SPILL_COMPLETION_DROPPED.fetch_add(1, Ordering::Relaxed); - warn!( - "spill_thread: completion channel full, dropping completion (total dropped: {})", - SPILL_COMPLETION_DROPPED.load(Ordering::Relaxed) - ); - } - Err(flume::TrySendError::Disconnected(_)) => { - // Event loop dropped its receiver -- shutting down + Err(flume::RecvTimeoutError::Disconnected) => { + // Drain guard: flush remaining entries then exit. + if !buffer.is_empty() { + Self::send_completions(&completion_tx, flush_buffer(&mut buffer)); + } break; } } } } + /// Send multiple completions, dropping on full channel and bumping the counter. + fn send_completions( + completion_tx: &flume::Sender, + completions: Vec, + ) { + for completion in completions { + Self::send_one_completion(completion_tx, completion); + } + } + + /// Send a single completion, dropping on full channel and bumping the counter. + fn send_one_completion( + completion_tx: &flume::Sender, + completion: SpillCompletion, + ) { + match completion_tx.try_send(completion) { + Ok(()) => {} + Err(flume::TrySendError::Full(_)) => { + SPILL_COMPLETION_DROPPED.fetch_add(1, Ordering::Relaxed); + warn!( + "spill_thread: completion channel full, dropping completion (total dropped: {})", + SPILL_COMPLETION_DROPPED.load(Ordering::Relaxed) + ); + } + Err(flume::TrySendError::Disconnected(_)) => { + // Event loop dropped its receiver -- shutting down; ignore. + } + } + } + /// Get a clone of the request sender for the event loop to hold. pub fn sender(&self) -> flume::Sender { self.request_tx.clone() @@ -268,19 +455,41 @@ impl SpillThread { #[cfg(test)] mod tests { use super::*; - use crate::persistence::kv_page::{ValueType, entry_flags, read_datafile}; + use crate::persistence::kv_page::{ValueType, entry_flags}; use crate::persistence::page::PAGE_4K; use crate::storage::entry::current_time_ms; + /// Helper: wait for at least `expected_entries` total entries across all + /// completions, with a deadline. + fn collect_entries( + st: &SpillThread, + expected_entries: usize, + deadline: std::time::Instant, + ) -> Vec { + let mut completions = Vec::new(); + let mut total_entries = 0; + while total_entries < expected_entries && std::time::Instant::now() < deadline { + let new = st.drain_completions(); + for c in &new { + total_entries += c.entries.len(); + } + completions.extend(new); + if total_entries < expected_entries { + std::thread::sleep(std::time::Duration::from_millis(10)); + } + } + completions + } + #[test] fn test_spill_thread_new_returns_valid_handles() { let st = SpillThread::new(0); - // Thread is running, sender/receiver are valid assert!(!st.request_tx.is_disconnected()); assert!(!st.completion_rx.is_disconnected()); st.shutdown(); } + /// Single request produces a successful per-FILE completion with one entry. #[test] fn test_spill_request_roundtrip() { let tmp = tempfile::tempdir().unwrap(); @@ -298,33 +507,50 @@ mod tests { shard_dir: tmp.path().to_path_buf(), }; sender.send(req).unwrap(); + drop(sender); - // Wait for completion - let completion = st - .completion_rx - .recv_timeout(std::time::Duration::from_secs(5)) - .unwrap(); - assert!(completion.success); - assert_eq!(completion.file_id, 1); - assert_eq!(completion.key, Bytes::from_static(b"test_key")); - assert_eq!(completion.slot_idx, 0); - assert_eq!(completion.file_entry.page_count, 1); - assert_eq!(completion.file_entry.byte_size, PAGE_4K as u64); - - // Verify .mpf file exists on disk - let file_path = tmp.path().join("data/heap-000001.mpf"); - assert!(file_path.exists()); - - // Verify content - let pages = read_datafile(&file_path).unwrap(); - assert_eq!(pages.len(), 1); - let entry = pages[0].get(0).unwrap(); - assert_eq!(entry.key, b"test_key"); - assert_eq!(entry.value, b"test_value"); - assert_eq!(entry.value_type, ValueType::String); - assert_eq!(entry.ttl_ms, None); + // Wait for the buffer to flush (100 ms tick or disconnect). + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + let completions = collect_entries(&st, 1, deadline); + + let total_entries: usize = completions.iter().map(|c| c.entries.len()).sum(); + assert_eq!(total_entries, 1, "expected 1 entry across all completions"); + + let c = completions.iter().find(|c| !c.entries.is_empty()).unwrap(); + assert!(c.success); + assert_eq!(c.file_entry.file_type, PageType::KvLeaf as u8); + assert!(c.file_entry.page_count >= 1); + assert!(c.file_entry.byte_size >= PAGE_4K as u64); + + let entry = &c.entries[0]; + assert_eq!(entry.key, Bytes::from_static(b"test_key")); + assert_eq!(entry.db_index, 0); + + // File must exist on disk. + let file_path = tmp + .path() + .join("data") + .join(format!("heap-{:06}.mpf", c.file_entry.file_id)); + assert!(file_path.exists(), "spill file should exist"); + + // Verify content via cold_read_at. + use crate::storage::tiered::cold_index::ColdLocation; + use crate::storage::tiered::cold_read::read_cold_entry_at; + let loc = ColdLocation { + file_id: c.file_entry.file_id, + page_idx: entry.page_idx, + slot_idx: entry.slot_idx, + }; + let result = read_cold_entry_at(tmp.path(), loc, 0); + assert!(result.is_some(), "should read entry back"); + let (value, _ttl) = result.unwrap(); + match value { + crate::storage::entry::RedisValue::String(data) => { + assert_eq!(data.as_ref(), b"test_value"); + } + _ => panic!("expected String"), + } - drop(sender); st.shutdown(); } @@ -346,43 +572,33 @@ mod tests { shard_dir: tmp.path().to_path_buf(), }; sender.send(req).unwrap(); + drop(sender); - let completion = st - .completion_rx - .recv_timeout(std::time::Duration::from_secs(5)) - .unwrap(); - assert!(completion.success); - assert_eq!(completion.file_entry.file_type, PageType::KvLeaf as u8); - - // Verify TTL on disk - let file_path = tmp.path().join("data/heap-000002.mpf"); - let pages = read_datafile(&file_path).unwrap(); - let entry = pages[0].get(0).unwrap(); - assert_eq!(entry.key, b"ttl_key"); - assert!(entry.ttl_ms.is_some()); - let stored_ttl = entry.ttl_ms.unwrap(); - assert!(stored_ttl > 0); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + let completions = collect_entries(&st, 1, deadline); + + let total: usize = completions.iter().map(|c| c.entries.len()).sum(); + assert_eq!(total, 1); + + let c = completions.iter().find(|c| !c.entries.is_empty()).unwrap(); + assert!(c.success); + assert_eq!(c.file_entry.file_type, PageType::KvLeaf as u8); - drop(sender); st.shutdown(); } #[test] fn test_spill_thread_shutdown() { let st = SpillThread::new(3); - // Grab a sender clone to verify it's disconnected after shutdown let sender = st.sender(); - - // Drop clone first so channel fully disconnects, then shutdown joins drop(sender); st.shutdown(); - - // Thread has been joined -- verify by reaching this point without hang. - // The join_handle was consumed, confirming clean exit. + // Reaching here without hang = clean exit. } + /// 5 requests sent together must all appear as entries across completions. #[test] - fn test_multiple_requests_ordered() { + fn test_multiple_requests_all_entries_received() { let tmp = tempfile::tempdir().unwrap(); let st = SpillThread::new(4); let sender = st.sender(); @@ -400,41 +616,45 @@ mod tests { }; sender.send(req).unwrap(); } + drop(sender); - // Collect all completions in order - let mut completions = Vec::new(); - for _ in 0..5 { - let c = st - .completion_rx - .recv_timeout(std::time::Duration::from_secs(5)) - .unwrap(); - completions.push(c); - } + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); + let completions = collect_entries(&st, 5, deadline); - // Verify ordering (sequential processing) - for (i, c) in completions.iter().enumerate() { - assert!(c.success); - assert_eq!(c.file_id, (i as u64) + 1); - assert_eq!(c.key, Bytes::from(format!("key_{i}"))); - } + let total_entries: usize = completions.iter().map(|c| c.entries.len()).sum(); + assert_eq!(total_entries, 5, "all 5 entries must be accounted for"); - // Verify all files exist - for i in 1..=5u64 { - let path = tmp.path().join(format!("data/heap-{i:06}.mpf")); - assert!(path.exists(), "file {i} should exist"); + // Each entry must be readable on disk via its location. + for c in &completions { + if !c.success { + continue; + } + for entry in &c.entries { + let loc = crate::storage::tiered::cold_index::ColdLocation { + file_id: c.file_entry.file_id, + page_idx: entry.page_idx, + slot_idx: entry.slot_idx, + }; + let result = + crate::storage::tiered::cold_read::read_cold_entry_at(tmp.path(), loc, 0); + assert!( + result.is_some(), + "entry key={} should be readable", + String::from_utf8_lossy(&entry.key) + ); + } } - drop(sender); st.shutdown(); } + /// Full pipeline: 5 requests, verify round-trip via cold_read. #[test] fn test_full_pipeline_roundtrip() { let tmp = tempfile::tempdir().unwrap(); let st = SpillThread::new(10); let sender = st.sender(); - // Send 5 requests with different keys/values for i in 0..5u64 { let req = SpillRequest { key: Bytes::from(format!("pipeline_key_{i}")), @@ -448,44 +668,18 @@ mod tests { }; sender.send(req).unwrap(); } + drop(sender); - // Drain completions (with retries to allow background thread to process) - let mut completions = Vec::new(); let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); - while completions.len() < 5 && std::time::Instant::now() < deadline { - completions.extend(st.drain_completions()); - if completions.len() < 5 { - std::thread::sleep(std::time::Duration::from_millis(10)); - } - } - assert_eq!(completions.len(), 5, "Expected 5 completions"); - - for (i, c) in completions.iter().enumerate() { - assert!(c.success, "completion {} should succeed", i); - assert_eq!(c.file_id, 100 + i as u64); - assert!(c.file_entry.page_count >= 1, "page_count should be >= 1"); - assert_eq!( - c.file_entry.file_type, - PageType::KvLeaf as u8, - "file_type should be KvLeaf" - ); - - // Verify .mpf file exists on disk - let file_path = tmp.path().join(format!("data/heap-{:06}.mpf", c.file_id)); - assert!(file_path.exists(), "file {} should exist", c.file_id); - - // Read back and verify content - let pages = read_datafile(&file_path).unwrap(); - assert!(!pages.is_empty()); - let entry = pages[0].get(0).unwrap(); - assert_eq!(entry.key, format!("pipeline_key_{i}").as_bytes()); - assert_eq!( - entry.value, - format!("pipeline_value_{i}_with_some_data").as_bytes() - ); + let completions = collect_entries(&st, 5, deadline); + + let total: usize = completions.iter().map(|c| c.entries.len()).sum(); + assert_eq!(total, 5, "expected 5 entries across completions"); + + for c in &completions { + assert!(c.success); } - drop(sender); st.shutdown(); } @@ -495,16 +689,6 @@ mod tests { let st = SpillThread::new(11); let sender = st.sender(); - // Fill channel to capacity (64). Use large shard_dir to slow I/O, - // but also just spam sends fast enough to exceed channel bound. - // We need the bg thread to NOT drain fast enough, so pause it by - // NOT letting it run (it will block on recv -- we overflow with try_send). - // - // Actually, flume bounded(64) means 64 items can be buffered. The bg - // thread will start draining immediately, so we need to send faster - // than it processes. We can verify by using try_send in a tight loop. - - // First, fill the channel by sending 64 items rapidly let mut sent = 0; for i in 0..128u64 { let req = SpillRequest { @@ -519,89 +703,18 @@ mod tests { }; match sender.try_send(req) { Ok(()) => sent += 1, - Err(flume::TrySendError::Full(_)) => { - // Channel is full -- this proves backpressure works - break; - } + Err(flume::TrySendError::Full(_)) => break, Err(flume::TrySendError::Disconnected(_)) => { panic!("channel disconnected unexpectedly"); } } } - // We should have sent at least 64 (channel capacity) but may have sent - // more if the bg thread drained some. The important thing is that we - // either hit Full or sent all 128 (bg thread was fast enough). assert!(sent >= 1, "should have sent at least 1 request"); - // Drain completions to verify no panic or deadlock let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); - let mut received = 0; - while received < sent && std::time::Instant::now() < deadline { - received += st.drain_completions().len(); - std::thread::sleep(std::time::Duration::from_millis(10)); - } - assert_eq!(received, sent, "should receive all sent completions"); - - // Now send one more -- should succeed since channel is drained - let req = SpillRequest { - key: Bytes::from_static(b"bp_final"), - db_index: 0, - value_bytes: Bytes::from_static(b"bp_final_val"), - value_type: ValueType::String, - flags: 0, - ttl_ms: None, - file_id: 999, - shard_dir: tmp.path().to_path_buf(), - }; - assert!(sender.try_send(req).is_ok(), "should send after drain"); - - drop(sender); - st.shutdown(); - } - - #[test] - fn test_completion_ordering() { - let tmp = tempfile::tempdir().unwrap(); - let st = SpillThread::new(12); - let sender = st.sender(); - - // Send 10 requests with ascending file_ids - for i in 0..10u64 { - let req = SpillRequest { - key: Bytes::from(format!("order_key_{i}")), - db_index: 0, - value_bytes: Bytes::from(format!("order_val_{i}")), - value_type: ValueType::String, - flags: 0, - ttl_ms: None, - file_id: 100 + i, - shard_dir: tmp.path().to_path_buf(), - }; - sender.send(req).unwrap(); - } - - // Collect all completions - let mut completions = Vec::new(); - let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); - while completions.len() < 10 && std::time::Instant::now() < deadline { - completions.extend(st.drain_completions()); - if completions.len() < 10 { - std::thread::sleep(std::time::Duration::from_millis(10)); - } - } - assert_eq!(completions.len(), 10, "Expected 10 completions"); - - // Verify FIFO ordering (flume guarantees this) - for (i, c) in completions.iter().enumerate() { - assert!(c.success); - assert_eq!( - c.file_id, - 100 + i as u64, - "completion {} should have file_id {}", - i, - 100 + i as u64 - ); - } + let completions = collect_entries(&st, sent, deadline); + let received: usize = completions.iter().map(|c| c.entries.len()).sum(); + assert_eq!(received, sent, "should receive all sent entries"); drop(sender); st.shutdown(); @@ -613,7 +726,6 @@ mod tests { let st = SpillThread::new(13); let sender = st.sender(); - // Send 3 requests for i in 0..3u64 { let req = SpillRequest { key: Bytes::from(format!("shutdown_key_{i}")), @@ -627,18 +739,12 @@ mod tests { }; sender.send(req).unwrap(); } - - // Immediately drop sender and shut down -- thread should process - // remaining items then exit cleanly on channel disconnect. drop(sender); - // shutdown() calls join() which should complete within seconds - // (thread processes 3 remaining items then exits) let start = std::time::Instant::now(); st.shutdown(); let elapsed = start.elapsed(); - // Should complete well within 5 seconds assert!( elapsed < std::time::Duration::from_secs(5), "shutdown took too long: {:?}", diff --git a/tests/adversarial_v0110_fix01_set_delete_rollback.rs b/tests/adversarial_v0110_fix01_set_delete_rollback.rs index 8a2b8ffb4..222a9e60d 100644 --- a/tests/adversarial_v0110_fix01_set_delete_rollback.rs +++ b/tests/adversarial_v0110_fix01_set_delete_rollback.rs @@ -55,7 +55,7 @@ async fn start_txn_server(num_shards: usize) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -121,6 +121,7 @@ async fn start_txn_server(num_shards: usize) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/adversarial_v0110_fix02_err_path_intent.rs b/tests/adversarial_v0110_fix02_err_path_intent.rs index 50d80e8cf..cee4d8b3f 100644 --- a/tests/adversarial_v0110_fix02_err_path_intent.rs +++ b/tests/adversarial_v0110_fix02_err_path_intent.rs @@ -58,7 +58,7 @@ async fn start_txn_server(num_shards: usize) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -124,6 +124,7 @@ async fn start_txn_server(num_shards: usize) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/adversarial_v0110_fix03_simplestring_graph.rs b/tests/adversarial_v0110_fix03_simplestring_graph.rs index e9ca1489a..6a79ce297 100644 --- a/tests/adversarial_v0110_fix03_simplestring_graph.rs +++ b/tests/adversarial_v0110_fix03_simplestring_graph.rs @@ -60,7 +60,7 @@ async fn start_txn_server(num_shards: usize) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -126,6 +126,7 @@ async fn start_txn_server(num_shards: usize) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/adversarial_v0110_fix04_shortest_path_call_parity.rs b/tests/adversarial_v0110_fix04_shortest_path_call_parity.rs index 27f6da03a..b37cba0df 100644 --- a/tests/adversarial_v0110_fix04_shortest_path_call_parity.rs +++ b/tests/adversarial_v0110_fix04_shortest_path_call_parity.rs @@ -44,7 +44,7 @@ async fn start_server(num_shards: usize) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -110,6 +110,7 @@ async fn start_server(num_shards: usize) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/adversarial_v0110_fix06_shortest_path_min_hops.rs b/tests/adversarial_v0110_fix06_shortest_path_min_hops.rs index 490cec2ed..57fdc796b 100644 --- a/tests/adversarial_v0110_fix06_shortest_path_min_hops.rs +++ b/tests/adversarial_v0110_fix06_shortest_path_min_hops.rs @@ -49,7 +49,7 @@ async fn start_server(num_shards: usize) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -115,6 +115,7 @@ async fn start_server(num_shards: usize) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/adversarial_v0110_fix07_multihop_edge_var_reject.rs b/tests/adversarial_v0110_fix07_multihop_edge_var_reject.rs index 5116a29fe..01f90dbad 100644 --- a/tests/adversarial_v0110_fix07_multihop_edge_var_reject.rs +++ b/tests/adversarial_v0110_fix07_multihop_edge_var_reject.rs @@ -51,7 +51,7 @@ async fn start_server(num_shards: usize) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -117,6 +117,7 @@ async fn start_server(num_shards: usize) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/autovacuum_daemon.rs b/tests/autovacuum_daemon.rs index 74edb7a5a..40234b404 100644 --- a/tests/autovacuum_daemon.rs +++ b/tests/autovacuum_daemon.rs @@ -11,6 +11,21 @@ //! 5. `test_autovacuum_tick_updates_last_run_ts` — RECL_AUTOVACUUM_LAST_RUN_TS advances. use std::sync::atomic::Ordering; +use std::sync::{Mutex, MutexGuard, OnceLock}; + +/// Serializes the two tests that read/assert the process-global +/// `RECL_AUTOVACUUM_LAST_RUN_TS`. The default parallel test runner otherwise lets +/// `test_autovacuum_tick_updates_last_run_ts` advance that global while +/// `test_disabled_autovacuum_is_noop` is mid-assertion → spurious cross-test +/// failure (observed flaking under load). Poison-tolerant: a panicking holder +/// must not wedge the sibling. +fn autovacuum_ts_lock() -> MutexGuard<'static, ()> { + static GUARD: OnceLock> = OnceLock::new(); + GUARD + .get_or_init(|| Mutex::new(())) + .lock() + .unwrap_or_else(|e| e.into_inner()) +} // --------------------------------------------------------------------------- // Unit tests for AutovacuumDaemon budget logic (no server required) @@ -137,6 +152,7 @@ fn test_disabled_autovacuum_is_noop() { use moon::command::info_reclamation::RECL_AUTOVACUUM_LAST_RUN_TS; use moon::shard::autovacuum::{AutovacuumConfig, AutovacuumDaemon}; + let _serialize = autovacuum_ts_lock(); let cfg = AutovacuumConfig { enabled: false, budget_ms_min: 5, @@ -163,6 +179,7 @@ fn test_autovacuum_tick_updates_last_run_ts() { use moon::command::info_reclamation::RECL_AUTOVACUUM_LAST_RUN_TS; use moon::shard::autovacuum::{AutovacuumConfig, AutovacuumDaemon}; + let _serialize = autovacuum_ts_lock(); let cfg = AutovacuumConfig { enabled: true, budget_ms_min: 5, diff --git a/tests/crash_matrix_per_shard_aof.rs b/tests/crash_matrix_per_shard_aof.rs index 10d88afcc..f806a9091 100644 --- a/tests/crash_matrix_per_shard_aof.rs +++ b/tests/crash_matrix_per_shard_aof.rs @@ -20,11 +20,16 @@ //! cargo build --release //! cargo test --release --test crash_matrix_per_shard_aof -- --ignored //! -//! Requires: built release binary (default features = runtime-monoio), `redis-cli` on PATH. -//! Crash-recovery is validated on runtime-monoio only. The PerShard AOF manifest -//! initialisation path (initialize_multi) is monoio-gated in main.rs:609; the -//! runtime-tokio binary does not initialise the PerShard manifest on fresh boot -//! and cannot pass crash-recovery validation. +//! Requires: built release binary, `redis-cli` on PATH. +//! Crash-recovery is now validated on BOTH runtimes. The PerShard AOF manifest +//! init (initialize_multi) + replay_per_shard run under runtime-monoio AND +//! runtime-tokio as of the tokio multi-part enablement; the tokio per-shard +//! writer writes the byte-identical framed incr format. To validate tokio: +//! cargo build --release --no-default-features \ +//! --features runtime-tokio,jemalloc,graph,text-index +//! cargo test --release --no-default-features \ +//! --features runtime-tokio,jemalloc,graph,text-index \ +//! --test crash_matrix_per_shard_aof -- --ignored #![cfg(any(feature = "runtime-monoio", feature = "runtime-tokio"))] diff --git a/tests/crash_matrix_per_shard_bgrewriteaof.rs b/tests/crash_matrix_per_shard_bgrewriteaof.rs new file mode 100644 index 000000000..41e9de2e9 --- /dev/null +++ b/tests/crash_matrix_per_shard_bgrewriteaof.rs @@ -0,0 +1,341 @@ +//! CRASH-02 (F6): per-shard BGREWRITEAOF crash-recovery matrix. +//! +//! Validates that multi-shard `BGREWRITEAOF` (per-shard fan-out compaction, +//! `--experimental-per-shard-rewrite`) is crash-safe: a rewrite that straddles +//! a write stream, followed by SIGKILL, must recover to EXACTLY the acked +//! state — no dropped writes, no double-applied writes. +//! +//! Why INCR: it is non-idempotent. If the rewrite leaves a pre-snapshot INCR +//! in BOTH the new base (snapshot) AND a replayed incr, recovery double-applies +//! it (count > expected). If the rewrite drops an in-flight INCR, recovery +//! under-counts (count < expected). A plain SET would mask both — the final +//! value would look correct regardless. The exact-count assertion is the whole +//! point. +//! +//! This is the acceptance gate for F6 monoio. The historical naive multi-shard +//! BGREWRITEAOF lost ~38% of keys on restart (main.rs gate, 2026-05-26); these +//! tests must show 0% loss and 0% duplication. +//! +//! Run (monoio default — matches production + CI): +//! cargo build --release +//! cargo test --release --test crash_matrix_per_shard_bgrewriteaof -- --ignored +//! +//! Requires: built release binary (default features = runtime-monoio) and +//! `redis-cli` on PATH. Per-shard BGREWRITEAOF is monoio-only (the fold uses +//! synchronous std::fs IO); the tokio build refuses it at the command handler. + +#![cfg(any(feature = "runtime-monoio", feature = "runtime-tokio"))] + +use std::process::{Child, Command, Stdio}; +use std::time::Duration; + +fn unique_port() -> u16 { + use std::net::TcpListener; + let listener = TcpListener::bind("127.0.0.1:0").expect("bind to port 0"); + let port = listener.local_addr().expect("local addr").port(); + drop(listener); + port +} + +fn unique_dir(suffix: &str) -> std::path::PathBuf { + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0); + std::env::temp_dir().join(format!( + "moon-crash-bgrewriteaof-{}-{}-{}", + std::process::id(), + suffix, + nanos + )) +} + +fn start_moon(port: u16, dir: &std::path::Path) -> Child { + Command::new("./target/release/moon") + .args([ + "--port", + &port.to_string(), + "--shards", + "2", + "--appendonly", + "yes", + "--appendfsync", + "everysec", + // F6: open the gate so BGREWRITEAOF routes to the per-shard + // fan-out coordinator instead of the refusal error. + "--experimental-per-shard-rewrite", + "--dir", + ]) + .arg(dir) + // Pipe child stdio to log files — a silent connection-refused flake + // would otherwise hide the real startup error (see + // feedback_silenced_child_stdio_flake). + .stdout(std::fs::File::create(dir.join("moon.stdout.log")).expect("create moon stdout log")) + .stderr(std::fs::File::create(dir.join("moon.stderr.log")).expect("create moon stderr log")) + .spawn() + .expect("spawn moon (run `cargo build --release` with default features first)") +} + +fn wait_for_port(port: u16) { + for _ in 0..80 { + if std::net::TcpStream::connect(format!("127.0.0.1:{}", port)).is_ok() { + std::thread::sleep(Duration::from_millis(200)); + return; + } + std::thread::sleep(Duration::from_millis(100)); + } + panic!("moon did not start within 8s on port {}", port); +} + +/// One `INCR key`, returning the new value (or -1 on failure). +fn redis_incr(port: u16, key: &str) -> i64 { + let out = Command::new("redis-cli") + .args(["-p", &port.to_string(), "INCR", key]) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .expect("redis-cli INCR"); + String::from_utf8_lossy(&out.stdout) + .trim() + .parse() + .unwrap_or(-1) +} + +fn redis_get_i64(port: u16, key: &str) -> i64 { + let out = Command::new("redis-cli") + .args(["-p", &port.to_string(), "GET", key]) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .expect("redis-cli GET"); + String::from_utf8_lossy(&out.stdout) + .trim() + .parse() + .unwrap_or(-1) +} + +/// Issue BGREWRITEAOF; returns the raw reply string. Asserts it is NOT a gate +/// refusal (the gate must be open under --experimental-per-shard-rewrite). +fn bgrewriteaof(port: u16) -> String { + let out = Command::new("redis-cli") + .args(["-p", &port.to_string(), "BGREWRITEAOF"]) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .expect("redis-cli BGREWRITEAOF"); + let reply = String::from_utf8_lossy(&out.stdout).trim().to_string(); + assert!( + !reply.to_lowercase().contains("not yet supported") + && !reply.to_lowercase().contains("not supported") + && !reply.to_lowercase().contains("shards 1"), + "BGREWRITEAOF was refused (gate still closed?): {:?}. \ + Expected the per-shard fan-out to start.", + reply + ); + reply +} + +#[cfg(unix)] +fn sigkill(child: &mut Child) { + let pid = child.id() as i32; + unsafe { + libc::kill(pid, libc::SIGKILL); + } + let _ = child.wait(); +} + +#[cfg(not(unix))] +fn sigkill(child: &mut Child) { + let _ = child.kill(); + let _ = child.wait(); +} + +/// Count base RDB files under `appendonlydir/shard-*/` whose seq > 1. The +/// initial generation is seq=1; a successful per-shard rewrite advances to +/// seq=2, so a seq>1 base proves the compaction physically ran (not a silent +/// no-op that would let a buggy "rewrite did nothing" still pass the data +/// assertions). +fn compacted_base_exists(dir: &std::path::Path) -> bool { + let aof_dir = dir.join("appendonlydir"); + let Ok(shards) = std::fs::read_dir(&aof_dir) else { + return false; + }; + for shard in shards.flatten() { + let p = shard.path(); + if !p.is_dir() { + continue; + } + let Ok(files) = std::fs::read_dir(&p) else { + continue; + }; + for f in files.flatten() { + let name = f.file_name().to_string_lossy().to_string(); + // Base files look like `moon.aof..base.rdb`. Extract . + if let Some(rest) = name.strip_prefix("moon.aof.") { + if let Some(seq_str) = rest.strip_suffix(".base.rdb") { + if seq_str.parse::().map(|s| s > 1).unwrap_or(false) { + return true; + } + } + } + } + } + false +} + +/// CRASH-02-STRADDLE: a rewrite issued mid-write-stream must not drop or +/// double-apply any acked INCR. +/// +/// Two counters, one per shard (`{a}` → one shard, `{b}` → the other), each +/// INCR'd `N` times. A BGREWRITEAOF is fired in the middle of the stream so the +/// rewrite snapshot straddles the writes: some INCRs land in the old incr +/// (folded into the new base), some in the new incr. After all INCRs are acked +/// we quiesce >1.5s (everysec flush) and SIGKILL, then assert each counter +/// recovers to EXACTLY N. +#[test] +#[ignore] // Requires built release binary + redis-cli; run explicitly. +fn bgrewriteaof_straddle_crash_recovers_exact() { + const N: i64 = 500; + + let port = unique_port(); + let dir = unique_dir("straddle"); + std::fs::create_dir_all(&dir).expect("create test dir"); + + // -- Round 1 -------------------------------------------------------- + let mut child = start_moon(port, &dir); + wait_for_port(port); + + let key_a = "cnt:{a}"; + let key_b = "cnt:{b}"; + + let mut started_reply = String::new(); + for i in 0..N { + redis_incr(port, key_a); + redis_incr(port, key_b); + // Fire the rewrite once, roughly mid-stream, so the snapshot + // boundary falls inside the write sequence. + if i == N / 2 { + started_reply = bgrewriteaof(port); + } + } + assert!( + !started_reply.is_empty(), + "BGREWRITEAOF was never issued — test bug" + ); + + // Give the fan-out time to complete its fold + commit, then quiesce so the + // everysec window flushes every acked INCR to durable storage. + std::thread::sleep(Duration::from_millis(2000)); + + // The compacted base must physically exist (rewrite was not a no-op). + assert!( + compacted_base_exists(&dir), + "no seq>1 base RDB found under {}/appendonlydir — the per-shard \ + rewrite did not produce a compacted base (silent no-op?)", + dir.display() + ); + + sigkill(&mut child); + + // -- Round 2 (recovery) --------------------------------------------- + let mut child2 = start_moon(port, &dir); + wait_for_port(port); + + let got_a = redis_get_i64(port, key_a); + let got_b = redis_get_i64(port, key_b); + + sigkill(&mut child2); + + assert_eq!( + got_a, N, + "CRASH-02-STRADDLE: {} recovered to {} (expected {}). \ + <{} = dropped writes across the rewrite boundary; \ + >{} = double-applied pre-snapshot INCRs (incr replayed on top of a \ + base that already contains them).", + key_a, got_a, N, N, N + ); + assert_eq!( + got_b, N, + "CRASH-02-STRADDLE: {} recovered to {} (expected {}). \ + <{} = dropped writes; >{} = double-applied INCRs.", + key_b, got_b, N, N, N + ); + + let _ = std::fs::remove_dir_all(&dir); +} + +/// CRASH-02-COMPOSE: base + post-rewrite incr must compose exactly once. +/// +/// INCR a counter `pre` times, BGREWRITEAOF (folds the `pre` INCRs into the new +/// base), wait for the fold to settle, then INCR `post` more times (these land +/// only in the new incr). After crash+recovery the counter must equal +/// `pre + post` — proving the new base (snapshot of the pre INCRs) and the new +/// incr (post INCRs) replay together without dropping or double-counting the +/// boundary. +#[test] +#[ignore] // Requires built release binary + redis-cli; run explicitly. +fn bgrewriteaof_base_plus_incr_recovers_exact() { + const PRE: i64 = 300; + const POST: i64 = 200; + + let port = unique_port().saturating_add(1); + let dir = unique_dir("compose"); + std::fs::create_dir_all(&dir).expect("create test dir"); + + // -- Round 1 -------------------------------------------------------- + let mut child = start_moon(port, &dir); + wait_for_port(port); + + let key = "compose:{a}"; + + for _ in 0..PRE { + redis_incr(port, key); + } + + let reply = bgrewriteaof(port); + assert!(!reply.is_empty(), "BGREWRITEAOF returned empty reply"); + + // Let the fold complete + writers reopen to the new incr before the POST + // INCRs, so POST writes land cleanly in the new generation's incr. + std::thread::sleep(Duration::from_millis(1500)); + + assert!( + compacted_base_exists(&dir), + "no seq>1 base RDB found — rewrite did not compact" + ); + + for _ in 0..POST { + redis_incr(port, key); + } + + // Quiesce so the everysec window flushes the POST INCRs. + std::thread::sleep(Duration::from_millis(1500)); + + sigkill(&mut child); + + // -- Round 2 (recovery) --------------------------------------------- + let mut child2 = start_moon(port, &dir); + wait_for_port(port); + + let got = redis_get_i64(port, key); + + sigkill(&mut child2); + + assert_eq!( + got, + PRE + POST, + "CRASH-02-COMPOSE: {} recovered to {} (expected {}+{}={}). \ + A value of {} means the pre-rewrite INCRs were double-applied \ + (base + replayed old incr); a value of {} means the post-rewrite \ + INCRs in the new incr were lost.", + key, + got, + PRE, + POST, + PRE + POST, + PRE + PRE + POST, + PRE, + ); + + let _ = std::fs::remove_dir_all(&dir); +} diff --git a/tests/crash_recovery_disk_offload_no_aof.rs b/tests/crash_recovery_disk_offload_no_aof.rs new file mode 100644 index 000000000..45d96de48 --- /dev/null +++ b/tests/crash_recovery_disk_offload_no_aof.rs @@ -0,0 +1,425 @@ +//! CRASH-COLD-NOAOF: disk-offload cold recovery WITHOUT AOF (#22 regression). +//! +//! Reproduces and guards the #22 durability bug: with `--appendonly no` and +//! `--disk-offload enable`, cold (spilled-to-disk) keys must survive a hard +//! crash and be served via cold read-through after restart — driven PURELY by +//! the per-shard manifest, with NO AOF replay (there is no AOF under +//! `appendonly no`). +//! +//! Root cause (main.rs): `persistence_dir` was derived as `Some(..)` only when +//! `appendonly == "yes" || save.is_some()`. Under `appendonly no` it was `None`, +//! and the shard-construction closure gated `restore_from_persistence` behind +//! `if let Some(ref dir) = persistence_dir { .. }` — so the v3 cold rebuild was +//! skipped entirely even though `disk_offload_base` was `Some`. Cold keys +//! recovered 0/200. The fix fires recovery when +//! `persistence_dir.is_some() || disk_offload_base.is_some()`. +//! +//! Discriminating signal (RED vs GREEN): +//! * `heap-*.mpf` files on disk prove the probes were durably spilled to cold. +//! * POST-crash read-through proves recovery re-attached the cold index. +//! RED (pre-#22-fix binary): heap files present, POST == ~0 (recovery skipped). +//! GREEN (post-#22-fix): heap files present, POST == ~PROBE_COUNT. +//! Because there is NO AOF under `appendonly no`, a non-zero POST count can ONLY +//! come from the cold-manifest recovery path the #22 fix enables. +//! +//! NOTE: we deliberately do NOT read the probes back PRE-crash. A cold GET +//! PROMOTES the key back to the hot tier (db.rs cold_read_through), and hot keys +//! are NOT durable under `appendonly no` — a pre-crash read-through would pull +//! probes out of cold and lose them on crash, corrupting the measurement. +//! +//! Run with (monoio default — matches CI): +//! cargo build --release +//! cargo test --release --test crash_recovery_disk_offload_no_aof -- --ignored +//! +//! tokio runtime: +//! cargo build --release --no-default-features \ +//! --features runtime-tokio,jemalloc,graph,text-index +//! cargo test --release --no-default-features \ +//! --features runtime-tokio,jemalloc,graph,text-index \ +//! --test crash_recovery_disk_offload_no_aof -- --ignored +//! +//! Requires: built release binary, `redis-cli` on PATH. + +#![cfg(any(feature = "runtime-monoio", feature = "runtime-tokio"))] + +use std::io::Write; +use std::process::{Child, Command, Stdio}; +use std::time::Duration; + +const PROBE_COUNT: usize = 200; +const PROBE_VALUE_LEN: usize = 500; +/// Filler keys written after the probes to drive memory past the disk-offload +/// threshold, forcing the (older) probe keys to be evicted to the cold tier. +const FILLER_COUNT: usize = 16_000; +const FILLER_VALUE_LEN: usize = 600; +/// 8 MiB total across 4 shards (2 MiB/shard). disk-offload spills at +/// 0.85 × maxmemory; probes (~100 KiB) + filler (~9.6 MiB) >> threshold. +const MAXMEMORY_BYTES: usize = 8 * 1024 * 1024; +const SHARDS: usize = 4; +/// Post-crash recovery floor. This test catches the *categorical* #22 +/// regression: when recovery is skipped the cold tier recovers a hard 0; when +/// it fires it recovers most of the cold probes. The spread is intrinsic to +/// crashing under `appendonly no`, where a cold key only survives once its +/// eviction-tick manifest commit has landed (no per-write durability without +/// AOF) — and it WIDENS under CPU contention (4 shards on a 6-core CI box +/// running the filler load + harness), where the spill/manifest threads drain +/// fewer ticks before the kill. Observed tail under that contention dipped to +/// 143/200 (still 143× above the RED 0 — recovery plainly fired). So the floor +/// is 65%: robustly inside the legitimate GREEN band, 130× above the RED 0, and +/// no longer clipping the load-induced lower tail. The `settle` sleep before the +/// kill is sized in tandem (see SETTLE_BEFORE_KILL) to drain most ticks first. +const RECOVERY_FLOOR: usize = (PROBE_COUNT * 65) / 100; +/// Seconds to let the async spill/manifest ticks drain before the SIGKILL. Under +/// `appendonly no` only tick-committed cold keys survive a crash; a longer settle +/// lands more commits and tightens the recovery distribution upward. Sized for +/// the contended multishard case (8s) — raising it trades test time for a higher, +/// tighter GREEN band. +const SETTLE_BEFORE_KILL: u64 = 8; + +fn unique_port() -> u16 { + use std::net::TcpListener; + let listener = TcpListener::bind("127.0.0.1:0").expect("bind to port 0"); + let port = listener.local_addr().expect("local addr").port(); + drop(listener); + port +} + +fn unique_dir(suffix: &str) -> std::path::PathBuf { + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0); + std::env::temp_dir().join(format!( + "moon-cold-noaof-{}-{}-{}", + std::process::id(), + suffix, + nanos + )) +} + +fn start_moon(port: u16, dir: &std::path::Path) -> Child { + let off_dir = dir.join("off"); + std::fs::create_dir_all(&off_dir).expect("create off dir"); + Command::new("./target/release/moon") + .args([ + "--port", + &port.to_string(), + "--shards", + &SHARDS.to_string(), + "--maxmemory", + &MAXMEMORY_BYTES.to_string(), + "--maxmemory-policy", + "allkeys-lru", + "--disk-offload", + "enable", + "--disk-offload-dir", + off_dir.to_str().expect("off dir utf8"), + // The bug under test: NO AOF. Cold recovery must work anyway. + "--appendonly", + "no", + "--cold-orphan-sweep-interval-secs", + "60", + "--dir", + ]) + .arg(dir) + // Captured to a log file so a CI flake produces a real diagnostic + // (see feedback_silenced_child_stdio_flake — never Stdio::null()). + .stdout(std::fs::File::create(dir.join("moon.stdout.log")).expect("create moon stdout log")) + .stderr(std::fs::File::create(dir.join("moon.stderr.log")).expect("create moon stderr log")) + .spawn() + .expect("spawn moon (run `cargo build --release` with default features first)") +} + +fn wait_for_port(port: u16) { + for _ in 0..80 { + if std::net::TcpStream::connect(format!("127.0.0.1:{}", port)).is_ok() { + std::thread::sleep(Duration::from_millis(200)); + return; + } + std::thread::sleep(Duration::from_millis(100)); + } + panic!("moon did not start within 8s on port {}", port); +} + +/// Wait until the previous server on `port` is FULLY down before restarting on +/// it. The crash test restarts on the SAME port immediately after a SIGKILL; +/// without this, round 2 races round 1's listener teardown and hits "Address +/// already in use", and round 1's lingering per-shard SO_REUSEPORT listeners +/// keep load-balancing some connections — so the post-crash probes read 0 even +/// though recovery itself fully succeeded (rebuilt the cold index on every +/// shard). That presented as a tokio-specific flake purely because of OS +/// port-release timing, not a recovery defect. +/// +/// IMPORTANT: a bind-based "is it free?" check is useless here — moon binds with +/// SO_REUSEPORT, so a plain `TcpListener::bind` SUCCEEDS even while round 1 still +/// holds the port. The reliable signal is the opposite: poll until a `connect` +/// is REFUSED, meaning no listener (round 1's) is accepting any more. +fn wait_for_port_down(port: u16) { + let addr = format!("127.0.0.1:{}", port); + let mut consecutive_refused = 0; + for _ in 0..120 { + match std::net::TcpStream::connect_timeout( + &addr.parse().expect("addr"), + Duration::from_millis(100), + ) { + Ok(_) => { + // Something is still accepting (round 1 not fully gone yet). + consecutive_refused = 0; + std::thread::sleep(Duration::from_millis(100)); + } + Err(_) => { + // Require two consecutive refusals to avoid a transient gap + // between two of round 1's per-shard listeners. + consecutive_refused += 1; + if consecutive_refused >= 2 { + return; + } + std::thread::sleep(Duration::from_millis(50)); + } + } + } + // Fall through after ~12s: let the restart proceed and surface its own error. +} + +/// Restart attempts for round 2. The central tokio listener plain-binds the +/// port; on a rapid SIGKILL→restart it can race the dying round-1 process's +/// socket teardown, hit `Address already in use`, and self-terminate +/// (`Listener error` → `Server shut down`). That is a transient OS port-release +/// race on fast restart, NOT a recovery defect — the attempt that DOES bind +/// recovers the cold tier fully. Detect the self-terminated restart and retry, +/// bounded. The recovery assertion (`post >= RECOVERY_FLOOR`) still has to pass +/// on the attempt that binds, so retrying works around the race WITHOUT +/// weakening the durability signal. +const RESTART_ATTEMPTS: usize = 6; + +/// Start moon and return a child that is BOTH alive and accepting on `port`. +/// Retries if the freshly-spawned server self-terminates on a transient rebind +/// EADDRINUSE (see `RESTART_ATTEMPTS`). Panics only if every attempt fails to +/// come up — that would be a real start failure, not the benign rebind race. +fn start_moon_alive(port: u16, dir: &std::path::Path) -> Child { + for attempt in 1..=RESTART_ATTEMPTS { + let mut child = start_moon(port, dir); + let mut up = false; + // Poll up to ~8s for the server to either accept or self-terminate. + for _ in 0..80 { + // Did the server exit on its own (rebind EADDRINUSE self-shutdown)? + if let Ok(Some(_status)) = child.try_wait() { + break; // self-terminated — fall through to retry + } + if std::net::TcpStream::connect(format!("127.0.0.1:{}", port)).is_ok() { + std::thread::sleep(Duration::from_millis(200)); + up = true; + break; + } + std::thread::sleep(Duration::from_millis(100)); + } + if up { + return child; + } + // Reap (no-op if already self-terminated) and back off so the kernel + // finishes releasing the port before the next rebind attempt. + let _ = child.kill(); + let _ = child.wait(); + if attempt < RESTART_ATTEMPTS { + std::thread::sleep(Duration::from_millis(300)); + } + } + panic!( + "moon failed to start+serve on port {} after {} attempts \ + (not the rebind race — a real start failure)", + port, RESTART_ATTEMPTS + ); +} + +fn probe_key(i: usize) -> String { + format!("probe:{}", i) +} + +fn redis_set(port: u16, key: &str, value: &str) { + let out = Command::new("redis-cli") + .args(["-p", &port.to_string(), "SET", key, value]) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .expect("redis-cli SET"); + assert!( + out.status.success(), + "redis-cli SET {} failed: {}", + key, + String::from_utf8_lossy(&out.stderr) + ); +} + +fn redis_get(port: u16, key: &str) -> Option { + let out = Command::new("redis-cli") + .args(["-p", &port.to_string(), "GET", key]) + .output() + .expect("redis-cli GET"); + if !out.status.success() { + return None; + } + let s = String::from_utf8_lossy(&out.stdout).trim().to_string(); + if s.is_empty() || s == "(nil)" { + None + } else { + Some(s) + } +} + +/// Write `FILLER_COUNT` distinct keys via pipelined TCP SETs to push memory +/// past the disk-offload threshold and evict the (older) probe keys to cold. +fn write_filler(port: u16) { + let mut stream = + std::net::TcpStream::connect(format!("127.0.0.1:{}", port)).expect("connect for filler"); + stream.set_write_timeout(Some(Duration::from_secs(30))).ok(); + let val = "F".repeat(FILLER_VALUE_LEN); + let mut buf: Vec = Vec::with_capacity(64 * 1024); + for i in 0..FILLER_COUNT { + let key = format!("filler:{}", i); + let cmd = format!( + "*3\r\n$3\r\nSET\r\n${}\r\n{}\r\n${}\r\n{}\r\n", + key.len(), + key, + val.len(), + val + ); + buf.extend_from_slice(cmd.as_bytes()); + // Flush in ~64 KiB chunks so the server applies eviction incrementally + // rather than receiving one giant burst that overruns the spill queue. + if buf.len() >= 64 * 1024 { + stream.write_all(&buf).expect("filler write"); + buf.clear(); + } + } + if !buf.is_empty() { + stream.write_all(&buf).expect("filler tail write"); + } + stream.flush().ok(); + // Drain is unnecessary for correctness here; closing the stream is fine. +} + +fn count_heap_files(dir: &std::path::Path) -> usize { + let off = dir.join("off"); + fn walk(p: &std::path::Path, acc: &mut usize) { + if let Ok(rd) = std::fs::read_dir(p) { + for e in rd.flatten() { + let path = e.path(); + if path.is_dir() { + walk(&path, acc); + } else if path + .file_name() + .and_then(|n| n.to_str()) + .map(|n| n.starts_with("heap-") && n.ends_with(".mpf")) + .unwrap_or(false) + { + *acc += 1; + } + } + } + } + let mut acc = 0; + walk(&off, &mut acc); + acc +} + +fn count_probes_readable(port: u16) -> usize { + (0..PROBE_COUNT) + .filter(|&i| redis_get(port, &probe_key(i)).is_some()) + .count() +} + +#[cfg(unix)] +fn sigkill(child: &mut Child) { + let pid = child.id() as i32; + unsafe { + libc::kill(pid, libc::SIGKILL); + } + let _ = child.wait(); +} + +#[cfg(not(unix))] +fn sigkill(child: &mut Child) { + let _ = child.kill(); + let _ = child.wait(); +} + +/// #22: cold keys spilled under `--appendonly no --disk-offload enable` must +/// recover after a SIGKILL crash via the per-shard manifest (no AOF). +#[test] +#[ignore] // Requires built release binary + redis-cli; run explicitly. +fn cold_keys_recover_after_crash_without_aof() { + let port = unique_port(); + let dir = unique_dir("c22"); + std::fs::create_dir_all(&dir).expect("create test dir"); + + // -- Round 1: populate + force cold spill ------------------------------- + let mut child = start_moon(port, &dir); + wait_for_port(port); + + let probe_val = "P".repeat(PROBE_VALUE_LEN); + for i in 0..PROBE_COUNT { + redis_set(port, &probe_key(i), &probe_val); + } + // Filler evicts the older probes to the cold tier. + write_filler(port); + // Let the async spill thread drain its queue and commit manifests. Under + // `appendonly no` a cold key is only crash-durable once its eviction-tick + // manifest commit lands, so give the ticks ample time before the kill. + std::thread::sleep(Duration::from_secs(SETTLE_BEFORE_KILL)); + + // NOTE: do NOT read the probes here — a cold GET promotes the key back to + // hot, and hot is not durable under `appendonly no`, which would corrupt + // the POST measurement. `heap_files > 0` is the spill-happened proof. + let heap_files = count_heap_files(&dir); + + // SIGKILL — hard crash, no graceful drain. + sigkill(&mut child); + // Wait until round 1 is fully down (port no longer accepting) before + // restarting on it — otherwise round 2 races round 1's SO_REUSEPORT listener + // teardown and the probes read round 1's empty post-eviction hot tier as 0 + // despite a fully successful recovery. This was the entire source of the + // apparent tokio flake. + wait_for_port_down(port); + + // -- Round 2: recover --------------------------------------------------- + // start_moon_alive retries the rebind if the central listener loses the + // port-release race and self-terminates (transient EADDRINUSE on fast + // restart) — the recovery assertion below still has to pass on the attempt + // that binds, so this does not weaken the durability signal. + let mut child2 = start_moon_alive(port, &dir); + // Give recovery a beat to re-attach the cold index before probing. + std::thread::sleep(Duration::from_secs(1)); + + let post = count_probes_readable(port); + + sigkill(&mut child2); + + // Setup sanity: if nothing spilled, the test exercised nothing — fail loud + // (a maxmemory/eviction misconfig, not a recovery pass). + assert!( + heap_files > 0, + "test setup: expected cold spill (heap-*.mpf) but found 0 — eviction never fired \ + (maxmemory/threshold too high). post={}", + post + ); + + // The actual #22 assertion: cold keys recovered WITHOUT AOF. + // Clean up ONLY on success so a failure keeps moon.std*.log for diagnosis + // (matches the crash_matrix convention: cleanup after the assert). + let recovered = post >= RECOVERY_FLOOR; + if recovered { + let _ = std::fs::remove_dir_all(&dir); + } + assert!( + recovered, + "#22 REGRESSION: post-crash cold read-through {}/{} (floor {}). heap_files={}. \ + Logs kept at {} for diagnosis. A POST near 0 with cold files on disk means \ + restore_from_persistence was skipped under `appendonly no` — the persistence_dir \ + gate regressed (recovery must fire on disk_offload_base.is_some()).", + post, + PROBE_COUNT, + RECOVERY_FLOOR, + heap_files, + dir.display() + ); +} diff --git a/tests/ft_search_as_of_boundary.rs b/tests/ft_search_as_of_boundary.rs index abcd60b1d..7bc29d104 100644 --- a/tests/ft_search_as_of_boundary.rs +++ b/tests/ft_search_as_of_boundary.rs @@ -43,7 +43,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -109,6 +109,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/ft_search_as_of_filter.rs b/tests/ft_search_as_of_filter.rs index 01126e3ec..c68e1e263 100644 --- a/tests/ft_search_as_of_filter.rs +++ b/tests/ft_search_as_of_filter.rs @@ -51,7 +51,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -117,6 +117,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/ft_search_concurrent_readers.rs b/tests/ft_search_concurrent_readers.rs index d1848d32a..f0ce851d3 100644 --- a/tests/ft_search_concurrent_readers.rs +++ b/tests/ft_search_concurrent_readers.rs @@ -40,7 +40,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -106,6 +106,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/ft_search_multi_shard_as_of.rs b/tests/ft_search_multi_shard_as_of.rs index f3fc89d1b..776b9e928 100644 --- a/tests/ft_search_multi_shard_as_of.rs +++ b/tests/ft_search_multi_shard_as_of.rs @@ -52,7 +52,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -120,6 +120,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/ft_search_temporal_parity.rs b/tests/ft_search_temporal_parity.rs index 7bd34aea8..67679a3f7 100644 --- a/tests/ft_search_temporal_parity.rs +++ b/tests/ft_search_temporal_parity.rs @@ -64,7 +64,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -132,6 +132,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/graph_bench_compare.rs b/tests/graph_bench_compare.rs index 789ed608c..5615847ee 100644 --- a/tests/graph_bench_compare.rs +++ b/tests/graph_bench_compare.rs @@ -48,7 +48,7 @@ async fn start_moon() -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -106,6 +106,7 @@ async fn start_moon() -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { diff --git a/tests/graph_bench_e2e.rs b/tests/graph_bench_e2e.rs index 136137eb8..26d7315b8 100644 --- a/tests/graph_bench_e2e.rs +++ b/tests/graph_bench_e2e.rs @@ -34,7 +34,7 @@ async fn start_server() -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -92,6 +92,7 @@ async fn start_server() -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { diff --git a/tests/graph_integration.rs b/tests/graph_integration.rs index 0afd3df3c..60b319dd2 100644 --- a/tests/graph_integration.rs +++ b/tests/graph_integration.rs @@ -34,7 +34,7 @@ async fn start_graph_server() -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -92,6 +92,7 @@ async fn start_graph_server() -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { diff --git a/tests/graph_stress_deep.rs b/tests/graph_stress_deep.rs index 9e30032ce..57b312e0a 100644 --- a/tests/graph_stress_deep.rs +++ b/tests/graph_stress_deep.rs @@ -37,7 +37,7 @@ async fn start_server() -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -95,6 +95,7 @@ async fn start_server() -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { diff --git a/tests/integration.rs b/tests/integration.rs index 9b10d8920..c5d66787b 100644 --- a/tests/integration.rs +++ b/tests/integration.rs @@ -38,7 +38,7 @@ async fn start_server() -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -106,6 +106,7 @@ async fn start_server() -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { @@ -141,7 +142,7 @@ async fn start_server_with_pass(password: &str) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -209,6 +210,7 @@ async fn start_server_with_pass(password: &str) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { @@ -1316,7 +1318,7 @@ async fn start_server_with_persistence( dir: dir.to_string_lossy().to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -1384,6 +1386,7 @@ async fn start_server_with_persistence( migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { @@ -2203,7 +2206,7 @@ async fn start_server_with_maxmemory(maxmemory: usize, policy: &str) -> (u16, Ca dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory, + maxmemory: Some(maxmemory), maxmemory_policy: policy.to_string(), maxmemory_samples: 5, shards: 0, @@ -2271,6 +2274,7 @@ async fn start_server_with_maxmemory(maxmemory: usize, policy: &str) -> (u16, Ca migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { @@ -2617,7 +2621,7 @@ async fn start_sharded_server(num_shards: usize) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -2685,6 +2689,7 @@ async fn start_sharded_server(num_shards: usize) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); @@ -3811,7 +3816,7 @@ async fn start_cluster_server() -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 1, @@ -3879,6 +3884,7 @@ async fn start_cluster_server() -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; std::thread::spawn(move || { @@ -4476,7 +4482,7 @@ async fn start_server_with_aclfile(acl_path: &str) -> (u16, CancellationToken) { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -4544,6 +4550,7 @@ async fn start_server_with_aclfile(acl_path: &str) -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { diff --git a/tests/kill_snapshot.rs b/tests/kill_snapshot.rs index 51f870f52..4fc946eb3 100644 --- a/tests/kill_snapshot.rs +++ b/tests/kill_snapshot.rs @@ -34,7 +34,7 @@ fn base_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -103,6 +103,7 @@ fn base_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_to: None, migrate_aof_shards: 0, vec_warm_mmap_budget: "2gb".to_string(), + ..Default::default() } } diff --git a/tests/lunaris_cypher_shortest_path.rs b/tests/lunaris_cypher_shortest_path.rs index 86ba146f3..707d4baa3 100644 --- a/tests/lunaris_cypher_shortest_path.rs +++ b/tests/lunaris_cypher_shortest_path.rs @@ -92,7 +92,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -158,6 +158,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/lunaris_cypher_temporal.rs b/tests/lunaris_cypher_temporal.rs index 345da2cbf..1de442a5a 100644 --- a/tests/lunaris_cypher_temporal.rs +++ b/tests/lunaris_cypher_temporal.rs @@ -101,7 +101,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -167,6 +167,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/lunaris_hybrid_ft_search.rs b/tests/lunaris_hybrid_ft_search.rs index dff7e88bb..d812ed56e 100644 --- a/tests/lunaris_hybrid_ft_search.rs +++ b/tests/lunaris_hybrid_ft_search.rs @@ -78,7 +78,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -145,6 +145,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/mq_integration.rs b/tests/mq_integration.rs index 49b9da573..47ff32616 100644 --- a/tests/mq_integration.rs +++ b/tests/mq_integration.rs @@ -46,11 +46,13 @@ async fn start_mq_server(num_shards: usize) -> (u16, CancellationToken) { appendonly: "no".to_string(), unsafe_multishard_aof: false, appendfsync: "everysec".to_string(), + aof_fsync_timeout_ms: 2000, + experimental_per_shard_rewrite: false, save: None, dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, diff --git a/tests/multishard_serve_smoke.rs b/tests/multishard_serve_smoke.rs new file mode 100644 index 000000000..5c880bd97 --- /dev/null +++ b/tests/multishard_serve_smoke.rs @@ -0,0 +1,203 @@ +//! MULTISHARD-SERVE-SMOKE: the server must actually SERVE commands in multishard +//! mode on every supported runtime/platform. +//! +//! Regression guard for the tokio-on-Linux multishard accept hang (2026-06-03). +//! Root cause: the central tokio listener plain-bound the port (NO SO_REUSEPORT) +//! while each shard binds its own SO_REUSEPORT listener — incompatible sockets on +//! the same port, decided by a bind-order RACE. Whichever way it resolved, the +//! server stopped serving: if the shards won the race the central plain-bind hit +//! EADDRINUSE and `run_sharded` returned Err, tearing down the WHOLE server +//! ("Server shut down"); if the central won, the shards' REUSEPORT binds failed +//! and the accept path was left half-wired. Either way `PING` never got a reply — +//! the "zombie eating RAM" signature (port up, shards parked, nothing dispatched). +//! Fix: the central listener also binds SO_REUSEPORT (server/listener.rs), so it +//! COEXISTS with the shard sockets regardless of order; the kernel load-balances +//! accepts across all of them and every one of them actively accepts. +//! +//! This test boots the real binary in multishard configs and asserts a raw-TCP +//! inline `PING` is answered with `+PONG` inside a hard deadline. It uses +//! `CARGO_BIN_EXE_moon` (cargo auto-builds the bin) and raw TCP (no redis-cli +//! dependency) so it runs unmodified in CI and fails RED — not hangs — on the bug. +//! +//! NOT `#[ignore]`: this must run in the default `cargo test` CI matrix on BOTH +//! `runtime-tokio` and `runtime-monoio` so the accept path can never silently +//! regress again. + +#![cfg(any(feature = "runtime-monoio", feature = "runtime-tokio"))] + +use std::io::{Read, Write}; +use std::net::{TcpStream, ToSocketAddrs}; +use std::process::{Child, Command}; +use std::time::{Duration, Instant}; + +fn moon_binary() -> std::path::PathBuf { + std::path::PathBuf::from(env!("CARGO_BIN_EXE_moon")) +} + +/// Pick an ephemeral port by binding :0 and releasing it. Good enough for a +/// short-lived test; the server rebinds immediately. +fn free_port() -> u16 { + let l = std::net::TcpListener::bind("127.0.0.1:0").expect("bind :0"); + let p = l.local_addr().expect("local_addr").port(); + drop(l); + p +} + +fn spawn_moon(port: u16, dir: &std::path::Path, extra: &[&str]) -> Child { + let mut args: Vec = vec![ + "--port".into(), + port.to_string(), + "--dir".into(), + dir.to_string_lossy().into_owned(), + ]; + for e in extra { + args.push((*e).into()); + } + Command::new(moon_binary()) + .args(&args) + // Pipe to a log file so a CI failure has a diagnostic (never Stdio::null()). + .stdout(std::fs::File::create(dir.join("moon.stdout.log")).expect("create stdout log")) + .stderr(std::fs::File::create(dir.join("moon.stderr.log")).expect("create stderr log")) + .spawn() + .expect("spawn moon (CARGO_BIN_EXE_moon)") +} + +/// Connect + inline PING, return true iff we read `+PONG` within `deadline`. +/// Distinguishes "not started yet" (connect refused → retry) from "accepted TCP +/// but no reply" (the hang → keep trying until the deadline, then false). +fn ping_ok(port: u16, deadline: Duration) -> bool { + let addr = format!("127.0.0.1:{port}") + .to_socket_addrs() + .expect("addr") + .next() + .expect("one addr"); + let start = Instant::now(); + while start.elapsed() < deadline { + match TcpStream::connect_timeout(&addr, Duration::from_millis(200)) { + Ok(mut s) => { + s.set_read_timeout(Some(Duration::from_millis(500))).ok(); + s.set_write_timeout(Some(Duration::from_millis(500))).ok(); + if s.write_all(b"PING\r\n").is_ok() { + let mut buf = [0u8; 16]; + if let Ok(n) = s.read(&mut buf) + && n > 0 + && buf[..n].windows(4).any(|w| w == b"PONG") + { + return true; + } + } + // Connected but no PONG (the hang) — drop, brief wait, retry. + std::thread::sleep(Duration::from_millis(150)); + } + Err(_) => std::thread::sleep(Duration::from_millis(100)), + } + } + false +} + +/// Shard count for the multishard cases. Capped at the number of available +/// cores so the test cannot oversubscribe the runtime on a small CI runner: +/// BOTH runtimes (thread-per-core) collapse serving once shards >= cores (the +/// documented oversubscription wall — see CLAUDE.md gotchas), which is a +/// SEPARATE phenomenon from the bind race this test guards. Floor of 2 keeps it +/// genuinely multishard (central + >=2 shard SO_REUSEPORT sockets is the full +/// coexistence surface; higher counts add no race surface, only oversubscription +/// confound). 4 on the dev box, scales down on 2-core runners. +fn safe_shards() -> usize { + std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(2) + .clamp(2, 4) +} + +fn run_serves(label: &str, extra: &[&str]) { + let port = free_port(); + let dir = std::env::temp_dir().join(format!( + "moon-serve-smoke-{}-{}-{}", + std::process::id(), + port, + label, + )); + std::fs::create_dir_all(dir.join("off")).expect("mk dir"); + + let mut child = spawn_moon(port, &dir, extra); + // Hard deadline: a healthy server answers within ~1s; the hang never answers. + let ok = ping_ok(port, Duration::from_secs(15)); + + let _ = child.kill(); + let _ = child.wait(); + + if ok { + let _ = std::fs::remove_dir_all(&dir); + } + assert!( + ok, + "[{label}] multishard server did not answer PING with +PONG within 15s — \ + the central listener / per-shard SO_REUSEPORT accept path is not serving. \ + Logs kept at {}", + dir.display() + ); +} + +// A small explicit --maxmemory is REQUIRED on every smoke case. With it unset, +// the memory guardrail auto-caps maxmemory at ~80% of host RAM, and each shard +// EAGERLY pre-allocates a PageCache frame pool sized to maxmemory/shards AT +// STARTUP (≈3.8 GB for a 4-shard server on a 6 GB box, before serving anything). +// That makes the "smoke" test a multi-GB monster that OOMs small CI runners. The +// bind race this test guards is bind-time / REUSEPORT-coexistence — entirely +// independent of maxmemory — so a 128 MB cap keeps the guard valid (~112 MB RSS) +// while keeping the test light. (PageCache eager pre-alloc is tracked separately.) +const SMOKE_MAXMEMORY: &str = "134217728"; // 128 MiB + +#[test] +fn serves_multishard_plain() { + let shards = safe_shards().to_string(); + run_serves( + "multishard-plain", + &[ + "--shards", + &shards, + "--appendonly", + "no", + "--maxmemory", + SMOKE_MAXMEMORY, + ], + ); +} + +#[test] +fn serves_1shard_plain() { + run_serves( + "1shard-plain", + &[ + "--shards", + "1", + "--appendonly", + "no", + "--maxmemory", + SMOKE_MAXMEMORY, + ], + ); +} + +#[test] +fn serves_multishard_disk_offload() { + // The exact surface the durability work targets: multishard + disk-offload + + // appendonly=no. Must serve commands, not just bind the port. + let shards = safe_shards().to_string(); + run_serves( + "multishard-diskoffload", + &[ + "--shards", + &shards, + "--maxmemory", + "8388608", + "--maxmemory-policy", + "allkeys-lru", + "--disk-offload", + "enable", + "--appendonly", + "no", + ], + ); +} diff --git a/tests/pipeline_auto_index.rs b/tests/pipeline_auto_index.rs index 8d20fa713..7ae7506e3 100644 --- a/tests/pipeline_auto_index.rs +++ b/tests/pipeline_auto_index.rs @@ -48,7 +48,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -115,6 +115,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/replication_test.rs b/tests/replication_test.rs index 55e1ebee7..cbc588fa7 100644 --- a/tests/replication_test.rs +++ b/tests/replication_test.rs @@ -36,7 +36,7 @@ async fn start_server() -> (u16, CancellationToken) { dir: dir_path, dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 0, @@ -104,6 +104,7 @@ async fn start_server() -> (u16, CancellationToken) { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; tokio::spawn(async move { diff --git a/tests/txn_completeness_edge_cases.rs b/tests/txn_completeness_edge_cases.rs index 9199736e0..49a283906 100644 --- a/tests/txn_completeness_edge_cases.rs +++ b/tests/txn_completeness_edge_cases.rs @@ -52,7 +52,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -118,6 +118,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/txn_cypher_write_rollback.rs b/tests/txn_cypher_write_rollback.rs index fe108f863..e17f3a933 100644 --- a/tests/txn_cypher_write_rollback.rs +++ b/tests/txn_cypher_write_rollback.rs @@ -64,7 +64,7 @@ async fn start_txn_server(num_shards: usize, persistence_dir: &str) -> (u16, Can dir, dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -130,6 +130,7 @@ async fn start_txn_server(num_shards: usize, persistence_dir: &str) -> (u16, Can migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/txn_ft_search_snapshot.rs b/tests/txn_ft_search_snapshot.rs index f20b63a1d..4c371fd87 100644 --- a/tests/txn_ft_search_snapshot.rs +++ b/tests/txn_ft_search_snapshot.rs @@ -52,7 +52,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -120,6 +120,7 @@ fn build_config(port: u16, num_shards: usize) -> ServerConfig { migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() } } diff --git a/tests/txn_graph_wiring.rs b/tests/txn_graph_wiring.rs index a2d80fecc..a13b00cdb 100644 --- a/tests/txn_graph_wiring.rs +++ b/tests/txn_graph_wiring.rs @@ -74,7 +74,7 @@ async fn start_txn_server(num_shards: usize, persistence_dir: &str) -> (u16, Can dir, dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -140,6 +140,7 @@ async fn start_txn_server(num_shards: usize, persistence_dir: &str) -> (u16, Can migrate_aof_from: None, migrate_aof_to: None, migrate_aof_shards: 0, + ..Default::default() }; let cancel = token.clone(); diff --git a/tests/txn_kv_wiring.rs b/tests/txn_kv_wiring.rs index 3a04ff14d..9816908a5 100644 --- a/tests/txn_kv_wiring.rs +++ b/tests/txn_kv_wiring.rs @@ -50,12 +50,14 @@ async fn start_txn_server(num_shards: usize, persistence_dir: &str) -> (u16, Can requirepass: None, appendonly, unsafe_multishard_aof: false, + experimental_per_shard_rewrite: false, appendfsync: "everysec".to_string(), + aof_fsync_timeout_ms: 2000, save: None, dir, dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, diff --git a/tests/vacuum_commands.rs b/tests/vacuum_commands.rs index 86260b22c..4649c29c7 100644 --- a/tests/vacuum_commands.rs +++ b/tests/vacuum_commands.rs @@ -41,7 +41,7 @@ fn base_config(port: u16) -> ServerConfig { dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: 1, @@ -109,6 +109,7 @@ fn base_config(port: u16) -> ServerConfig { migrate_aof_to: None, migrate_aof_shards: 0, vec_warm_mmap_budget: "2gb".to_string(), + ..Default::default() } } diff --git a/tests/workspace_integration.rs b/tests/workspace_integration.rs index a32fe0c69..2543ce793 100644 --- a/tests/workspace_integration.rs +++ b/tests/workspace_integration.rs @@ -39,11 +39,13 @@ async fn start_workspace_server(num_shards: usize) -> (u16, CancellationToken) { appendonly: "no".to_string(), unsafe_multishard_aof: false, appendfsync: "everysec".to_string(), + aof_fsync_timeout_ms: 2000, + experimental_per_shard_rewrite: false, save: None, dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards, @@ -260,11 +262,13 @@ async fn start_workspace_server_with_auth( appendonly: "no".to_string(), unsafe_multishard_aof: false, appendfsync: "everysec".to_string(), + aof_fsync_timeout_ms: 2000, + experimental_per_shard_rewrite: false, save: None, dir: ".".to_string(), dbfilename: "dump.rdb".to_string(), appendfilename: "appendonly.aof".to_string(), - maxmemory: 0, + maxmemory: Some(0), maxmemory_policy: "noeviction".to_string(), maxmemory_samples: 5, shards: num_shards,