perf: RSS/CPU remediation wave 5 — mmap f16 sidecar, idle-adaptive AOF writer, hygiene batch

CHANGELOG.md

@@ -6,6 +6,99 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed — RSS/CPU remediation wave 5 (PR #TBD)
+
+- **Item A — mmap the exact-rerank f16 sidecar on segment reload**
+  (`src/vector/segment/raw_f16_store.rs`, new `RawF16Store` enum): a segment
+  reloaded from disk used to `fs::read` the entire `raw_f16.bin` sidecar into
+  a second heap `Vec<u16>`, doubling resident vector memory for
+  reload-heavy deployments (warm starts, segment promotion). Reload now
+  memory-maps the file (`memmap2`, already a workspace dependency) and hands
+  out a zero-copy `&[u16]` view backed by the kernel page cache — RSS only
+  grows for pages the rerank path actually touches. Freshly-built segments
+  (compaction/merge) are unaffected — they keep their owned buffer. Rerank
+  parity (Owned vs Mapped, byte-identical sidecar + identical `search()`
+  output) is pinned by
+  `test_reload_raw_f16_sidecar_uses_mmap_and_matches_owned_rerank`.
+- **Item A follow-up — text posting-list capacity reclaim**
+  (`src/text/posting.rs`): `PostingList::term_freqs`/`positions` grow to the
+  peak document count ever seen for a term and, per the existing
+  `remove_doc` contract, the `postings` HashMap entry is kept forever even
+  once a term has zero live documents. The buffers now `shrink_to_fit()`
+  once the last document leaves a posting, releasing peak capacity for
+  terms that go idle without changing the "entry survives" contract.
+- **Item B — AOF writer idle wake made adaptive** (`src/persistence/aof/writer_task.rs`,
+  new `IdleWait` state machine): the 3 steady-state writer loops that need a
+  bounded channel poll to service the EverySec proactive-fsync deadline
+  (TopLevel tokio, PerShard tokio, PerShard monoio — TopLevel monoio blocks
+  on an untimed `rx.recv()` and needed no change) used to poll at a FIXED
+  cadence forever (50ms monoio / 200ms tokio), waking an idle server's AOF
+  writer thread 5-20 times a second doing nothing. The wait now escalates
+  50ms → 250ms → 1s once a poll times out with nothing queued, and resets to
+  the floor the instant any message arrives — a real write always wakes the
+  loop immediately regardless of the current timeout, since the poll races
+  a message against the deadline. Escalation is refused (pinned at the
+  floor) whenever a write is buffered under `FsyncPolicy::EverySec` without
+  an immediate fsync, or `last_fsync` was manually back-dated (the F6
+  post-fold drain trick) — the ~1.2s EverySec bound is provably unchanged.
+  `FsyncPolicy::Always`/`No` have no such deadline and escalate freely once
+  idle.
+- **Item C1 — WAL v3 write buffer shrinks after an oversized flush**
+  (`src/persistence/wal_v3/segment.rs`): a single large record (e.g. a
+  FullPageImage) grew the 8KB write buffer to fit it, and `clear()` alone
+  never released that capacity — the peak allocation was pinned for the
+  writer's lifetime. `flush_write`/`rotate_segment` now `shrink_to` the
+  8KB default once capacity exceeds 4x that, a no-op for the common
+  small-record case.
+- **Item C2 — SearchScratch visited-set: already bitset-based (SKIP)**
+  (`src/vector/hnsw/search.rs`): the per-query search hot path already uses
+  a word-based `BitVec` (u64 words, `test_and_set`/`clear_all` memset),
+  thread-cached and reused across queries — no change needed. The other
+  `Vec<bool>` visited sets found in the vector module are all build-time/
+  compaction/merge-oracle code, not the per-query path; `search_sq.rs` in
+  particular carries an explicit comment warning that a prior BitVec
+  conversion there caused correctness issues, so it was left untouched.
+- **Item C3 — SmallVec the per-tick elastic-budget shard snapshot**
+  (`src/shard/shared_databases.rs`): `recompute_elastic_budget` (called
+  from every shard's 100ms eviction tick) `collect()`ed a fresh
+  `Vec<usize>` snapshot of all shards' published memory on every call.
+  Switched to `SmallVec<[usize; 16]>` — stack-only for the common <=16
+  shard case, unchanged single heap allocation beyond that.
+- **Item C4 — Lua script-cache byte estimate exposed via INFO/MEMORY
+  DOCTOR** (`src/scripting/cache.rs`, `ScriptCache::resident_bytes()`): the
+  per-shard Lua cache was invisible to observability — its growth folded
+  silently into "allocator overhead." Added a byte-estimate accounting
+  method, published per-shard via the existing C5/M4 `ShardStoreMemory`
+  tick pattern (new `lua` atomic), and surfaced in both the Prometheus
+  `moon_memory_bytes{kind="lua_scripts"}` gauge and `MEMORY DOCTOR`'s text
+  report. The cache itself remains intentionally unbounded (Redis parity —
+  `SCRIPT FLUSH` is the only eviction path); this is observability only.
+- **Item C5 — removed dead `parse_single_frame_zc` RESP parser**
+  (`src/protocol/parse.rs`): a full ~150-line RESP2/RESP3 parser
+  superseded by the current `validate_frame` + `parse_frame_zerocopy`
+  pipeline, with zero external callers (only self-recursion) — silently
+  masked by the file's `#![allow(dead_code)]`. Removed along with its
+  exclusively-private helper `read_decimal_zc`.
+- **Item C6 — jemalloc decay policy audited, docs added (SKIP code
+  change)** (`CLAUDE.md`): the baked-in `_rjem_malloc_conf` static and the
+  `--memory-arenas-cap` re-spawn override already carry byte-identical
+  `dirty_decay_ms:1000,muzzy_decay_ms:5000,background_thread:true` tuning
+  — no drift to reconcile. Added the missing operator-facing
+  `_RJEM_MALLOC_CONF` documentation (docs-only, no code changed).
+- **Item C7 — tokio 1ms shard tick idle cost audited (SKIP)**
+  (`src/shard/event_loop.rs`, `src/shard/spsc_handler.rs`): the 1ms
+  `periodic_interval` tick's SPSC drain is already a non-blocking,
+  zero-allocation `try_pop()` loop, and every downstream side effect is
+  already gated behind a cheap conditional. The one unconditional cost
+  (`cached_clock.update()`, a single `clock_gettime`) is the documented
+  "Timestamp caching" design. Unlike item B's AOF writer poll, this 1ms
+  cadence IS the low-latency WAL-flush contract (CLAUDE.md), not
+  incidental idle waste — escalating it would widen that bound. No code
+  change; a real fix would be event-driven WAL triggering, an
+  architectural change out of scope here.
+- Item C8 (sigterm readiness deadline) landed early via the Windows-CI PR
+  (#229) — see the CI section below.
+
 ### CI — fix Windows main-push test failures (PR #TBD)
 
 - `test_poll_real_process_smoke` is now gated to Linux/macOS: `get_rss_bytes()`

CLAUDE.md

@@ -96,6 +96,7 @@ orb run -m moon-dev bash -c 'sudo apt-get update -qq && sudo apt-get install -y
 - `MOON_URING_SPIN_US`, `MOON_URING_SQPOLL[_CPU]`, `MOON_URING_PLAIN` — io_uring-side experiment gates kept as documented diagnostics; all dead ends for the p=1 path (see `tmp/KV-FULLPROOF.md` Round 2).
 - `MOON_XSHARD_SPIN_BUDGET` / `MOON_XSHARD_SPIN_GATE` / `MOON_XSHARD_SPIN_MAX_CONNS` — diagnostic overrides for the C2 reply-side spin (`src/shard/slice.rs`; defaults 4096 iters / gate 2 / **solo-conn 1**). Budget `0` disables the spin entirely (the same-instance A/B knob that proved the c8P1 convoy). ⚠ The solo-conn ceiling (spin only when the conn is ALONE on its shard thread) is the L1 convoy fix — raising `MAX_CONNS` re-creates the s4 c8P1 collapse (a spinning conn starves its sibling AND the shard's SPSC drain, 0.45× vs Redis; fixed = 2.75× better, see `tmp/MULTISHARD-REDESIGN.md`). Bench-only knobs: never set in production.
 - `RUSTFLAGS="-C target-cpu=native"` — enable CPU-specific optimizations for benchmarking
+- `_RJEM_MALLOC_CONF` — jemalloc's tuning knob (prefixed because `tikv-jemallocator` builds with the `_rjem_` symbol prefix; the unprefixed `MALLOC_CONF` has no effect). Moon bakes in `narenas:8,background_thread:true,metadata_thp:auto,dirty_decay_ms:1000,muzzy_decay_ms:5000,abort_conf:true` via a static `_rjem_malloc_conf` export (`src/main.rs`) — 1s dirty-page decay + a background reclaim thread so freed-but-idle pages return to the OS quickly instead of sitting in jemalloc's dirty/muzzy caches inflating RSS. `--memory-arenas-cap N` re-spawns the process (`execve`) with this exact string, narenas substituted, **before** jemalloc's one-time init reads it — `mallctl` after init is a documented no-op for `opt.narenas`. If `_RJEM_MALLOC_CONF` is already set in the environment, `--memory-arenas-cap` is a no-op (operator env wins, warns instead of clobbering). Only applies to `--features jemalloc` builds; `mimalloc` (the non-jemalloc default) has no equivalent decay knob in this codebase.
 
 ## Key Design Decisions
 

src/admin/metrics_setup.rs

@@ -1360,6 +1360,7 @@ fn update_moon_memory_bytes() {
     let mut csr: usize = 0;
     let wal: usize = 0; // WalWriterV3 is stack-owned; not reachable here
     let mut backlog: usize = 0;
+    let mut lua: usize = 0;
 
     if let Some(shard_dbs) = get_global_shard_databases() {
         // KV memory: sum of per-shard published atomics. Lock-free.
@@ -1372,6 +1373,8 @@ fn update_moon_memory_bytes() {
             hnsw += mem.vector.load(Ordering::Relaxed);
             // graph is cfg-gated at publish time; the atomic is always present.
             csr += mem.graph.load(Ordering::Relaxed);
+            // C4 (wave-5 hygiene): Lua script-cache byte estimate.
+            lua += mem.lua.load(Ordering::Relaxed);
         }
     }
 
@@ -1382,7 +1385,7 @@ fn update_moon_memory_bytes() {
         }
     }
 
-    let other_sum = dashtable + hnsw + csr + wal + sealed + backlog;
+    let other_sum = dashtable + hnsw + csr + wal + sealed + backlog + lua;
     let alloc_overhead = rss.saturating_sub(other_sum);
 
     gauge!("moon_memory_bytes", "kind" => "dashtable").set(dashtable as f64);
@@ -1391,6 +1394,7 @@ fn update_moon_memory_bytes() {
     gauge!("moon_memory_bytes", "kind" => "wal").set(wal as f64);
     gauge!("moon_memory_bytes", "kind" => "sealed").set(sealed as f64);
     gauge!("moon_memory_bytes", "kind" => "replication_backlog").set(backlog as f64);
+    gauge!("moon_memory_bytes", "kind" => "lua_scripts").set(lua as f64);
     gauge!("moon_memory_bytes", "kind" => "allocator_overhead").set(alloc_overhead as f64);
 
     // Update the existing RSS gauge in the same snapshot so the integration

src/command/server_admin.rs

@@ -410,6 +410,7 @@ fn memory_doctor() -> Frame {
     #[cfg_attr(not(feature = "graph"), allow(unused_variables))]
     let csr_bytes: usize;
     let wal_bytes: usize = 0;
+    let lua_bytes: usize;
 
     if let Some(shard_dbs) = crate::admin::metrics_setup::get_global_shard_databases() {
         // KV memory: sum of per-shard published atomics. Lock-free.
@@ -418,16 +419,21 @@ fn memory_doctor() -> Frame {
         // Store memory: sum published per-shard vector/graph atomics.
         let mut vec_total = 0usize;
         let mut csr_total = 0usize;
+        let mut lua_total = 0usize;
         for mem in shard_dbs.store_memory_per_shard.iter() {
             vec_total += mem.vector.load(Ordering::Relaxed);
             csr_total += mem.graph.load(Ordering::Relaxed);
+            // C4 (wave-5 hygiene): Lua script-cache byte estimate.
+            lua_total += mem.lua.load(Ordering::Relaxed);
         }
         hnsw_bytes = vec_total;
         csr_bytes = csr_total;
+        lua_bytes = lua_total;
     } else {
         dashtable_bytes = 0;
         hnsw_bytes = 0;
         csr_bytes = 0;
+        lua_bytes = 0;
     }
 
     // Replication backlog via global state (same pattern as INFO replication).
@@ -440,8 +446,13 @@ fn memory_doctor() -> Frame {
     let (allocator_name, arena_count) = allocator_info();
 
     // ── Computed overhead ────────────────────────────────────────────────
-    let tracked_sum =
-        dashtable_bytes + hnsw_bytes + csr_bytes + wal_bytes + sealed_bytes + repl_bytes;
+    let tracked_sum = dashtable_bytes
+        + hnsw_bytes
+        + csr_bytes
+        + wal_bytes
+        + sealed_bytes
+        + repl_bytes
+        + lua_bytes;
     let allocator_overhead = rss.saturating_sub(tracked_sum);
 
     // ── VSZ ratio recommendation ─────────────────────────────────────────
@@ -515,6 +526,12 @@ fn memory_doctor() -> Frame {
         humanize_bytes(repl_bytes),
         pct(repl_bytes, rss)
     );
+    let _ = writeln!(
+        out,
+        "  Lua scripts:            {}  ({:.1}%)",
+        humanize_bytes(lua_bytes),
+        pct(lua_bytes, rss)
+    );
     let _ = writeln!(
         out,
         "  Allocator overhead:     {}  ({:.1}%)",