feat(memory): K4 unified resident-bytes accounting — text/cold-index planes + elastic budget + tier-ladder types (kernel M2 stage 2) - #297
Merged
Conversation
Add TermDictionary::resident_bytes() -- term string length + u32 id + a fixed HashMap bucket-overhead constant per entry, mirroring the approximation style already used by Database::entry_overhead (WS6) and graph::index::PropertyIndex::resident_bytes (serialized_size()). Building block for TextStore::resident_bytes() (K4 stage 2): FTS memory is hard-coded 0 today in the elastic memory budget's used-term (persistence_tick.rs), which only counts kv+vector. This is the first of several owned structures TextIndex/TextStore must sum. author: Tin Dang
TextIndex::resident_bytes() sums posting lists (existing PostingStore::estimated_bytes), term dictionaries (previous commit), FST fuzzy/prefix sidecars, per-document bookkeeping maps (field lengths, key<->doc_id, MVCC insert/delete LSNs), and TAG/NUMERIC secondary indexes (RoaringBitmap::serialized_size(), same convention as graph::index::PropertyIndex::resident_bytes). TextStore::resident_bytes() sums across all indexes on the shard. Explicitly excluded (documented in the doc comment): AnalyzerPipeline (one stemmer + stop-word set per field, built once at FT.CREATE, not doc-scaling), FieldStats, and schema metadata (BM25Config, text_fields, key_prefixes, name) -- all O(field count), negligible. This closes the FTS side of the K4 audit-keeper gap: FT text engine memory was hard-coded 0 in persistence_tick.rs. Wiring the real value into ShardStoreMemory::text (elastic budget used-term + MEMORY DOCTOR / Prometheus) is the next commit. RED-first test `text_store_resident_bytes_grows_after_indexing` asserts an empty store reports 0 and an indexed store reports > 0. author: Tin Dang
Charge the KV disk-offload cold index's own in-RAM footprint, which
was invisible to every memory signal today: a spilled key is dropped
from the hot DashTable (removing its bytes from
Database::estimated_memory), but the ColdIndex entry tracking its
disk location (key bytes + ColdLocation + file_refs bookkeeping)
still consumes heap. At scale this looked like "free" RAM.
Deliberately an O(1) incremental accumulator updated at every map
mutation site (insert/remove/clear_all, plus the two sweep methods'
direct self.map.remove bypasses), NOT an O(n) walk computed at read
time. A cold index backing a disk-offloaded dataset is exactly the
structure G2 ("serve 10x RAM datasets") sizes up to tens of millions
of entries -- an O(n) walk every 100ms shard tick would regress the
workload this index exists for. This is a deliberate escalation past
the brief's "approximation via capacity is fine" baseline, justified
by G2's explicit scale target.
Wiring resident_bytes() into the shard's published memory (the
elastic budget used-term) is a separate commit, at the tick site
(persistence_tick.rs), NOT inside Database::estimated_memory()/
resident_bytes() -- those stay O(1) hot-path reads untouched, so
this change cannot regress the per-write eviction pre-gate.
author: Tin Dang
… (K4 steps 3-4) Wire the resident_bytes() accounting from the last three commits into the actual memory signals: - persistence_tick.rs: publish TextStore::resident_bytes() into store_memory.text (previously hard-coded 0 -- FTS memory was invisible to every observer). Charge each db's ColdIndex bytes into the per-shard KV publish (Database::estimated_memory() itself stays an untouched O(1) hot-path read; this is additive at the 100ms tick only, so it cannot change per-write eviction pre-gate behavior). - shared_databases.rs recompute_elastic_budget: the donor/hot classification's used-term now sums KV + vector + text + graph (was kv+vector only, per the K4 audit-keeper gap). Mirrors the A4 vector fix exactly -- a text- or graph-heavy/KV-light shard was misclassified as an idle donor, lending headroom to siblings while its true resident footprint was already over base. RED-first test `recompute_elastic_budget_text_and_graph_heavy_shards_not_donors` (mirrors the existing vector-heavy test) asserts the budget correctly excludes text/graph-heavy shards from donor classification. No policy semantics changed: this widens what the SAME donor/hot formula (compute_elastic_budget, unmodified) sees. try_evict_if_needed and the inline-write eviction pre-gate are untouched -- only the elastic budget (cross-shard headroom lending) and the disk-offload pressure cascade's aggregate (which reads the same published atomic) are affected, and only when text/graph/cold-index planes are actually in use (all zero for pure-KV workloads, matching existing eviction_parity test fixtures). author: Tin Dang
…(K4 step 5)
Text memory was hard-coded 0 everywhere it was published, so it never
appeared as an observable signal despite the store_memory.text atomic
existing since C5/M4. With the previous commits making that atomic
real, surface it at both existing cross-shard scatter-gather sites
(sum published per-shard atomics, additive, matching the FT.INFO
precedent):
- MEMORY DOCTOR: new "Text (FTS):" line + dominance recommendation,
folded into tracked_sum (so allocator_overhead stays accurate
instead of silently absorbing FTS bytes).
- Prometheus moon_memory_bytes{kind="text"}: new gauge, primed at
startup and updated every 15s alongside the existing hnsw/csr/lua
kinds.
Updates the two integration tests that hard-assert the exact kind/
label set (memory_prometheus_kinds.rs: 7 -> 8 kinds; memory_doctor_
response.rs needed no change -- it only asserts presence of a fixed
subset, and the new "Text (FTS):" line does not appear in its
required_kinds list, so the >=95%-of-RSS sum invariant is unaffected
when no text index exists in that test's workload).
author: Tin Dang
New src/storage/tier.rs: ResidencyTier enum (Hot / WarmReloadable / ColdStub) + TierPolicy trait skeleton, doc-commented. TYPES ONLY -- no plane adopts this in M2. Gives a shared name to the residency concept vector already implements ad hoc (mutable/immutable (HOT) vs warm segments), so KV disk-offload, graph, and FTS can adopt the same vocabulary incrementally in M4 instead of inventing three more bespoke tier concepts. Explicit non-goals (per the K4 brief): forcing one eviction policy on all planes, and adopting any plane onto the trait in this milestone -- that touches each plane's compact/demote/promote call sites, which is real behavior-affecting work belonging in M4 with its own red/green tests. Unit tests cover the enum's derives (Copy/Eq/Hash usability) and prove the trait is implementable/usable generically via a MockPlane -- not a real adoption, just confirming the skeleton's shape holds together. author: Tin Dang
… test
Pre-existing gap surfaced while wiring K4's "text" kind into the same
two call sites: update_moon_memory_bytes() has emitted
moon_memory_bytes{kind="lua_scripts"} since C4 (wave-5 hygiene), but
prime_moon_memory_bytes() never primed it, and
memory_prometheus_kinds.rs's EXPECTED_KINDS / count assertion never
included it either -- so the gauge silently existed only after the
first 15s update tick, and the integration test's own count check
would have passed 7 even though 8 kinds were actually emitted (the
kind-count assertion was verifying its own stale expectation, not the
real gauge set).
Fixed alongside the "text" addition since both land in this file:
EXPECTED_KINDS and the prime list are now the real 9-kind set
(dashtable, hnsw, text, csr, wal, sealed, replication_backlog,
lua_scripts, allocator_overhead). Verified end-to-end against the
release binary: sum/RSS ratio 1.0000.
author: Tin Dang
author: Tin Dang
TextStore/TextIndex::resident_bytes() was an unbounded O(doc-count + vocabulary) full-recompute walk -- every posting incl. nested positions, every term-dict entry, every TAG/NUMERIC entry, across every index -- called unconditionally every 100ms from run_eviction_tick's shard tick regardless of whether maxmemory is even set. Measured 6.4ms/call at 50K docs, 21.3ms at 200K (>20% of the tick budget), causing recurring P99 spikes for every command on that shard. This directly contradicted the ColdIndex commit's (ed3525a2) own stated reasoning for why an O(n) tick-time walk is wrong at scale. Replace the walk with O(1) incremental accumulators mirroring ColdIndex's COLD_ENTRY_OVERHEAD pattern, using a fixed-cost-per-occurrence/per-entry approximation model instead of exact RoaringBitmap::serialized_size() deltas (compressed bitmap size is non-linear/non-additive across arbitrary insert/remove patterns and cannot be delta-tracked in O(1)): - PostingStore (src/text/posting.rs): resident_bytes field updated in add_term_occurrence/remove_doc; estimated_bytes() is now a pure field read. - TermDictionary (src/text/term_dict.rs): resident_bytes field updated in get_or_insert's new-term branch (no deletion path exists). - TextIndex (src/text/store.rs): resident_bytes_extra field covers every other contributor (doc bookkeeping maps, TAG/NUMERIC secondary indexes, FST sidecars). Mutation sites audited symmetrically: index_document, ensure_doc_id, set_doc_insert_lsn, tag_index_document, numeric_index_document, build_fst, load_fst_sidecars, remove_doc_by_doc_id, and new_with_schema's TAG/NUMERIC field seeding (a mutation site missed on the first pass -- caught by the ground-truth test). FST rebuilds are a "re-sync at structural event" (fst::Map:: as_fst().size() is O(1) per call; the rebuild itself is already O(vocabulary)), not a periodic walk. TAG/NUMERIC revoke logic is deduplicated into shared tag_bitmap_revoke/numeric_bitmap_revoke helpers used by both the upsert-revoke loop and the hard-delete path, closing off the two call sites drifting apart independently. doc_id_to_delete_lsn currently has no insertion call site anywhere in the codebase (reserved for future v0.2 logical-delete wiring), so its uncharge is written defensively symmetric but is a no-op today. TDD, RED-first: kept the old O(n) walk as resident_bytes_ground_truth() (#[cfg(test)] only) using the SAME fixed-cost formulas as the incremental accumulators, and added mixed-mutation tests (index N docs across TEXT + TAG + NUMERIC fields, upsert, FST build, partial hard-delete, drain to empty) asserting the incremental value matches the ground-truth walk at every step, plus a wall-clock test proving resident_bytes()/ estimated_bytes() are O(1) reads against a 5,000-doc / 100K-posting index. No eviction/budget behavior changes: this only fixes how the existing text resident_bytes() value is computed, not what donor/hot decisions see. author: Tin Dang
Update the [Unreleased] "memory + tier accounting spine (kernel M2 stage 2 / K4)" entry to describe TextStore/TextIndex::resident_bytes() as an O(1) incremental accumulator (same contract as ColdIndex/graph), and record the measured tick-time regression the initial O(n) walk had before the P0 fix landed. author: Tin Dang
release_binary() hardcoded CARGO_MANIFEST_DIR/target/release/moon, which inside the OrbStack Linux VM can resolve to a stale macOS Mach-O binary on the shared checkout -- it gets silently host-proxied back to the Mac, the port never binds VM-side, and the test times out after 30s waiting to accept a connection with no obvious cause (gotcha_orbstack_macho_binary_ trap). Fall back to the hardcoded path only when MOON_BIN is unset, so CI and local VM runs can pin an explicitly-built, ELF-verified binary. author: Tin Dang
… artifact) Rebasing feat/kernel-m2-k4-accounting onto origin/main (K3, 21081ef) auto-merged cleanly at the text level but concatenated the tail of a pre-existing bullet directly against K3's new section header with no blank line separator, breaking Markdown heading rendering. Caught by the post-rebase CHANGELOG spot-check; no other file needed a similar fix (git diff origin/main --stat only touches the K4 source files). author: Tin Dang
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughThis PR adds O(1) resident-memory accounting for text and cold indexes, includes all storage planes in elastic budgets, publishes text memory in shard metrics, adds residency-tier types, and updates administrative reporting, tests, and changelog entries. ChangesResident memory accounting
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant TextStore
participant ColdIndex
participant ShardMemory
participant MemoryMetrics
TextStore->>ShardMemory: publish text resident_bytes()
ColdIndex->>ShardMemory: publish cold-index resident_bytes()
ShardMemory->>ShardMemory: recompute elastic budget using all planes
ShardMemory->>MemoryMetrics: emit per-kind memory gauges
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
tests/memory_prometheus_kinds.rs (1)
188-206: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExercise the new text metric, not just its label.
This test now requires
kind="text"to exist, but it only inserts ordinary string keys. A broken text publication path can still pass because the series is primed at zero. Add a minimal real text-index workload and assert that the parsedtextvalue becomes positive.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/memory_prometheus_kinds.rs` around lines 188 - 206, Update metrics_endpoint_emits_nine_memory_kinds to create a minimal real text-index workload before scraping metrics, rather than only inserting ordinary string keys. Parse the emitted metrics and assert that the value for kind="text" is positive, while preserving the existing nine-kind label/count assertions.src/text/store.rs (1)
85-192: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoff
store.rsexceeds the repository file-size limit.This file is now ~2,200+ lines and mixes the index write path (
index_document,tag_index_document,numeric_index_document, accounting helpers) with the read/search path (search_field*,expand_terms,search_numeric_range). The K4 additions push it further past the limit. Consider splitting into a directory module (e.g. write/read/accounting submodules) as a follow-up.As per coding guidelines: "No single Rust file should exceed 1500 lines. Split command-group files into directory modules when approaching the limit; split read and write implementations above 1000 lines."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/text/store.rs` around lines 85 - 192, The TextStore implementation exceeds the 1500-line repository limit and combines unrelated responsibilities. Split store.rs into a directory module with focused write/indexing, read/search, and accounting submodules, moving symbols such as index_document, tag_index_document, numeric_index_document, search_field*, expand_terms, search_numeric_range, and the charge_/revoke_ helpers while preserving the existing public API and behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Around line 15-28: Update TextStore::resident_bytes() so it returns a
store-level cached aggregate instead of summing field_postings and
field_term_dicts on each call. Maintain that aggregate incrementally at every
mutation site affecting those structures, preserving consistency with the
existing PostingStore, TermDictionary, and TextIndex cached totals so the
documented O(1) behavior is accurate.
In `@src/shard/persistence_tick.rs`:
- Around line 328-335: Update TextStore::resident_bytes() to provide a cached
O(1) aggregate instead of scanning field_postings and field_term_dicts on each
call. Add or reuse a store-level resident-bytes counter, update it at all
relevant mutation sites, and keep the existing persistence_tick read unchanged.
In `@tests/memory_prometheus_kinds.rs`:
- Around line 29-39: Update release_binary() to treat an empty MOON_BIN value as
unset by only returning the override when it contains a non-empty path;
otherwise continue to the existing fallback binary resolution.
---
Nitpick comments:
In `@src/text/store.rs`:
- Around line 85-192: The TextStore implementation exceeds the 1500-line
repository limit and combines unrelated responsibilities. Split store.rs into a
directory module with focused write/indexing, read/search, and accounting
submodules, moving symbols such as index_document, tag_index_document,
numeric_index_document, search_field*, expand_terms, search_numeric_range, and
the charge_/revoke_ helpers while preserving the existing public API and
behavior.
In `@tests/memory_prometheus_kinds.rs`:
- Around line 188-206: Update metrics_endpoint_emits_nine_memory_kinds to create
a minimal real text-index workload before scraping metrics, rather than only
inserting ordinary string keys. Parse the emitted metrics and assert that the
value for kind="text" is positive, while preserving the existing nine-kind
label/count assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 18f204e4-b1cc-4e46-bc80-a107e6f93ec5
📒 Files selected for processing (12)
CHANGELOG.mdsrc/admin/metrics_setup.rssrc/command/server_admin.rssrc/shard/persistence_tick.rssrc/shard/shared_databases.rssrc/storage/mod.rssrc/storage/tier.rssrc/storage/tiered/cold_index.rssrc/text/posting.rssrc/text/store.rssrc/text/term_dict.rstests/memory_prometheus_kinds.rs
…g + empty MOON_BIN guard
CHANGELOG: "O(1) incremental accumulator" for the text plane overstated the
accessor — the publish-site read sums cached per-structure totals, which is
O(schema field count) (bounded by FT.CREATE definitions), not a single load.
Reworded to "data-size-independent"; the P0 property (no corpus-size scaling
on the 100ms shard tick) is unchanged. A store-level single counter was
considered and declined: it would add cross-structure drift risk to shave a
schema-bounded sub-microsecond sum.
tests/memory_prometheus_kinds: treat MOON_BIN="" as unset — an empty export
previously resolved to PathBuf::from("") (the current directory) and
spawn_moon() tried to execute a directory instead of falling back to
target/release/moon.
author: Tin Dang
TinDang97
added a commit
that referenced
this pull request
Jul 14, 2026
…(v0.7.0 prep) Owner reconciliation 2026-07-14 of docs/PRODUCTION-CONTRACT.md against the tree (main is 67 commits past the v0.6.0 tag, carrying the full v0.6.1 hygiene scope and the v0.7 replication workstreams untagged): Re-ticked with verified evidence: - FUZZ-01: graph_props_record.rs restored, all 12 declared targets exist - ACL-REG-01: TXN/WS/MQ/TEMPORAL/CDC in metadata.rs + monoio intercept reorder (PR #258) - COLD-TTL-01: ColdIndex::sweep_expired + info_reclamation counters - SEC-07: SECURITY.md now states a release-agnostic supported-versions policy (latest minor line pre-1.0, LTS from v1.0.0) Updated honestly: - FT-PARITY-01: FT.AGGREGATE shipped (ft_aggregate.rs); FT.ALTER still absent -- stays unticked - CRASH-01: caveats recorded -- graph-durability g1-g3 never actually ran before PRs #322/#324 (WAL v2 flat-file probe + legacy replay no-op); crash_recovery_disk_offload_no_aof residual red (task #44) - SUPPLY-01: supply-chain.yml workflow in flight (task #63), tick on merge New rows for shipped-but-previously-untracked guarantees: - CRASH-02: 37-cell cross-plane kill-9 matrix (v0.8 G1, PR #298 + the 4 RED-group fixes #52/#53/#57/#60) - MEM-10X-01: 10x RAM datasets G2 acceptance (restart readiness 157s->3.7s, PRs #319/#297/#320) - REPL-PLANES-01: all-plane replication (Waves A/B, PRs #285/#294) - REPL-SOAK-01 (unticked): 24h replication soak gates the v0.7.0 tag Header refreshed (milestone = v0.7.0 Replication GA, tag gated on soak). scripts/check-production-contract.sh parses all new rows; GA-blocking gap is now an honest 17 rows. Refs: tasks #61-#65 author: Tin Dang
pilotspacex-byte
added a commit
that referenced
this pull request
Jul 14, 2026
…(v0.7.0 prep) (#325) Owner reconciliation 2026-07-14 of docs/PRODUCTION-CONTRACT.md against the tree (main is 67 commits past the v0.6.0 tag, carrying the full v0.6.1 hygiene scope and the v0.7 replication workstreams untagged): Re-ticked with verified evidence: - FUZZ-01: graph_props_record.rs restored, all 12 declared targets exist - ACL-REG-01: TXN/WS/MQ/TEMPORAL/CDC in metadata.rs + monoio intercept reorder (PR #258) - COLD-TTL-01: ColdIndex::sweep_expired + info_reclamation counters - SEC-07: SECURITY.md now states a release-agnostic supported-versions policy (latest minor line pre-1.0, LTS from v1.0.0) Updated honestly: - FT-PARITY-01: FT.AGGREGATE shipped (ft_aggregate.rs); FT.ALTER still absent -- stays unticked - CRASH-01: caveats recorded -- graph-durability g1-g3 never actually ran before PRs #322/#324 (WAL v2 flat-file probe + legacy replay no-op); crash_recovery_disk_offload_no_aof residual red (task #44) - SUPPLY-01: supply-chain.yml workflow in flight (task #63), tick on merge New rows for shipped-but-previously-untracked guarantees: - CRASH-02: 37-cell cross-plane kill-9 matrix (v0.8 G1, PR #298 + the 4 RED-group fixes #52/#53/#57/#60) - MEM-10X-01: 10x RAM datasets G2 acceptance (restart readiness 157s->3.7s, PRs #319/#297/#320) - REPL-PLANES-01: all-plane replication (Waves A/B, PRs #285/#294) - REPL-SOAK-01 (unticked): 24h replication soak gates the v0.7.0 tag Header refreshed (milestone = v0.7.0 Replication GA, tag gated on soak). scripts/check-production-contract.sh parses all new rows; GA-blocking gap is now an honest 17 rows. Refs: tasks #61-#65 author: Tin Dang Co-authored-by: Tin Dang <tindang.ht97@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Kernel M2 stage 2 (K4): unified
resident_bytes()accounting across every storage plane, folded into the elastic memory budget's used-term. Completes storage-kernel M2 (stage 1 was K3, PR #296).0everywhere it's published (elastic budget, MEMORY DOCTOR, Prometheus).PostingStore/TermDictionary/TextIndexeach carry an O(1) incremental accumulator — mutation sites (index_document,ensure_doc_id, TAG/NUMERIC insert+revoke, FST rebuild, hard-delete, schema-time field seeding) update a cached total directly; the public accessor is a pure field read.resident_bytes()was an O(doc-count + vocabulary) full-recompute walk, invoked unconditionally every 100ms from the shard eviction tick regardless ofmaxmemory— reviewer measured 6.4ms/call at 50K docs, 21.3ms at 200K (>20% of the tick budget), recurring P99 spikes for every command on that shard. Fixed before merge: replaced with the incremental design above, verified against a#[cfg(test)]ground-truth full-walk (same fixed-cost formulas) across a mixed mutation sequence — index/upsert/TAG-NUMERIC-update/FST-build/partial-delete/drain-to-empty — plus a wall-clock test proving the accessor is O(1) against a 5,000-doc/50,000-term index.ColdIndex::resident_bytes()(KV disk-offload bookkeeping): same O(1) incremental-accumulator contract, sized for the G2 "10x RAM" scale target.ShardDatabases::recompute_elastic_budget, GAP-1/PR feat(shard): elastic per-shard memory budgets (hot-shard headroom borrowing) #170) now sumskv + vector + text + graph— previously kv+vector only. RED-firstrecompute_elastic_budget_text_and_graph_heavy_shards_not_donorstest proves text/graph-heavy shards are correctly excluded from donor classification.Text (FTS):line + Prometheusmoon_memory_bytes{kind="text"}gauge. Also fixes a pre-existing gap wheremoon_memory_bytes{kind="lua_scripts"}was emitted but never primed (EXPECTED_KINDS7→9).src/storage/tier.rs:ResidencyTier(Hot/WarmReloadable/ColdStub) +TierPolicytrait skeleton — types only, no plane adoption yet (that's M4).tests/memory_prometheus_kinds.rs'srelease_binary()now honorsMOON_BIN, closing the OrbStack Mach-O host-proxy trap (a stale macOS binary on the shared checkout gets silently host-proxied back to the Mac inside the Linux VM — the port never binds VM-side, producing a 30s accept timeout with no obvious cause).Reference:
.planning/reviews/kernel-m2-brief-2026-07-12.md. This completes kernel M2 stage 2 of 2.Hot-path note
Every publish-site read (
persistence_tick.rs's 100ms tick, MEMORY DOCTOR, Prometheus scrape) is now O(1) — a cached field load, never a walk. Eviction pre-gate semantics are unchanged: the eviction decision itself still uses KV-onlyestimated_memoryas before; this PR only widens what the elastic budget's used-term sees, not how eviction decides.Gates
MOON_BINharness fix) landed and re-verified before this PR opened.cargo fmt --checkclean;cargo clippy -- -D warningsclean on both default and--no-default-features --features runtime-tokio,jemallocmatrices; fullcargo test --profile release-fast --lib— 4209 passed, 0 failed, 1 ignored (pre-existing perf-flake ignore, unrelated).odmagic-byte check):memory_prometheus_kindsgreen withMOON_BINpinned to the VM-built binary (previously would have hit the Mach-O host-proxy trap);eviction_parity_shards1,eviction_parity_shards4,eviction_parity_hash_disk_offload_shards1,eviction_parity_hash_disk_offload_shards4— 4/4 pass, confirming no eviction behavior change.git diff origin/main --statconfirms only K4 source files changed; caught and fixed one CHANGELOG whitespace artifact from the auto-merge (missing blank line between adjacent[Unreleased]entries) before pushing.Test plan
cargo fmt --checkcargo clippy --profile release-fast -- -D warnings(default features)cargo clippy --profile release-fast --no-default-features --features runtime-tokio,jemalloc -- -D warningscargo test --profile release-fast --lib(full suite, 4209 passed)memory_prometheus_kindswithMOON_BINpinned to a VM-built ELF binaryeviction_parity_{shards1,shards4}+eviction_parity_hash_disk_offload_{shards1,shards4}(4/4)Summary by CodeRabbit