feat(text): v3-1 FTS hardening — posting-rank TF + query combinators (OR/intersect fix)

.add/milestones/v3-1-fts-hardening/MILESTONE.md

@@ -0,0 +1,78 @@
+# MILESTONE: FTS Hardening
+
+goal: Moon's full-text search is trustworthy and competitive: every query combinator (term, AND, OR, TEXT+TAG, NUMERIC) returns correct results with true total-matched counts, and indexing plus high-DF queries run without the O(V)/O(M^2) cliffs the 2026-06-16 benchmark exposed.
+rationale: new-major (split 1/3) — headline slice of the v3 "secondary-engine correctness & parity" theme opened from the 2026-06-16 deep review + 4-feature GCloud benchmark (FTS was the benchmark's biggest gap vs RediSearch and was previously un-benchmarked). No closed milestone (v1 shared-nothing, v2/v2-1 throughput) covers FTS correctness/parity. This slice owns FTS's O(M^2)/O(V) perf cliffs + the OR/combo/count correctness defects + a latent query-routing panic. Graph (v3-2) and Vector/KV (v3-3) are sibling slices.
+stage: production · status: active · created: 2026-06-16
+
+> SDD living doc for this milestone. Keep it THIN: breadth, shared decisions, and
+> exit criteria only — per-task detail lives in each `.add/tasks/<slug>/TASK.md`,
+> written just-in-time. Update this doc whenever a task reveals a milestone gap.
+
+## Scope
+In:
+- High-DF term query no longer O(M^2): replace the O(N) `.position(|id| id == doc_id)` TF lookup
+  with a rank-based lookup (e.g. `RoaringBitmap::rank`). `src/text/store.rs:507,773`.
+  (bench: term_hi 419 ms / 19 qps vs RediSearch 2 ms / 3,813 qps.)
+- Bulk indexing no longer O(V) per doc: replace the per-doc posting upsert scan with an
+  incremental update. `src/text/posting.rs:139`. (bench: 376 vs 18,052 docs/s, ~48×.)
+- `OR` (`|`) returns the union of matched docs; `TEXT+TAG` combined query returns the intersection
+  (non-empty when matches exist). (bench: OR total 10 vs 2,072; combo 0 vs 253.)
+- FT.SEARCH reply reports the true total-matched count, not the returned-page count.
+  (bench: TAG 10 vs 5,064 — an FT.SEARCH protocol deviation.)
+- Query routing + robustness: `is_text_query()` recognizes `SPARSE` (no misroute of sparse-vector
+  queries to the BM25 path) and the FT query path removes its 3× `expect()` panics.
+  `src/command/vector_search/ft_text_search.rs:1004`, `src/text/store.rs:463`.
+
+Out:
+- BM25 ranking-quality tuning / new analyzers — this is correctness of result SETS + counts + speed,
+  not relevance-score changes.
+- Aggregation/GROUPBY speed (bench 6 vs 11 qps) and NUMERIC-range query speed (~3× slower) — noted,
+  deferred perf, not correctness defects.
+- Vector-search QPS/recall (v3-3 / a later effort) and graph (v3-2).
+
+## Shared decisions & glossary deltas   (living — every task must honor these)
+- FT.SEARCH count semantics = RediSearch's: the integer reply is the TOTAL matched; paging is
+  separate. Every query task honors this once `fts-search-count-semantics` freezes it.
+- TDD red/green: each correctness defect lands a FAILING test first (wrong union / zero combo /
+  wrong count / SPARSE misroute), then the fix (CLAUDE.md Rule 3).
+- New/changed parser/eval paths get a fuzz target (CLAUDE.md Fuzzing) and NEVER panic on malformed
+  input — return `Frame::Error`, no `expect`/`unwrap` on the query path.
+- No new hot-path allocations on the dispatch/query path; no new `unsafe` without approved SAFETY.
+
+## Shared / risky contracts (freeze these first)
+- FT.SEARCH total-count semantics (matched vs returned) — a wire-visible reply contract every query
+  path shares and clients depend on. Freeze first.            -> owning task `fts-search-count-semantics`
+- Posting-list TF lookup API (rank-based) — the shape the high-DF fix and the count/eval paths both
+  call; wrong here re-does downstream query work.             -> owning task `fts-posting-rank-tf`
+
+## Tasks (breadth-first decomposition; detail lives in each TASK.md)
+- [x] fts-posting-rank-tf           depends-on: none                      — replace O(N) TF `.position()`
+      with rank-based lookup; kills the high-DF O(M^2) cliff. DONE 2026-06-16 (gate PASS, commits
+      45b3db8+a7e816d). Also fixed a latent BM25-misalignment bug; froze the rank-aligned PostingList
+      TF contract (term_freqs/positions sorted-doc_id-aligned, tf()/positions_for() via rank).
+- [ ] fts-upsert-incremental        depends-on: none                      — replace O(V) per-doc posting
+      upsert scan with incremental update; fast bulk indexing.
+- [ ] fts-query-combinators         depends-on: none                      — `OR` (`|`) unions and
+      `TEXT+TAG`/`TEXT+NUMERIC` combos intersect; correct matched sets. Code trace upgraded this from
+      two point-bugs to a missing PARSER layer; froze a query GRAMMAR+AST+eval_set contract @ v1
+      (2026-06-16) and SPLIT the build into 2a+2b:
+  - [ ] fts-query-combinators (2a)  depends-on: none                      — recursive-descent
+        `parse_query` → `QueryNode`/`QueryError` + grammar + parser fuzz target. Owns the frozen contract.
+  - [ ] fts-query-eval-dispatch (2b) depends-on: fts-query-combinators    — `eval_set` (RoaringBitmap
+        union/intersect) + ft_text_search dispatch rewrite + wire reply. Inherits the frozen contract.
+- [ ] fts-search-count-semantics    depends-on: fts-query-eval-dispatch  — FT.SEARCH reply = true
+      total-matched count = `eval_set(root).len()` (RediSearch semantics). Counts over 2b's matched set.
+- [ ] fts-query-routing-robustness  depends-on: none                      — `is_text_query()` recognizes
+      `SPARSE`; remove 3× `expect()` on the FT query path.
+
+## Exit criteria (observable; map each to the task that delivers it)
+- [ ] A high-DF term (~5% of docs) returns without the O(M^2) cliff — a 100K-doc query-latency test
+      holds well under the old 419 ms (rank-based, not linear scan).            (← fts-posting-rank-tf)
+- [ ] Bulk indexing scales ~linearly (no O(V) per-doc scan) — an indexing-rate test shows the cliff
+      gone vs the 376 docs/s baseline.                                          (← fts-upsert-incremental)
+- [ ] `OR` returns |A ∪ B| matched docs and `TEXT+TAG` returns the non-empty intersection —
+      correctness tests over a known corpus.                                    (← fts-query-combinators)
+- [ ] FT.SEARCH's integer reply equals the true matched count (not the page size) — test asserts the
+      total over a corpus larger than the returned page.                        (← fts-search-count-semantics)
+- [ ] A `SPARSE @field …` query routes to the vector path (not BM25), and malformed FT input returns
+      an error frame (no panic) — routing + fuzz/negative tests.                (← fts-query-routing-robustness)

.add/milestones/v3-2-graph-correctness/MILESTONE.md

@@ -0,0 +1,57 @@
+# MILESTONE: Graph Correctness & Cypher Filtering
+
+goal: Moon's graph queries are correct: Cypher MATCH narrows on inline node-property predicates instead of full-scanning the label, directional traversal covers incoming/Both edges post-compaction, and node labels >= 32 are no longer silently dropped.
+rationale: new-major (split 2/3) — part of the v3 "secondary-engine correctness & parity" theme from the 2026-06-16 deep review + 4-feature benchmark. No closed milestone (v1 shared-nothing, v2/v2-1 throughput) covers graph-query correctness. This slice owns the three graph defects the review/benchmark surfaced; FTS (v3-1) and Vector/KV (v3-3) are sibling slices. Correctness only — Moon's native graph build + 1-hop already beat FalkorDB (bench §11.4).
+stage: production · status: planned · created: 2026-06-16
+
+> SDD living doc for this milestone. Keep it THIN: breadth, shared decisions, and
+> exit criteria only — per-task detail lives in each `.add/tasks/<slug>/TASK.md`,
+> written just-in-time. Update this doc whenever a task reveals a milestone gap.
+
+## Scope
+In:
+- Cypher MATCH narrows on an inline node-property predicate (e.g. `MATCH (a {id:N})`) instead of
+  full-scanning the label. (bench: `cypher_match_rows` = 14,991 ≈ |E| vs FalkorDB's filtered 4;
+  Moon Cypher point-query 40 qps.)
+- `Direction::Incoming` and `Direction::Both` traversals return incoming edges after compaction —
+  CSR currently stores only outgoing. `src/graph/traversal.rs:188–191`.
+- Node label storage supports labels with id >= 32 (the current 32-bit bitmap truncates).
+  `src/graph/csr/mod.rs:161` (`if label < 32`).
+
+Out:
+- Full openCypher coverage / general predicate pushdown beyond inline node-property equality — only
+  the point-filter the benchmark exercised is in scope.
+- Graph throughput — native GRAPH.ADDNODE/ADDEDGE build + native 1-hop already lead FalkorDB
+  (bench §11.4); this milestone is correctness, not speed.
+- Cross-shard graph — single-keyspace per CLAUDE.md, unchanged here.
+
+## Shared decisions & glossary deltas   (living — every task must honor these)
+- TDD red/green: each defect lands a FAILING test first (wrong rows / missing incoming edge /
+  dropped label), then the fix (CLAUDE.md Rule 3).
+- Malformed Cypher must never panic — return an error frame (parser/eval defensiveness).
+- New/changed graph commands keep their `scripts/test-consistency.sh` + `scripts/test-commands.sh`
+  entries (CLAUDE.md New Commands).
+- No new `unsafe` without explicit user approval + a `// SAFETY:` comment.
+
+## Shared / risky contracts (freeze these first)
+- Cypher inline-predicate evaluation semantics — how `{prop:val}` narrows the candidate set (index
+  probe vs filtered scan) and what it returns. Wrong shape re-does every downstream Cypher query.
+  -> owning task `graph-cypher-inline-filter`
+- Incoming-edge representation — reverse adjacency vs on-demand scan; a layout choice the label and
+  traversal work both read.                                   -> owning task `graph-incoming-edges`
+
+## Tasks (breadth-first decomposition; detail lives in each TASK.md)
+- [ ] graph-cypher-inline-filter   depends-on: none   — Cypher MATCH narrows on inline node-property
+      predicate instead of full-scanning the label; point-query returns only the matching node's edges.
+- [ ] graph-incoming-edges         depends-on: none   — CSR traversal returns incoming / Both-direction
+      edges post-compaction (reverse adjacency or incoming index).
+- [ ] graph-label-bitmap-overflow  depends-on: none   — node label storage supports id >= 32 without
+      silent truncation.
+
+## Exit criteria (observable; map each to the task that delivers it)
+- [ ] A Cypher point-query `MATCH (a {id:N})-[]->(b)` returns only node N's edges (returned rows ==
+      expected, not ≈|E|) — test asserts the narrowed row count.            (← graph-cypher-inline-filter)
+- [ ] On a compacted graph, `Direction::Incoming` and `Both` return the incoming edges (non-empty
+      where they should be) — test on a post-compaction graph.              (← graph-incoming-edges)
+- [ ] A node assigned label id >= 32 (e.g. a 40-label graph) is matched by its label query — no
+      silent drop; test with >= 33 distinct labels.                         (← graph-label-bitmap-overflow)

.add/milestones/v3-3-vector-kv-polish/MILESTONE.md

@@ -0,0 +1,65 @@
+# MILESTONE: Vector & KV Latent-Correctness + Hot-Path Polish
+
+goal: Moon's vector segments decode at the correct code length (SQ8 included) and FT.INFO reports true cross-segment doc counts, and the KV/vector command hot paths honor the no-alloc and parking_lot lock-discipline rules.
+rationale: new-major (split 3/3) — the v3 "secondary-engine correctness & parity" theme's latent-correctness + rule-compliance slice from the 2026-06-16 deep review. Bundles the two latent vector-correctness traps (none block the current call paths, hence they survived to review) with the small hot-path-alloc / lock-discipline list. FTS (v3-1) and Graph (v3-2) are sibling slices.
+stage: production · status: planned · created: 2026-06-16
+
+> SDD living doc for this milestone. Keep it THIN: breadth, shared decisions, and
+> exit criteria only — per-task detail lives in each `.add/tasks/<slug>/TASK.md`,
+> written just-in-time. Update this doc whenever a task reveals a milestone gap.
+
+## Scope
+In:
+- SQ8 segment `code_len = bytes_per_code - 4` is wrong for SQ8's 8-byte trailer — corrected decode
+  across the segment lifecycle (search / merge / persistence). `src/vector/segment/compaction.rs:554`
+  + `immutable.rs` / `mutable.rs`. (Latent P0; same family as the v0.3.0-deferred code_len note.)
+- FT.INFO `num_docs` sums mutable + immutable segments, not just the mutable one.
+  `src/command/vector_search/ft_info.rs:42`.
+- FT.SEARCH avoids the ~3.2 MB `key_hash_to_key` clone per query (borrow / `Arc` the map).
+- KV `INCR`/`DECR` write the integer via `itoa` to a buffer — no per-op `String` alloc on the hot
+  path. `src/command/string/string_write.rs:312,317`.
+- The command-dispatch path uses `parking_lot::RwLock`, not `std::sync::RwLock`, for the ACL table.
+  `src/shard/event_loop.rs:65`, `src/command/connection.rs:374,460`.
+
+Out:
+- Re-quantization / new vector codecs — only the existing SQ8 decode length is in scope, not new
+  formats.
+- Vector-search QPS/recall competitiveness vs RediSearch (bench §10.5) — a larger HNSW/quant effort,
+  deferred.
+- Broader allocation/lock audit beyond these named sites.
+
+## Shared decisions & glossary deltas   (living — every task must honor these)
+- On-disk segment compatibility: the `code_len` fix MUST either decode existing SQ8 segments
+  correctly or bump the segment format version with a documented migration — never silently mis-read
+  persisted data (CLAUDE.md persistence/reload care; cf. the CWD-reload trap).
+- No new hot-path allocations in command / protocol / event_loop / io (CLAUDE.md Allocations):
+  `itoa` / `SmallVec` / borrow only.
+- `parking_lot` locks only; never hold a lock across `.await` (CLAUDE.md Lock Handling).
+- TDD red/green per fix; no new `unsafe` without approved SAFETY.
+
+## Shared / risky contracts (freeze these first)
+- SQ8 on-disk `code_len` / segment-trailer layout — the byte format read at decode. Wrong or
+  un-versioned here corrupts persisted indexes. Freeze the decode contract (and migration stance)
+  before touching the read path.                              -> owning task `vector-sq8-code-len`
+
+## Tasks (breadth-first decomposition; detail lives in each TASK.md)
+- [ ] vector-sq8-code-len           depends-on: none   — fix `code_len` for SQ8's 8-byte trailer;
+      correct decode across search / merge / persistence (latent P0).
+- [ ] vector-ftinfo-num-docs        depends-on: none   — FT.INFO `num_docs` sums all segments
+      (mutable + immutable), not just mutable.
+- [ ] vector-search-keyhash-noclone depends-on: none   — drop the ~3.2 MB `key_hash_to_key` clone per
+      FT.SEARCH (borrow / `Arc`).
+- [ ] kv-incr-itoa                  depends-on: none   — `INCR`/`DECR` via `itoa`-to-buffer; no
+      `String` alloc on the hot path.
+- [ ] kv-dispatch-lock-discipline   depends-on: none   — ACL / dispatch `std::sync::RwLock` ->
+      `parking_lot::RwLock`.
+
+## Exit criteria (observable; map each to the task that delivers it)
+- [ ] An SQ8 immutable segment decodes vectors at the correct length — recall parity with the
+      pre-compaction mutable segment (within quant tolerance) across search / merge / reload; test.  (← vector-sq8-code-len)
+- [ ] FT.INFO `num_docs` == total docs across mutable + immutable after a compaction; test.          (← vector-ftinfo-num-docs)
+- [ ] FT.SEARCH performs no per-query 3 MB `key_hash` clone — allocation probe / throughput test
+      shows the clone gone.                                                                          (← vector-search-keyhash-noclone)
+- [ ] `INCR`/`DECR` allocate no `String` on the hot path — `itoa` path asserted (test / no-alloc check). (← kv-incr-itoa)
+- [ ] No `std::sync::RwLock` remains on the command-dispatch path — ACL table is `parking_lot`;
+      audit/grep + test.                                                                             (← kv-dispatch-lock-discipline)

.add/state.json

@@ -1,8 +1,8 @@
 {
   "project": "moon",
   "stage": "production",
-  "active_task": "ft-yield-costfree-monoio",
-  "active_milestone": "v2-1-throughput-polish",
+  "active_task": "fts-query-combinators",
+  "active_milestone": "v3-1-fts-hardening",
   "tasks": {
     "hotpath-lock-quickwins": {
       "title": "Eliminate per-command global locks & syscall-level quick wins",
@@ -88,6 +88,39 @@
       "created": "2026-06-15T11:57:05+00:00",
       "updated": "2026-06-15T14:39:07+00:00",
       "flag_verified": true
+    },
+    "fts-posting-rank-tf": {
+      "title": "Rank-based posting TF lookup (kill the high-DF O(M^2) cliff)",
+      "phase": "done",
+      "gate": "PASS",
+      "milestone": "v3-1-fts-hardening",
+      "depends_on": [
+        "none"
+      ],
+      "created": "2026-06-16T04:51:36+00:00",
+      "updated": "2026-06-16T05:40:44+00:00",
+      "flag_verified": true
+    },
+    "fts-query-combinators": {
+      "title": "OR unions + TEXT+TAG intersects \u2014 correct combinator result sets",
+      "phase": "done",
+      "gate": "PASS",
+      "milestone": "v3-1-fts-hardening",
+      "depends_on": [],
+      "created": "2026-06-16T05:44:57+00:00",
+      "updated": "2026-06-16T09:07:25+00:00",
+      "flag_verified": true
+    },
+    "fts-query-eval-dispatch": {
+      "title": "Evaluate the query AST to matched sets + wire FT.SEARCH dispatch (2b)",
+      "phase": "done",
+      "gate": "PASS",
+      "milestone": "v3-1-fts-hardening",
+      "depends_on": [
+        "fts-query-combinators"
+      ],
+      "created": "2026-06-16T06:21:47+00:00",
+      "updated": "2026-06-16T09:07:25+00:00"
     }
   },
   "milestones": {
@@ -114,10 +147,34 @@
       "status": "done",
       "created": "2026-06-15T11:57:00+00:00",
       "updated": "2026-06-15T14:49:42+00:00"
+    },
+    "v3-1-fts-hardening": {
+      "title": "FTS Hardening",
+      "goal": "Moon's full-text search is trustworthy and competitive: every query combinator (term, AND, OR, TEXT+TAG, NUMERIC) returns correct results with true total-matched counts, and indexing plus high-DF queries run without the O(V)/O(M^2) cliffs the 2026-06-16 benchmark exposed.",
+      "stage": "production",
+      "status": "active",
+      "created": "2026-06-16T04:42:21+00:00",
+      "updated": "2026-06-16T04:42:21+00:00"
+    },
+    "v3-2-graph-correctness": {
+      "title": "Graph Correctness & Cypher Filtering",
+      "goal": "Moon's graph queries are correct: Cypher MATCH narrows on inline node-property predicates instead of full-scanning the label, directional traversal covers incoming/Both edges post-compaction, and node labels >= 32 are no longer silently dropped.",
+      "stage": "production",
+      "status": "planned",
+      "created": "2026-06-16T04:42:42+00:00",
+      "updated": "2026-06-16T04:42:42+00:00"
+    },
+    "v3-3-vector-kv-polish": {
+      "title": "Vector & KV Latent-Correctness + Hot-Path Polish",
+      "goal": "Moon's vector segments decode at the correct code length (SQ8 included) and FT.INFO reports true cross-segment doc counts, and the KV/vector command hot paths honor the no-alloc and parking_lot lock-discipline rules.",
+      "stage": "production",
+      "status": "planned",
+      "created": "2026-06-16T04:42:42+00:00",
+      "updated": "2026-06-16T04:42:42+00:00"
     }
   },
   "created": "2026-06-11T03:18:21+00:00",
-  "updated": "2026-06-15T14:49:42+00:00",
+  "updated": "2026-06-16T09:07:25+00:00",
   "setup": {
     "locked": true,
     "locked_at": "2026-06-11T03:28:00+00:00",