feat(text): v3-1 FTS hardening — upsert / count / routing (final 3 tasks)

.add/state.json

@@ -1,7 +1,7 @@
 {
   "project": "moon",
   "stage": "production",
-  "active_task": "fts-query-combinators",
+  "active_task": "fts-query-routing-robustness",
   "active_milestone": "v3-1-fts-hardening",
   "tasks": {
     "hotpath-lock-quickwins": {
@@ -121,6 +121,40 @@
       ],
       "created": "2026-06-16T06:21:47+00:00",
       "updated": "2026-06-16T09:07:25+00:00"
+    },
+    "fts-upsert-incremental": {
+      "title": "Incremental posting-list upsert (kill O(V) per-doc scan)",
+      "phase": "done",
+      "gate": "PASS",
+      "milestone": "v3-1-fts-hardening",
+      "depends_on": [],
+      "created": "2026-06-16T10:34:55+00:00",
+      "updated": "2026-06-16T12:46:23+00:00",
+      "flag_verified": true
+    },
+    "fts-search-count-semantics": {
+      "title": "FT.SEARCH integer reply = true total-matched (RediSearch count semantics)",
+      "phase": "done",
+      "gate": "PASS",
+      "milestone": "v3-1-fts-hardening",
+      "depends_on": [
+        "fts-query-eval-dispatch"
+      ],
+      "created": "2026-06-16T12:48:50+00:00",
+      "updated": "2026-06-16T13:13:21+00:00",
+      "flag_verified": true
+    },
+    "fts-query-routing-robustness": {
+      "title": "FT.SEARCH routing robustness: SPARSE detection + no expect() on the BM25 AND path",
+      "phase": "done",
+      "gate": "PASS",
+      "milestone": "v3-1-fts-hardening",
+      "depends_on": [
+        "fts-query-eval-dispatch"
+      ],
+      "created": "2026-06-16T13:14:26+00:00",
+      "updated": "2026-06-16T13:41:08+00:00",
+      "flag_verified": true
     }
   },
   "milestones": {
@@ -174,7 +208,7 @@
     }
   },
   "created": "2026-06-11T03:18:21+00:00",
-  "updated": "2026-06-16T09:07:25+00:00",
+  "updated": "2026-06-16T13:41:08+00:00",
   "setup": {
     "locked": true,
     "locked_at": "2026-06-11T03:28:00+00:00",

CHANGELOG.md

@@ -6,6 +6,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed — FT.SEARCH routing: prose "knn" searches as text, standalone SPARSE reaches the vector engine, no panic on the BM25 AND path (PR #192)
+
+`is_text_query` uppercased the query and matched the bare substring `KNN `, so a
+text search whose terms merely contained the word *knn* (e.g.
+`FT.SEARCH idx "knn tutorial"`) was misclassified as a vector query and fell to
+the KNN parser, returning `ERR invalid KNN query syntax` instead of text
+results. It now keys on the canonical `[KNN` vector-query bracket — prose
+containing "knn" searches as text while `*=>[KNN …]` still routes to the vector
+engine. A standalone `SPARSE @field $param` clause paired with a text-looking
+query string was silently dropped onto the text path (`is_text_query` only sees
+`args[1]`); a `has_sparse_clause` guard now defers any SPARSE-carrying query to
+the vector engine at every text-route gate. The three `.expect("posting exists")`
+on `TextStore::search_field`'s BM25 AND + scoring path are replaced with
+defensive control flow, so a vanished posting yields empty results instead of
+panicking the server. Output is byte-identical for all existing queries.
+
+### Fixed — FT.SEARCH integer reply is the true total-matched, not the page size (PR #192)
+
+The first element of an `FT.SEARCH` reply (the match count) was capped at the
+`LIMIT`/`top_k` page size: `FT.SEARCH idx "term" LIMIT 0 5` over 100 matches
+reported `5`, not `100`, because the count was read from the already-truncated
+result page. On multi-shard indexes the coordinator compounded it by counting the
+merged-and-truncated returned docs rather than each shard's true matched count.
+The reply now reports the true number of matched, key-resolvable documents
+(RediSearch semantics): the evaluator surfaces the count before truncation, and
+the multi-shard merge sums each shard's local matched count (keys partition to
+exactly one shard, so the sum is exact; errored shards contribute 0). Verified
+identical on 1- and 4-shard servers. Returned document pages (count, order,
+scores, keys) are unchanged.
+
+### Performance — FT.SEARCH upsert / bulk re-index no longer O(V) per document (PR #192)
+
+Re-indexing a document (an HSET upsert, or a bulk re-index pass) called
+`PostingStore::remove_doc`, which scanned EVERY term's posting list to clear one
+document — O(total-vocabulary) per doc — so per-upsert cost grew with the corpus
+(the indexing-rate cliff: ~376 vs ~18,052 docs/s as vocabulary V grew).
+`PostingStore` now keeps a reverse `doc_id → term_ids` index, populated once per
+(doc, term) edge as terms are added, so `remove_doc` visits only the terms the
+document actually contributed — O(terms-in-doc), independent of V. Search output
+is byte-identical (the rank-aligned posting contract is unchanged): matched doc
+sets, BM25 scores, `doc_freq`, `num_docs`, and avgdl are unaffected. A missing or
+already-cleared reverse entry is skipped defensively — never an unwrap/panic.
+
 ### Fixed — FT.SEARCH `OR` and multi-clause queries return correct result sets (PR #190)
 
 `FT.SEARCH` query combinators were silently broken: `OR` (`alpha | beta`)