diff --git a/docs/spec/todos/TODO-0106.md b/docs/spec/todos/TODO-0106.md index 011d13c..1e19fd0 100644 --- a/docs/spec/todos/TODO-0106.md +++ b/docs/spec/todos/TODO-0106.md @@ -4,7 +4,7 @@ title: "Link graph from internal links and wikilinks" status: todo priority: medium created: 2026-03-14 -updated: 2026-06-19 +updated: 2026-07-03 depends_on: [] blocks: [] related: @@ -18,213 +18,108 @@ related: ## Summary -Extract internal links (markdown `[text](path.md)`, GitHub-style relative links) and wikilinks (Obsidian-style `[[page]]` / `[[page|alias]]` / `[[page#heading]]`) and frontmatter-declared references (`parent = "[[index]]"`) into a first-class relationship layer. The storage strategy is **layered**: a single canonical edge table in Lance for all edge data and 1-hop attribute-filtered queries; derived adjacency indexes added later as performance optimisations when (and only when) measured workload demands them. The killer capability is composing graph queries with semantic / FTS / hybrid search in one query — at any vault scale, with a clear escalation path to scale further when needed. +Extract the links a markdown corpus already contains — markdown `[text](path.md#slug)`, Obsidian wikilinks `[[page]]` / `[[page#Heading]]`, and frontmatter references (`parent = [[index]]` via `is_reference = true`) — into a queryable relationship layer, and fold it into the commands mdvs already has: `search` (relationship filters), `check` (broken / ambiguous links), and `info` (stats). No new top-level command, no separate dataset. -The unbounded-scaling goal stays. What's deferred is *when* we pay for the scaling layer: the architecture is structured so derived adjacency indexes can be added without breaking changes, but Phase 1 ships the canonical store only. +**Nodes are files and sections** (a per-document heading hierarchy). **Edges are Referential** (the extracted authored links) **and Structural** (the heading tree — parent / child / sibling). **Semantic similarity is not part of the graph** — it is discovery-only and lives in [TODO-0171](TODO-0171.md). Explicit links are stored as a column on the chunk row; petgraph handles traversal at query time. -## Architecture decision: canonical edge table + (deferred) derived adjacency layer +## Scope — extracted links only -### The single load-bearing decision +This TODO covers links **extracted from markdown content** — connections *derivable from the files themselves* (a wikilink, a markdown link, an `is_reference` frontmatter value, the heading structure). They are build state: recreatable, gitignored, never a source of truth. -**`edges.lance` is the source of truth**. Every other graph data structure is a derived index built from it. This invariant survives across all phases of the rollout and is what makes phased scaling possible: each new layer is purely additive, never replacing or migrating existing storage. +**Out of scope:** (1) *declared*, out-of-band references — e.g. a link a user or agent authors between a doc and a specific code symbol, maintained in a committed sidecar with drift / staleness detection over code; that is a separate concern (a distinct traceability project), not mdvs. (2) *semantic* edges — a cosine score is a hypothesis, not a connection; semantic similarity is discovery, handled by TODO-0171. The dividing line for edges: **extracted-and-explicit belongs here; declared-code and inferred-semantic do not.** -``` -edges.lance ← canonical edge storage (Phase 1) - Lance dataset. Mutable. Queryable. Columnar pushdown. - -[ optional derived layers, added when measurement shows they're needed ] - -adjacency.redb ← KV-backed adjacency (Phase 2, only when Phase 1 hits a ceiling) - Derived from edges.lance at build time. O(log n) per step. - -adjacency.mdix ← Custom CSR mmap'd sidecar (Phase 3, only if redb proves inadequate - for a measured workload — extremely unlikely at any realistic scale). -``` - -The architecture is intentionally designed so that adding a Phase 2 layer is a purely additive change: a new builder pass reads `edges.lance`, populates the index, and the query layer routes through the index when it exists. No schema migration. No breaking changes. - -### Why this shape, not the previous designs - -This TODO has had three storage proposals over its lifetime; the current design is the third. The reasoning for the latest revision (2026-06-19): - -1. **Denormalised List columns on chunk rows** (original 2026-03-14 / 2026-05-30 design). Pros: single LanceDB query composes graph filter with search. Cons: O(corpus_edges) projection per multi-hop query; rich edge attributes don't fit. **Rejected** because the multi-hop scaling story is structural — it gets worse with corpus growth — and rich edge attributes (anchor text, line numbers, content hash) want their own home. - -2. **Custom CSR mmap'd sidecar baked in from day one** (2026-06-16 design). Pros: constant-time per traversal step at any vault scale. Cons: 5-6 weeks of careful implementation work (writer + reader + bounds checks + version migration + corruption recovery + cross-platform mmap quirks + compaction logic); a binary format mdvs would own with all the long-tail costs; **bus-factor-of-1 maintenance risk** that the `lance-graph` case illustrates concretely. **Rejected** because the maintenance investment is not justified by current evidence: no user has reported the multi-hop wall, and even at 100K files an in-memory petgraph projection finishes traversals in under a second. - -3. **Canonical edge table + deferred adjacency layer** (current 2026-06-19 design). Phase 1 ships only the canonical store and uses petgraph at query time for multi-hop. Phase 2 adds redb-backed adjacency when measurement shows the petgraph projection is the bottleneck. Phase 3 — custom CSR — only if redb falls short, which is extremely unlikely given the O(log n) lookup cost is microseconds at any realistic scale. - -This walk through the design history is preserved in the spec because both rejected designs taught load-bearing lessons. - -### Why not depend on an existing graph layer +## Nodes — files and sections (internal hierarchy) -`lance-graph` ([github.com/lance-format/lance-graph](https://github.com/lance-format/lance-graph)) is an official Lance project that does Cypher → DataFusion translation over Lance node/edge tables. Evaluated 2026-06-19 and **not adopted**: +Nodes are at two granularities: -- Bus factor of 1 — top contributor authored 57% of recent commits and isn't a LanceDB employee -- 90-day velocity collapse since March 2026; PRs sitting unreviewed for 27+ days -- Not listed on lance.org's official project page -- Maintainer's own LDBC benchmarks show ~10x slower than Kuzu/Ladybug -- Locked to Lance 1.0.0 exact and edition 2021; mdvs is on edition 2024 -- The pieces it delivers (Cypher parser, Cypher → SQL translator) aren't what we want; the pieces we want (storage primitives, traversal) it doesn't really have — its "graph storage" is a HashMap of label → table name +- **File** — the document (root node). +- **Section** — a heading and its content, identified by `(file, heading-path)`. -The lance-graph experience is also instructive in the other direction: it's a case study of what happens to small graph projects on top of columnar engines (rapid build-up, then maintenance evaporation). Building our own custom CSR sidecar would expose us to the same fate, just with us holding the bag. The Phase 1 / canonical-store-only strategy is the antidote — minimal new code, no custom binary format, all heavy lifting deferred until evidence justifies it. +A markdown document has an implicit **heading tree**: H1 ⊃ H2 ⊃ H3; headings at the same level under the same parent are siblings; a paragraph belongs to its nearest preceding heading; the file is the root. Markdown nesting is positional, so the tree is reconstructed with a stack pass over the headings (via `tree-sitter-markdown` or a plain heading scan) — deterministic, model-free, cheap, so it is **derived on demand, never stored**. -`KuzuDB` (archived Oct 2025) and `CozoDB` (no commits since Dec 2024) are also dead. The category of embedded columnar-graph databases is in active decline. The decision is to stay on Lance's own infrastructure and grow our graph layer on top of it, deferring complexity until it's measurably needed. +The heading tree produces **Structural edges** — `contains` / `parent-of` / `sibling` — a distinct edge origin from Referential links. Keeping them distinct matters: a `--backlinks-of` query must not accidentally return the section next door. Structural edges are derived at query / validate time, never persisted. -## What to extract +**Sections only for v0.** No block / paragraph (`^blockid`) nodes — that finer tier is deferred (and is Obsidian-only anyway, see notations below). -Three link forms, all resolved to file paths within the scanned root; external `http(s)://` links are ignored. +**Validation win:** with the section tree, `check` detects **broken heading anchors** — `[[doc#Ghost]]` where the doc exists but the heading doesn't — a sharper integrity signal than "file missing." -1. **Markdown links**: `[text](relative/path.md)` — standard GitHub / mdBook style. `relation_type = "markdown_link"`. -2. **Wikilinks**: `[[page]]`, `[[page|display text]]`, `[[page#heading]]` — Obsidian / wiki style. `relation_type = "wikilink"`. `display text` → `anchor_text`; `heading` → `heading_anchor`. -3. **Frontmatter references**: any frontmatter field whose schema declares `is_reference = true` produces edges of type `fm_reference` with the field name as `relation_tag`. Lets users write `parent = "[[index]]"` or `related = ["[[a]]", "[[b]]"]` and have those participate in the graph. +## Link addressing, types, and notations -Resolution rules (configurable under `[graph]`): +A link is `(source-node, target-node)`, each file-or-section. The two granularities fall out of two independent axes: -- **Wikilink resolution.** Default strict: ambiguous wikilinks become `LinkAmbiguous` violations in `check` with one resolution applied (lexicographic-first by default). `[graph].resolve = "proximity"` opts into Obsidian-style closest-match resolution. -- **Broken links**: surface as `LinkBroken` in `check`. Soft by default (warning); strict opt-in via `[graph].strict = true`. Edge still stored with `broken = true` for `mdvs graph broken` to find. +- **Source granularity — from *where the link is authored*:** frontmatter → *file* source (no body position → the file root); body → *section* source (the section enclosing the link, derived from the link's position + the heading tree). Because links are stored per-chunk, the source section is **derivable**, not separately stored. +- **Target granularity — from *the target syntax*:** no anchor → the file root; a `#heading` / `#slug` anchor → a section. -## Storage +The four link types are one uniform model: -### `edges.lance` — the canonical edge table +| | → file | → section | +|---|---|---| +| **file** source (frontmatter link) | `parent = [[index]]` | `see = [[guide#Setup]]` | +| **section** source (body link) | body `[[guide]]` | body `[[guide#Setup]]` | -Single Lance dataset alongside `index.lance`. Schema (top-level columns): +**Both notations resolve to the same section node.** Each heading node carries two keys — its raw **text** and its canonical **slug** — so a heading link resolves identically whether written: -``` -source_file_id: Utf8 // hash-derived file id (matches index.lance's file_id) -target_file_id: Utf8 // resolved target's file_id, or sentinel for broken -relation_type: Utf8 // "wikilink" | "markdown_link" | "fm_reference" | "similarity" -relation_tag: Utf8 (nullable) // frontmatter field name for fm_reference; null otherwise -anchor_text: Utf8 (nullable) // wikilink display text -heading_anchor: Utf8 (nullable) // for [[file#Section]] -source_chunk_id: Utf8 (nullable) // chunk this edge was extracted from (null for fm/similarity) -source_line: UInt32 (nullable)// line within the source chunk -score: Float32 (nullable)// for relation_type = "similarity"; null otherwise -content_hash_at_extraction: Utf8 // chunk content hash when extracted (cache key) -declared_at: Timestamp // build timestamp when first observed -broken: Boolean // target couldn't be resolved at build time -ambiguous: Boolean // multiple resolution candidates at extraction time -``` +- Obsidian — `[[file#Heading]]`, nested `[[file#H1#H2]]` (a *path* of heading texts), disambiguated by hierarchy; or +- GitHub / standard markdown — `[text](file.md#slug)` (a *flat slug*), disambiguated by order (`slug`, `slug-1`, `slug-2`). -**Edge identity is content-addressed**, not monotonic. There is no explicit `edge_id` column. The natural key is `(source_file_id, target_file_id, relation_type, relation_tag, heading_anchor)`. Deleting and re-adding the same edge produces the same row identity; no risk of id reuse for semantically different edges. This was a specific finding from the 2026-06-17 adversarial review. +Pick **one canonical slugger** (github-slugger is the de facto standard) and document it — heading→slug differs slightly across renderers on unicode / punctuation. Because blocks are out of scope, the only Obsidian-exclusive capability (block refs) is gone, so within section-only the two notations reach **full parity**. -Standard Lance dataset. Queryable via DataFusion. The existing `--where` translator extends to it trivially. Lance scalar indexes on `source_file_id` and `target_file_id` make 1-hop queries cheap regardless of total edge count. +## Storage — a `links` column on the chunk row (no `edges.lance`) -**Why a separate Lance dataset rather than columns on `index.lance`**: edges are file-level (relationships between files), not chunk-level. Denormalising them across chunks bloats the chunk table, complicates its schema, and mixes two different access patterns. A dedicated edge dataset is the cleaner shape — and matches how TODO-0016 already separates "the thing the user has" (markdown files / chunks) from "indexes mdvs computes on top" (FTS, vector). `edges.lance` is in the second category. +A separate persisted edge dataset was considered and dropped. Persisting `.mdvs/` state is justified by *recompute cost* (embeddings need model inference); link extraction is a cheap scan — cheaper than the validation pass `check` already runs. A second Lance dataset (its own manifest, fragments, incremental-write story to keep in lockstep with `index.lance`) is a sledgehammer for a list of edges. -### Phase 1 query model (no derived adjacency layer) +Instead: store the outbound links extracted from each chunk as a **column on that chunk's row in `index.lance`**. -All graph queries route through one of two paths: - -**1-hop queries: direct DataFusion on `edges.lance`** with columnar pushdown. - -```bash -mdvs graph backlinks # WHERE target_file_id = -mdvs graph neighbors # WHERE source_file_id = -mdvs graph broken # WHERE broken = true -``` +- Proposed shape: `links: List`. `target_file_id` = `None` when unresolved (keep `raw` so `info` can list broken links); `target_heading` = the anchor as authored (heading-path for Obsidian, slug for GitHub) or `None` for a whole-file link; `notation` tells resolution which key to match. This folds the would-be `broken` / `ambiguous` / `anchor` / `heading` / `source_chunk_id` columns into one nested column. Lance/Arrow handles `List` (same machinery as the existing nested `data` frontmatter struct). +- **Per-chunk granularity is correct**: links physically live in a chunk, so storing them there is faithful and non-redundant. File→file / section edges come from a **group-by `file_id`** during the petgraph projection — part of the O(corpus) projection that already feeds petgraph, so it costs nothing extra. The source section is recovered from the chunk's line-range + heading tree. +- **Frontmatter `is_reference` links are nearly free**: those fields are already stored per-file in the `data` struct. Don't duplicate them into the `links` column — the projection parses the wikilink out of `data.` on read (file-source by definition). Only *body* links need the new column. +- Rides TODO-0173's incremental writes for free: the column is part of the chunk row, so rewriting a file's chunks rewrites its links in the same operation — consistency is automatic, no second incremental path. -Microsecond response at any scale. Indexes on `source_file_id` / `target_file_id` make these effectively O(log n) lookups. +**The one tradeoff, stated honestly:** a column on `index.lance` only exists after `mdvs build`, so the *persisted, queryable* graph couples to the search / build layer. The two-layer invariant is preserved by splitting responsibilities: `check` computes broken / ambiguous links (including broken heading anchors) **on the fly** during its existing scan (model-free, no build required → validation stands alone); `mdvs build` persists the `links` column; the query axes read it. -**Multi-hop queries: petgraph projection at query time**. +## Traversal — petgraph at query time -For `mdvs graph neighbors --depth N` and `mdvs search --connected-to --depth N`: +Project `(file_id, links)` + group-by + build `petgraph::DiGraph` over the Referential (and, when asked, Structural) edges, per query. Estimated: ~10ms at 1K files, ~100ms at 10K, ~1s at 100K, ~10s at 1M. At the corpora mdvs actually runs (Refractions 549, example_kb 45, K8s 1,669) this is sub-second and moot. The only thing that would bite is composing a graph filter into *every* search (rebuild petgraph per query) — that's the signal to add a deferred disposable cache (content-hash-keyed, rebuilt from the column). On-the-fly is the v0 answer; the cache stays unbuilt until a real corpus crosses ~100K files. -1. Project `(source_file_id, target_file_id)` from `edges.lance` into memory -2. Build a `petgraph::DiGraph` -3. Run BFS from the anchor file to depth N -4. Return the visited set, or pass it as a `WHERE file_id IN (...)` clause to a downstream search query +## UX — folds into `search` / `check` / `info` (no `mdvs graph` command) -Per-query cost is O(corpus_edges) for the projection. Concretely: +The graph is "just another part of the data," so it doesn't get its own verb. A **search is a set of composable narrowing operators over the document set**; each operator is optional, any subset composes: -| Vault | Edges | Projection + petgraph build | Verdict | -|---|---|---|---| -| 1K files | 20K | <10ms | Fine | -| 10K files | 200K | ~100ms | Fine | -| 100K files | 2M | ~1s | Annoying but workable | -| 1M files | 20M | ~10s | Unworkable — escalate to Phase 2 | - -Phase 1 is sufficient up to ~100K files. By the time mdvs is in regular use at that scale (no current user is), we'll have specific workload data to inform Phase 2. - -### Phase 2: derived adjacency via redb (deferred) - -When Phase 1's petgraph projection becomes the measured bottleneck, add a derived adjacency index. The recommended implementation is **redb** ([github.com/cberner/redb](https://github.com/cberner/redb)) — a pure-Rust embedded B-tree KV store, ACID, single-writer / multi-reader, mmap'd for zero-copy reads, ~3k stars, used by Iroh and others in production. - -The schema is intentionally minimal: +| Axis | What it does | Surface | +|---|---|---| +| Relevance | rank by similarity to an anchor | query string · `--related-to ` (semantic *discovery*, see TODO-0171 — **not a graph edge**) | +| Attributes | filter by typed frontmatter | `--where ""` (existing) | +| Relationships | filter by graph connection | `--connected-to --depth N` · `--backlinks-of ` (file- or section-level, e.g. `--backlinks-of guide#Setup`) | ``` -adjacency.redb table "forward": - key: xxh3(source_file_id) - value: bincode-encoded Vec<(target_file_id_hash: u64, edges_lance_row_id: u64)> - -adjacency.redb table "reverse": - key: xxh3(target_file_id) - value: bincode-encoded Vec<(source_file_id_hash: u64, edges_lance_row_id: u64)> - -adjacency.redb table "meta": - key: "edges_lance_revision" - value: u64 — the Lance dataset version this index was built from - key: "node_id_table" - value: bincode-encoded Vec<(file_id_hash: u64, file_id: Utf8)> — for round-tripping hashes back to file_ids +mdvs search --backlinks-of guide.md#Setup # who links into that section +mdvs search "calibration" \ + --where "status = 'active'" \ + --connected-to protocols/spec.md --depth 2 --mode hybrid ``` -Build pass at the end of `mdvs build`: scan `edges.lance`, group by source and target, write the two tables. The `edges_lance_row_id` in each entry lets traversal resolve back to the full edge row when attribute filters are involved. - -Read path: O(log n) B-tree lookup per traversal step. Microseconds at any realistic scale. ACID semantics protect against partial writes. No custom format to maintain. Single crate dependency. - -**Crash-recovery invariant**: if `edges_lance_revision` in the meta table doesn't match `edges.lance`'s current revision, the adjacency index is considered stale and queries fall back to petgraph projection. Builders rewrite the index atomically via redb's transaction semantics. The index is fully disposable — `rm adjacency.redb` and a subsequent `mdvs build` regenerates it. +The would-be `mdvs graph` namespace dissolves entirely: -### Phase 3: custom CSR sidecar (only if Phase 2 is measurably inadequate) +- **Navigation / filtering** → flags on `search` (above). A REPL "walk" mode is out — it would violate "no interactive prompts until 1.0." Human spatial navigation belongs to a future TUI / GUI, not the CLI; the CLI stays one-shot + `--output json`, the right shape for the agent audience. +- **Integrity** (broken links, broken heading anchors, ambiguous) → *violations*, so they belong in `check`, model-free. +- **Stats** (edge counts, most-linked files/sections) → `info`. -If — extremely unlikely — redb's O(log n) per-step cost becomes a measured problem at some scale we actually operate at, a custom CSR `.mdix` sidecar is the next escalation. This is the design from the 2026-06-16 revision (`docs/spec/archive/adjacency-sidecar-assessment.md` if it survives), with the adversarial-review fixes applied: +## Semantic is not part of the graph -- Atomic writes via `write-tmp + fsync + rename` -- Revision-mismatch behaviour: ignore the sidecar, fall back to redb (or petgraph) -- `file_id` stored directly in entries; no `node_id_table` indirection -- Sidecar is fully disposable; always rebuildable from `edges.lance` +A cosine score identifies a *possible* connection; it is never *a* connection. The graph asserts only **Referential** (explicit authored links) and **Structural** (heading-tree) edges — both deterministic. Semantic similarity lives entirely in **search as discovery**: `search --related-to ` surfaces similar docs to *suggest* links a human or agent then makes explicit. The workflows that turn similarity into explicit links are [TODO-0171](TODO-0171.md). Nothing semantic is stored as an edge, and `--connected-to` never traverses a semantic "edge" — there is none. -Phase 3 is deliberately not specified in detail because the conditions that would trigger it (vault size beyond 1M files / 10M edges with traversal-heavy workloads) are speculative. - -## Commands - -Three subcommands in v1, all powered by Phase 1's `edges.lance` (and Phase 2's index when present): - -- `mdvs graph backlinks ` — direct query on `edges.lance` -- `mdvs graph neighbors [--depth N] [--type T] [--tag G]` — depth 1 = direct query; depth > 1 = petgraph projection (Phase 1) or redb traversal (Phase 2) -- `mdvs graph broken` — direct query: `WHERE broken = true` - -Additions in `search`: - -- `mdvs search --connected-to [--depth N]` — graph filter composed with the existing search modes. Sidecar/petgraph produces the candidate file set; search runs `WHERE file_id IN (...)`. **Benchmark required** on a 50K+ file synthetic corpus before promoting beyond `--depth 1`. - -Additions in `check`: - -- `LinkAmbiguous` and `LinkBroken` violation kinds, emitted alongside frontmatter violations. Cheap (no model needed); reuses the existing validation pipeline. - -Additions in `info`: - -- Total edges, broken-link count, ambiguous-link count, top-N most-linked files. Reads `edges.lance` directly. - -Deferred to later phases: - -- `mdvs graph orphans` — needs efficient anti-join; reasonable for Phase 2 -- `mdvs graph related ` — depends on similarity edges from TODO-0171; ships when both TODOs have Phase 1 in -- `mdvs graph dot` — visualisation; nice-to-have; Phase 2+ -- `mdvs graph compact` — operational; Phase 3 only (if custom CSR ever lands) -- `mdvs graph edges` — paginated raw edge table; covered by `mdvs search --where` against `edges.lance` if anyone needs it +## What to extract -The original 8-command surface was overreach for v1 (per the adversarial review). Three is enough to deliver the user-visible value. +Link forms, all resolved to file paths (and optional section anchors) within the scanned root; external `http(s)://` links are ignored. -## Config +1. **Markdown links**: `[text](path.md)` → file; `[text](path.md#slug)` → section (GitHub slug). Relation `markdown_link`. +2. **Wikilinks**: `[[page]]` → file; `[[page#Heading]]` / `[[page#H1#H2]]` → section (Obsidian heading text / path); `[[page|alias]]` display text kept in `raw`. Relation `wikilink`. +3. **Frontmatter references**: fields with `is_reference = true` → `fm_reference` edges (file-source), tagged with the field name; target may be file or section. Derived from the stored `data` struct, not re-scanned. -New `[graph]` section in `mdvs.toml`: +Resolution rules (configurable under `[graph]`): -```toml -[graph] -enabled = true # whole graph layer; default false in Phase 1 to ease rollout -resolve = "strict" # "strict" | "proximity" -strict = false # broken links = violation when true (otherwise warning) -adjacency_index = "auto" # "auto" | "off" | "redb" — auto triggers redb when corpus crosses threshold -``` +- **Wikilink resolution.** Default strict: ambiguous file matches become `LinkAmbiguous` violations in `check` (lexicographic-first applied). `resolve = "proximity"` opts into Obsidian-style closest-match. +- **Broken links.** `LinkBroken` in `check` — covers both a missing *file* and a missing *heading anchor* (file resolves, heading not in its tree). Warning by default; `strict = true` makes them violations. The unresolved `raw` is retained in the column so `info` can list them. ## Frontmatter reference declaration @@ -234,7 +129,7 @@ New attribute on `[[fields.field]]` entries: [[fields.field]] name = "parent" type = "String" -is_reference = true # parses the value as a wikilink, produces fm_reference edges +is_reference = true # parse the value as a wikilink → fm_reference edge [[fields.field]] name = "related" @@ -242,70 +137,70 @@ type = "Array(String)" is_reference = true # array form: each element parsed as a wikilink ``` -When `is_reference = true`, the field's string value(s) are parsed as wikilinks. The result is one or more `fm_reference` edges in `edges.lance` with `relation_tag = `. Frontmatter reference parsing happens during the existing frontmatter validation pass — no separate scan. +When `is_reference = true`, the field's string value(s) are parsed as wikilinks during the existing frontmatter validation pass — no separate scan, and no separate storage (the edges derive from the `data` struct on read). -## Open decisions +## Config -1. **Target vault scale.** Phase 1 is good up to ~100K files. Phase 2 (redb) is good to any vault size you'd plausibly operate at. **Confirm**: are we comfortable with Phase 1 as the v0 shipping target? -2. **Edge attribute richness.** The current schema includes `anchor_text`, `heading_anchor`, `source_line`, `content_hash_at_extraction`. Each unlocks distinct capability. **Trim or keep.** -3. **Node granularity = file_id only.** Heading-anchor wikilinks carry `heading_anchor` as an attribute but resolve to file_id at the node level. Chunk-level addressing was explicitly considered and rejected as too much complexity for too little visible benefit. **Confirm.** -4. **`mdvs graph` v1 surface.** Three commands (`backlinks`, `neighbors`, `broken`) plus `search --connected-to`. Adversarial review wanted three; previous draft had eight. **Confirm.** -5. **`--connected-to` composition needs a benchmark.** At depth > 1 the candidate set can grow large; LanceDB `WHERE file_id IN (...)` with thousands of literals may not push down cleanly. **Schedule the benchmark before promoting beyond depth 1.** -6. **Frontmatter reference declaration via `is_reference = true`.** Cleanly extends the existing schema; no other use case overloads the attribute name. **Confirm.** -7. **`[graph].adjacency_index = "auto"` default.** The auto-trigger threshold for redb is something we set based on Phase 1 measurement. **Pick after Phase 1 ships.** +New `[graph]` section in `mdvs.toml`: -## Interaction with other TODOs +```toml +[graph] +enabled = false # whole graph layer; default off in v0 to ease rollout +resolve = "strict" # "strict" | "proximity" +strict = false # broken links become violations when true (otherwise warnings) +``` + +## Prior art — filter vs rank is the real organizing principle -- **TODO-0016 (Lance swap, done).** This TODO sits as a sibling Lance dataset (`edges.lance`) next to `index.lance`. Same architectural pattern — separate datasets for separable concerns, single storage stack. Does not revisit 0016's chunk-table shape. -- **TODO-0170 (incremental cache).** `edges.lance` mutations need to be tracked for incremental adjacency rebuild (Phase 2). For Phase 1 the entire petgraph projection is rebuilt per query — no incremental concern. Coordinate with 0170 once Phase 2 work begins. -- **TODO-0171 (similarity-edge graph).** Companion. Rewritten 2026-06-19 to share this same storage architecture: similarity edges live in `edges.lance` with `relation_type = "similarity"` and the `score` column populated. The `mdvs graph related ` command merges explicit and similarity edges by walking `edges.lance` and grouping by `relation_type`. -- **TODO-0156 (Array of structured items).** Sidestepped — edges live in their own Lance dataset, not as `Array(Object)` columns on chunks. +Surveyed how production systems put relevance + attribute + relationship in one query. Two philosophies: **(A) one query representation** with operators for all three — Cypher/GQL, SurrealQL (SQL + graph arrows), SQL/PGQ `GRAPH_TABLE(... MATCH ...)`, Weaviate GraphQL, Vespa YQL (`nearestNeighbor()` + `rank()`); **(B) multiple retrievers + fusion** — GraphRAG merges vector/graph/keyword retrievers with Reciprocal Rank Fusion (`Σ 1/(k+rank_i)`, rank-based so scores needn't share a scale). -## Scope and rollout +The cross-cutting lesson, more important than syntax: **each layer is either a hard filter (narrows, boolean) or a soft ranker (orders the survivors)** — Vespa's `rank()` ("retrieve by A, score by B"), Neo4j's pre/in/post-filter taxonomy. Attributes = filter; relevance = ranker; relationship = *either* (filter: "≤N hops"; rank: link-proximity boost). **mdvs already embodies this** — `--where` is a filter, query+`--mode` is a ranker, `hybrid` is already RRF fusion. So graph-as-filter via flags is the natural v0; graph-as-*boost* folds into the existing hybrid/RRF machinery later. -Three phases. Phase 1 alone delivers ~80% of the user-visible graph value. Phase 2 and Phase 3 are escalations gated on measured Phase 1 evidence — **do not commit to them in advance**. +Sources: [Neo4j vector+filter](https://neo4j.com/blog/genai/vector-search-with-filters-in-neo4j-v2026-01-preview/), [SurrealDB KG-RAG](https://surrealdb.com/blog/knowledge-graph-rag-two-query-patterns-for-smarter-ai-agents), [DuckPGQ SQL/PGQ](https://duckpgq.org/documentation/sql_pgq/), [Weaviate hybrid](https://docs.weaviate.io/weaviate/search/hybrid), [Vespa NN search](https://docs.vespa.ai/en/querying/nearest-neighbor-search), [RAG patterns 2026 / RRF](https://ailearningguides.com/rag-production-patterns-2026/). -### Phase 1 — `edges.lance` + petgraph at query time (~2 weeks) +## Comparative note — graphify -- Add `edges.lance` schema to `.mdvs/` layout -- Extract markdown links, wikilinks, frontmatter references during `mdvs build` -- Add `LinkAmbiguous` and `LinkBroken` violations to `check` -- Implement `mdvs graph backlinks`, `mdvs graph neighbors --depth 1`, `mdvs graph broken` -- Implement `mdvs search --connected-to --depth 1` -- Multi-hop fallback via petgraph projection (no caching) -- Surface graph stats in `mdvs info` -- Schema attribute `is_reference = true` on `[[fields.field]]` -- Config section `[graph]` with `enabled`, `resolve`, `strict` +The closest neighbouring tool, [graphify](https://github.com/safishamsi/graphify), persists an in-memory NetworkX graph to a single `graph.json` (node-link JSON) plus a derived `graph.html` — because that file *is* its database (its nodes are LLM-extracted concepts with nowhere else to live). mdvs's graph derives from data it already holds, so a standalone graph file would be a redundant second store; mdvs keeps a `links` column and treats `graph.json` / `graph.html` as a *derived export* (`mdvs export`, also giving Obsidian interop). Two robustness points worth carrying, which graphify independently reached: derived graph state should refuse to silently shrink, and directional edges should use a directed representation (`petgraph::DiGraph`). And a design contrast that confirms mdvs's choice: graphify's implicit edges are an **LLM judgment** (`semantically_similar_to`), whereas mdvs keeps implicit similarity out of the graph entirely (discovery-only, TODO-0171) — lighter, deterministic, and no LLM in the index path. -This is the entire shipping target for v0. Use it for several months on real corpora before deciding whether Phase 2 is needed. +## Online search — a separate source axis, deferred -### Phase 2 — redb adjacency (deferred, ~1 week, only if Phase 1 hits a measured ceiling) +"Search online" is another *source* (`--source local|web|both`), not another operator. It changes mdvs's identity (breaks offline / single-binary; needs API keys; web results aren't markdown-with-typed-frontmatter, so `--where` / schema / validation don't apply). Explicitly opt-in, decided as its own thing later; it must not shape the local unification. -- Add `redb` crate dependency -- Builder pass at end of `mdvs build` populates `adjacency.redb` from `edges.lance` -- Crash-recovery invariant: revision-mismatch falls back to petgraph projection -- Multi-hop traversal queries switch from petgraph to redb when the index exists -- `[graph].adjacency_index = "auto" | "off" | "redb"` config switch -- Promote `mdvs graph neighbors --depth N` and `mdvs search --connected-to --depth N` beyond depth 1 once benchmarks confirm composability +## Still open (do not implement until settled) -Gating criterion for starting Phase 2: real workload measurements showing the petgraph projection is the bottleneck. **No work on Phase 2 starts before that evidence exists.** +1. **Representation for v0**: orthogonal flags (zero parser work, no cross-layer `OR`) vs graph predicates embedded in `--where` (full cross-layer boolean, grows the translator). Leaning flags. +2. **Filter vs rank per graph axis**, and whether graph-as-boost (link-proximity / PageRank fused via RRF) is in v0 or later. +3. **Composition order** for `--connected-to` + a relevance query (pre- / in- / post-filter, per Neo4j). +4. **Structural edges in traversal**: does `--connected-to` walk `parent` / `sibling`, or is hierarchy a separate lens (breadcrumb / `--within `)? Leaning **separate** — keep origins distinct so `--backlinks-of` never returns a sibling section. +5. **Final `links` struct shape** — the exact representation of `target_heading` (store raw anchor + `notation`, or normalize to a resolved heading-path?). +6. **Canonical slugger** — confirm github-slugger and document the algorithm. +7. **GUI** (separate, not-yet-written TODO): overview-poster (static HTML, force-directed) vs navigation-cockpit (ratatui TUI). mdvs should offer `mdvs export --format graphjson|html` as a *derived* artifact from the `links` column. -### Phase 3 — custom CSR sidecar (extremely deferred, only if Phase 2 is measurably inadequate) +## Scope (v0) — what ships -Spec deliberately thin. The conditions that would trigger Phase 3 are speculative. +- Extract markdown links, wikilinks (both with optional heading anchors, both notations), and `is_reference` frontmatter refs during `mdvs build`; persist as the `links` column. +- Build the per-document heading tree (derived) → section nodes + Structural edges. +- `check`: `LinkBroken` (missing file **or** missing heading anchor) and `LinkAmbiguous`, computed on the fly (model-free, no build required). +- `search`: `--connected-to [--depth N]`, `--backlinks-of ` (file- or section-level), composed with the existing modes and `--where`. +- `info`: edge count, broken / ambiguous counts, top-N most-linked files/sections. +- `[graph]` config (`enabled` default false, `resolve`, `strict`) and the `is_reference` schema attribute. +- Multi-hop via petgraph projection, no cache. **Sections only** (blocks deferred). -### Phase 4 (independent) — PageRank as ranking signal +Semantic `--related-to` and the discovery workflows are **not here** — they are TODO-0171. Deferred, gated on evidence: a disposable traversal cache (only past ~100K files); graph-as-rank-boost. **`--connected-to --depth > 1` needs a benchmark** on a 50K+ synthetic corpus before promotion. -- Compute PageRank at build time via `petgraph` over `edges.lance` -- Store as `pagerank: Float32` denormalised column on `index.lance` chunk rows -- Expose via `search --rank-by score*pagerank` or always-on weighting per `[graph].pagerank_weight` +## Interaction with other TODOs -Independent of Phases 1-3. Ships when there's a real signal that semantic ranking alone leaves quality on the table. +- **TODO-0016 (Lance swap, done).** The `links` column lives on the existing chunk table — no new dataset, same single storage stack. +- **TODO-0173 (incremental writes, done).** The `links` column rides the existing incremental write path. +- **TODO-0171 (semantic-assisted link authoring).** Companion, reversed dependency: 0171's discovery workflows (`--related-to`, missing-links, clustering) *consume* this graph (to know what's already linked) and *feed* it (committed suggestions become edges here). Semantic never becomes an edge in this TODO. +- **TODO-0170 (incremental cache).** Only relevant if the deferred disposable traversal cache is ever built. +- **TODO-0156 (Array of structured items).** Sidestepped — links are an internal nested `List` storage column, not a user-facing `Array(Object)` frontmatter field. ## Design history - **2026-03-14**: TODO opened with markdown links + wikilinks extraction goal. -- **2026-05-30**: Redesigned for Lance with denormalised List columns on chunk rows (commit `953f98d`). TODO-0171 opened as companion. -- **2026-06-16**: Redesigned again for hybrid storage (edges table + custom CSR sidecar) after a separate architectural assessment surfaced multi-hop scaling concerns and the dead-graph-DB platform risk. -- **2026-06-17**: Adversarial review flagged the custom-CSR scope as overreach and identified specific correctness hazards (drift semantics, edge_id stability, node_id_table indirection). Parallel research evaluated `lance-graph` and found it not viable as a dependency. -- **2026-06-19**: Redesigned to the current layered shape — Phase 1 canonical edge table only, Phase 2 redb-backed adjacency as the natural escalation, Phase 3 custom CSR only if measurably necessary. The unbounded-scaling goal is preserved; the implementation work is deferred until measured evidence justifies each layer. +- **2026-05-30**: Redesigned for Lance with denormalised List columns on chunk rows. TODO-0171 opened as companion. +- **2026-06-16 / 06-17**: Redesigned for a canonical edge table + custom CSR sidecar; adversarial review flagged the CSR scope as overreach and evaluated `lance-graph` (Cypher → DataFusion over Lance) as non-viable (bus factor of 1, locked to Lance 1.0.0, ~10× slower than Kuzu). `KuzuDB` / `CozoDB` are dead — the embedded columnar-graph category is in decline. +- **2026-06-19**: Layered shape — a canonical `edges.lance` table with petgraph at query time, redb/CSR adjacency as deferred escalations. +- **2026-06-25**: Superseded storage and UX. Dropped `edges.lance` for a `links` column on the chunk row; dissolved the standalone `mdvs graph` command (folds into `search` / `check` / `info`); moved semantic on-the-fly. Captured prior art (filter-vs-rank) and situated against graphify. +- **2026-07-03**: Cleaned up (removed the superseded `edges.lance` / phased-adjacency sections) and **finalized the graph model**: nodes = files **and** sections (internal heading hierarchy → Structural edges + broken-heading detection); both Obsidian and GitHub heading notations resolve to the same section node (one canonical slugger); the file/section × file/section link-type model; **semantic excluded from the graph** (discovery-only, moved to TODO-0171, which was reframed from a stored similarity-edge graph into semantic-assisted link-authoring workflows). Sections only; blocks deferred. diff --git a/docs/spec/todos/TODO-0171.md b/docs/spec/todos/TODO-0171.md index f201fc6..10768a8 100644 --- a/docs/spec/todos/TODO-0171.md +++ b/docs/spec/todos/TODO-0171.md @@ -1,175 +1,97 @@ --- id: 171 -title: "Similarity-edge graph: top-K semantic neighbors per file" +title: "Semantic-assisted link authoring: turn similarity into explicit links" status: todo priority: medium created: 2026-05-30 -updated: 2026-06-19 +updated: 2026-07-03 depends_on: [106] blocks: [] related: - 16 - 106 - - 157 - 170 --- -# TODO-0171: Similarity-edge graph — top-K semantic neighbors per file +# TODO-0171: Semantic-assisted link authoring — turn similarity into explicit links ## Summary -Augment the link graph from [TODO-0106](TODO-0106.md) with a parallel **implicit edge** source: for every file, materialise the top-K most cosine-similar other files using the existing vector index, and store the result as rows in the same `edges.lance` table that TODO-0106 introduces — with `relation_type = "similarity"` and the cosine score in the existing `score` column. The high-value capability is surfacing connections that no one wrote down explicitly — semantic neighbours a reader didn't bother to wikilink but a recommender system would. - -This TODO depends on TODO-0106's Phase 1 landing first: the `edges.lance` storage layer is shared. +Semantic search is a **discovery** tool, not a graph. This TODO defines the **workflows** that use semantic similarity — nearest-neighbour and clustering over the embeddings mdvs already stores — to *surface candidate connections*, which a user or agent then **commits as explicit links** (wikilinks / `is_reference` frontmatter values). Those explicit links are the durable graph edges from [TODO-0106](TODO-0106.md); the semantic signal itself is never stored as an edge. This is mdvs's expression of *discover semantically → commit referentially*: similarity proposes, the author disposes. ## Motivation -The explicit-link graph in TODO-0106 captures only the structure a writer remembered to encode. Real notebooks are full of related material that's never linked — different vocabulary for the same topic, parallel projects, shared references — and surfacing these is the main thing a search-aware notes tool can offer that hand-curated graphs cannot. - -The existing semantic search machinery already computes the answer for one query at a time. Pre-computing the answer for every file as a fixed top-K neighbourhood turns "related notes" from a query into an edge-table read, and makes it possible to compose semantic adjacency with link-graph operations. - -## Storage - -Similarity edges share `edges.lance` with explicit links from TODO-0106 — no separate column, no separate dataset. Each similarity edge is a row: - -``` -source_file_id: -target_file_id: -relation_type: "similarity" -relation_tag: null -anchor_text: null -heading_anchor: null -source_chunk_id: null -source_line: null -score: 0.0..1.0 // cosine similarity -content_hash_at_extraction: -declared_at: -broken: false -ambiguous: false -``` - -K is config-driven (`[graph].similar_top_k`, default 10). For a 10K-file vault with K=10 that's 100K similarity edges in `edges.lance` alongside however many explicit edges exist. Storage is trivial — Lance's columnar compression makes the cost of repetitive `relation_type = "similarity"` values negligible. - -Identity follows TODO-0106's content-addressed scheme. The natural key is `(source_file_id, target_file_id, relation_type, relation_tag, heading_anchor)`. Recomputing similarity for the same pair produces the same row; replacing one with an updated `score` is a normal Lance update. - -### What was rejected - -The 2026-05-30 draft of this TODO proposed a separate `similar_files: List>` column denormalised on every chunk row of `index.lance`. That design is **superseded** by TODO-0106's edges-table architecture: - -- Mixing similarity edges into chunk rows complicated the chunk schema with file-level data -- It couldn't compose with explicit links cleanly — `mdvs graph related ` would have to read two different storage shapes and merge in code -- The denormalisation argument (List columns RLE-compress nicely across chunks of the same file) is moot once edges live in a separate dataset designed for that access pattern - -Putting similarity edges in `edges.lance` solves both: same storage shape as explicit links, unified `mdvs graph related ` view, single query path. - -## How it's computed at build time +The explicit-link graph in TODO-0106 captures only the structure a writer remembered to encode. Real notebooks are full of related-but-unlinked material — different vocabulary for the same topic, parallel projects, shared references. Surfacing those and helping the author turn them into real links is the main thing a search-aware notes tool can offer that a hand-curated graph cannot. But the connection only becomes trustworthy and queryable once it is *written down* — a cosine score is a hypothesis, not a fact. -The existing vector index (cosine, `nearest_to`) already supports the query. The build-time pass: +## Principle — semantic proposes, the author commits -1. After the chunk Lance table is written and the vector index (or exact flat scan path) is available, iterate every chunk row. -2. For each chunk, run `nearest_to(chunk.embedding).limit(K * OVER_FETCH_FACTOR)`. -3. Filter out same-file hits; group by `file_id`; keep the max chunk-similarity per file (same logic `search` already uses). -4. Take top K, write K rows into `edges.lance` with `relation_type = "similarity"` and the cosine score. +- **Semantic similarity is ephemeral discovery.** Computed on the fly from stored embeddings; never persisted as an edge, never counted by the graph, never a source of truth. +- **Explicit links are the truth.** A suggestion becomes a graph edge only when a human or agent writes a `[[wikilink]]` or an `is_reference` frontmatter value. TODO-0106 then picks it up. +- The stored graph therefore contains only what someone deliberately linked. Recall is bounded by author diligence — which is the point: the graph asserts only vetted connections. -Build-time cost is O(N_chunks × K_search), exactly what `search` pays per query — paid once at build time instead of per query. For a 10K-chunk corpus with K=10 and the IVF-PQ index active (above `VECTOR_INDEX_MIN_ROWS = 10_000`), this is comfortably sub-minute. For smaller corpora on the flat-scan path it's O(N²) but N is small (<10K), still seconds. +## Workflows -### Incremental story +Three, in rough order of value. -Similarity is corpus-wide: when a file's body changes, its outgoing similarity edges change, but its *incoming* similarity edges may also change (it might enter or leave other files' top-K). For v0, accept that the similarity pass is corpus-wide and triggered when: +### 1. Find related — "what should this link to?" -- the embedding model / revision changed (forces rebuild anyway), or -- more than some threshold of files changed in a build (e.g. >1%), or -- the user passes `--rebuild-similarity`. +`search --related-to ` (from TODO-0106) → a ranked list of the most cosine-similar other files → the author reviews and adds `[[links]]` to the ones that are genuinely related. One `nearest_to` over the existing index; model-free for an existing file (reads its stored embedding). This is the primitive the other two build on. -For small per-build deltas, leave the similarity edges stale; document the staleness. This is acceptable because similarity is a soft signal — staleness degrades quality but doesn't break correctness, unlike `links_in_file` which has correctness implications. +### 2. Suggest missing links — "similar but not linked" (the high-value one) -Coordinate with TODO-0170's reverse-dependency story when Phase 2 of TODO-0106 introduces the redb adjacency index. For Phase 1 (no derived adjacency layer), staleness in `edges.lance` is the only concern. +The gap between *semantic proximity* and the *explicit graph*: pairs of files that are highly similar yet have **no** explicit link between them. Computed as `top-K nearest neighbours` **minus** `already-linked` (the explicit edges from TODO-0106). Surfaces "these two are clearly about the same thing and nobody connected them" — the suggestions most worth acting on. Needs both signals: embeddings (proximity) and the 0106 link graph (what's already linked). -## Query model +### 3. Cluster the corpus — "what groups exist, and are they cross-linked?" -### Read similarity edges directly (Phase 1) +Group documents by embedding similarity (k-means, or community detection over the on-the-fly k-NN graph) → surface clusters of related documents → suggest either a hub / MOC note or intra-cluster cross-links for clusters that are internally under-linked. Clusters are a **render / analysis-time artifact**, computed on demand and discarded — never a stored `community` label (notes belong to multiple topics; partition labels are unstable across builds). -```bash -# files most similar to a specific file — direct edges.lance query -mdvs graph similar +## The agent loop -# search filtered by being a similarity neighbour of X -mdvs search "calibration" --where "EXISTS (SELECT 1 FROM edges WHERE source = file_id AND target = '' AND relation_type = 'similarity')" -``` +Because mdvs's callers are agents, these workflows are naturally an **author-assist loop**, entirely within mdvs: -The `--where` syntax exact shape depends on how TODO-0106's `--where` translator handles cross-dataset references. The simpler invocation `mdvs graph similar ` is the user-facing surface; the `--where` form is plumbing. +1. mdvs **suggests** (`--related-to` / missing-links / clusters), output as JSON. +2. The agent (or human) **commits** the chosen ones by writing `[[links]]` / frontmatter refs into the files. +3. mdvs **validates + surfaces** them via `check` (broken / ambiguous) and the 0106 link graph. -### Composed with explicit links via `mdvs graph related` +mdvs suggests, the agent commits, mdvs maintains — no LLM inside mdvs, no stored semantic edges. -The whole point. `mdvs graph related ` walks `edges.lance` for both edge kinds and merges: +## No storage -- Outgoing explicit edges (this file links to them) -- Incoming explicit edges (they link to this file) -- Outgoing similarity edges (cosine neighbours) +Nothing here is persisted. Nearest-neighbour and clustering are computed on demand from the embeddings already in `index.lance`. The only persisted graph is TODO-0106's explicit `links` column. This is a deliberate reversal of the earlier design (below). -Ranking weights are configurable but a reasonable default is `1.0` for explicit link presence + `score` for similarity. Bucketed output ("Explicit" / "Semantic neighbours" sections) is an alternative; UX decision when implementing. +## Command surface (open) -## What this explicitly does NOT do +Candidates, to settle when implementing: -(Same boundary list as the 2026-05-30 draft. The rejections still hold under the new storage.) +- `search --related-to ` — exists in TODO-0106 (nearest-neighbour). +- Missing-links — a flag on `--related-to` (e.g. `--unlinked`, restrict to non-linked neighbours) or a dedicated `mdvs suggest-links []`. +- Clustering — `mdvs cluster` / `mdvs suggest`, or folded into `info` as a corpus overview. +- All support `--output json` for the agent loop. -- **No `community: Int32` label per file.** Hard-partition clustering (Leiden / Louvain) is the wrong output shape for a notes corpus: notes naturally belong to multiple topics; partition labels are unstable across incremental builds; the continuous top-K neighborhood carries strictly more information than any single label. If a clustered overview view becomes valuable later, compute it at query time over the materialised edges in petgraph and discard the labels — same scratchpad pattern TODO-0106 uses for traversal during Phase 1. -- **No frontmatter co-occurrence as a stored edge type.** "Both files have `topic = X`" is expressible via `--where` against `index.lance`'s `data.*`; doesn't need materialisation as edges. -- **No named-entity / concept extraction.** Would require either an NER ONNX model in the build path or LLM calls at index time. Changes the project's character (build no longer offline / model-free at the validation layer; binary size grows; index time multiplies). Out of scope for v0; revisit once the simpler implicit-edge work has been used in anger. -- **No co-citation edges as a stored type.** "A and B both link to C" is a derived query over `edges.lance` (self-join or DataFusion equivalent) and doesn't need its own row type. Surface in `mdvs graph related` output if it's wanted. +## What was rejected -## Commands +Earlier drafts stored the similarity signal as graph edges: -Same `mdvs graph` surface as TODO-0106 plus: +- **2026-05-30**: a `similar_files: List>` column denormalised on chunk rows. +- **2026-06-19**: top-K similarity edges as rows in a shared `edges.lance` table (`relation_type = "similarity"`), merged with explicit links in a `mdvs graph related` view. -- `mdvs graph similar ` — direct query: `WHERE source_file_id = AND relation_type = 'similarity'`, ordered by `score DESC`. -- `mdvs graph related ` — merges explicit and similarity edges (ships once both this TODO and TODO-0106's Phase 1 have landed). - -Additions in `info`: - -- Average neighbour score, count of files with no neighbours above a threshold, count of similarity edges total. - -## Open design questions - -1. **Default K.** 10 is the starting point; could be lower (5) for tighter storage, or higher (20) for downstream consumers to threshold. **Pick after the storage cost is measured on a real corpus.** -2. **Similarity floor.** Files with no truly close neighbours pull noise at K=10. Optional `[graph].similar_min_score` truncates. **Default 0 for v0 (no truncation); users opt in.** -3. **Stability across embedding model swaps.** Changing the embedding model invalidates the similarity edges. The schema-hash mechanism already covers this — model identity change triggers full rebuild. **Document the implication; no code change needed.** -4. **Symmetry.** Cosine is symmetric but top-K isn't: A in B's top-10 doesn't imply B in A's top-10. "Files I'm similar to" reads the forward direction; "files for which I'm in their top-K" requires a reverse pass. For v0, ship only the forward direction; reverse becomes free once TODO-0106's Phase 2 redb adjacency exists (just query reverse edges by `relation_type = "similarity"`). -5. **Interaction with hybrid / FTS modes.** Similarity is purely vector-based. Materialising FTS-based "lexically similar" neighbours has marginal value and complicates the schema. **Skip for v0.** +Both are **rejected**. Storing similarity as an edge freezes a fuzzy, unstable, unverified hypothesis into structural truth — it pollutes the graph with connections nobody vetted, conflates discovery with commitment, and needs a corpus-wide recompute + staleness story for a signal that should simply be recomputed on demand. Semantic stays ephemeral; only explicit links are stored. This also drops the `edges.lance` dataset, the `[graph].similar_*` knobs, the build-time similarity pass, and `--rebuild-similarity` from those drafts. ## Interaction with other TODOs -- **TODO-0106 (link graph).** Hard dependency. This TODO ships after TODO-0106's Phase 1 — same `edges.lance` storage, same `mdvs graph` command surface, same `[graph]` config section. `mdvs graph related ` ships once both TODOs have Phase 1 done. -- **TODO-0170 (incremental cache).** Similarity recomputation is global (a changed file can shift other files' top-K). v0 strategy is "rebuild on model change / large delta, manual flag otherwise." Coordinate with 0170's reverse-dependency story when designing the rebuild trigger. -- **TODO-0157 (incremental ANN optimize).** Independent; performance is correlated because the similarity pass benefits from a well-optimised ANN index, but the storage layer doesn't couple. -- **TODO-0016 (Lance swap, done).** This TODO is a direct application of 0016's "single Lance stack, separate datasets for separable concerns" architecture. - -## Scope and rollout - -Single slice, but gated on TODO-0106's Phase 1 completing. - -### Slice 1 — similarity pass writes to `edges.lance` (~1 week, after TODO-0106 Phase 1) - -- After the chunk Lance table is built and the vector index (or flat-scan) is ready, run the similarity pass -- Write K rows per file to `edges.lance` with `relation_type = "similarity"` and `score` -- Add `[graph].similar = true` and `[graph].similar_top_k = 10` config knobs -- Implement `mdvs graph similar ` (direct query) -- Implement `mdvs graph related ` (merges with explicit edges from TODO-0106) -- Surface stats in `mdvs info` - -### Slice 2 — incremental rebuild policy (~0.5 week) - -- Track when a corpus-wide similarity rebuild is needed -- Surface staleness in `mdvs info` ("similarity edges are stale; run `mdvs build --rebuild-similarity`") -- `--rebuild-similarity` flag on `mdvs build` - -### Slice 3 (optional) — `--boost-by` ranking +- **TODO-0106 (link graph).** Hard dependency. 0171 *consumes* 0106's explicit link graph (to know what's already linked, for missing-link detection) and mdvs's embeddings (for proximity), and *feeds back* into it (committed suggestions become 0106 edges). `--related-to` is defined in 0106. +- **TODO-0170 / TODO-0157 (incremental cache / ANN optimize).** Relevant only to the *latency* of the on-the-fly nearest-neighbour queries; no storage coupling. +- **TODO-0016 (Lance swap, done).** Reuses the existing vector index; adds no dataset. -- `mdvs search ... --boost-by similar_files ` to bias results toward the reference's neighbourhood. Reads similarity edges from `edges.lance` and weights the search scores. +## Open questions -Default `[graph].similar = false` until storage cost and quality have been measured on a real corpus (Refractions or the K8s benchmark corpus). +1. **Missing-link threshold.** Above what cosine score is a similar-but-unlinked pair worth suggesting? A default plus a `--min-score` override; pick on a real corpus (Refractions). +2. **Clustering algorithm** (k-means vs community detection over the k-NN graph), and whether clustering ships in v0 or after the two nearest-neighbour workflows. +3. **Agent-mode JSON shape** — what a suggestion record contains so an agent can act (source, target, score, already-linked?, suggested relation). +4. **Symmetry.** top-K isn't symmetric (A in B's top-K ≠ B in A's). For missing-links, treat a pair as a candidate if either direction is a near neighbour. ## Design history -- **2026-05-30**: Opened with a `similar_files: List>` column denormalised on chunk rows of `index.lance`. -- **2026-06-19**: Rewritten to align with TODO-0106's edges-table storage. Similarity edges now live as rows in `edges.lance` with `relation_type = "similarity"` and the `score` column. The original column-on-chunks proposal is preserved here as a rejected alternative. +- **2026-05-30**: Opened as a similarity-edge graph — a `similar_files` column on chunk rows. +- **2026-06-19**: Rewritten to store similarity edges in TODO-0106's `edges.lance` table. +- **2026-07-03**: Reframed. Semantic similarity is **discovery only, never a stored edge** (consistent with excluding semantic from the graph in TODO-0106). This TODO is now the set of **workflows** that use nearest-neighbour + clustering to *suggest* connections a user or agent then *commits as explicit links*. Dropped the stored-edge design, `edges.lance`, the `[graph].similar_*` knobs, the build-time pass, and `--rebuild-similarity`. diff --git a/docs/spec/todos/TODO-0193.md b/docs/spec/todos/TODO-0193.md new file mode 100644 index 0000000..5a274f1 --- /dev/null +++ b/docs/spec/todos/TODO-0193.md @@ -0,0 +1,61 @@ +--- +id: 193 +title: Support .mdx files — free validation, gated search-body stripping +status: todo +priority: medium +created: 2026-07-06 +depends_on: [] +blocks: [] +--- + +# TODO-0193: Support `.mdx` files — free validation, gated search-body stripping + +## Summary +Teach mdvs to ingest `.mdx` files. MDX frontmatter is byte-identical to Markdown frontmatter, so the validation layer (`init` / `check` / `update`) works the moment the scanner accepts the extension — one line, zero new logic. The search layer needs more: MDX bodies carry `import`/`export` statements and `{expr}` JavaScript expressions that leak into embeddings as noise, so a fence-aware pre-chunk strip stage is required before `.mdx` bodies produce good search results. Ship the two halves as separate increments; the validation half unlocks docs-site frontmatter linting (MDX's actual unmet pain point) on its own. + +## Motivation +MDX (Markdown + JSX) is the dominant content format for docs-site pipelines — Docusaurus, Astro, Nextra, Contentlayer — where frontmatter *drives* the build (routing, sidebar order, tags). Nothing in that ecosystem validates frontmatter against a schema before the build breaks. There is no "Obsidian for MDX" either; MDX note-taking is done via Obsidian plugins that render `.mdx` as `.md`. mdvs's validation layer maps onto this niche almost for free. See the design conversation that produced this TODO for the landscape survey. + +## Current behavior (grounding) +- **Ingest gate.** The default scan glob is `**` (`../schema/config.rs` default) — it matches every path — so the extension filter in `discover/scan.rs` is the *sole* markdown-only gate: + ```rust + .filter(|e| e.path().extension().is_some_and(|ext| ext == "md" || ext == "markdown")) + ``` + Everything downstream is extension-agnostic. +- **Frontmatter.** Engine selection keys off the leading delimiter (`detect_engine(&raw)` probing `---` / `+++` / `{`), never the extension. An MDX frontmatter block is byte-identical to a Markdown one → validation is unaffected by the body. +- **Body → search.** `index/chunk.rs::extract_plain_text` collects only `Event::Text` from pulldown-cmark. JSX *tag markup* (``) parses as `Event::Html` / `Event::InlineHtml` and is already dropped for free; text *between* tags (`Important` → `Important`) survives as `Event::Text`, which is desirable. + +## What leaks into embeddings (the search problem) +Parsed as `Event::Text` by a CommonMark parser, therefore embedded as noise: +1. **`import` / `export` statements** — every MDX file opens with a block of these; they are plain paragraphs to pulldown-cmark, not HTML. +2. **Expression braces** — `{frontmatter.title}`, `{2026 - startYear}`, `{items.map(...)}`. +3. **Splitter confusion** — JSX inside a paragraph can blur where `MarkdownSplitter` (`index/chunk.rs`) places semantic boundaries (quality wobble, not garbage). + +## Proposed work + +### Increment A — ingest + validation (trivial, ship first) +- Add `|| ext == "mdx"` to the extension filter in `discover/scan.rs`. +- Add `.mdx` fixtures to the scan tests; confirm frontmatter detection + `check` behave identically to `.md`. +- No search changes. `build` / `search` will "work" on `.mdx` but with the noise described above — acceptable for an initial validation-focused release, documented as a known limitation. + +### Increment B — search-body strip (the real work, gated) +- New **fence-aware** pre-chunk strip stage, wired upstream of `Chunks::new` in `index/chunk.rs` (mirrors how `strip_wikilinks` runs as a targeted pass inside `extract_plain_text`). It must: + - remove leading/inline `import` / `export` lines, + - remove `{expr}` expressions **outside** fenced code blocks, + - **never** touch content inside code fences — the `plain_text_preserves_code_block_content` test in `chunk.rs` guards this and it is desirable content. +- Because a fence-aware `{…}` strip is not a trivial regex (expressions nest and span lines; `{` appears in prose and code), this is the one genuine unit of implementation work. + +## Design decisions to settle before coding +1. **Opt-in vs automatic strip.** Stripping `{…}` from a plain-Markdown `.md` file could eat legitimate prose. Recommend gating the strip by extension (`.mdx` gets stripped, `.md` does not) rather than a global transform; optionally back it with an explicit `[scan]` flag. Pure-Markdown vaults must not pay MDX cost or risk false strips. +2. **Strip aggressiveness.** Minimal (drop `import`/`export` lines + inline `{…}` outside fences) removes ~90% of the noise cheaply. Full JSX-tree removal is more correct but wants a real MDX AST (e.g. `markdown-rs` MDX mode) — heavier, likely not worth it for v0. Recommend minimal. +3. **Config surface.** If gated by a flag, decide the `[scan]` field name and default. If gated purely by extension, no config change. + +## Non-goals +- No JSX/React rendering, component resolution, or evaluation — mdvs indexes prose, it does not build a site. +- No new heavy MDX-AST dependency in v0 unless decision (2) chooses full-tree removal. + +## Files likely touched +- `crates/mdvs/src/discover/scan.rs` — extension gate (A) + tests. +- `crates/mdvs/src/index/chunk.rs` — strip stage + tests (B). +- `crates/mdvs/src/schema/shared.rs` / `config.rs` — only if a `[scan]` flag is chosen (B). +- Spec: `../architecture.md` (pipeline note), possibly a `book/` page on MDX support. diff --git a/docs/spec/todos/index.md b/docs/spec/todos/index.md index 14c5949..038c4ea 100644 --- a/docs/spec/todos/index.md +++ b/docs/spec/todos/index.md @@ -172,7 +172,7 @@ | [0168](TODO-0168.md) | Compact-snippet output mode for LLM-friendly search results | todo | medium | 2026-05-27 | | [0169](TODO-0169.md) | Investigate Lance encoding panic on large markdown corpora | done | medium | 2026-05-28 | | [0170](TODO-0170.md) | Incremental check cache to keep auto-validation cheap at scale | deferred | low | 2026-05-28 | -| [0171](TODO-0171.md) | Similarity-edge graph: top-K semantic neighbors per file | todo | medium | 2026-05-30 | +| [0171](TODO-0171.md) | Semantic-assisted link authoring: turn similarity into explicit links | todo | medium | 2026-05-30 | | [0172](TODO-0172.md) | Cheap wins in check::validate — precompile globs, hoist conversions, fast-path validators | done | high | 2026-05-29 | | [0173](TODO-0173.md) | Incremental Lance writes — stop nuking the index on every build | done | high | 2026-05-29 | | [0174](TODO-0174.md) | content_hash should cover frontmatter, not just the parsed body | todo | medium | 2026-05-29 | @@ -194,3 +194,4 @@ | [0190](TODO-0190.md) | Design `mdvs scaffold` — unified agent-harness integration command surface | done | high | 2026-06-22 | | [0191](TODO-0191.md) | Auto-rewrite array-field comparisons in `--where` (parser-based, with translation note) | done | medium | 2026-06-23 | | [0192](TODO-0192.md) | Don't persist the mock-embedder default to `mdvs.toml` | done | high | 2026-06-23 | +| [0193](TODO-0193.md) | Support .mdx files — free validation, gated search-body stripping | todo | medium | 2026-07-06 |