Fix/hybrid search id space

CLAUDE.md

@@ -105,7 +105,7 @@ docker logs <container-id>
 
 1. **Modular Trait-Based Design**: Each major component is defined by a trait (EmbeddingProvider, VectorDatabase) with concrete implementations, enabling easy swapping of backends.
 
-2. **MCP Protocol Integration**: Uses `rmcp` macros (`#[tool]`, `#[prompt]`, `#[tool_router]`, `#[prompt_router]`) to define 9 MCP tools and 9 slash commands. The server communicates over stdio following MCP spec.
+2. **MCP Protocol Integration**: Uses `rmcp` macros (`#[tool]`, `#[prompt]`, `#[tool_router]`, `#[prompt_router]`) to define 13 MCP tools and 12 slash commands. The server communicates over stdio following MCP spec.
 
 3. **Async-First Architecture**: Built on Tokio runtime with async traits. File walking runs on blocking threads via `tokio::task::spawn_blocking` to avoid blocking the async runtime.
 
@@ -117,7 +117,7 @@ docker logs <container-id>
 
 ```
 src/
-├── mcp_server.rs           # Main MCP server with 9 tools + 9 prompts
+├── mcp_server.rs           # Main MCP server with 13 tools + 12 prompts
 │   ├── RagMcpServer        # Server state (embedding provider, vector DB, chunker, hash cache)
 │   ├── Tool handlers       # index_codebase (smart), query_codebase, find_definition, etc.
 │   └── Prompt handlers     # Slash commands for each tool
@@ -142,10 +142,12 @@ src/
 │   ├── repomap/            # AST-based symbol extraction (fallback provider)
 │   │   ├── mod.rs          # RepoMapProvider implementing RelationsProvider
 │   │   ├── symbol_extractor.rs  # Extract definitions from AST nodes
+│   │   ├── import_extractor.rs  # Extract import/use/include bindings (SymbolKind::Import)
 │   │   └── reference_finder.rs  # Find references via identifier matching
 │   ├── storage/            # Relations storage layer
 │   │   ├── mod.rs          # RelationsStore trait
-│   │   └── lance_store.rs  # LanceDB storage for definitions/references
+│   │   └── lance_store/    # LanceDB storage (definitions + references tables,
+│   │                       #   populated with definitions during indexing)
 │   └── stack_graphs/       # Optional: High-precision name resolution (feature-gated)
 │       └── mod.rs          # StackGraphsProvider for Python, TypeScript, Java, Ruby
 ├── bm25_search.rs          # Tantivy BM25 keyword search with RRF fusion
@@ -220,6 +222,8 @@ src/
 - **Hybrid Architecture**: Uses stack-graphs (high precision, ~95%) for Python, TypeScript, Java, Ruby when feature enabled; RepoMap fallback (~70%) for all other languages
 - **RelationsProvider Trait**: Abstraction for extracting definitions and references from source files
 - **SymbolExtractor**: Uses tree-sitter AST to extract function, class, method, struct definitions
+- **ImportExtractor**: Extracts import/use/include statements as `SymbolKind::Import` definitions, one per bound name (`use a::{B, C as D}` yields `B` and `D`); glob/side-effect imports are reported as skipped definitions
+- **LanceRelationsStore**: Definitions are persisted to LanceDB tables (`relations_definitions`, `relations_references`) during indexing; idempotent per file, cleaned up on file removal and clear_index
 - **ReferenceFinder**: Text-based identifier matching with context analysis (call, read, write, import, etc.)
 - **PrecisionLevel**: High (stack-graphs), Medium (AST-based RepoMap), Low (text-based)
 - **Symbol Types**: Function, Method, Class, Struct, Interface, Trait, Enum, Module, Variable, Constant, etc.
@@ -286,11 +290,12 @@ When adding new tools:
 All tools return JSON responses conforming to types defined in `types.rs`. Key response types:
 - `IndexResponse`: files_indexed, chunks_created, embeddings_generated, duration_ms, errors, mode (full or incremental)
 - `QueryResponse`: results (SearchResult[] with vector_score, keyword_score, combined_score), duration_ms
-- `StatisticsResponse`: total_files, total_chunks, language_breakdown
+- `StatisticsResponse`: total_files, total_chunks, language_breakdown, total_definitions, total_references, files_with_definitions
 - `ClearResponse`: success, message
 - `FindDefinitionResponse`: definitions (DefinitionResult[] with file_path, line, symbol info), precision_level
 - `FindReferencesResponse`: references (ReferenceResult[] with file_path, line, reference_kind), precision_level
 - `GetCallGraphResponse`: node (CallGraphNode with callers/callees), precision_level
+- `FindUnusedResponse`: candidates (UnusedCandidate[] with confidence high/medium/low and reason), unverifiable_imports, probes_exhausted, truncated
 
 ### Prompt (Slash Command) Pattern
 Prompts in `#[prompt_router]` expand to user messages that instruct the AI to call the corresponding tool. Example:
@@ -302,7 +307,7 @@ async fn index_prompt(&self, Parameters(args): Parameters<serde_json::Value>)
 
 ### Server Capabilities
 Defined in `ServerHandler::get_info()`:
-- Tools: Enabled (9 tools available):
+- Tools: Enabled (13 tools available):
   - `index_codebase` - Index a codebase with smart full/incremental detection
   - `query_codebase` - Semantic search across indexed code
   - `get_statistics` - Get index statistics
@@ -312,7 +317,11 @@ Defined in `ServerHandler::get_info()`:
   - `find_definition` - Find where a symbol is defined (LSP-like)
   - `find_references` - Find all references to a symbol (LSP-like)
   - `get_call_graph` - Get callers/callees for a function
-- Prompts: Enabled (9 slash commands: /project:index, /project:query, /project:stats, /project:clear, /project:search, /project:git-search, /project:definition, /project:references, /project:callgraph)
+  - `list_symbols` - List every symbol defined in one file
+  - `read_file` - Read a slice (or all) of a file inside an indexed project root
+  - `edit_file` - Replace a line range (or the whole file), then auto-reindex the affected root
+  - `find_unused` - Find unused imports and dead-code candidates (report-only, confidence-rated)
+- Prompts: Enabled (12 slash commands: /project:index, /project:query, /project:stats, /project:clear, /project:search, /project:git-search, /project:definition, /project:references, /project:callgraph, /project:read, /project:edit, /project:unused)
 - Resources: Not implemented
 - Sampling: Not implemented
 

benches/indexing_benchmark.rs

@@ -100,6 +100,7 @@ fn benchmark_indexing(c: &mut Criterion) {
                                 1_048_576,
                                 None,
                                 None,
+                                None,
                             )
                             .await
                             .unwrap()

examples/basic_indexing.rs

@@ -16,9 +16,7 @@ async fn main() -> anyhow::Result<()> {
     tracing_subscriber::fmt::init();
 
     // Get codebase path from command line or use current directory
-    let codebase_path = env::args()
-        .nth(1)
-        .unwrap_or_else(|| ".".to_string());
+    let codebase_path = env::args().nth(1).unwrap_or_else(|| ".".to_string());
 
     println!("=== Project RAG - Basic Indexing Example ===\n");
 

examples/mcp_client.rs

@@ -11,7 +11,7 @@
 //! Run with: cargo run --example mcp_client
 //! (The server will wait for MCP protocol messages on stdin)
 
-use project_rag::{mcp_server::RagMcpServer, Config, RagClient};
+use project_rag::{Config, RagClient, mcp_server::RagMcpServer};
 use std::sync::Arc;
 
 #[tokio::main]

src/bm25_search/mod.rs

@@ -10,7 +10,7 @@ use tantivy::{Index, IndexWriter, ReloadPolicy, TantivyDocument, doc};
 /// BM25-based keyword search using Tantivy
 pub struct BM25Search {
     index: Index,
-    id_field: Field,
+    chunk_id_field: Field,
     content_field: Field,
     file_path_field: Field,
     /// Path to the index directory (needed for lock cleanup)
@@ -20,9 +20,13 @@ pub struct BM25Search {
 }
 
 /// Search result from BM25
+///
+/// `chunk_id` is the chunk's stable `file_path:start_line` identifier -- the same value
+/// stored in the vector table's `id` column. Both retrieval arms must key on it, or
+/// Reciprocal Rank Fusion silently fuses nothing. See the note on `add_documents`.
 #[derive(Debug, Clone)]
 pub struct BM25Result {
-    pub id: u64,
+    pub chunk_id: String,
     pub score: f32,
 }
 
@@ -31,26 +35,58 @@ impl BM25Search {
     pub fn new<P: AsRef<Path>>(index_path: P) -> Result<Self> {
         let index_path = index_path.as_ref().to_path_buf();
 
-        // Create schema with ID, content, and file_path fields
-        let mut schema_builder = Schema::builder();
-        let id_field = schema_builder.add_u64_field("id", STORED | INDEXED);
-        let content_field = schema_builder.add_text_field("content", TEXT);
-        let file_path_field = schema_builder.add_text_field("file_path", STRING | STORED);
-        let schema = schema_builder.build();
+        // Schema keyed by the chunk's stable `file_path:start_line` id.
+        let build_schema = || {
+            let mut b = Schema::builder();
+            b.add_text_field("chunk_id", STRING | STORED);
+            b.add_text_field("content", TEXT);
+            b.add_text_field("file_path", STRING | STORED);
+            b.build()
+        };
 
         // Create or open index
         std::fs::create_dir_all(&index_path).context("Failed to create BM25 index directory")?;
 
-        let index = if index_path.join("meta.json").exists() {
+        let mut index = if index_path.join("meta.json").exists() {
             Index::open_in_dir(&index_path).context("Failed to open existing BM25 index")?
         } else {
-            Index::create_in_dir(&index_path, schema.clone())
+            Index::create_in_dir(&index_path, build_schema())
                 .context("Failed to create BM25 index")?
         };
 
+        // Older builds keyed documents by a u64 `id` holding a table row number. Field
+        // handles are positional, so opening one of those directories with the new schema
+        // would read a u64 field as text and corrupt every lookup. Detect and rebuild --
+        // the BM25 index is a derived artifact, so throwing it away costs only a re-index.
+        if index.schema().get_field("chunk_id").is_err() {
+            tracing::warn!(
+                "BM25 index at {:?} predates chunk_id keying; rebuilding it",
+                index_path
+            );
+            drop(index);
+            std::fs::remove_dir_all(&index_path)
+                .context("Failed to remove stale BM25 index directory")?;
+            std::fs::create_dir_all(&index_path)
+                .context("Failed to recreate BM25 index directory")?;
+            index = Index::create_in_dir(&index_path, build_schema())
+                .context("Failed to recreate BM25 index")?;
+        }
+
+        // Take handles from the schema actually on disk, never from a locally built one.
+        let schema = index.schema();
+        let chunk_id_field = schema
+            .get_field("chunk_id")
+            .context("BM25 schema is missing the chunk_id field")?;
+        let content_field = schema
+            .get_field("content")
+            .context("BM25 schema is missing the content field")?;
+        let file_path_field = schema
+            .get_field("file_path")
+            .context("BM25 schema is missing the file_path field")?;
+
         Ok(Self {
             index,
-            id_field,
+            chunk_id_field,
             content_field,
             file_path_field,
             index_path,
@@ -111,8 +147,14 @@ impl BM25Search {
     /// Add documents to the index
     ///
     /// Arguments:
-    /// * `documents` - Vec of (id, content, file_path) tuples
-    pub fn add_documents(&self, documents: Vec<(u64, String, String)>) -> Result<()> {
+    /// * `documents` - Vec of (chunk_id, content, file_path) tuples
+    ///
+    /// `chunk_id` MUST be the same `file_path:start_line` value stored in the vector
+    /// table's `id` column. It used to be a `count_rows()`-derived row number, which
+    /// matched the vector arm's batch-relative index only for the first insert into a
+    /// fresh table -- which is exactly the shape the unit tests create, so the mismatch
+    /// never showed up there while fusion was dead in every real index.
+    pub fn add_documents(&self, documents: Vec<(String, String, String)>) -> Result<()> {
         // Lock to ensure only one writer at a time (within this process)
         let _guard = self
             .writer_lock
@@ -161,9 +203,9 @@ impl BM25Search {
             }
         };
 
-        for (id, content, file_path) in documents {
+        for (chunk_id, content, file_path) in documents {
             let doc = doc!(
-                self.id_field => id,
+                self.chunk_id_field => chunk_id,
                 self.content_field => content,
                 self.file_path_field => file_path,
             );
@@ -206,18 +248,22 @@ impl BM25Search {
                 .doc(doc_address)
                 .context("Failed to retrieve document")?;
 
-            if let Some(id_value) = retrieved_doc.get_first(self.id_field)
-                && let Some(id) = id_value.as_u64()
+            if let Some(id_value) = retrieved_doc.get_first(self.chunk_id_field)
+                && let Some(chunk_id) = id_value.as_str()
             {
-                results.push(BM25Result { id, score });
+                results.push(BM25Result {
+                    chunk_id: chunk_id.to_string(),
+                    score,
+                });
             }
         }
 
         Ok(results)
     }
 
-    /// Delete all documents for a specific ID
-    pub fn delete_by_id(&self, id: u64) -> Result<()> {
+    /// Delete all documents for a specific chunk ID
+    #[allow(dead_code)]
+    pub fn delete_by_chunk_id(&self, chunk_id: &str) -> Result<()> {
         // Lock to ensure only one writer at a time
         let _guard = self
             .writer_lock
@@ -229,7 +275,7 @@ impl BM25Search {
             .writer(50_000_000)
             .context("Failed to create index writer")?;
 
-        let term = Term::from_field_u64(self.id_field, id);
+        let term = Term::from_field_text(self.chunk_id_field, chunk_id);
         index_writer.delete_term(term);
 
         index_writer.commit().context("Failed to commit deletion")?;
@@ -318,12 +364,15 @@ pub const RRF_K_CONSTANT: f32 = 60.0;
 /// This is a convenience wrapper around `reciprocal_rank_fusion_generic` for the common case
 /// of combining vector search results (u64 IDs) with BM25 results.
 pub fn reciprocal_rank_fusion(
-    vector_results: Vec<(u64, f32)>,
+    vector_results: Vec<(String, f32)>,
     bm25_results: Vec<BM25Result>,
     k: usize,
-) -> Vec<(u64, f32)> {
+) -> Vec<(String, f32)> {
     // Convert BM25 results to the same format as vector results
-    let bm25_tuples: Vec<(u64, f32)> = bm25_results.into_iter().map(|r| (r.id, r.score)).collect();
+    let bm25_tuples: Vec<(String, f32)> = bm25_results
+        .into_iter()
+        .map(|r| (r.chunk_id, r.score))
+        .collect();
 
     // Use the generic implementation
     reciprocal_rank_fusion_generic([vector_results, bm25_tuples], k)