From e393515c7c94e0f127bcbac633052ee457c0527e Mon Sep 17 00:00:00 2001 From: Agent IX Date: Sat, 29 Aug 2026 02:50:47 -0700 Subject: [PATCH 1/2] fix: prevent workstation path publication --- .github/workflows/path-hygiene.yml | 36 +++++ AGENTS.md | 11 ++ .../install-pgvectorscale-release.log | 8 +- .../feedback/2026-04-08-01-reviewer.md | 3 +- crates/ecaz-cli/README.md | 15 +- crates/ecaz-cli/src/commands/dev/install.rs | 16 +-- ...re-phase11-paper-parity-production-gate.md | 2 +- plan/tasks/240-artifact-path-sanitization.md | 42 ++++++ ...87-candidate-batched-scoring-across-ams.md | 4 +- .../88-streaming-ann-result-iteration.md | 6 +- ...9-turboquant-tqplus-cross-am-validation.md | 2 +- plan/tasks/README.md | 2 + .../request.md | 2 +- .../feedback/2026-04-09-01-reviewer.md | 12 +- .../request.md | 3 +- .../request.md | 3 +- .../artifacts/manifest.md | 2 +- .../feedback/2026-07-30-02-reviewer.md | 2 +- .../feedback/2026-07-30-02-reviewer.md | 2 +- .../artifacts/paper-algorithm-citations.md | 2 +- .../003-ab/feedback/2026-07-30-02-reviewer.md | 2 +- .../request.md | 8 +- .../feedback/2026-04-23-01-reviewer.md | 2 +- .../feedback.md | 2 +- .../feedback/2026-05-09-01-reviewer.md | 2 +- .../artifacts/paper-parity-seed.md | 2 +- .../artifacts/paper-source-outline.md | 2 +- .../artifacts/manifest.md | 2 +- .../artifacts/turbovec-tq-analysis.md | 40 +++--- .../artifacts/source-scoring-map.md | 2 +- scripts/check_workstation_paths.py | 136 ++++++++++++++++++ scripts/tests/test_check_workstation_paths.py | 74 ++++++++++ spec/adr/ADR-006-own-quantizer.md | 3 +- spec/adr/ADR-024-fwht-transform-strategy.md | 5 +- ...ymphonyqg-quantized-graph-access-method.md | 3 +- .../ADR-085-ec-distann-single-global-graph.md | 3 +- spec/reviews/evidence.md | 2 +- 37 files changed, 379 insertions(+), 86 deletions(-) create mode 100644 .github/workflows/path-hygiene.yml create mode 100644 plan/tasks/240-artifact-path-sanitization.md create mode 100644 scripts/check_workstation_paths.py create mode 100644 scripts/tests/test_check_workstation_paths.py diff --git a/.github/workflows/path-hygiene.yml b/.github/workflows/path-hygiene.yml new file mode 100644 index 0000000000..db36376d0d --- /dev/null +++ b/.github/workflows/path-hygiene.yml @@ -0,0 +1,36 @@ +name: Path hygiene + +on: + pull_request: + push: + branches: [main] + workflow_dispatch: + +permissions: + contents: read + +jobs: + added-lines: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Reject newly added workstation paths + env: + EVENT_NAME: ${{ github.event_name }} + BASE_SHA: ${{ github.event.pull_request.base.sha }} + BEFORE_SHA: ${{ github.event.before }} + run: | + if [ "$EVENT_NAME" = "pull_request" ]; then + diff_range="${BASE_SHA}...HEAD" + elif [ -n "$BEFORE_SHA" ] && ! printf '%s' "$BEFORE_SHA" | grep -Eq '^0+$'; then + diff_range="${BEFORE_SHA}..HEAD" + elif git rev-parse HEAD^ >/dev/null 2>&1; then + diff_range="HEAD^..HEAD" + else + empty_tree="$(git hash-object -t tree /dev/null)" + diff_range="${empty_tree}..HEAD" + fi + python3 scripts/check_workstation_paths.py --diff-range "$diff_range" diff --git a/AGENTS.md b/AGENTS.md index d3f120aeb6..4d90dd5a5e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -94,6 +94,17 @@ Packet directories inside a task bucket must sort in chronological order. - `request.md` should summarize the result and point at the packet-local artifact files. +### Workstation Path Hygiene + +- New or changed tracked content must not introduce a machine-specific absolute + workstation home path. Use an environment variable, repository-relative path, + or durable public URL instead. +- `scripts/check_workstation_paths.py` enforces this on added diff lines without + echoing the sensitive value into CI output. +- Historical evidence that predates this control must not be bulk-rewritten merely + to make the legacy corpus look clean. Any redaction requires an explicit audit + trail and a separate history-remediation decision. + ### Never Commit: Corpus Data, Operational Logs, and Polling Cruft A review packet is decision-grade evidence, not a capture of everything the run diff --git a/benchmarks/11105-task29-release-latency-refresh/artifacts/install-pgvectorscale-release.log b/benchmarks/11105-task29-release-latency-refresh/artifacts/install-pgvectorscale-release.log index 0a2d74006a..eacdbe86e3 100644 --- a/benchmarks/11105-task29-release-latency-refresh/artifacts/install-pgvectorscale-release.log +++ b/benchmarks/11105-task29-release-latency-refresh/artifacts/install-pgvectorscale-release.log @@ -118,7 +118,7 @@ Script started on 2026-04-30 16:13:15-07:00 [] Compiling ptr_meta_derive v0.1.4 Compiling bytecheck_derive v0.6.12 Compiling rkyv_derive v0.7.46 - Compiling pgvectorscale_derive v0.1.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale/pgvectorscale_derive) + Compiling pgvectorscale_derive v0.1.0 ([local-checkout]/pgvectorscale/pgvectorscale_derive) Compiling ptr_meta v0.1.4 Compiling synstructure v0.13.2 Compiling futures-executor v0.3.32 @@ -157,7 +157,7 @@ Script started on 2026-04-30 16:13:15-07:00 [] Compiling pgrx-bindgen v0.16.1 Compiling pgrx-pg-sys v0.16.1 Compiling pgrx v0.16.1 - Compiling vectorscale v0.9.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale) + Compiling vectorscale v0.9.0 ([local-checkout]/pgvectorscale) Finished `release` profile [optimized] target(s) in 54.98s  Installing extension  Copying control file to /home/peter/.pgrx/18.3/pgrx-install/share/postgresql/extension/vectorscale.control @@ -279,7 +279,7 @@ Script started on 2026-04-30 16:13:15-07:00 [] Compiling ptr_meta_derive v0.1.4 Compiling bytecheck_derive v0.6.12 Compiling rkyv_derive v0.7.46 - Compiling pgvectorscale_derive v0.1.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale/pgvectorscale_derive) + Compiling pgvectorscale_derive v0.1.0 ([local-checkout]/pgvectorscale/pgvectorscale_derive) Compiling futures-executor v0.3.32 Compiling ptr_meta v0.1.4 Compiling regex v1.12.3 @@ -319,7 +319,7 @@ Script started on 2026-04-30 16:13:15-07:00 [] Compiling pgrx-bindgen v0.16.1 Compiling pgrx-pg-sys v0.16.1 Compiling pgrx v0.16.1 - Compiling vectorscale v0.9.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale) + Compiling vectorscale v0.9.0 ([local-checkout]/pgvectorscale) Finished `dev` profile [unoptimized + debuginfo] target(s) in 37.88s  Writing SQL entities to /home/peter/.pgrx/18.3/pgrx-install/share/postgresql/extension/vectorscale--0.9.0.sql  Copying extension schema upgrade file to /home/peter/.pgrx/18.3/pgrx-install/share/postgresql/extension/vectorscale--0.2.0--0.3.0.sql diff --git a/benchmarks/200-a4-recall-gate-rerun/feedback/2026-04-08-01-reviewer.md b/benchmarks/200-a4-recall-gate-rerun/feedback/2026-04-08-01-reviewer.md index f6d03916aa..a91ea3e810 100644 --- a/benchmarks/200-a4-recall-gate-rerun/feedback/2026-04-08-01-reviewer.md +++ b/benchmarks/200-a4-recall-gate-rerun/feedback/2026-04-08-01-reviewer.md @@ -72,7 +72,8 @@ Well-framed. Updated assessment after cross-referencing reviews 195-198 and 201: ## Additional Findings -Verified against reference implementations in `~/dev_bak/hnsw-references/`: +Verified against local reference checkouts of `hnsw`, `hnswlib-rs`, +`instant-distance`, and `swarc`: 1. **Layer indexing**: hnswlib-rs `get_neighborhood_id()` iterates `0..nb_layer` — layer 0 = index 0. Matches `pack_point_neighbor_slots`. No reversal bug. diff --git a/crates/ecaz-cli/README.md b/crates/ecaz-cli/README.md index d57a589c3b..4081136d22 100644 --- a/crates/ecaz-cli/README.md +++ b/crates/ecaz-cli/README.md @@ -47,14 +47,14 @@ cargo install --path crates/ecaz-cli That installs the binary under Cargo's bin directory, usually `$HOME/.cargo/bin/ecaz`. Interactive shells may also put that directory on `$PATH`; when running from an agent or other sandboxed tool session, prefer the -absolute installed path (for example `/Users/peter/.cargo/bin/ecaz`) so one +absolute installed path (for example `${HOME}/.cargo/bin/ecaz`) so one approval rule can cover the operator surface consistently. All commands accept `--database`, `--host`, `--port`, `--user`, `--password`, and `--log-file`; each also falls back to the matching libpq environment variable (`PGDATABASE`, `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`) when the flag is omitted. `--host` may be either a TCP host name or a Unix socket -directory such as `/home/peter/.pgrx`. +directory supplied through `PGRX_HOME` or `PGHOST`. `--log-file` mirrors the CLI's stdout/stderr into a packet-local artifact file so review runs do not need shell `tee` wrappers. When `--log-file` is set, transient progress bars are suppressed so the artifact stays stable. @@ -67,8 +67,8 @@ generation, load/list/inspect, benchmark, storage, scratch, and SQL checks through that binary: ```sh -/Users/peter/.cargo/bin/ecaz dev sql --pg 18 --db postgres \ - --socket-dir /Users/peter/.pgrx --raw \ +${HOME}/.cargo/bin/ecaz dev sql --pg 18 --db postgres \ + --socket-dir "${PGRX_HOME}" --raw \ --sql "select version()" \ --log-output reviews/task-{id}/001-example/artifacts/pg18-status.log ``` @@ -82,10 +82,11 @@ libpq options, and approval scope in one place. External comparison extensions can use the same setup surface: ```sh -/Users/peter/.cargo/bin/ecaz dev install pgvector --pg 18 -/Users/peter/.cargo/bin/ecaz dev install vectorscale \ +${HOME}/.cargo/bin/ecaz dev install pgvector --pg 18 \ + --repo "${PGVECTOR_REPO}" +${HOME}/.cargo/bin/ecaz dev install vectorscale \ --pg 18 \ - --repo "$HOME/dev_bak/pgvectorscale/pgvectorscale" \ + --repo "${PGVECTORSCALE_REPO}" \ --cargo-pgrx /tmp/pgvectorscale-cargo-pgrx-0.16.1/bin/cargo-pgrx ``` diff --git a/crates/ecaz-cli/src/commands/dev/install.rs b/crates/ecaz-cli/src/commands/dev/install.rs index 5a7a269782..cc4adbe3c8 100644 --- a/crates/ecaz-cli/src/commands/dev/install.rs +++ b/crates/ecaz-cli/src/commands/dev/install.rs @@ -50,7 +50,7 @@ pub struct InstallEcazPgTestArgs { #[derive(Args, Debug)] pub struct InstallPgvectorArgs { /// pgvector repository checkout. - #[arg(long, env = "PGVECTOR_REPO", default_value_os_t = default_pgvector_repo())] + #[arg(long, env = "PGVECTOR_REPO")] repo: PathBuf, /// PostgreSQL major version to install against. @@ -69,11 +69,7 @@ pub struct InstallPgvectorArgs { #[derive(Args, Debug)] pub struct InstallVectorscaleArgs { /// pgvectorscale extension crate checkout. - #[arg( - long, - env = "PGVECTORSCALE_REPO", - default_value_os_t = default_vectorscale_repo() - )] + #[arg(long, env = "PGVECTORSCALE_REPO")] repo: PathBuf, /// cargo-pgrx binary to use for the install. @@ -284,14 +280,6 @@ fn home_dir() -> PathBuf { .unwrap_or_else(|| PathBuf::from("/")) } -fn default_pgvector_repo() -> PathBuf { - home_dir().join("dev_bak/pgvector") -} - -fn default_vectorscale_repo() -> PathBuf { - home_dir().join("dev_bak/pgvectorscale/pgvectorscale") -} - fn default_cargo_pgrx_bin() -> PathBuf { home_dir().join(".cargo/bin/cargo-pgrx") } diff --git a/plan/design/spire-phase11-paper-parity-production-gate.md b/plan/design/spire-phase11-paper-parity-production-gate.md index a6b51e31e8..9293aa8240 100644 --- a/plan/design/spire-phase11-paper-parity-production-gate.md +++ b/plan/design/spire-phase11-paper-parity-production-gate.md @@ -2,7 +2,7 @@ Status: active Phase 11.1 gate Task: Task 30 Phase 11 -Paper basis: `/home/peter/dev_bak/papers/2512.17264v1.pdf` +Paper basis: arXiv:2512.17264v1 ## Source Basis diff --git a/plan/tasks/240-artifact-path-sanitization.md b/plan/tasks/240-artifact-path-sanitization.md new file mode 100644 index 0000000000..61d929f674 --- /dev/null +++ b/plan/tasks/240-artifact-path-sanitization.md @@ -0,0 +1,42 @@ +# Task 240: Artifact Path Sanitization + +Status: **implementation in progress** (2026-08-29). Priority: P0 privacy and +evidence hygiene. Origin: private Engineering Assurance gap audit. + +## Why + +The public default branch contains machine-specific workstation paths in authored +configuration and historical evidence. GitHub code search exposed only a subset; a +local tracked-file audit found 6,396 affected files. The values disclose local layout, +are non-portable, and can accidentally identify rights-unreviewed source locations. + +## Goal + +Remove the explicitly identified internal scratch locator from the current tree, make +external-source references durable, and prevent any newly added workstation path from +reaching GitHub without rewriting historical benchmark evidence blindly. + +## Scope + +1. Replace current authored internal-source locators with durable public identifiers or + explicit operator inputs. +2. Remove machine-specific default comparison-repository locations from `ecaz dev + install`; require `--repo` or the documented environment variable. +3. Add a diff-aware guard that detects Unix, root, and Windows workstation home paths, + reports only file/line/category, and never repeats the matched value in CI logs. +4. Run the guard on every pull request and main-branch push in a lightweight workflow. +5. Preserve a transparent distinction between current-tree redaction and historical Git + reachability. Do not rewrite history or silently change digest-bound evidence. + +## Acceptance + +1. The current tree contains no reference to the internal scratch locator identified by + the audit. +2. Synthetic tests prove added Unix, root, and Windows workstation paths fail while + removals and portable environment-based paths pass. +3. Failure output contains the file, line, and category but not the sensitive value. +4. The CLI comparison installers have no machine-specific repository default. +5. The lightweight GitHub workflow enforces the guard without invoking benchmark or + PostgreSQL lanes. +6. A review packet records the legacy-corpus count, redaction boundary, test evidence, + and the separate unresolved history-remediation decision. diff --git a/plan/tasks/87-candidate-batched-scoring-across-ams.md b/plan/tasks/87-candidate-batched-scoring-across-ams.md index 787437e502..b3e5a5a04a 100644 --- a/plan/tasks/87-candidate-batched-scoring-across-ams.md +++ b/plan/tasks/87-candidate-batched-scoring-across-ams.md @@ -379,7 +379,7 @@ fix, not Task 87. the calibration prototype already produces `Prepared…Query` shapes that batch naturally. - **pgvectorscale** is a read-only reference: clone at - `/Users/peter/dev_bak/pgvectorscale/`; key files + the `timescale/pgvectorscale` repository; key files `access_method/scan.rs` (resort_buffer pattern) and `access_method/graph/mod.rs` (streaming iteration). @@ -426,7 +426,7 @@ fix, not Task 87. - Task 86 packet 001 transferability matrix (per-AM block- kernel fit ranking) - pgvectorscale resort_buffer pattern: - `/Users/peter/dev_bak/pgvectorscale/pgvectorscale/src/access_method/scan.rs` + `pgvectorscale/src/access_method/scan.rs` - FR-038 (benchmark provenance): every suite checked-in JSON - ADR-075 (Task 65b stepping stone framing) — similar staged-rollout pattern diff --git a/plan/tasks/88-streaming-ann-result-iteration.md b/plan/tasks/88-streaming-ann-result-iteration.md index 49b828b13b..e660707794 100644 --- a/plan/tasks/88-streaming-ann-result-iteration.md +++ b/plan/tasks/88-streaming-ann-result-iteration.md @@ -8,7 +8,7 @@ Priority: 2 (hybrid-search support; follows Task 87) Task 86 investigation surfaced pgvectorscale's `StreamingDiskANN` pattern (see -`/Users/peter/dev_bak/pgvectorscale/pgvectorscale/src/access_method/graph/mod.rs` +`timescale/pgvectorscale:pgvectorscale/src/access_method/graph/mod.rs` `greedy_search_streaming_init` + `greedy_search_iterate`). The pattern yields candidates lazily to the PostgreSQL executor in approximate-score order, with a resort buffer + @@ -200,7 +200,7 @@ the two. the post-filter use case justification — don't expand Task 88's scope. - **pgvectorscale** is the reference implementation - (`/Users/peter/dev_bak/pgvectorscale/`). Key files: + (`timescale/pgvectorscale`). Key files: - `pgvectorscale/src/access_method/scan.rs` (`resort_buffer`, `StreamingStats`) - `pgvectorscale/src/access_method/graph/mod.rs` @@ -243,7 +243,7 @@ the two. (surfaced the streaming-vs-batching distinction) - Task 87 (predecessor): `plan/tasks/87-candidate-batched-scoring-across-ams.md` - pgvectorscale `StreamingDiskANN` reference: - `/Users/peter/dev_bak/pgvectorscale/` + `timescale/pgvectorscale` - pgvectorscale README streaming section: *"The post-filtering implementation, while slower, is streaming and correct…"* diff --git a/plan/tasks/89-turboquant-tqplus-cross-am-validation.md b/plan/tasks/89-turboquant-tqplus-cross-am-validation.md index 183216f2f6..ed73f43a86 100644 --- a/plan/tasks/89-turboquant-tqplus-cross-am-validation.md +++ b/plan/tasks/89-turboquant-tqplus-cross-am-validation.md @@ -293,7 +293,7 @@ permission to skip the AM. (includes reviewer feedback on the original closeout) - Reverted Task 86 commits (TQ+ work, preserved in git history): `e0ae9fe7d`, `c7e85e8ac`, `16f1e6104`, etc. -- pgvectorscale reference: `/Users/peter/dev_bak/pgvectorscale/` +- pgvectorscale reference: `timescale/pgvectorscale` ## Estimated size diff --git a/plan/tasks/README.md b/plan/tasks/README.md index a9443567d9..8e7ae62171 100644 --- a/plan/tasks/README.md +++ b/plan/tasks/README.md @@ -279,6 +279,8 @@ These pre-lane task files are retained only for historical context under 238. `238-ec-distann-retry-snapshot-use-after-free.md` — **implementation and evidence merged; packet 001 outside closeout review-open** (updated 2026-08-27, P0 correctness): the equivalent `RegisteredSnapshotGuard` lifetime fix remains on main via `15f7fcf5f` / Task 167 PR #77. Checkpoint `7d4103885` restores the deterministic forced-retry regression and explicit loopback-only plaintext test conninfo; both PG18 callers pass on base `3c81319a3`. PR #91 merged the implementation and evidence as `bcc9b0bfa`; only the outside ACCEPT remains open. Evidence and request: `reviews/task-238/001-retry-snapshot-uaf/`. 239. `239-ec-distann-bounded-read-overfetch.md` — **complete — review-closed ACCEPT; HARNESS REGRESSION CORRECTED, EXACT-MAIN LAZY-10 SEMANTIC PATH RESTORED TO 10/10; no rerun; packet 004 10k/50k/100k NOT triggered; Task 229 / 230--233 semantic-surface blocker LIFTED** (updated 2026-08-26, P1 correctness/performance): packet 001's 12/10 result was a shared-session batch-size GUC leak in the benchmark harness — a variant at batch size 10 emitted no `SET`, so the "lazy-10" arm inherited the eager control's 0 — not production overfetch, and the bound was never widened. Packet 002 corrected the harness but its cross-SHA runner aborted pre-semantics on 40-versus-37 stage rows. Packet 003 removed that confounder with exact-main runner/extension plus only the harness fix; its sole C1--C5 run passed every preregistered gate — nine scenarios exactly once in both logs, seven core rows at `control_batch_size=0 candidate_batch_size=10` with exact identity and zero duplicates, `exactly_one_window` 6 remote + 4 local = 10 reads/bound 10, mixed and outage pass, routed DELETE+VACUUM pass, both recall arms 0.9990 over 200 queries / 2,000 trials with predictions byte-identical to packet 001 — and the stopped 1.2 GB fixture was removed after capture. `git diff 41392c011 def565270 -- src` is empty, so no production runtime behavior changed and the 10k/50k/100k closeout matrix does not apply. Determinism rests on the same 6/4/10 split and digests at three independent extension SHAs (task-191 `7883cfcf`, task-198 `2ff72b3e`, this run `4ab2aa9a9`), so no rerun was needed. Carried follow-ups for whichever task next touches the multinode semantic harness: mixed/outage failures emit no structured `pass=false` row; `owner_payload_plan_cache` retains the same conditional-`SET` leak shape; recall/latency child batch size stays inferred. Closeout verdict: `reviews/task-239/003-main-baseline-semantic-proof/feedback/2026-08-26-02-reviewer.md`; result decision: `reviews/task-239/003-main-baseline-semantic-proof/artifacts/live-run-decision.md`; review request: `reviews/task-239/003-main-baseline-semantic-proof/request.md`; live authorization: `reviews/task-239/003-main-baseline-semantic-proof/feedback/2026-08-26-01-reviewer.md`. +240. `240-artifact-path-sanitization.md` — **implementation in progress** (2026-08-29, P0 privacy/evidence hygiene): remove current-tree internal scratch locators, require portable comparison-repository inputs, and add a non-echoing diff guard for newly introduced workstation paths; historical Git remediation remains a separate explicit decision. + ## Coordination rules - Freeze binary datum layout before downstream work expands. diff --git a/reviews/task-08/014-205-a4-quantizer-impl-mismatch-audit/request.md b/reviews/task-08/014-205-a4-quantizer-impl-mismatch-audit/request.md index f0b4815e94..db2912237d 100644 --- a/reviews/task-08/014-205-a4-quantizer-impl-mismatch-audit/request.md +++ b/reviews/task-08/014-205-a4-quantizer-impl-mismatch-audit/request.md @@ -1,7 +1,7 @@ # Review Request: A4 Quantizer Implementation Mismatch Audit Basis: `main` working tree after review `204`, plus upstream reference at -`~/dev_bak/TurboQuantDB/` +`https://github.com/jyunming/TurboQuantDB` ## Why This Packet Exists diff --git a/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/feedback/2026-04-09-01-reviewer.md b/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/feedback/2026-04-09-01-reviewer.md index 8a76bc7c7e..154a1fbc30 100644 --- a/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/feedback/2026-04-09-01-reviewer.md +++ b/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/feedback/2026-04-09-01-reviewer.md @@ -16,14 +16,15 @@ Request: ## Additional Reference: prism-mcp TurboQuant Implementation -A third independent TurboQuant implementation exists in `~/dev_bak/prism-mcp/`. It is a pure +A third independent TurboQuant implementation exists in +`https://github.com/dcostenco/prism-mcp`. It is a pure TypeScript port used for Prism MCP's "Quantized Agentic Memory" feature (v5.0). Comparing it alongside TurboQuantDB strengthens several conclusions from review 204 and surfaces one new finding. ### prism-mcp Architecture Summary -Source: `~/dev_bak/prism-mcp/src/utils/turboquant.ts` (~840 lines) +Source: `dcostenco/prism-mcp:src/utils/turboquant.ts` (~840 lines) - **Rotation**: QR decomposition (Householder) of a random Gaussian matrix — NOT FWHT/SRHT - **Dimension**: 768 (Gemini text-embedding-004), no power-of-2 constraint @@ -150,7 +151,7 @@ quality lever. ## Paper Reference: Rotation Is QR, Not FWHT -Source: `~/dev_bak/turboquant-2504.19874.pdf` (arXiv:2504.19874, ICLR 2026) +Source: arXiv:2504.19874 (ICLR 2026) The TurboQuant paper specifies QR decomposition, not FWHT: @@ -216,7 +217,7 @@ contract within tqvector's SRHT-based architecture. ## TurboQuantDB Published Recall Benchmarks -Source: `~/dev_bak/TurboQuantDB/README.md`, section "Benchmarks" +Source: `jyunming/TurboQuantDB:README.md`, section "Benchmarks" TurboQuantDB publishes recall numbers on **real embeddings** (50k × 1536, DBpedia OpenAI embeddings, top_k=10): @@ -245,7 +246,8 @@ is already good enough that the residual is small and the 1-bit correction is ef ## TurboQuantDB Deep-Dive: Code-Level Architecture -Full source read of `~/dev_bak/TurboQuantDB/src/quantizer/` and `src/linalg/hadamard.rs`. +Full source read of `jyunming/TurboQuantDB:src/quantizer/` and +`src/linalg/hadamard.rs`. ### README Describes QR, Code Uses SRHT diff --git a/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/request.md b/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/request.md index 0d3cf20422..3864abb828 100644 --- a/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/request.md +++ b/reviews/task-16/001-204-a4-full-vs-tiled-fwht-turboquantdb/request.md @@ -1,6 +1,7 @@ # Review Request: A4 Full vs Tiled FWHT — TurboQuantDB Comparison -Basis: `main` working tree, with reference to `~/dev_bak/TurboQuantDB/` +Basis: `main` working tree, with reference to +`https://github.com/jyunming/TurboQuantDB` ## Summary diff --git a/reviews/task-16/054-363-c1-adr030-v2-pgvector-size-and-runtime-baseline/request.md b/reviews/task-16/054-363-c1-adr030-v2-pgvector-size-and-runtime-baseline/request.md index becbc2ddc0..d6d2a242c8 100644 --- a/reviews/task-16/054-363-c1-adr030-v2-pgvector-size-and-runtime-baseline/request.md +++ b/reviews/task-16/054-363-c1-adr030-v2-pgvector-size-and-runtime-baseline/request.md @@ -53,7 +53,7 @@ Updated: Concrete changes: 1. added a reproducible scratch install wrapper for a local pgvector checkout - (`/home/peter/dev_bak/pgvector` by default) + (the checkout supplied through `PGVECTOR_REPO` or `--repo`) 2. the script installs pgvector into the same pg17 pgrx/Postgres tree already used by the scratch cluster: - `PG_CONFIG=/home/peter/.pgrx/17.9/pgrx-install/bin/pg_config` @@ -195,4 +195,3 @@ The next useful follow-up is not more size work. It is one of: isolated grouped index without the special verified harness 3. make the product call explicitly: latency-first compressed ANN versus higher-recall full-vector ANN - diff --git a/reviews/task-203/001-decision-reaudit/artifacts/manifest.md b/reviews/task-203/001-decision-reaudit/artifacts/manifest.md index a335b0b0c5..6276d5e0b1 100644 --- a/reviews/task-203/001-decision-reaudit/artifacts/manifest.md +++ b/reviews/task-203/001-decision-reaudit/artifacts/manifest.md @@ -24,7 +24,7 @@ existing packets, task files, spec files, or source. `DISTRIBUTEDANN: Efficient Scaling of a Single DISKANN Graph Across Thousands of Computers`, arXiv:2509.06046v1, 7 Sep 2025. Local copy: -`~/dev_bak/papers/distributedann-2509.06046.pdf` (8 pages). Sections cited: +DistributedANN, arXiv:2509.06046 (8 pages). Sections cited: §2.2 (index layout modifications, head index), §2.3 (near-data computation, Algorithm 1), §2.4 (orchestration service, Algorithm 2), §3 (graph construction), §4 (evaluation parameters), §4.1 (scaling), §4.2 (reliability). diff --git a/reviews/task-204/001-arm-fidelity/feedback/2026-07-30-02-reviewer.md b/reviews/task-204/001-arm-fidelity/feedback/2026-07-30-02-reviewer.md index 7df0172a39..c3e5d0cce1 100644 --- a/reviews/task-204/001-arm-fidelity/feedback/2026-07-30-02-reviewer.md +++ b/reviews/task-204/001-arm-fidelity/feedback/2026-07-30-02-reviewer.md @@ -12,7 +12,7 @@ Short addendum to `2026-07-30-01-reviewer.md`, narrowing one instruction in it. My seq-01 "Open item 1" asked for two things under Phase 3: a computed cross-scale growth row, and a gate that fails the run on breach. After reading -`~/dev_bak/papers/distributedann-2509.06046.pdf` directly, I want to split those +DistributedANN (arXiv:2509.06046) directly, I want to split those apart. Full reasoning and citations are in `reviews/task-205/003-ab/feedback/2026-07-30-02-reviewer.md` and `reviews/task-205/003-ab/artifacts/paper-algorithm-citations.md`. diff --git a/reviews/task-205/001-contract/feedback/2026-07-30-02-reviewer.md b/reviews/task-205/001-contract/feedback/2026-07-30-02-reviewer.md index 4847deba11..ea5df63074 100644 --- a/reviews/task-205/001-contract/feedback/2026-07-30-02-reviewer.md +++ b/reviews/task-205/001-contract/feedback/2026-07-30-02-reviewer.md @@ -9,7 +9,7 @@ seq: 02 # Task 205 / 001-contract — the paper's actual contract values Addendum to `2026-07-30-01-reviewer.md`. I read the paper directly -(`~/dev_bak/papers/distributedann-2509.06046.pdf`) and transcribed the two +(DistributedANN, arXiv:2509.06046) and transcribed the two algorithms verbatim into `reviews/task-205/003-ab/artifacts/paper-algorithm-citations.md` so these citations are checkable from inside the packet. diff --git a/reviews/task-205/003-ab/artifacts/paper-algorithm-citations.md b/reviews/task-205/003-ab/artifacts/paper-algorithm-citations.md index bf011ea087..79ec3a9dc3 100644 --- a/reviews/task-205/003-ab/artifacts/paper-algorithm-citations.md +++ b/reviews/task-205/003-ab/artifacts/paper-algorithm-citations.md @@ -2,7 +2,7 @@ Source: `DISTRIBUTEDANN: Efficient Scaling of a Single DISKANN Graph Across Thousands of Computers`, arXiv:2509.06046, 8 pages. Local copy read at -`~/dev_bak/papers/distributedann-2509.06046.pdf`. Text extracted with `pypdf`; +DistributedANN (arXiv:2509.06046). Text extracted with `pypdf`; line breaks and hyphenation are as extracted. These are transcribed here so the review's citations are checkable from inside diff --git a/reviews/task-205/003-ab/feedback/2026-07-30-02-reviewer.md b/reviews/task-205/003-ab/feedback/2026-07-30-02-reviewer.md index 992289ffcb..97ddbbab61 100644 --- a/reviews/task-205/003-ab/feedback/2026-07-30-02-reviewer.md +++ b/reviews/task-205/003-ab/feedback/2026-07-30-02-reviewer.md @@ -9,7 +9,7 @@ seq: 02 # Task 205 / 003-ab — paper grounding for the re-run and the NFR-021 verdict Addendum to `2026-07-30-01-reviewer.md`, after reading -`~/dev_bak/papers/distributedann-2509.06046.pdf` directly. Verbatim algorithm and +DistributedANN (arXiv:2509.06046) directly. Verbatim algorithm and results text is transcribed in `artifacts/paper-algorithm-citations.md`. Both seq-01 P1s survive the reading, and both get sharper. One correction to my diff --git a/reviews/task-25/014-20013-task25-symphony-prerequisite-finding/request.md b/reviews/task-25/014-20013-task25-symphony-prerequisite-finding/request.md index bf02225aee..5c8dc22701 100644 --- a/reviews/task-25/014-20013-task25-symphony-prerequisite-finding/request.md +++ b/reviews/task-25/014-20013-task25-symphony-prerequisite-finding/request.md @@ -12,9 +12,9 @@ the task-25 closing posture and the task-27 start date: wrong configuration for Symphony. Artifacts: -- `~/dev_bak/papers/symphonyqg-2025-sigmod-arxiv-2411.12229.pdf` -- `~/dev_bak/papers/rabitq-2024-sigmod-arxiv-2405.12497.pdf` -- `~/dev_bak/papers/extended-rabitq-2025-sigmod-arxiv-2409.09913.pdf` +- SymphonyQG, arXiv:2411.12229 +- RaBitQ, arXiv:2405.12497 +- Extended RaBitQ, arXiv:2409.09913 Branch: `task25-rabitq-stage1-phase0` (slice 14 builds on `3431476`). @@ -137,7 +137,7 @@ trait surface exactly as today. subsection to the same section with the API shape, the equation-6 decomposition, and the rationale for keeping the centered path off the `Quantizer` trait. -- Papers archived in `~/dev_bak/papers/` (not committed to repo +- Papers inspected from local authorized copies (not committed to repo since outside the project tree and large binary). ## What this slice does NOT do diff --git a/reviews/task-25/015-20014-task25-centered-api/feedback/2026-04-23-01-reviewer.md b/reviews/task-25/015-20014-task25-centered-api/feedback/2026-04-23-01-reviewer.md index bad56eca16..a61b3cf883 100644 --- a/reviews/task-25/015-20014-task25-centered-api/feedback/2026-04-23-01-reviewer.md +++ b/reviews/task-25/015-20014-task25-centered-api/feedback/2026-04-23-01-reviewer.md @@ -65,7 +65,7 @@ seq: 01 ## Validation - Read the task-25 and task-27 docs, the slice-6/14/15 packets, and ADR-045. -- Verified the Symphony paper passages locally from `~/dev_bak/papers/`. +- Verified the Symphony paper passages from an authorized local copy. - `cargo test` - `cargo check -p ecaz-cli` diff --git a/reviews/task-29/014-11095-task29-diskann-pgvectorscale-comparison/feedback.md b/reviews/task-29/014-11095-task29-diskann-pgvectorscale-comparison/feedback.md index 17660bcb8c..42e0c9efaf 100644 --- a/reviews/task-29/014-11095-task29-diskann-pgvectorscale-comparison/feedback.md +++ b/reviews/task-29/014-11095-task29-diskann-pgvectorscale-comparison/feedback.md @@ -7,7 +7,7 @@ Recipient: coder-1 This is a code-level evaluation of `src/am/ec_diskann/` and a comparison to pgvectorscale's Vamana implementation. It is informed by reading every probe packet `11087..11094` and the relevant ec_diskann source plus -`~/dev_bak/pgvectorscale/pgvectorscale/src/access_method/` (cloned at HEAD). +`timescale/pgvectorscale:pgvectorscale/src/access_method/` (cloned at HEAD). The review excludes pgvectorscale's streaming-iterator amgettuple shape per the request; everything else (graph build, prune, search, distance handling, diff --git a/reviews/task-30/581-30658-spire-phase9-routing-plan/feedback/2026-05-09-01-reviewer.md b/reviews/task-30/581-30658-spire-phase9-routing-plan/feedback/2026-05-09-01-reviewer.md index 0be6ca41fe..9cebef3805 100644 --- a/reviews/task-30/581-30658-spire-phase9-routing-plan/feedback/2026-05-09-01-reviewer.md +++ b/reviews/task-30/581-30658-spire-phase9-routing-plan/feedback/2026-05-09-01-reviewer.md @@ -11,7 +11,7 @@ seq: 01 Review pass on `661d5008`, focused on graph structure, composition across hierarchy levels, multi-NVMe / multi-instance behavior, the query path, and the current custom libpq surfaces. Paper cross-check was against -`/home/peter/dev_bak/papers/2512.17264v1.pdf`, which describes SPIRE as a +arXiv:2512.17264v1, which describes SPIRE as a hierarchical index with top/root graph routing, level-by-level top-m descent, boundary replication, disaggregated index storage, and stateless query engines that fetch relevant index data from the store. diff --git a/reviews/task-30/613-30691-spire-phase11-production-parity-plan/artifacts/paper-parity-seed.md b/reviews/task-30/613-30691-spire-phase11-production-parity-plan/artifacts/paper-parity-seed.md index e47e5ec566..3ddcc6fe15 100644 --- a/reviews/task-30/613-30691-spire-phase11-production-parity-plan/artifacts/paper-parity-seed.md +++ b/reviews/task-30/613-30691-spire-phase11-production-parity-plan/artifacts/paper-parity-seed.md @@ -4,7 +4,7 @@ Reference basis: - Paper cross-check from `reviews/task-30/581-30658-spire-phase9-routing-plan/feedback/2026-05-09-01-reviewer.md` - against `/home/peter/dev_bak/papers/2512.17264v1.pdf`. + against arXiv:2512.17264v1. - Phase 9 and 10 local architecture closeout in: - `plan/tasks/task30-phase9-spire-graph-architecture.md` - `plan/tasks/task30-phase10-spire-execution-performance.md` diff --git a/reviews/task-30/614-30692-spire-phase11-paper-parity-gate/artifacts/paper-source-outline.md b/reviews/task-30/614-30692-spire-phase11-paper-parity-gate/artifacts/paper-source-outline.md index 052a6bd6dd..5872d6453e 100644 --- a/reviews/task-30/614-30692-spire-phase11-paper-parity-gate/artifacts/paper-source-outline.md +++ b/reviews/task-30/614-30692-spire-phase11-paper-parity-gate/artifacts/paper-source-outline.md @@ -1,6 +1,6 @@ # SPire Paper Source Outline -Source file: `/home/peter/dev_bak/papers/2512.17264v1.pdf` +Source: arXiv:2512.17264v1 The local PDF metadata identifies the paper as "Scalable Distributed Vector Search via Accuracy Preserving Index Construction". The decoded outline used diff --git a/reviews/task-86/001-turbovec-tq-analysis/artifacts/manifest.md b/reviews/task-86/001-turbovec-tq-analysis/artifacts/manifest.md index 2266a11456..87a46ba412 100644 --- a/reviews/task-86/001-turbovec-tq-analysis/artifacts/manifest.md +++ b/reviews/task-86/001-turbovec-tq-analysis/artifacts/manifest.md @@ -12,7 +12,7 @@ - `sed` / `rg` / `nl` source inspection against local TurboVec and current repo sources - TurboVec source snapshot: - - Local path: `/Users/peter/dev_bak/turbovec` + - Source identity: TurboVec repository snapshot - SHA: `efe29a184986cbf562a9847c2ac52a2990bfaca2` - Status: clean `main` diff --git a/reviews/task-86/001-turbovec-tq-analysis/artifacts/turbovec-tq-analysis.md b/reviews/task-86/001-turbovec-tq-analysis/artifacts/turbovec-tq-analysis.md index be01a6d7aa..0f7c2d76c6 100644 --- a/reviews/task-86/001-turbovec-tq-analysis/artifacts/turbovec-tq-analysis.md +++ b/reviews/task-86/001-turbovec-tq-analysis/artifacts/turbovec-tq-analysis.md @@ -4,7 +4,7 @@ Scope: compare TurboVec's TurboQuant implementation to our TurboQuant implementation only. This report intentionally does not evaluate other quantizers or external leaderboards. -TurboVec source inspected: `/Users/peter/dev_bak/turbovec` at +TurboVec source inspected at `efe29a184986cbf562a9847c2ac52a2990bfaca2`. Our source inspected: `/Users/peter/dev/tqvector` at @@ -30,17 +30,17 @@ build lookup tables. The query is not packed into the same database code format. Relevant source: -- `/Users/peter/dev_bak/turbovec/turbovec/src/lib.rs:96` defines +- `TurboVec@efe29a1:turbovec/src/lib.rs:96` defines `TurboQuantIndex` with `packed_codes`, `scales`, TQ+ shift/scale arrays, and lazy blocked-code caches. -- `/Users/peter/dev_bak/turbovec/turbovec/src/lib.rs:250` through `:283` +- `TurboVec@efe29a1:turbovec/src/lib.rs:250` through `:283` initializes rotation/codebook state and calls `encode::encode`. -- `/Users/peter/dev_bak/turbovec/turbovec/src/encode.rs:1` through `:27` +- `TurboVec@efe29a1:turbovec/src/encode.rs:1` through `:27` documents normalize, rotate, TQ+ calibrate, quantize, pack, and scale. -- `/Users/peter/dev_bak/turbovec/turbovec/src/encode.rs:65` through `:133` +- `TurboVec@efe29a1:turbovec/src/encode.rs:65` through `:133` implements normalize, dense rotation, calibration, packed-code allocation, and per-row fused quantize/scale/pack. -- `/Users/peter/dev_bak/turbovec/turbovec/src/encode.rs:245` through `:352` +- `TurboVec@efe29a1:turbovec/src/encode.rs:245` through `:352` explains and implements the per-vector renormalization scalar. Encoding shape: @@ -61,16 +61,16 @@ the same calibration, so all vectors live in one calibrated coordinate system. Relevant source: -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1466` through `:1530` +- `TurboVec@efe29a1:turbovec/src/search.rs:1466` through `:1530` implements full query search setup. -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1492` through `:1508` +- `TurboVec@efe29a1:turbovec/src/search.rs:1492` through `:1508` rotates all queries with a batched GEMM. -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1510` through `:1518` +- `TurboVec@efe29a1:turbovec/src/search.rs:1510` through `:1518` applies inverse TQ+ calibration. -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1419` through `:1455` +- `TurboVec@efe29a1:turbovec/src/search.rs:1419` through `:1455` defines `q_calib[d] = q_rot[d] / scale_tq[d]` and bias `-sum(q_rot[d] * shift[d])`. -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1165` through `:1280` +- `TurboVec@efe29a1:turbovec/src/search.rs:1165` through `:1280` builds per-query `u8` nibble lookup tables with per-subtable mins, one shared scale, and a bias. @@ -83,20 +83,20 @@ decompressed into dense float vectors at query time. Relevant source: -- `/Users/peter/dev_bak/turbovec/turbovec/src/lib.rs:357` through `:475` +- `TurboVec@efe29a1:turbovec/src/lib.rs:357` through `:475` exposes `search` and `search_with_mask`, materializes a blocked-code cache, packs optional filters into a slot bitset, and calls `search::search`. -- `/Users/peter/dev_bak/turbovec/turbovec/src/pack.rs:1` through `:60` +- `TurboVec@efe29a1:turbovec/src/pack.rs:1` through `:60` repacks per-vector bit-plane codes into SIMD-blocked layout. -- `/Users/peter/dev_bak/turbovec/turbovec/src/pack.rs:62` through `:88` +- `TurboVec@efe29a1:turbovec/src/pack.rs:62` through `:88` uses an x86 FAISS-style 32-vector block layout with split hi/lo nibbles. -- `/Users/peter/dev_bak/turbovec/turbovec/src/pack.rs:90` through `:114` +- `TurboVec@efe29a1:turbovec/src/pack.rs:90` through `:114` uses a sequential 32-vector block layout on non-x86. -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1293` through `:1335` +- `TurboVec@efe29a1:turbovec/src/search.rs:1293` through `:1335` implements block and block-pair filter skips. -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1337` through `:1417` +- `TurboVec@efe29a1:turbovec/src/search.rs:1337` through `:1417` shows scalar scoring over all blocks and lanes with fused heap maintenance. -- `/Users/peter/dev_bak/turbovec/turbovec/src/search.rs:1532` onward dispatches +- `TurboVec@efe29a1:turbovec/src/search.rs:1532` onward dispatches platform-specific scoring and top-k. There is no routing layer in the inspected implementation: @@ -118,10 +118,10 @@ candidate generation, graph traversal, page layout, heap rerank, or tuple I/O. TurboVec file layout source: -- `/Users/peter/dev_bak/turbovec/turbovec/src/io.rs:178` through `:213` writes +- `TurboVec@efe29a1:turbovec/src/io.rs:178` through `:213` writes bit width, dimension, vector count, packed codes, per-vector scales, and TQ+ calibration. -- `/Users/peter/dev_bak/turbovec/turbovec/src/io.rs:262` through `:276` reads +- `TurboVec@efe29a1:turbovec/src/io.rs:262` through `:276` reads `packed_bytes = (dim / 8) * bit_width * n_vectors` and `n_vectors` f32 scales. diff --git a/reviews/task-87/001-phase1-design/artifacts/source-scoring-map.md b/reviews/task-87/001-phase1-design/artifacts/source-scoring-map.md index 82ee26f8a7..37374a117e 100644 --- a/reviews/task-87/001-phase1-design/artifacts/source-scoring-map.md +++ b/reviews/task-87/001-phase1-design/artifacts/source-scoring-map.md @@ -126,7 +126,7 @@ Task 87 mapping: Reference file: -- `/home/peter/dev_bak/pgvectorscale/pgvectorscale/src/access_method/scan.rs` +- `timescale/pgvectorscale:pgvectorscale/src/access_method/scan.rs` Relevant resort-buffer lines: diff --git a/scripts/check_workstation_paths.py b/scripts/check_workstation_paths.py new file mode 100644 index 0000000000..6c64bc5c65 --- /dev/null +++ b/scripts/check_workstation_paths.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +"""Reject newly added machine-specific workstation paths without echoing them.""" + +from __future__ import annotations + +import argparse +import re +import subprocess +import sys +from dataclasses import dataclass + + +HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@") +PATH_PATTERNS = ( + ("tilde workstation home path", re.compile(r"(?:^|(?<=\s)|(?<=[\"'`]))~/(?=\S)")), + ( + "Unix workstation home path", + re.compile(r"/(?:home|Users)/[^/\s\"'`]+(?=/|\s|[\"'`]|$)"), + ), + ("root workstation path", re.compile(r"/root(?=/|\s|[\"'`]|$)")), + ( + "Windows workstation home path", + re.compile( + r"(?i)\b[A-Z]:[\\/](?:Users|Documents and Settings)[\\/]" + r"[^\\/\s\"'`]+(?=[\\/]|\s|[\"'`]|$)" + ), + ), +) + + +@dataclass(frozen=True) +class AddedLine: + path: str + line_number: int + text: str + + +@dataclass(frozen=True) +class Violation: + path: str + line_number: int + label: str + + +def parse_added_lines(diff: str) -> list[AddedLine]: + """Return added text lines from a zero-context unified diff.""" + + path: str | None = None + line_number: int | None = None + added: list[AddedLine] = [] + + for raw in diff.splitlines(): + if raw.startswith("+++ "): + header_path = raw[4:].strip() + path = header_path[2:] if header_path.startswith("b/") else header_path + if path == "/dev/null": + path = None + continue + + match = HUNK.match(raw) + if match: + line_number = int(match.group(1)) + continue + + if path is None or line_number is None: + continue + if raw.startswith("+"): + added.append(AddedLine(path, line_number, raw[1:])) + line_number += 1 + elif raw.startswith("-") or raw.startswith("\\ No newline"): + continue + else: + line_number += 1 + + return added + + +def violations_for_diff(diff: str) -> list[Violation]: + violations: list[Violation] = [] + for line in parse_added_lines(diff): + for label, pattern in PATH_PATTERNS: + if pattern.search(line.text): + violations.append(Violation(line.path, line.line_number, label)) + return violations + + +def render_violations(violations: list[Violation]) -> str: + lines = [ + "workstation-path check failed: newly added machine-specific paths found" + ] + lines.extend( + f"{violation.path}:{violation.line_number}: {violation.label}" + for violation in violations + ) + lines.append("Use an environment variable, repository-relative path, or durable URL.") + return "\n".join(lines) + + +def git_diff(diff_range: str) -> str: + completed = subprocess.run( + [ + "git", + "diff", + "--unified=0", + "--no-color", + "--no-ext-diff", + diff_range, + "--", + ], + check=True, + stdout=subprocess.PIPE, + text=True, + ) + return completed.stdout + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--diff-range", + required=True, + help="Git revision range to inspect, for example BASE...HEAD", + ) + args = parser.parse_args() + + violations = violations_for_diff(git_diff(args.diff_range)) + if violations: + print(render_violations(violations), file=sys.stderr) + return 1 + + print("workstation-path check passed") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/tests/test_check_workstation_paths.py b/scripts/tests/test_check_workstation_paths.py new file mode 100644 index 0000000000..d511a376a7 --- /dev/null +++ b/scripts/tests/test_check_workstation_paths.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 + +from __future__ import annotations + +import importlib.util +import sys +import unittest +from pathlib import Path + + +SCRIPT = Path(__file__).parents[1] / "check_workstation_paths.py" +SPEC = importlib.util.spec_from_file_location("check_workstation_paths", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +MODULE = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = MODULE +SPEC.loader.exec_module(MODULE) + + +class WorkstationPathCheckTests(unittest.TestCase): + def test_added_workstation_path_is_reported_without_its_value(self) -> None: + private_path = "/" + "home" + "/example/project/output.log" + diff = "\n".join( + [ + "diff --git a/report.md b/report.md", + "--- a/report.md", + "+++ b/report.md", + "@@ -2,0 +3 @@", + f'+artifact = "{private_path}"', + ] + ) + + violations = MODULE.violations_for_diff(diff) + self.assertEqual(len(violations), 1) + self.assertEqual(violations[0].path, "report.md") + self.assertEqual(violations[0].line_number, 3) + rendered = MODULE.render_violations(violations) + self.assertNotIn(private_path, rendered) + + def test_removed_paths_and_portable_values_are_ignored(self) -> None: + private_path = "/" + "Users" + "/example/project/output.log" + diff = "\n".join( + [ + "diff --git a/report.md b/report.md", + "--- a/report.md", + "+++ b/report.md", + "@@ -3 +3 @@", + f'-artifact = "{private_path}"', + '+artifact = "${ARTIFACT_ROOT}/output.log"', + ] + ) + + self.assertEqual(MODULE.violations_for_diff(diff), []) + + def test_root_tilde_and_windows_home_paths_are_detected(self) -> None: + root_path = "/" + "root" + "/work/output.log" + tilde_path = "~" + "/work/output.log" + windows_path = "C:" + "\\" + "Users\\example\\output.log" + diff = "\n".join( + [ + "diff --git a/report.md b/report.md", + "--- a/report.md", + "+++ b/report.md", + "@@ -0,0 +1,3 @@", + f"+{root_path}", + f"+{tilde_path}", + f"+{windows_path}", + ] + ) + + self.assertEqual(len(MODULE.violations_for_diff(diff)), 3) + + +if __name__ == "__main__": + unittest.main() diff --git a/spec/adr/ADR-006-own-quantizer.md b/spec/adr/ADR-006-own-quantizer.md index f274b1456e..756913817c 100644 --- a/spec/adr/ADR-006-own-quantizer.md +++ b/spec/adr/ADR-006-own-quantizer.md @@ -24,7 +24,8 @@ The `turbo-quant` crate (v0.1) was the original plan. Investigation revealed thr ## Decision -**Drop `turbo-quant` crate.** Extract the quantizer core from `~/dev_bak/TurboQuantDB/` and adapt for pgrx. +**Drop `turbo-quant` crate.** Extract the quantizer core from +`jyunming/TurboQuantDB` and adapt for pgrx. ### What to extract diff --git a/spec/adr/ADR-024-fwht-transform-strategy.md b/spec/adr/ADR-024-fwht-transform-strategy.md index 23164d0eb4..d871c8f970 100644 --- a/spec/adr/ADR-024-fwht-transform-strategy.md +++ b/spec/adr/ADR-024-fwht-transform-strategy.md @@ -43,7 +43,7 @@ Configuration A is retired. The decision is between B and C. ### What TurboQuantDB Does Differently -The upstream TurboQuantDB implementation (`~/dev_bak/TurboQuantDB/`) uses configuration C: +The upstream `jyunming/TurboQuantDB` implementation uses configuration C: 1. **Pad to `n = next_power_of_two(d)`** — for d=1536, n=2048 2. **Full FWHT on n dimensions** — complete decorrelation across all coordinates @@ -156,4 +156,5 @@ No code path should produce configuration A going forward. - Review 202: A4 1536 tail-truncation probes — transform-tail truncation identified as dominant loss - Review 203: A4 1536 tiled-FWHT quantizer — production tiled FWHT implementation - Review 204: A4 full vs tiled FWHT — TurboQuantDB comparison and three-configuration analysis -- TurboQuantDB source: `~/dev_bak/TurboQuantDB/src/quantizer/prod.rs`, `mse.rs`, `../linalg/hadamard.rs` +- TurboQuantDB source: `src/quantizer/prod.rs`, `mse.rs`, and + `src/linalg/hadamard.rs` in `jyunming/TurboQuantDB` diff --git a/spec/adr/ADR-045-symphonyqg-quantized-graph-access-method.md b/spec/adr/ADR-045-symphonyqg-quantized-graph-access-method.md index 1136952ff9..735aef6d1a 100644 --- a/spec/adr/ADR-045-symphonyqg-quantized-graph-access-method.md +++ b/spec/adr/ADR-045-symphonyqg-quantized-graph-access-method.md @@ -257,8 +257,7 @@ added a pragmatic q ∈ {1, 2, 4, 8} extension to `RaBitQQuantizer` so non-Symphony consumers (DiskANN in-memory tier, general prefilter, offline eval) can dial the recall/storage tradeoff. The slice-12 q-bit implementation is not paper-faithful to -**Extended RaBitQ** (Gao et al., SIGMOD 2025, arXiv:2409.09913 — -`~/dev_bak/papers/extended-rabitq-2025-sigmod-arxiv-2409.09913.pdf`): +**Extended RaBitQ** (Gao et al., SIGMOD 2025, arXiv:2409.09913): 1. **Scalar quantizer.** Slice 12 uses uniform binning on `±2σ`; Extended RaBitQ uses a Lloyd-Max-like codebook diff --git a/spec/adr/ADR-085-ec-distann-single-global-graph.md b/spec/adr/ADR-085-ec-distann-single-global-graph.md index 9c8d783839..76ab444abb 100644 --- a/spec/adr/ADR-085-ec-distann-single-global-graph.md +++ b/spec/adr/ADR-085-ec-distann-single-global-graph.md @@ -37,8 +37,7 @@ branch + packet path): The root cause is architectural: a lossy partition-level routing decision must be hedged wider as recall targets rise and corpora grow. DistributedANN -(**arXiv:2509.06046**, the durable citation; convenience local copy -`~/dev_bak/papers/distributedann-2509.06046.pdf`; same research group as +(**arXiv:2509.06046**, the durable citation; same research group as SPANN/SPIRE, replaced partition-routing in Bing production) inverts the design: distribute the storage, keep the index whole. One global Vamana graph; records hash-placed; query cost = beam × hops, corpus-independent. diff --git a/spec/reviews/evidence.md b/spec/reviews/evidence.md index afd1a21a23..73e1954a46 100644 --- a/spec/reviews/evidence.md +++ b/spec/reviews/evidence.md @@ -57,4 +57,4 @@ commit" convention. | FND-009 | low | NFR-017's "p50 latency at matched recall" does not define the operating-point selection rule (interpolation across sweep points vs nearest cell at ≥ 0.999), and the p50/p95 rows are specified only at 100k while the recall row spans 10k/50k/100k — pre-register the matched-recall protocol (the Task 146 packet's rule, if that is the intent) in the Verification section so the four-way table is not assembled ad hoc | NFR-017, TC-044 | | FND-010 | low | NFR-020's fault taxonomy includes "network partition", but the gate substrate is loopback multi-instance (ADR-085 D2) where a true partition is not injectable with the reused connection-level drill machinery (reset/timeout/termination are); name the injection mechanism for the partition case or scope it to connection-level manifestations so the "100% drill matrix" row is executable | NFR-020, TC-042 | | FND-011 | low | FR-077-CON-4 / ADR-085 D8 bound stitch memory by "one vec_id group plus prune working set" with validation "Analysis + build instrumentation", but no numeric bound and no named analysis artifact path — non-test verification methods must name their artifact (analysis doc + which epoch-manifest/instrumentation field). As written the CON is not falsifiable | FR-077, ADR-085 | -| FND-012 | low | ADR-085 cites the DistributedANN paper by a local home-directory path (`~/dev_bak/papers/distributedann-2509.06046.pdf`) — a non-durable, single-machine reference; the arXiv id (2509.06046) alone is the durable citation. Minor because it is background literature, not gate evidence, but the packet-discipline rule against citing local-only paths applies to ADRs too | ADR-085 | +| FND-012 | resolved | ADR-085 now cites DistributedANN by its durable arXiv identifier rather than a local home-directory path. | ADR-085; arXiv:2509.06046 | From a21e844bf05ddd68a1ca895be3d08fa11a213080 Mon Sep 17 00:00:00 2001 From: Agent IX Date: Sat, 29 Aug 2026 02:52:47 -0700 Subject: [PATCH 2/2] docs: request Task 240 path hygiene review --- plan/tasks/240-artifact-path-sanitization.md | 5 +- plan/tasks/README.md | 2 +- .../artifacts/manifest.md | 57 +++++++++++++ .../artifacts/redaction-audit.md | 37 +++++++++ .../001-current-tree-sanitization/request.md | 82 +++++++++++++++++++ 5 files changed, 180 insertions(+), 3 deletions(-) create mode 100644 reviews/task-240/001-current-tree-sanitization/artifacts/manifest.md create mode 100644 reviews/task-240/001-current-tree-sanitization/artifacts/redaction-audit.md create mode 100644 reviews/task-240/001-current-tree-sanitization/request.md diff --git a/plan/tasks/240-artifact-path-sanitization.md b/plan/tasks/240-artifact-path-sanitization.md index 61d929f674..99cd853425 100644 --- a/plan/tasks/240-artifact-path-sanitization.md +++ b/plan/tasks/240-artifact-path-sanitization.md @@ -1,7 +1,8 @@ # Task 240: Artifact Path Sanitization -Status: **implementation in progress** (2026-08-29). Priority: P0 privacy and -evidence hygiene. Origin: private Engineering Assurance gap audit. +Status: **implementation complete; packet 001 review-open; history remediation +pending explicit decision** (2026-08-29). Priority: P0 privacy and evidence +hygiene. Origin: private Engineering Assurance gap audit. ## Why diff --git a/plan/tasks/README.md b/plan/tasks/README.md index 8e7ae62171..a89928ade9 100644 --- a/plan/tasks/README.md +++ b/plan/tasks/README.md @@ -279,7 +279,7 @@ These pre-lane task files are retained only for historical context under 238. `238-ec-distann-retry-snapshot-use-after-free.md` — **implementation and evidence merged; packet 001 outside closeout review-open** (updated 2026-08-27, P0 correctness): the equivalent `RegisteredSnapshotGuard` lifetime fix remains on main via `15f7fcf5f` / Task 167 PR #77. Checkpoint `7d4103885` restores the deterministic forced-retry regression and explicit loopback-only plaintext test conninfo; both PG18 callers pass on base `3c81319a3`. PR #91 merged the implementation and evidence as `bcc9b0bfa`; only the outside ACCEPT remains open. Evidence and request: `reviews/task-238/001-retry-snapshot-uaf/`. 239. `239-ec-distann-bounded-read-overfetch.md` — **complete — review-closed ACCEPT; HARNESS REGRESSION CORRECTED, EXACT-MAIN LAZY-10 SEMANTIC PATH RESTORED TO 10/10; no rerun; packet 004 10k/50k/100k NOT triggered; Task 229 / 230--233 semantic-surface blocker LIFTED** (updated 2026-08-26, P1 correctness/performance): packet 001's 12/10 result was a shared-session batch-size GUC leak in the benchmark harness — a variant at batch size 10 emitted no `SET`, so the "lazy-10" arm inherited the eager control's 0 — not production overfetch, and the bound was never widened. Packet 002 corrected the harness but its cross-SHA runner aborted pre-semantics on 40-versus-37 stage rows. Packet 003 removed that confounder with exact-main runner/extension plus only the harness fix; its sole C1--C5 run passed every preregistered gate — nine scenarios exactly once in both logs, seven core rows at `control_batch_size=0 candidate_batch_size=10` with exact identity and zero duplicates, `exactly_one_window` 6 remote + 4 local = 10 reads/bound 10, mixed and outage pass, routed DELETE+VACUUM pass, both recall arms 0.9990 over 200 queries / 2,000 trials with predictions byte-identical to packet 001 — and the stopped 1.2 GB fixture was removed after capture. `git diff 41392c011 def565270 -- src` is empty, so no production runtime behavior changed and the 10k/50k/100k closeout matrix does not apply. Determinism rests on the same 6/4/10 split and digests at three independent extension SHAs (task-191 `7883cfcf`, task-198 `2ff72b3e`, this run `4ab2aa9a9`), so no rerun was needed. Carried follow-ups for whichever task next touches the multinode semantic harness: mixed/outage failures emit no structured `pass=false` row; `owner_payload_plan_cache` retains the same conditional-`SET` leak shape; recall/latency child batch size stays inferred. Closeout verdict: `reviews/task-239/003-main-baseline-semantic-proof/feedback/2026-08-26-02-reviewer.md`; result decision: `reviews/task-239/003-main-baseline-semantic-proof/artifacts/live-run-decision.md`; review request: `reviews/task-239/003-main-baseline-semantic-proof/request.md`; live authorization: `reviews/task-239/003-main-baseline-semantic-proof/feedback/2026-08-26-01-reviewer.md`. -240. `240-artifact-path-sanitization.md` — **implementation in progress** (2026-08-29, P0 privacy/evidence hygiene): remove current-tree internal scratch locators, require portable comparison-repository inputs, and add a non-echoing diff guard for newly introduced workstation paths; historical Git remediation remains a separate explicit decision. +240. `240-artifact-path-sanitization.md` — **implementation complete; packet 001 review-open; history remediation pending explicit decision** (2026-08-29, P0 privacy/evidence hygiene): current-tree internal scratch locators are removed, comparison-repository inputs are explicit, and a non-echoing diff guard rejects newly introduced workstation paths; packet `reviews/task-240/001-current-tree-sanitization/` requests outside review, while historical Git remediation remains a separate explicit decision. ## Coordination rules diff --git a/reviews/task-240/001-current-tree-sanitization/artifacts/manifest.md b/reviews/task-240/001-current-tree-sanitization/artifacts/manifest.md new file mode 100644 index 0000000000..4d0317b9b1 --- /dev/null +++ b/reviews/task-240/001-current-tree-sanitization/artifacts/manifest.md @@ -0,0 +1,57 @@ +# Task 240 packet 001 artifact manifest + +- Task/packet: `task-240/001-current-tree-sanitization` +- Baseline: `23fb9b7ba1f0803be5dfc700d9865f80fbf60862` +- Implementation: `e393515c7c94e0f127bcbac633052ee457c0527e` +- Review-request checkpoint: this packet commit +- Scope: current tracked tree and future added diff lines +- Excluded: Git-history rewrite, hosted hidden refs/caches, and bulk + normalization of unrelated legacy workstation paths + +## Audit results + +- Baseline files containing the identified internal scratch locator: 31. +- Implementation files containing that locator: 0. +- Baseline files containing the audited absolute workstation prefix: 6,396. +- Implementation files containing that exact prefix: 6,390. +- Added-lines guard across baseline to implementation: pass. +- Whitespace/diff validation: pass. + +The sensitive locator and matching source text are intentionally absent from +this public packet. The audit reports counts and categories only. + +## Functional validation + +- `python3 scripts/tests/test_check_workstation_paths.py`: 3 tests pass. +- `python3 scripts/check_workstation_paths.py --diff-range + 23fb9b7ba1f0803be5dfc700d9865f80fbf60862...e393515c7c94e0f127bcbac633052ee457c0527e`: + pass. +- `cargo fmt --all -- --check`: pass with existing stable-rustfmt warnings. +- Isolated `cargo build -p ecaz-cli`: pass with one existing dead-code + warning for `LoadedDistributedPlacementConfig.path`. +- `cargo test -p ecaz-cli commands::dev::install`: compile pass; 546 tests + filtered out and zero selected. +- Freshly built CLI with each repository variable unset: + - `ecaz dev install pgvector --pg 18`: exit 2; `--repo` required. + - `ecaz dev install vectorscale --pg 18`: exit 2; `--repo` required. + +No installer runtime path was reached during the missing-argument checks. + +## Known baseline test-runner failures + +`scripts/tests/run.sh` reports: + +- three failures in + `scripts/tests/test_pg17_scratch_psql_socket_resolution.py`; +- four errors in + `scripts/tests/test_resolve_scratch_socket_dir.py`. + +Both groups arise because the unchanged baseline references already-deleted +scripts. They are outside Task 240 and are recorded without being reclassified +as a Task 240 regression. + +## Generated artifact cleanup + +The isolated Cargo target directory used for the behavioral CLI checks was +removed after validation. It contained generated build output only; no tracked +file or installed extension was removed. diff --git a/reviews/task-240/001-current-tree-sanitization/artifacts/redaction-audit.md b/reviews/task-240/001-current-tree-sanitization/artifacts/redaction-audit.md new file mode 100644 index 0000000000..32b8faafc1 --- /dev/null +++ b/reviews/task-240/001-current-tree-sanitization/artifacts/redaction-audit.md @@ -0,0 +1,37 @@ +# Task 240 current-tree redaction audit + +## Classification + +The 31 baseline files containing the identified internal scratch locator were: + +| Category | Files | Treatment | +|---|---:|---| +| CLI source and documentation | 2 | Removed implicit local defaults; documented explicit operator inputs | +| Benchmark/reviewer evidence | 2 | Replaced only source/build locator text | +| Planning and design | 4 | Replaced local locations with durable public identities | +| Review packets | 18 | Replaced citation/source locators; retained findings and measurements | +| Specifications, ADRs, and findings | 5 | Replaced local citations and closed the resolved citation finding | +| **Total** | **31** | **Current-tree locator count is zero** | + +## Evidence-integrity check + +The diff was reviewed line by line. Existing evidence edits are confined to: + +- paper-location to arXiv-identifier substitutions; +- checkout-location to public repository/source-identity substitutions; +- local build-path components to a neutral `[local-checkout]` marker; and +- prose wrapping needed by those substitutions. + +No measured value, result row, duration, source SHA, algorithm description, +review verdict, or acceptance conclusion changed. The edited historical build +log is described by its manifest but is not content-digest-bound there. + +## Residual exposure + +This is a current-tree repair, not a history purge. Existing Git objects retain +the old bytes, and the legacy corpus still contains other workstation paths. +The new guard prevents newly added instances; it does not retroactively reject +or silently rewrite pre-control evidence. + +Any later history remediation must have its own authorization, recovery plan, +force-push coordination, hidden-ref/cache verification, and post-rewrite audit. diff --git a/reviews/task-240/001-current-tree-sanitization/request.md b/reviews/task-240/001-current-tree-sanitization/request.md new file mode 100644 index 0000000000..94198e7d93 --- /dev/null +++ b/reviews/task-240/001-current-tree-sanitization/request.md @@ -0,0 +1,82 @@ +--- +task: 240 +packet: 001-current-tree-sanitization +agent: Codex +role: coder +model: gpt-5 +date: 2026-08-29 +seq: 01 +--- + +# Task 240 current-tree sanitization review request + +Please review the current-tree privacy repair and future-publication guard at +`e393515c7c94e0f127bcbac633052ee457c0527e`, based on public main +`23fb9b7ba1f0803be5dfc700d9865f80fbf60862`. + +## Requested decision + +Accept the implementation if the reviewer agrees that: + +1. the 31 current-tree references to the identified internal scratch locator + were replaced with durable public citations, source identities, explicit + operator inputs, or a neutral local-checkout marker; +2. those edits changed locator/citation metadata only and did not alter + benchmark measurements, source SHAs, findings, or technical conclusions; +3. the CLI must require `--repo` or the documented environment variable for + both external comparison-extension installers; +4. the added-lines guard safely reports file, line, and category without + echoing the matched value; and +5. Git-history remediation remains explicitly out of scope until a separate + destructive-history decision is approved. + +## Implementation + +- Removed the identified internal scratch locator from every tracked file in + the current tree. +- Replaced paper locations with arXiv identifiers and source checkouts with + durable repository identities plus pinned SHAs where already available. +- Removed machine-specific comparison-repository defaults from + `ecaz dev install pgvector` and `ecaz dev install vectorscale`. +- Added `scripts/check_workstation_paths.py`, focused synthetic tests, and a + lightweight pull-request/main-push workflow. +- Added the workstation-path rule to `AGENTS.md`. + +The guard covers added Unix, root, Windows, and tilde-home paths. It inspects +only added diff lines so pre-existing immutable evidence does not cause every +unrelated pull request to fail. + +## Deliberate boundary + +The baseline contains 6,396 tracked files with one common absolute workstation +prefix. This checkpoint reduces that exact-prefix count to 6,390 but does not +bulk-rewrite the legacy evidence corpus. It does reduce the specifically +identified internal scratch locator from 31 tracked files to zero. + +Old values remain reachable from existing Git objects. No branch, tag, pull +request ref, commit, or repository history was rewritten or deleted. A +history-wide purge would require a coordinated force-push, collaborator +re-clone procedure, and hosting-provider cache/ref handling; that is not +authorized by this packet. + +## Validation + +- Focused Python tests: 3/3 pass. +- Added-lines guard over `23fb9b7...e393515c7`: pass. +- Current-tree exact-locator audit: zero matches. +- `git diff --check`: pass. +- `cargo fmt --all -- --check`: pass (existing stable-rustfmt warnings only). +- Isolated `cargo build -p ecaz-cli`: pass (one existing unread-field warning). +- `cargo test -p ecaz-cli commands::dev::install`: compile pass; the filter + selected zero tests, so this is not claimed as behavioral coverage. +- New binary, repository variables unset: + - pgvector installer: exit 2, missing required `--repo `; + - vectorscale installer: exit 2, missing required `--repo `. + +The repository-wide `scripts/tests/run.sh` is not green on the unchanged +baseline: it reports three failures and four errors in the socket-resolution +tests because two referenced shell scripts are already absent. This task does +not modify that unrelated surface and does not claim the full runner as a pass. + +See `artifacts/manifest.md` and `artifacts/redaction-audit.md` for the exact +scope and evidence classification.