Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .github/workflows/path-hygiene.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Path hygiene

on:
pull_request:
push:
branches: [main]
workflow_dispatch:

permissions:
contents: read

jobs:
added-lines:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Reject newly added workstation paths
env:
EVENT_NAME: ${{ github.event_name }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
BEFORE_SHA: ${{ github.event.before }}
run: |
if [ "$EVENT_NAME" = "pull_request" ]; then
diff_range="${BASE_SHA}...HEAD"
elif [ -n "$BEFORE_SHA" ] && ! printf '%s' "$BEFORE_SHA" | grep -Eq '^0+$'; then
diff_range="${BEFORE_SHA}..HEAD"
elif git rev-parse HEAD^ >/dev/null 2>&1; then
diff_range="HEAD^..HEAD"
else
empty_tree="$(git hash-object -t tree /dev/null)"
diff_range="${empty_tree}..HEAD"
fi
python3 scripts/check_workstation_paths.py --diff-range "$diff_range"
11 changes: 11 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,17 @@ Packet directories inside a task bucket must sort in chronological order.
- `request.md` should summarize the result and point at the packet-local
artifact files.

### Workstation Path Hygiene

- New or changed tracked content must not introduce a machine-specific absolute
workstation home path. Use an environment variable, repository-relative path,
or durable public URL instead.
- `scripts/check_workstation_paths.py` enforces this on added diff lines without
echoing the sensitive value into CI output.
- Historical evidence that predates this control must not be bulk-rewritten merely
to make the legacy corpus look clean. Any redaction requires an explicit audit
trail and a separate history-remediation decision.

### Never Commit: Corpus Data, Operational Logs, and Polling Cruft

A review packet is decision-grade evidence, not a capture of everything the run
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Script started on 2026-04-30 16:13:15-07:00 [<not executed on terminal>]
Compiling ptr_meta_derive v0.1.4
Compiling bytecheck_derive v0.6.12
Compiling rkyv_derive v0.7.46
Compiling pgvectorscale_derive v0.1.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale/pgvectorscale_derive)
Compiling pgvectorscale_derive v0.1.0 ([local-checkout]/pgvectorscale/pgvectorscale_derive)
Compiling ptr_meta v0.1.4
Compiling synstructure v0.13.2
Compiling futures-executor v0.3.32
Expand Down Expand Up @@ -157,7 +157,7 @@ Script started on 2026-04-30 16:13:15-07:00 [<not executed on terminal>]
Compiling pgrx-bindgen v0.16.1
Compiling pgrx-pg-sys v0.16.1
Compiling pgrx v0.16.1
Compiling vectorscale v0.9.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale)
Compiling vectorscale v0.9.0 ([local-checkout]/pgvectorscale)
Finished `release` profile [optimized] target(s) in 54.98s
 Installing extension
 Copying control file to /home/peter/.pgrx/18.3/pgrx-install/share/postgresql/extension/vectorscale.control
Expand Down Expand Up @@ -279,7 +279,7 @@ Script started on 2026-04-30 16:13:15-07:00 [<not executed on terminal>]
Compiling ptr_meta_derive v0.1.4
Compiling bytecheck_derive v0.6.12
Compiling rkyv_derive v0.7.46
Compiling pgvectorscale_derive v0.1.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale/pgvectorscale_derive)
Compiling pgvectorscale_derive v0.1.0 ([local-checkout]/pgvectorscale/pgvectorscale_derive)
Compiling futures-executor v0.3.32
Compiling ptr_meta v0.1.4
Compiling regex v1.12.3
Expand Down Expand Up @@ -319,7 +319,7 @@ Script started on 2026-04-30 16:13:15-07:00 [<not executed on terminal>]
Compiling pgrx-bindgen v0.16.1
Compiling pgrx-pg-sys v0.16.1
Compiling pgrx v0.16.1
Compiling vectorscale v0.9.0 (/home/peter/dev_bak/pgvectorscale/pgvectorscale)
Compiling vectorscale v0.9.0 ([local-checkout]/pgvectorscale)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 37.88s
 Writing SQL entities to /home/peter/.pgrx/18.3/pgrx-install/share/postgresql/extension/vectorscale--0.9.0.sql
 Copying extension schema upgrade file to /home/peter/.pgrx/18.3/pgrx-install/share/postgresql/extension/vectorscale--0.2.0--0.3.0.sql
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@ Well-framed. Updated assessment after cross-referencing reviews 195-198 and 201:

## Additional Findings

Verified against reference implementations in `~/dev_bak/hnsw-references/`:
Verified against local reference checkouts of `hnsw`, `hnswlib-rs`,
`instant-distance`, and `swarc`:

1. **Layer indexing**: hnswlib-rs `get_neighborhood_id()` iterates `0..nb_layer` — layer 0 = index 0.
Matches `pack_point_neighbor_slots`. No reversal bug.
Expand Down
15 changes: 8 additions & 7 deletions crates/ecaz-cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,14 @@ cargo install --path crates/ecaz-cli
That installs the binary under Cargo's bin directory, usually
`$HOME/.cargo/bin/ecaz`. Interactive shells may also put that directory on
`$PATH`; when running from an agent or other sandboxed tool session, prefer the
absolute installed path (for example `/Users/peter/.cargo/bin/ecaz`) so one
absolute installed path (for example `${HOME}/.cargo/bin/ecaz`) so one
approval rule can cover the operator surface consistently.

All commands accept `--database`, `--host`, `--port`, `--user`, `--password`,
and `--log-file`; each also falls back to the matching libpq environment
variable (`PGDATABASE`, `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`) when the
flag is omitted. `--host` may be either a TCP host name or a Unix socket
directory such as `/home/peter/.pgrx`.
directory supplied through `PGRX_HOME` or `PGHOST`.
`--log-file` mirrors the CLI's stdout/stderr into a packet-local artifact
file so review runs do not need shell `tee` wrappers. When `--log-file` is
set, transient progress bars are suppressed so the artifact stays stable.
Expand All @@ -67,8 +67,8 @@ generation, load/list/inspect, benchmark, storage, scratch, and SQL checks
through that binary:

```sh
/Users/peter/.cargo/bin/ecaz dev sql --pg 18 --db postgres \
--socket-dir /Users/peter/.pgrx --raw \
${HOME}/.cargo/bin/ecaz dev sql --pg 18 --db postgres \
--socket-dir "${PGRX_HOME}" --raw \
--sql "select version()" \
--log-output reviews/task-{id}/001-example/artifacts/pg18-status.log
```
Expand All @@ -82,10 +82,11 @@ libpq options, and approval scope in one place.
External comparison extensions can use the same setup surface:

```sh
/Users/peter/.cargo/bin/ecaz dev install pgvector --pg 18
/Users/peter/.cargo/bin/ecaz dev install vectorscale \
${HOME}/.cargo/bin/ecaz dev install pgvector --pg 18 \
--repo "${PGVECTOR_REPO}"
${HOME}/.cargo/bin/ecaz dev install vectorscale \
--pg 18 \
--repo "$HOME/dev_bak/pgvectorscale/pgvectorscale" \
--repo "${PGVECTORSCALE_REPO}" \
--cargo-pgrx /tmp/pgvectorscale-cargo-pgrx-0.16.1/bin/cargo-pgrx
```

Expand Down
16 changes: 2 additions & 14 deletions crates/ecaz-cli/src/commands/dev/install.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ pub struct InstallEcazPgTestArgs {
#[derive(Args, Debug)]
pub struct InstallPgvectorArgs {
/// pgvector repository checkout.
#[arg(long, env = "PGVECTOR_REPO", default_value_os_t = default_pgvector_repo())]
#[arg(long, env = "PGVECTOR_REPO")]
repo: PathBuf,

/// PostgreSQL major version to install against.
Expand All @@ -69,11 +69,7 @@ pub struct InstallPgvectorArgs {
#[derive(Args, Debug)]
pub struct InstallVectorscaleArgs {
/// pgvectorscale extension crate checkout.
#[arg(
long,
env = "PGVECTORSCALE_REPO",
default_value_os_t = default_vectorscale_repo()
)]
#[arg(long, env = "PGVECTORSCALE_REPO")]
repo: PathBuf,

/// cargo-pgrx binary to use for the install.
Expand Down Expand Up @@ -284,14 +280,6 @@ fn home_dir() -> PathBuf {
.unwrap_or_else(|| PathBuf::from("/"))
}

fn default_pgvector_repo() -> PathBuf {
home_dir().join("dev_bak/pgvector")
}

fn default_vectorscale_repo() -> PathBuf {
home_dir().join("dev_bak/pgvectorscale/pgvectorscale")
}

fn default_cargo_pgrx_bin() -> PathBuf {
home_dir().join(".cargo/bin/cargo-pgrx")
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Status: active Phase 11.1 gate
Task: Task 30 Phase 11
Paper basis: `/home/peter/dev_bak/papers/2512.17264v1.pdf`
Paper basis: arXiv:2512.17264v1

## Source Basis

Expand Down
43 changes: 43 additions & 0 deletions plan/tasks/240-artifact-path-sanitization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Task 240: Artifact Path Sanitization

Status: **implementation complete; packet 001 review-open; history remediation
pending explicit decision** (2026-08-29). Priority: P0 privacy and evidence
hygiene. Origin: private Engineering Assurance gap audit.

## Why

The public default branch contains machine-specific workstation paths in authored
configuration and historical evidence. GitHub code search exposed only a subset; a
local tracked-file audit found 6,396 affected files. The values disclose local layout,
are non-portable, and can accidentally identify rights-unreviewed source locations.

## Goal

Remove the explicitly identified internal scratch locator from the current tree, make
external-source references durable, and prevent any newly added workstation path from
reaching GitHub without rewriting historical benchmark evidence blindly.

## Scope

1. Replace current authored internal-source locators with durable public identifiers or
explicit operator inputs.
2. Remove machine-specific default comparison-repository locations from `ecaz dev
install`; require `--repo` or the documented environment variable.
3. Add a diff-aware guard that detects Unix, root, and Windows workstation home paths,
reports only file/line/category, and never repeats the matched value in CI logs.
4. Run the guard on every pull request and main-branch push in a lightweight workflow.
5. Preserve a transparent distinction between current-tree redaction and historical Git
reachability. Do not rewrite history or silently change digest-bound evidence.

## Acceptance

1. The current tree contains no reference to the internal scratch locator identified by
the audit.
2. Synthetic tests prove added Unix, root, and Windows workstation paths fail while
removals and portable environment-based paths pass.
3. Failure output contains the file, line, and category but not the sensitive value.
4. The CLI comparison installers have no machine-specific repository default.
5. The lightweight GitHub workflow enforces the guard without invoking benchmark or
PostgreSQL lanes.
6. A review packet records the legacy-corpus count, redaction boundary, test evidence,
and the separate unresolved history-remediation decision.
4 changes: 2 additions & 2 deletions plan/tasks/87-candidate-batched-scoring-across-ams.md
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,7 @@ fix, not Task 87.
the calibration prototype already produces `Prepared…Query`
shapes that batch naturally.
- **pgvectorscale** is a read-only reference: clone at
`/Users/peter/dev_bak/pgvectorscale/`; key files
the `timescale/pgvectorscale` repository; key files
`access_method/scan.rs` (resort_buffer pattern) and
`access_method/graph/mod.rs` (streaming iteration).

Expand Down Expand Up @@ -426,7 +426,7 @@ fix, not Task 87.
- Task 86 packet 001 transferability matrix (per-AM block-
kernel fit ranking)
- pgvectorscale resort_buffer pattern:
`/Users/peter/dev_bak/pgvectorscale/pgvectorscale/src/access_method/scan.rs`
`pgvectorscale/src/access_method/scan.rs`
- FR-038 (benchmark provenance): every suite checked-in JSON
- ADR-075 (Task 65b stepping stone framing) — similar
staged-rollout pattern
Expand Down
6 changes: 3 additions & 3 deletions plan/tasks/88-streaming-ann-result-iteration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Priority: 2 (hybrid-search support; follows Task 87)

Task 86 investigation surfaced pgvectorscale's
`StreamingDiskANN` pattern (see
`/Users/peter/dev_bak/pgvectorscale/pgvectorscale/src/access_method/graph/mod.rs`
`timescale/pgvectorscale:pgvectorscale/src/access_method/graph/mod.rs`
`greedy_search_streaming_init` + `greedy_search_iterate`).
The pattern yields candidates lazily to the PostgreSQL
executor in approximate-score order, with a resort buffer +
Expand Down Expand Up @@ -200,7 +200,7 @@ the two.
the post-filter use case justification — don't expand
Task 88's scope.
- **pgvectorscale** is the reference implementation
(`/Users/peter/dev_bak/pgvectorscale/`). Key files:
(`timescale/pgvectorscale`). Key files:
- `pgvectorscale/src/access_method/scan.rs`
(`resort_buffer`, `StreamingStats`)
- `pgvectorscale/src/access_method/graph/mod.rs`
Expand Down Expand Up @@ -243,7 +243,7 @@ the two.
(surfaced the streaming-vs-batching distinction)
- Task 87 (predecessor): `plan/tasks/87-candidate-batched-scoring-across-ams.md`
- pgvectorscale `StreamingDiskANN` reference:
`/Users/peter/dev_bak/pgvectorscale/`
`timescale/pgvectorscale`
- pgvectorscale README streaming section: *"The
post-filtering implementation, while slower, is streaming
and correct…"*
Expand Down
2 changes: 1 addition & 1 deletion plan/tasks/89-turboquant-tqplus-cross-am-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ permission to skip the AM.
(includes reviewer feedback on the original closeout)
- Reverted Task 86 commits (TQ+ work, preserved in git
history): `e0ae9fe7d`, `c7e85e8ac`, `16f1e6104`, etc.
- pgvectorscale reference: `/Users/peter/dev_bak/pgvectorscale/`
- pgvectorscale reference: `timescale/pgvectorscale`

## Estimated size

Expand Down
2 changes: 2 additions & 0 deletions plan/tasks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,8 @@ These pre-lane task files are retained only for historical context under
238. `238-ec-distann-retry-snapshot-use-after-free.md` — **implementation and evidence merged; packet 001 outside closeout review-open** (updated 2026-08-27, P0 correctness): the equivalent `RegisteredSnapshotGuard` lifetime fix remains on main via `15f7fcf5f` / Task 167 PR #77. Checkpoint `7d4103885` restores the deterministic forced-retry regression and explicit loopback-only plaintext test conninfo; both PG18 callers pass on base `3c81319a3`. PR #91 merged the implementation and evidence as `bcc9b0bfa`; only the outside ACCEPT remains open. Evidence and request: `reviews/task-238/001-retry-snapshot-uaf/`.
239. `239-ec-distann-bounded-read-overfetch.md` — **complete — review-closed ACCEPT; HARNESS REGRESSION CORRECTED, EXACT-MAIN LAZY-10 SEMANTIC PATH RESTORED TO 10/10; no rerun; packet 004 10k/50k/100k NOT triggered; Task 229 / 230--233 semantic-surface blocker LIFTED** (updated 2026-08-26, P1 correctness/performance): packet 001's 12/10 result was a shared-session batch-size GUC leak in the benchmark harness — a variant at batch size 10 emitted no `SET`, so the "lazy-10" arm inherited the eager control's 0 — not production overfetch, and the bound was never widened. Packet 002 corrected the harness but its cross-SHA runner aborted pre-semantics on 40-versus-37 stage rows. Packet 003 removed that confounder with exact-main runner/extension plus only the harness fix; its sole C1--C5 run passed every preregistered gate — nine scenarios exactly once in both logs, seven core rows at `control_batch_size=0 candidate_batch_size=10` with exact identity and zero duplicates, `exactly_one_window` 6 remote + 4 local = 10 reads/bound 10, mixed and outage pass, routed DELETE+VACUUM pass, both recall arms 0.9990 over 200 queries / 2,000 trials with predictions byte-identical to packet 001 — and the stopped 1.2 GB fixture was removed after capture. `git diff 41392c011 def565270 -- src` is empty, so no production runtime behavior changed and the 10k/50k/100k closeout matrix does not apply. Determinism rests on the same 6/4/10 split and digests at three independent extension SHAs (task-191 `7883cfcf`, task-198 `2ff72b3e`, this run `4ab2aa9a9`), so no rerun was needed. Carried follow-ups for whichever task next touches the multinode semantic harness: mixed/outage failures emit no structured `pass=false` row; `owner_payload_plan_cache` retains the same conditional-`SET` leak shape; recall/latency child batch size stays inferred. Closeout verdict: `reviews/task-239/003-main-baseline-semantic-proof/feedback/2026-08-26-02-reviewer.md`; result decision: `reviews/task-239/003-main-baseline-semantic-proof/artifacts/live-run-decision.md`; review request: `reviews/task-239/003-main-baseline-semantic-proof/request.md`; live authorization: `reviews/task-239/003-main-baseline-semantic-proof/feedback/2026-08-26-01-reviewer.md`.

240. `240-artifact-path-sanitization.md` — **implementation complete; packet 001 review-open; history remediation pending explicit decision** (2026-08-29, P0 privacy/evidence hygiene): current-tree internal scratch locators are removed, comparison-repository inputs are explicit, and a non-echoing diff guard rejects newly introduced workstation paths; packet `reviews/task-240/001-current-tree-sanitization/` requests outside review, while historical Git remediation remains a separate explicit decision.

## Coordination rules

- Freeze binary datum layout before downstream work expands.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Review Request: A4 Quantizer Implementation Mismatch Audit

Basis: `main` working tree after review `204`, plus upstream reference at
`~/dev_bak/TurboQuantDB/`
`https://github.com/jyunming/TurboQuantDB`

## Why This Packet Exists

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,15 @@ Request:

## Additional Reference: prism-mcp TurboQuant Implementation

A third independent TurboQuant implementation exists in `~/dev_bak/prism-mcp/`. It is a pure
A third independent TurboQuant implementation exists in
`https://github.com/dcostenco/prism-mcp`. It is a pure
TypeScript port used for Prism MCP's "Quantized Agentic Memory" feature (v5.0). Comparing it
alongside TurboQuantDB strengthens several conclusions from review 204 and surfaces one new
finding.

### prism-mcp Architecture Summary

Source: `~/dev_bak/prism-mcp/src/utils/turboquant.ts` (~840 lines)
Source: `dcostenco/prism-mcp:src/utils/turboquant.ts` (~840 lines)

- **Rotation**: QR decomposition (Householder) of a random Gaussian matrix — NOT FWHT/SRHT
- **Dimension**: 768 (Gemini text-embedding-004), no power-of-2 constraint
Expand Down Expand Up @@ -150,7 +151,7 @@ quality lever.

## Paper Reference: Rotation Is QR, Not FWHT

Source: `~/dev_bak/turboquant-2504.19874.pdf` (arXiv:2504.19874, ICLR 2026)
Source: arXiv:2504.19874 (ICLR 2026)

The TurboQuant paper specifies QR decomposition, not FWHT:

Expand Down Expand Up @@ -216,7 +217,7 @@ contract within tqvector's SRHT-based architecture.

## TurboQuantDB Published Recall Benchmarks

Source: `~/dev_bak/TurboQuantDB/README.md`, section "Benchmarks"
Source: `jyunming/TurboQuantDB:README.md`, section "Benchmarks"

TurboQuantDB publishes recall numbers on **real embeddings** (50k × 1536, DBpedia OpenAI
embeddings, top_k=10):
Expand Down Expand Up @@ -245,7 +246,8 @@ is already good enough that the residual is small and the 1-bit correction is ef

## TurboQuantDB Deep-Dive: Code-Level Architecture

Full source read of `~/dev_bak/TurboQuantDB/src/quantizer/` and `src/linalg/hadamard.rs`.
Full source read of `jyunming/TurboQuantDB:src/quantizer/` and
`src/linalg/hadamard.rs`.

### README Describes QR, Code Uses SRHT

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Review Request: A4 Full vs Tiled FWHT — TurboQuantDB Comparison

Basis: `main` working tree, with reference to `~/dev_bak/TurboQuantDB/`
Basis: `main` working tree, with reference to
`https://github.com/jyunming/TurboQuantDB`

## Summary

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Updated:
Concrete changes:

1. added a reproducible scratch install wrapper for a local pgvector checkout
(`/home/peter/dev_bak/pgvector` by default)
(the checkout supplied through `PGVECTOR_REPO` or `--repo`)
2. the script installs pgvector into the same pg17 pgrx/Postgres tree already
used by the scratch cluster:
- `PG_CONFIG=/home/peter/.pgrx/17.9/pgrx-install/bin/pg_config`
Expand Down Expand Up @@ -195,4 +195,3 @@ The next useful follow-up is not more size work. It is one of:
isolated grouped index without the special verified harness
3. make the product call explicitly: latency-first compressed ANN versus
higher-recall full-vector ANN

Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ existing packets, task files, spec files, or source.

`DISTRIBUTEDANN: Efficient Scaling of a Single DISKANN Graph Across Thousands of
Computers`, arXiv:2509.06046v1, 7 Sep 2025. Local copy:
`~/dev_bak/papers/distributedann-2509.06046.pdf` (8 pages). Sections cited:
DistributedANN, arXiv:2509.06046 (8 pages). Sections cited:
§2.2 (index layout modifications, head index), §2.3 (near-data computation,
Algorithm 1), §2.4 (orchestration service, Algorithm 2), §3 (graph
construction), §4 (evaluation parameters), §4.1 (scaling), §4.2 (reliability).
Expand Down
Loading
Loading