Migrate remaining workspaces and finalize CI to Bun-native (Fixes #2847)

.github/workflows/ci.yml

@@ -811,27 +811,34 @@ jobs:
       - name: 'Run test orchestrator on a single workspace'
         # Runs the orchestrator against one workspace. --skip-pretest skips
         # pretest hooks (like the agents API-surface guard) to keep the smoke
-        # fast. The workspace's test phase still runs (vitest run), but
-        # test-utils has a tiny test suite. --skip-scripts skips the root
-        # script harness tests. The only failure modes that matter here are
-        # bugs in the orchestrator itself (discovery, parsing, routing).
+        # fast. The workspace's test phase still runs, but test-utils has a
+        # tiny test suite. --skip-scripts skips the root script harness tests.
+        # The only failure modes that matter here are bugs in the orchestrator
+        # itself (discovery, parsing, routing).
         run: |-
           bun scripts/test.ts --workspace test-utils --skip-scripts
 
       - name: 'Run script harness tests for the orchestrator'
         run: |-
-          bunx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/test-orchestrator.test.ts
+          bun test --preload ./test-setup/augment-bun-vi.ts --preload ./scripts/tests/test-setup.ts scripts/tests/test-orchestrator.test.ts
 
   #
-  # Bun-native test compatibility (issue #2475)
+  # Bun-native manifest discovery gate (issues #2475, #2847)
   #
-  # Runs representative tests from workspaces with supported Bun-native
-  # compatibility. Vitest-only workspaces remain covered by the primary test
-  # jobs and are intentionally excluded from this smoke gate.
+  # Resolves every root declared in scripts/bun-test-manifest.ts without
+  # executing anything: globs are expanded, and every selected file, preload,
+  # tsconfig override and global-setup module must exist. That is what proves
+  # no test file was dropped.
+  #
+  # It deliberately does NOT run the suite. `test_shard` already executes every
+  # root exactly once by invoking each workspace's own `test` script, so a
+  # second full run would double the CI bill for no extra signal. The one-owner
+  # invariant is enforced by scripts/tests/bun-manifest-root-ownership.bun.test.ts,
+  # which fails if a root gains a second executor or loses its only one.
   bun_native_test_parity:
     name: 'Bun Native Test Compatibility'
     runs-on: 'ubuntu-latest'
-    timeout-minutes: 30
+    timeout-minutes: 10
     needs:
       - 'skip_check'
     if: ${{ needs.skip_check.outputs.should_skip != 'true' }}
@@ -860,9 +867,9 @@ jobs:
           bun install
           git checkout -- bun.lock
 
-      - name: 'Run Bun-native tests (manifest-based)'
+      - name: 'Resolve every Bun-native root (no execution)'
         run: |-
-          bun scripts/run_bun_tests.ts --timeout 30000
+          bun scripts/run_bun_tests.ts --dry-run
 
   #
   # Test: Node (Linux only — issue #2876)
@@ -1023,8 +1030,9 @@ jobs:
           # so local runs keep the signal; the env var is only set in CI.
           LLXPRT_COVERAGE: ${{ (matrix.shard == 'cli' || matrix.shard == 'core') && matrix.os == 'ubuntu-latest' && 'true' || 'false' }}
         # The orchestrator expands --shard to the shard's workspaces (or runs
-        # npm run test:scripts for the scripts shard). The tools, mcp, and storage
-        # workspaces run native Bun; all other packages use their configured runner.
+        # npm run test:scripts for the scripts shard). Each workspace's tests
+        # run under whichever runner its own `test` script selects — Bun-native
+        # for the migrated workspaces, Vitest for the rest (issue #2578).
         run: 'bun scripts/test.ts --shard "${{ matrix.shard }}"'
 
       - name: 'Report harness toolchain versions'
@@ -1057,10 +1065,9 @@ jobs:
         run: 'sleep 2'
 
       - name: 'Publish Test Report (for non-forks)'
-        # The scripts shard runs the root script harness, which has no
-        # per-workspace junit.xml output (scripts/tests/vitest.config.ts does
-        # not configure the junit reporter). Skip it to avoid a "no test
-        # report files found" error.
+        # The scripts shard runs the root script harness, which produces no
+        # per-workspace junit.xml. Skip it to avoid a "no test report files
+        # found" error.
         if: |-
           ${{ always() && matrix.shard != 'scripts' && (github.event.pull_request.head.repo.full_name == github.repository) }}
         uses: 'dorny/test-reporter@a43b3a5f7366b97d083190328d2c652e1a8b6aa2' # ratchet:dorny/test-reporter@v3

CONTRIBUTING.md

@@ -187,21 +187,29 @@ Type checking uses `tsc --noEmit`:
 bun run typecheck
 ```
 
-#### Bun-backed Test Orchestration
+#### The canonical test command
 
-The project also provides a Bun-backed root test entry point that preserves
-the same coverage and setup guarantees as `npm run test` (issue #2463):
+One command runs the complete suite:
 
 ```bash
 bun run test:bun
 ```
 
 This script (`scripts/test.ts`) orchestrates testing across all workspace
-packages using Bun as the runtime. Each workspace's tests still run under
-Vitest — not Bun's native test runner — so all Vitest-specific APIs
-(`vi.stubEnv`, `vi.unstubAllEnvs`, `vi.mocked`, `vi.setSystemTime`,
-`it.runIf`, etc.) and per-package `vitest.config.ts` configuration remain
-fully available.
+packages using Bun as the runtime, then runs the script harness. Migrated
+workspaces execute under **Bun's native test runner** via
+`scripts/run_bun_tests.ts` (one isolated process per test file); the
+workspaces still finishing their migration run under Vitest, with the
+Vitest-compatibility shim (`test-setup/augment-bun-vi.ts`) supplying the
+`vi.*` and `it.*` helpers Bun lacks.
+
+Two roots are excluded from that run because they call a real provider and
+consume quota — run them explicitly when you have credentials:
+
+```bash
+npm run test:integration:sandbox:none
+npm run test:all_evals
+```
 
 The script explicitly runs each workspace's `pretest` lifecycle hook before
 its test phase (npm does this automatically; Bun does not), so the agents

bun-junit-KAaFo3/0.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuites name="bun test" tests="1" assertions="0" failures="1" skipped="0" time="0.156603">
+  <testsuite name="save_memory.eval.ts" file="save_memory.eval.ts" tests="1" assertions="0" failures="1" skipped="0" time="0" hostname="Andrews-MacBook-Pro.local">
+    <testsuite name="save_memory" file="save_memory.eval.ts" line="14" tests="1" assertions="0" failures="1" skipped="0" time="0.009" hostname="Andrews-MacBook-Pro.local">
+      <testcase name="should be able to save to memory" classname="save_memory" time="0.009433" file="save_memory.eval.ts" line="115" assertions="0">
+        <failure type="AssertionError" />
+      </testcase>
+    </testsuite>
+  </testsuite>
+</testsuites>

dev-docs/bun.md

@@ -496,13 +496,14 @@ This script is a Bun-backed orchestrator that mirrors `npm run test
    (currently only `packages/agents`). This preserves the agents
    API-surface guard and any future pretest hooks.
 
-3. **Runs each workspace's `test` script** (typically `vitest run`) in the
-   workspace directory with `node_modules/.bin` on `PATH`. Tests still run
-   under **Vitest**, not Bun's native test runner, so all Vitest APIs and
-   per-package `vitest.config.ts` configuration remain fully available.
+3. **Runs each workspace's `test` script** in the workspace directory with
+   `node_modules/.bin` on `PATH`. Migrated workspaces point that script at
+   `scripts/run_bun_tests.ts` (Bun's native runner, one isolated process per
+   file); the workspaces still finishing their migration run under Vitest.
 
 4. **Runs the script harness tests** (`scripts/tests/`) after workspace
-   tests, matching the root `test:scripts` script.
+   tests, matching the root `test:scripts` script. These run natively under
+   Bun via the `scripts-tests` and `scripts-tests-slow` manifest roots.
 
 ### CLI flags
 
@@ -516,22 +517,34 @@ This script is a Bun-backed orchestrator that mirrors `npm run test
 The `--workspace` flag matches by directory name (`core`), relative path
 (`packages/core`), or package name (`@vybestack/llxprt-code-core`).
 
-### Relationship to `npm run test`
+### The canonical command
 
-Both paths run the same workspace Vitest suites and preserve the same
-pretest guards. The npm path (`npm run test`) remains the primary CI
-verification path; `test:bun` is the supported Bun-backed alternative. The
-npm path is not removed until the Bun-backed path is proven equivalent
-across all CI matrix legs.
+One command runs the complete suite:
+
+```bash
+bun run test:bun
+```
+
+It orchestrates every workspace plus the script harness, honouring pretest
+hooks. `npm run test` remains available and runs the same per-workspace
+`test` scripts, so the two agree by construction.
+
+Two roots are deliberately **not** part of that run because they call a real
+provider and consume quota. Request them by name when you have credentials:
+
+```bash
+npm run test:integration:sandbox:none   # bun scripts/run_bun_tests.ts --root integration-tests
+npm run test:all_evals                  # bun scripts/run_bun_tests.ts --root evals
+```
 
 ## Native Bun Test Runner Migration (issue #2475)
 
 ### Overview
 
-In addition to the Bun-backed Vitest orchestration (`test:bun`), each workspace
-is being incrementally migrated to run tests natively under Bun's test runner
-(`bun test`). Native Bun tests run faster (no Vitest overhead) and provide
-better integration with Bun's module system.
+Bun's own test runner (`bun test`) is the primary runner for every workspace
+except `agents` and `cli`, which are still on Vitest and tracked by #2578.
+Native Bun tests run faster (no Vitest overhead) and integrate better with
+Bun's module system.
 
 ### Workspace `test:bun` scripts
 
@@ -619,24 +632,50 @@ bun scripts/run_bun_tests.ts --workspace a2a-server
 npm run test:bun --workspace @vybestack/llxprt-code-a2a-server
 ```
 
-`run_bun_tests.ts` does not support `--exclude` — every file run must be
-explicitly listed in the manifest (`scripts/bun-test-manifest.ts`). Files
-that are not Bun-compatible are simply absent from the manifest rather than
-excluded at invocation time.
-
-Only `packages/a2a-server`, `packages/cli`, `packages/providers`,
-`packages/telemetry`, and `packages/test-utils` define a package-level
-`test:bun` script; each passes its exact manifest workspace name.
-The native `core` and `test-setup` entries are run through the root runner
-instead: `bun scripts/run_bun_tests.ts --workspace core` and
-`bun scripts/run_bun_tests.ts --workspace test-setup`. The root package's own
-`test:bun` command remains the separate Bun-backed Vitest orchestrator
-(`scripts/test.ts`), not the native runner. Use `bun scripts/run_bun_tests.ts`
-for all native manifest entries.
+### Test roots (`scripts/bun-test-manifest.ts`)
+
+Every file the native runner executes belongs to a **root** declared in
+`scripts/bun-test-manifest.ts`. A root selects its files in one of two ways:
+
+- **`include` / `exclude` globs** — used by fully migrated roots. This is the
+  Bun-native equivalent of a Vitest config's `include`, and it is what makes
+  "no test file can be silently dropped" mechanically true: a newly added
+  test file runs without any manifest edit.
+- **`files`** — an explicit list, used while a workspace is only partly
+  migrated and naming alone cannot tell a Bun-ready file from one still owned
+  by Vitest.
+
+A root may also declare:
+
+| Field          | Purpose                                                                        |
+| -------------- | ------------------------------------------------------------------------------ |
+| `preload`      | One or more Bun `--preload` scripts (the equivalent of Vitest `setupFiles`)    |
+| `tsconfig`     | A test-only `--tsconfig-override`, e.g. to stub the editor-injected `vscode`   |
+| `timeout`      | Per-test timeout, mirroring Vitest `testTimeout`                               |
+| `retries`      | Per-file retry budget, mirroring Vitest `retry`                                |
+| `globalSetup`  | `setup()` / `teardown()` run once in the runner process around the whole root  |
+| `credentialed` | Marks a root that calls a real provider; excluded unless requested by `--root` |
+
+`--root <name>` is an alias of `--workspace <name>`.
+
+An unfiltered `bun scripts/run_bun_tests.ts` runs every non-credentialed
+root — the complete offline suite. The `evals` and `integration-tests` roots
+are credentialed and must be named explicitly.
 
 ### CI parity
 
-The `bun_native_test_parity` CI job runs a representative sample of tests
-from each migrated workspace under Bun's native test runner to verify
-parity with Vitest results. The full Vitest suite remains the primary
-verification path; native Bun runs are additive parity checks.
+Bun's runner is no longer additive: it is the primary verification path for
+every migrated workspace, and `test_shard` executes it by invoking each
+workspace's own `test` script.
+
+Every root therefore runs **exactly once**. `test_shard` covers each workspace
+root; the scripts shard covers the roots that belong to no workspace, listed in
+`SCRIPTS_SHARD_ROOTS` in `scripts/test.ts`. A root with a second executor, or
+none at all, fails
+`scripts/tests/bun-manifest-root-ownership.bun.test.ts`.
+
+The `bun_native_test_parity` job does **not** execute tests. It resolves the
+manifest (`--dry-run`): globs expand and every selected file, preload, tsconfig
+override and global-setup module must exist. That is what proves no test file
+was dropped — re-running the whole suite a second time would double the CI bill
+for no extra signal.