highway: add a scalar feature so standalone cargo test binaries link; run native cargo test in CI

[robobun]

Repro

On a configured checkout (bun run build --configure-only), on main:

$ cargo test --locked -p bun_clap
ld.lld: error: undefined symbol: highway_index_of_char
>>> referenced by lib.rs:232 (src/highway/lib.rs:232)
>>>   libbun_core-*.rlib(...):(bun_core::string::immutable::index_of_char_usize)
error: could not compile `bun_clap` (lib test)

bun run rust:miri -p bun_clap passes (9 tests), so the crate's entry in MIRI_CRATES did not catch this.

Cause

A standalone test binary links the crate's Rust dependencies and nothing else; the highway C++ kernels behind bun_core::strings exist only in the full bun link. Since #37052, StreamingClap::normal splits --name=value with strings::index_of_char_usize, and the streaming tests reach normal, so the reference is live and lld reports it (lld only reports references from sections that survive --gc-sections, which is why this is a different problem from the dead-reference one #37575 handles on Windows). Miri kept passing because Miri never links and #37078 made bun_highway take scalar paths under cfg(miri).

Fix

That cfg(miri) switch is the right place to solve the native case too, so this keys it to a cargo feature as well:

• bun_highway gains a scalar feature; every miri gate in src/highway/lib.rs becomes any(miri, feature = "scalar"). With it on, no search wrapper references a kernel. cargo tree -p bun_bin -i bun_highway -e features shows only default, i.e. the bun build itself is unaffected (the flag is a constant-false cfg! in scalar_only, as before).
• bun_clap enables it from [dev-dependencies] (the one-line Cargo.lock change is that edge). Cargo only activates a crate's dev-dependencies when building that crate's own tests, so nothing else sees the feature. The test_err helper, which hand-rolled a substring search for exactly this reason (strings: route all byte search through highway, deny the paths around it #37052), now uses strings::contains, so the clap tests also exercise the memmem side of the gating natively.

An earlier revision of this PR instead added a per-crate #[no_mangle] stand-in for the kernel (the shape of src/parsers/native_test_shims.rs). Review of that revision pointed out that the central switch already exists, and that shims are hand-copied, name-matched reimplementations that every affected crate repeats; with the feature, a crate needs one manifest line and runs the same scalar code the Miri lane runs. The remaining in-tree shims (src/parsers/native_test_shims.rs, which does not currently link natively for other reasons; the highway half of #37592; the linear_fifo.rs byte-search lint exception) can move over separately.

CI coverage

No lane ran a native cargo test for any crate, which is how #37052 could break this: test/internal/rust-windows-sys-link.test.ts and friends skip on the test-only lanes (no configured workspace), clippy does not build tests, Miri does not link. So:

• scripts/rust-test.ts (bun run rust:test) runs cargo test --locked -p <crate> for the crates whose tests are expected to link natively. One invocation per crate on purpose: features unify within an invocation, so a combined -p a -p b run would turn scalar on for every crate and hide a missing dev-dependency. The list is the Miri set minus bun_ast (mimalloc symbols, tracked separately), bun_ptr (ptr: make cargo test -p bun_ptr link natively and run the Miri crate set natively in CI #37592) and bun_collections, whose pool tests turn out to race against thread-local pool teardown when libtest runs them in parallel (2 of 30 native runs fail on unmodified main; handed off separately, and it should be added back once fixed). The other 11 were stable over 15 consecutive runs here.
• .github/workflows/miri.yml runs it as a step before Miri: that job already has the configured tree and the clang/lld the generated .cargo/config.toml points at, and bun_core's is the only build script in these crates' graphs. src/highway/** and the new scripts are added to its path filter.
• The configure prerequisite logic that rust-miri.ts had inline moves to scripts/rust-workspace.ts so both scripts share it; rust-miri.ts is otherwise unchanged.
• test/internal/rust-native-cargo-test.test.ts runs the same script under bun bd test. It needs cargo and a configured checkout like its siblings, and is skipped on Windows, where every bun_core dependent's test binary fails to link for the dead-reference reason until build: make cargo test link on Windows hosts via /FORCE:UNRESOLVED in the generated .cargo/config.toml #37575 lands. Without the src/ changes it fails (on main proper with the link error above; with this PR's Cargo.lock and main's manifests, --locked rejects the lockfile first), with them it passes.

Verification

• cargo test --locked -p bun_clap: links, 9 passed. bun run rust:test: all 11 crates pass; bun run rust:test -p bun_ptr reports the failure correctly.
• bun run rust:miri (full set, through the refactored script): all 14 crates pass with the same counts as before.
• cargo fmt --all --check; cargo clippy -p bun_highway with and without --features scalar, and for bun_clap/bun_core: clean. bun test test/internal/source-lints/: 75 pass. bun bd rebuilt the debug binary with the feature off.
• bun bd test test/internal/rust-native-cargo-test.test.ts passes; fails with the src/ changes stashed.

---

no test proof · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/internal/rust-native-cargo-test.test.ts

[coderabbitai]

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: bd3400b1-a7bb-4036-871b-9e9d17d3c9e5

📥 Commits

Reviewing files that changed from the base of the PR and between 626034f and 610e6d7.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (10)

• .github/workflows/miri.yml
• package.json
• scripts/rust-miri.ts
• scripts/rust-test.ts
• scripts/rust-workspace.ts
• src/clap/Cargo.toml
• src/clap/streaming.rs
• src/highway/Cargo.toml
• src/highway/lib.rs
• test/internal/rust-native-cargo-test.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[robobun]

^{Updated 1:17 AM PT - Aug 12th, 2026}

✅ @robobun, your commit aa57c6420517abff10e16044f35eee3bc8d61b2b passed in Build #92996! 🎉

---

🧪   To try this PR locally:

bunx bun-pr 37599

That installs a local version of the PR into your bun-37599 executable, so you can run:

bun-37599 --bun

[robobun]

Status: ready for review. CI is green on aa57c64 (all Buildkite lanes, clippy, format, source lints; the Miri job now also runs the new native cargo test step, which passed in 39s before Miri itself). Review threads are all resolved.

Reproduced on main at 97e21e5 with cargo test --locked -p bun_clap on a configured linux-x64 checkout: ld.lld: error: undefined symbol: highway_index_of_char, referenced from bun_core::string::immutable::index_of_char_usize via StreamingClap::normal. With this branch the same command links and passes (9 tests), bun run rust:test passes for all 11 listed crates, and the full bun run rust:miri set still passes.

Gate test: test/internal/rust-native-cargo-test.test.ts runs scripts/rust-test.ts; it fails without the src/ changes and passes with them.

Related: #37592 (bun_ptr, same bug class; can switch to the feature once this lands), #37575 (Windows side). The bun_collections pool-test race found while building the crate list was handed off separately.

[claude]

LGTM — test-only shim following the existing src/parsers/native_test_shims.rs pattern; nothing here compiles into the shipping binary.

What was reviewed

• Shim signature/return-convention checked against src/highway/lib.rs (found_at: haystack_len = not found) and the C++ decl in highway_strings.cpp — matches.
• #[cfg(test)] gating on the module means the #[no_mangle] symbol only exists in the crate's test binary, so no collision with the real kernel in the full link.
• long_params addition: after stripping --, both new tokens exceed the 16-byte SCALAR_CUTOFF, so the shim is exercised on both found and not-found paths; expected Arg results line up.
• New .test.ts mirrors rust-windows-sys-link.test.ts (same skipIf gating, pipe draining, env spread).

Extended reasoning...

Overview

Fixes cargo test -p bun_clap failing to link on highway_index_of_char by adding a #[cfg(test)]-gated scalar shim (src/clap/native_test_shims.rs), extending the existing long_params unit test with a ≥16-byte flag so the shim is actually called, and adding a test/internal/ guard that runs the real cargo link. No production code paths change — StreamingClap::normal is untouched.

Security risks

None. The new symbol is compiled only into the crate's own test binary (module is behind #[cfg(test)]), the unsafe block reconstitutes a slice from a pointer/length pair the sole caller (bun_highway::index_of_char) derives from a live &[u8], and the SAFETY comment states this. The TS test spawns cargo locally with no network access.

Level of scrutiny

Low. This is test infrastructure following two in-tree precedents byte-for-byte: the shim body is identical to src/parsers/native_test_shims.rs:17-24, and the guard test is a near-copy of test/internal/rust-windows-sys-link.test.ts. Nothing user-facing or runtime-linked is touched.

Other factors

Verified the shim's contract against src/highway/lib.rs — scalar_only gates on len < SCALAR_CUTOFF (16), and found_at treats result == haystack_len as not-found, which the shim's .unwrap_or(haystack_len) honors. The new test tokens (ee-longer-than-sixteen-bytes, 28 bytes after prefix strip; and the =2 form, 30 bytes) both clear the cutoff, so the PR's claim that the shim is executed rather than merely linked holds. The 600s timeout on the TS test is unusual but explicitly justified (cold cargo build of bun_core) and matches the sibling test's rationale; the skipIf correctly gates on cargo availability and a configured workspace. The PR description documents the sibling issues (#37592 for bun_ptr, bun_ast tracked separately), so scope is deliberately bounded to bun_clap.

[robobun]

Heads-up for sequencing: #37592 adds scripts/rust-test.ts (run by the Miri workflow), which runs the Miri crate set as plain cargo test binaries and lists bun_clap in NATIVE_LINK_PENDING, re-linking it to check that it still fails. Whichever of the two lands second needs a one-line follow-up: if this PR lands after #37592, drop bun_clap from that list (the pending check fails CI until it is removed, and the crate is then covered by the workflow step); if it lands first, #37592 will drop it before merging.

[claude]

I reviewed this PR and didn't find any bugs. Because it introduces a new workspace-wide pattern (the bun_highway/scalar feature enabled via dev-dependencies), adds a new CI step to the Miri workflow, and has a stated sequencing dependency with #37592, a human look would still be worthwhile.

What was reviewed:

• src/highway/lib.rs: every cfg(miri) → cfg(any(miri, feature = "scalar")) site is a mechanical 1:1 rewrite; the scalar_only() predicate folds to a constant so DCE removes the FFI references, and the memmem wrappers use #[cfg] blocks — no kernel is reachable with the feature on.
• streaming.rs test_err: strings::contains is semantically equivalent to the hand-rolled search it replaces (quoted is always ≥2 bytes so index_of's empty-needle behaviour doesn't apply).
• rust-workspace.ts extraction is byte-for-byte the logic that was in rust-miri.ts; rust-miri.ts behaviour is unchanged.
• rust-test.ts runs one cargo test --locked -p per crate to avoid feature unification, collects failures, and exits nonzero — checked that a failing crate is reported and doesn't short-circuit the loop.

Extended reasoning...

Overview

Eleven files: a scalar cargo feature on bun_highway that reuses the existing Miri scalar-fallback paths so standalone cargo test binaries link without the highway C++ kernels; bun_clap enables it via [dev-dependencies]; the naive substring search in test_err is replaced with strings::contains (test-only). Tooling side: scripts/rust-workspace.ts extracts the configure-prerequisite check from rust-miri.ts, new scripts/rust-test.ts runs cargo test --locked -p <crate> per crate, .github/workflows/miri.yml gains a bun run rust:test step and path-filter entries, package.json gains rust:test, and test/internal/rust-native-cargo-test.test.ts runs the same script under bun bd test. Cargo.lock gains one edge (bun_clap → bun_highway).

Security risks

None. No runtime code path in the shipped binary changes: the scalar feature is off in bun_bin's dependency graph (only reachable from bun_clap's dev-dependencies), and cfg!(feature = "scalar") compiles to a constant false there. The streaming.rs change is inside #[cfg(test)]. The scripts spawn cargo/bun/ninja with fixed arguments in the repo directory.

Level of scrutiny

Medium. The src/ edits are mechanical (fifteen identical cfg rewrites plus a test helper simplification) and provably inert in the production build. The tooling is internal dev/CI infrastructure. What raises this above "rubber-stamp" is that it establishes a pattern the rest of the workspace is expected to adopt (per-crate [dev-dependencies] bun_highway = { features = ["scalar"] }), adds a step to a GitHub workflow that runs on every PR touching the listed crates, and the author flagged a merge-order interaction with #37592 that a maintainer should be aware of when landing.

Other factors

The PR description documents that an earlier revision (per-crate #[no_mangle] shims) was reviewed and this centralized approach was the suggested alternative, so the design has had prior input. The comment-cop bot flagged the scalar_only doc comment length; the author trimmed it to four lines pointing at Cargo.toml and those threads are resolved. The new test follows harness conventions (bunEnv, bunExe(), concurrent pipe drains, skipIf on missing prerequisites, stderr asserted before exit code). The rust-miri.ts refactor is behaviour-preserving by inspection. No CODEOWNERS cover the touched paths.