test-tonic: fail fast with stderr when cargo dies instead of a 150s hook timeout - #33970
Conversation
|
Updated 12:04 PM PT - Jul 15th, 2026
⏳ @robobun, your commit 9b00b06 is still building in |
WalkthroughChangesTonic test startup
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/js/third_party/grpc-js/test-tonic.test.ts`:
- Around line 39-43: Condense the explanatory comment immediately above the
cargo/toolchain probe to no more than three lines, preserving the key details
about Bun.which finding the rustup shim, missing default toolchains, and probing
with the same environment and outside-repository working directory.
- Around line 146-147: Update the afterAll teardown hook to await completion of
the child server shutdown and temporary-directory cleanup before returning. Use
the server teardown mechanism around server?.kill() and make the hook
asynchronous while preserving safe handling when no server exists.
- Around line 121-126: Update the readiness parsing in the server-startup helper
around the “Listening on” check to buffer incoming stdout chunks until a
complete newline-terminated readiness line is available. Only then extract and
trim the address, preserving the existing killServer return behavior and
avoiding undefined or partial addresses from split chunks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 97b1314a-56d8-479d-a4f2-ed3f141d7478
📒 Files selected for processing (1)
test/js/third_party/grpc-js/test-tonic.test.ts
There was a problem hiding this comment.
LGTM — test-only hardening of the tonic suite's cargo gate and startup readiness loop.
Checked that cargoWorks short-circuits on !cargoBin before spawning, and that the cargoBin! assertion is guarded by describe.skipIf(!cargoWorks ...).
Verified the fixture prints via println!("Listening on {}", addr) so the newline-terminated parse in the new read loop is sound.
Confirmed stderr.text() is started before the stdout read loop, so a piped stderr can't fill and wedge the child on the happy path even though the promise is only awaited on failure.
Extended reasoning...
Overview
Single-file, test-only change to test/js/third_party/grpc-js/test-tonic.test.ts. It replaces the Bun.which("cargo") presence check with a cargo --version probe (same stripped env, outside-repo cwd) so the suite skips when the rustup shim has no default toolchain; rewrites the stdout readiness loop to accumulate chunks, wait for a newline-terminated Listening on <addr> line, and throw with captured stderr + exit code if the child dies first; pipes and concurrently drains stderr; and null-guards/awaits server?.kill() in afterAll. RUSTUP_TOOLCHAIN is now forwarded and the env object is deduplicated between the probe and cargo run.
Security risks
None. No src/ changes, no new external inputs. The only new spawn is cargo --version with an explicit env allowlist and ignored stdio.
Level of scrutiny
Low. This is CI test infrastructure for a third-party integration test that already conditionally skipped when cargo was absent; the PR just tightens the skip predicate and turns a 150s hook-timeout into either a fast skip or a diagnosable throw. Worst case is a false skip on an agent where cargo works — strictly better than the current 150s red on every PR sharded to that lane.
Other factors
All three CodeRabbit review comments (comment length, buffer-to-newline, await teardown) were addressed in ca3add8 and confirmed resolved. I cross-checked fixtures/tonic-server/src/main.rs:33 — it uses println!, so the \n-terminated parse is correct. The stderrPromise is unawaited on the happy path but is actively draining the pipe, and --quiet keeps its volume bounded. No CODEOWNERS entry covers this path. The bug hunting system found nothing.
|
Updated in 9b00b06 (rebased onto main; diff unchanged) after the discussion with @dylan-conway: the The underlying break on CI on 9b00b06 (build 73331) finished:
Diff is test-only and ready to merge. |
There was a problem hiding this comment.
LGTM on the code — test-only hardening of startServer() that turns the 150s hook timeout into a fast, diagnosable failure. The one inline nit is about the PR title/description going stale after df96134 (it now fails fast rather than skips on an unusable toolchain); worth updating before squash-merge so the commit subject matches behavior.
Extended reasoning...
Overview
Single-file, test-only change to test/js/third_party/grpc-js/test-tonic.test.ts. The startServer() helper is rewritten to: forward RUSTUP_TOOLCHAIN; pipe and concurrently drain stderr via server.stderr.text(); accumulate stdout chunks and only return once the full newline-terminated Listening on <addr> line is present; and, if stdout closes without that line, await [stderrPromise, server.exited] and throw with the captured stderr and exit code. afterAll is now async and null-guards server?.kill(). No src/ changes.
Security risks
None. This is CI test scaffolding for a third-party gRPC interop test. No auth, crypto, or user-facing surface is touched; the only new env read (RUSTUP_TOOLCHAIN) is forwarded verbatim to a child cargo process, same as the existing CARGO_HOME/RUSTUP_HOME forwarding.
Level of scrutiny
Low. Test-infrastructure change that strictly improves failure diagnostics. The rewrite directly follows the repo's own testing guidance in CLAUDE.md: buffer raw stdout chunks to the protocol's framing before asserting, drain pipes concurrently so a chatty compile can't deadlock the child, and wire every failure path to throw rather than await a never-settled promise. I traced the new loop: done is checked after the marker scan so a final chunk that completes the line is still parsed; on the success path the un-awaited stderrPromise keeps draining in the background until the server is killed in afterAll, which is the intended behavior; on the error path server.exited is awaited before killServer() re-awaits it, which is a harmless no-op.
Other factors
All three CodeRabbit threads (comment length, wait-for-full-line, await teardown) were addressed in ca3add8 and confirmed resolved. CI on ca3add8 was green on the previously-failing darwin-aarch64-15.1 lane. df96134 only removes the cargoWorks probe added in 29d54ce, reverting the gate to the original skipIf(!cargoBin || !releases[release]) — a strict reduction relative to the CI-verified revision. The one open item is the inline nit: after df96134 the misconfigured-toolchain case fails fast rather than skips, so the PR title and first Fix/Verification bullets are stale. That's a metadata fix, not a code defect, and the inline comment spells out the suggested retitle.
…t on cargo errors
The darwin-aarch64-15.1 agent was reconfigured on Jul 9 to run as a user
with /opt/rust/bin in PATH but no RUSTUP_HOME, so Bun.which('cargo')
finds the rustup proxy while 'cargo run' in the throwaway tmpDir (which
has no rust-toolchain.toml and thus needs a rustup default) exits
immediately with:
error: rustup could not choose a version of cargo to run, because
one wasn't specified explicitly, and no default is configured.
startServer() then broke out of its stdout read loop on the 'done'
branch and awaited a never-settled promise until the 150s hook timeout,
after which afterAll threw on 'server.kill' with server still
undefined.
Probe 'cargo --version' with the same stripped env and an outside-the-
repo cwd and feed that into describe.skipIf, rewrite the read loop to
accumulate stdout and throw with the captured stderr when it closes
without a 'Listening on' line, drain stderr concurrently so a chatty
compile cannot wedge the pipe, and null-guard the afterAll kill.
Parse the address only after the terminating newline so a chunk boundary inside 'Listening on <addr>' cannot yield a partial address, make the afterAll hook await killServer(), and condense the cargo probe comment to three lines.
A cargo shim with no usable toolchain should fail loudly (now with the captured rustup stderr and exit code in seconds) rather than skip quietly, so a misconfigured agent is visible in CI. The .profile on darwin-test-arm64-1 has been fixed out of band so cargo works there again.
df96134 to
9b00b06
Compare
There was a problem hiding this comment.
LGTM — my earlier note about the stale title/description has been addressed, and the code is unchanged since then.
What was reviewed:
- stdout buffering waits for the newline-terminated
Listening on <addr>line (verifiedprintln!intonic-server/src/main.rs:33emits the trailing\n). - stdout-closed path awaits
[stderrPromise, server.exited]then throws with the captured stderr — no more never-settled promise. stderr: "pipe"is drained concurrently via.text();afterAllnull-guards and awaitskill().
Extended reasoning...
Overview
Test-only change to test/js/third_party/grpc-js/test-tonic.test.ts hardening startServer() so that when the spawned cargo run exits without printing the readiness line, beforeAll throws immediately with the captured stderr and exit code instead of hanging on a never-settled promise until the 150s hook timeout. Also: forwards RUSTUP_TOOLCHAIN, switches stderr from inherit to a concurrently-drained pipe, buffers stdout to a newline-terminated Listening on line before parsing the address, and null-guards/awaits server?.kill() in afterAll. The skipIf(!cargoBin || !releases[release]) gate is unchanged. No src/ changes.
Security risks
None. This is CI test harness plumbing for a third-party gRPC integration test; no user-facing code, no auth/crypto/permissions, no new external inputs beyond forwarding an existing env var to a child process.
Level of scrutiny
Low. Single test file, ~30 lines of diff, mechanical fix for a diagnosed hang. The success path is behaviorally identical (returns {address, kill} once the server prints its address); only the failure path changes from hang → diagnostic throw. I confirmed the fixture at tonic-server/src/main.rs:33 uses println!("Listening on {}", addr), so the trailing newline the parser now requires is always present.
Other factors
All three CodeRabbit findings (buffer-to-newline, await teardown, comment length) were addressed in follow-up commits and marked resolved. My prior review's only concern was that the PR title/description were stale after the cargoWorks probe was reverted; the author has since retitled the PR and rewritten the Fix/Verification sections to match the current behavior (fail-fast, not skip). The bug-hunting system found nothing. No outstanding reviewer comments remain.
|
The agent-user change on |
…onment (#37721) `test/js/third_party/grpc-js/test-tonic.test.ts` fails on every run of the `darwin 14 aarch64 - test-bun` lane since the tart agents came online (#37633), e.g. main builds [92687](https://buildkite.com/bun/bun/builds/92687) and [92739](https://buildkite.com/bun/bun/builds/92739): ``` error: tonic server exited (1) before reporting an address: error: rustup could not choose a version of cargo to run, because one wasn't specified explicitly, and no default is configured. ``` ### Cause That lane is served by the tart guests (`darwin-arm64-{challah,ciabatta,focaccia,sourdough}-tart-15-*`). `scripts/bootstrap.sh` installs rustup into `/opt/rust` (`RUSTUP_HOME=CARGO_HOME=/opt/rust`) and exports those two variables from the login profile. `scripts/darwin-ci/guest/job.sh` is run as a plain `/bin/bash ~/job.sh`, so the profile is never sourced; it puts `/opt/rust/bin` on `PATH` by hand but not the two variables. `Bun.which("cargo")` therefore finds the rustup proxy, and the proxy looks for a toolchain in `~/.rustup`, which does not exist in the guest. Checked inside a running guest on `darwin-arm64-focaccia`: `/opt/rust/settings.toml` has `default_toolchain = "stable-aarch64-apple-darwin"`, `cargo --version` with job.sh's environment fails with the message above, and the same command with `RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/rust` prints `cargo 1.97.1`. The bare agents are unaffected because `scripts/agent.mjs` runs jobs with `sh -elc`, which sources the profile; that is why the `darwin 26 aarch64` lane passes. ### Fix Export `RUSTUP_HOME` and `CARGO_HOME` in `job.sh` next to the `PATH` line that already hardcodes `/opt/rust/bin`. The test itself is left alone: it fails loudly on purpose (#33970), and that is what surfaced this. ### Rollout The hosts run the copy of `job.sh` installed in `/usr/local/share/darwin-ci` at provision time (`command.ts` pushes it into the guest per job), so this PR's own darwin 14 lane will still show the failure. The four tart hosts need the updated file copied into place (no agent restart needed; the next job picks it up), or a re-run of `main.ts provision`. cc @alii <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · build/CI scripts only; test-proof not applicable <!-- robobun:evidence:end -->
test/js/third_party/grpc-js/test-tonic.test.tswent red on thedarwin-aarch64-15.1lane in build 71849:Cause
startServer()read stdout chunk-by-chunk; when the process exits before printingListening on, the reader returns{done: true}, the loopbreaks, andawait promisewaits on a promise that is never settled. The 150s is the hook timeout, not a diagnosable error, andafterAllthen throws onserver.kill()withserverstill undefined.The trigger on that agent was an infra change: the host agent on
darwin-test-arm64-1was reconfigured on Jul 9 to run as a user with/opt/rust/bininPATHbut noRUSTUP_HOME/CARGO_HOMEin its profile, so the rustup shim resolved but had no toolchain. That has been fixed on the box (the agent user now has the same.profileexports the previous user had).Fix
This PR keeps the existing
skipIf(!cargoBin || ...)unchanged (no new skipping) and only hardensstartServer():Listening on <addr>line is present, so a chunk boundary mid-line cannot yield a partial address.tonic server exited (<code>) before reporting an address:followed by the captured stderr and exit code, instead of awaiting an unresolved promise until the hook timeout.RUSTUP_TOOLCHAINalongsideRUSTUP_HOME/CARGO_HOMEfor agents that pin via env.server?.kill()inafterAll.Verification
With
RUSTUP_HOMEpointed at an empty dir (reproduces the original CI state):beforeAllhangs to the hook timeout, thenafterAllthrowsundefined is not an object (evaluating 'server.kill')beforeAllthrows in ~14s withtonic server exited (1) before reporting an address:followed by therustup could not choose a version of cargo to run...messageWith a working toolchain the tonic server compiles,
Listening on 127.0.0.1:<port>is parsed, and the flow-control test runs as before.Test-only change; no
src/diff.no test proof · iteration 4 · Platform-specific test-only change; deferring to CI.