bun test: report a test that completes after its timeout as timed out

[robobun]

Problem

• A test whose callback resumes from the event loop (after an await, or in a timer callback) and then blocks synchronously past its timeout is reported as (pass). Blocking with a busy loop, Bun.sleepSync, or Bun.spawnSync all reproduce it, for plain test() as well as test.concurrent():

  test("c1", async () => { await Bun.sleep(50); Bun.sleepSync(3000); }, 300);  // (pass) c1 [3050ms]

• Cause: the timeout timer cannot fire while the callback is blocking. When the callback finishes, its completion (bun_test_then / done(), src/runtime/test_runner/bun_test.rs) is queued without looking at the entry's deadline, and Execution::step (src/runtime/test_runner/Execution.rs) advances the sequence as a pass. The timer fires afterwards and finds the sequence already finished. The synchronous-return path in step_sequence_one already calls evaluate_timeout after the callback returns; the asynchronous completion path had no equivalent.
• The spawnSync flavour inside a test.concurrent group with a sibling still running used to fail correctly in 1.3.14 (the wait loop fired the runner's timeout callback at the earliest deadline in the group). Since spawnSync: drain piped stdio to EOF after the direct child exits #33832, TestRunner::get_active_timeout returns the latest deadline in the group when the caller is no longer on the stack (src/runtime/test_runner/jest.rs), so it no longer kills a sibling's child, and the verdict was left to the completion path above, which dropped it. spawnSync: don't run the bun:test timeout callback while the isolated loop is active #38883 is not the cause: its deferred callback still reports the verdict whenever the wait loop does hit a deadline.

Fix

• BunTest::on_callback_completed, called from the promise handler and from done() right before the completion is queued, runs Execution::handle_callback_completed: if the completion still belongs to the running entry, evaluate_timeout is applied against the current time. This is the same check the synchronous-return path makes, made at the equivalent moment for asynchronous callbacks.
• It runs when the completion arrives rather than in step, because the queued completion may only be processed after other callbacks have run; a test that finished in time must not be blamed for a sibling that blocks afterwards (found in review on an earlier revision, covered by the last fixture below).
• Resulting semantics: a callback is judged against its deadline at the moment it finishes, whatever blocked until then. In a test.concurrent group this includes a test held up by a sibling blocking the event loop; its printed duration already exceeds its timeout in that case, and on main such a test passed or failed depending on the order in which its resume and the timeout timer happened to fire after the block. spawnSync in a multi-test concurrent group still waits for the child (there is no way to tell which test the child belongs to); the test is reported as timed out when the callback completes.
• A test ended by an unhandled rejection (on_unhandled_rejection, src/runtime/test_runner/jest.rs) is aborted rather than completed and is already a failure, so that path is intentionally left alone. Today done(err) also reports through it, so a sequential test that overruns and then calls done(err) still prints the error with a plain (fail) rather than the timed-out line (unchanged from main; the concurrent variant does get the line). bun test: fail the test or hook whose done() received an error #39112 changes done(err) to report directly against its own entry, after which the on_callback_completed call in bun_test_done_callback covers it too, so this PR does not restructure that block.
• Precedence is unchanged: the timeout replaces whatever the body produced, including a thrown error or a test.failing pass. The synchronous-return path and the timer path already behave this way on main (test.failing bodies that overrun are reported as timed out on both); the asynchronous completion path was the only one that did not.
• ExecutionEntry::evaluate_timeout takes a callback_completed flag. Both completion paths (synchronous return and the new one) record FailBecauseTimeout / FailBecauseHookTimeout; the ...WithDoneCallback variants, whose message is "timed out ..., before its done callback was called. If a done callback was not intended, remove the last parameter", are now only produced by the timer path, where the callback really is still waiting. Previously a done test that blocked and then called done() synchronously printed that hint although done() had been called. Every other consumer of those variants (junit, GitHub annotations, summary counts) already treats them identically to the plain ones.
• Tests: test/js/bun/test/test-test.test.ts, describe a test that completes after its timeout has passed is reported as timed out, five fixtures run through bun test: sequential Bun.sleepSync after an await (the next test still runs); test.concurrent busy loop with a sibling pending; test.concurrent Bun.spawnSync with a sibling pending; done() called late both from a timer callback and synchronously; and a test.concurrent test that returns in time while its sibling blocks for 300ms before the runner processes the completion, which must pass. On main the first three fixtures report 2 pass, the fourth reports the timer test as (pass) and the synchronous one with the done-callback hint, and the fifth passes; the fifth fails on the earlier revision of this branch that checked the deadline in step. With the fix all five match their snapshots (5 consecutive runs on the debug build). The overrunning fixtures block for 300ms against a 100ms timeout, and the timer heap fires in deadline order, so none of the outcomes depend on scheduling.
• Also run on the debug build: test/js/bun/test/{bun_test,concurrent,concurrent_immediate,done-async,test-retry-repeats-basic,test-on-test-finished,failure-skip,jest-hooks,test-error-code-done-callback,test-failing}.test.ts, test/cli/test/{test-timeout-behavior,retry-flag,rerun-each}.test.ts, test/cli/test/bun-test.test.ts -t timeout, test/js/node/test_runner/node-test.test.ts; cargo fmt --all -- --check is clean.
• Open PRs bun test --todo: report a todo body that times out as todo instead of a failure #38153 and bun test: a matcher blocked on a promise that never settles fails with the test timeout #39089 also touch evaluate_timeout (different bugs); whichever lands second has a one-line conflict in its signature or call sites.

Background

• Execution entry: one test callback or one hook, with a timespec deadline set when it starts (on_entry_started). A sequence is a test plus its beforeEach/afterEach entries; a concurrent group holds the sequences that may run at the same time (one sequence for a plain test()).
• How a completion arrives: run_test_callback either gets a result synchronously (returns Some, handled in step_sequence_one) or registers then/catch handlers on the returned promise and a done callback. When the last of those fires, it pushes a RefDataValue naming the entry onto the file's result queue and schedules a task; BunTest::run later drains the queue into Execution::step, which checks that the entry is still the sequence's active one (get_current_and_valid_execution_sequence) and advances it. The new hook sits at the push and reuses that same validity check; step is unchanged.
• Timeout timer: one timer per file, armed for the earliest deadline of the running entries. When it fires, handle_timeout pushes a Start onto the same queue and step_group re-walks the running sequences, calling evaluate_timeout on each one still executing. It is the only other place a timeout verdict comes from, and it cannot run while JavaScript is blocking.
• spawnSync in bun test: the wait loop ticks an isolated event loop and asks the runner for a deadline to wake up at (get_active_timeout); when that deadline passes it kills the child and, since spawnSync: don't run the bun:test timeout callback while the isolated loop is active #38883, reports the timeout once the loop is torn down. With several concurrent tests running and no caller on the stack it uses the latest deadline so that it never kills a child belonging to a test that still has time.

Behaviour on 1.3.14, main, and this branch for the reported repros

Body runs after await Bun.sleep(50) in a test with a 300ms timeout; "sibling" is a second test.concurrent with a 5s timeout.

body	1.3.14	main	this branch
busy loop, sequential	pass	pass	fail, timed out
busy loop, concurrent alone	pass	pass	fail, timed out
busy loop, concurrent with sibling	pass	pass	fail, timed out; sibling passes
spawnSync, sequential	fail at 300ms, child killed	fail at 300ms, child killed	unchanged
spawnSync, concurrent alone	fail at 300ms, child killed	fail at 300ms, child killed	unchanged
spawnSync, concurrent with sibling	fail at 300ms, child not killed	pass	fail when the child exits, child not killed; sibling passes

Earlier revision

The first revision evaluated the deadline inside Execution::step, i.e. when the queued completion was processed. A review comment pointed out that a sibling blocking between a test's completion being queued and the queue being drained then got that test reported as timed out; reproduced and replaced by the completion-time hook above, with the fifth fixture covering it.

---

no test proof · iteration 1 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/js/bun/test/test-test.test.ts

[robobun]

Status: reproduced on main with the three blocking bodies from the report (busy loop, Bun.sleepSync, Bun.spawnSync) in test() and test.concurrent(), and compared against a 1.3.14 binary (table in the description). Fix and tests are in this PR; all review threads resolved (the done(err) message nit is deferred to #39112, see description). Rebased onto main at 4400766 (trivial conflict, see the comment below). CI on the rebased code, build 100814: 178 of 179 jobs passed, with no failing test files; the one remaining job, Windows x64 verify-baseline, was canceled before it ran because robobun's Azure VM creation for it timed out, and the same code was fully green in build 100047 before the rebase. The retrigger for this PR has already been used, so that job needs a maintainer retry if it is required. Ready for review.

[coderabbitai]

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

Limit details: You’ve used all 5 included reviews currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: edf4156d-2087-4533-884e-7bc2c3f09bbb

📥 Commits

Reviewing files that changed from the base of the PR and between ec7a24b and 4400766.

📒 Files selected for processing (3)

• src/runtime/test_runner/Execution.rs
• src/runtime/test_runner/bun_test.rs
• test/js/bun/test/test-test.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[robobun]

Pushed 61ab141 for the one review comment (the fixture helper now drains stdout and snapshots it). The CodeRabbit "Review failed" notice and the two failed Actions jobs on the first push (lol-html cargo test died in bun install on an HTTP 4xx from api.github.com; claude-find-issues errored before doing anything) are unrelated to the diff; the new push re-runs them.

[claude]

I re-reviewed after 61ab141 (which addressed the stdout-drain nit) and didn't find any bugs. The fix mirrors the existing synchronous-return path's evaluate_timeout call and the callback_completed flag correctly gates the done-callback hint at all three call sites. Because this sits in Execution::step — the core verdict path every bun test goes through — and interacts with retry/repeat, test.failing, and concurrent-group siblings, a maintainer look would still be worthwhile.

Checked: the new &mut *sequence_ptr borrow is dead before advance_sequence re-derives (matches the pattern used throughout this file); evaluate_timeout overwriting sequence.result on the async path now matches what the sync path already did for .failing/rejected promises; maybe_skip = true routes through advance_sequence identically to the sync path so afterEach handling is unchanged.

The comment-cop flag on bun_test.rs:1949 is the new 4-line doc comment on evaluate_timeout — it documents the callback_completed parameter, not a workaround, so it reads as a false positive to me.

Extended reasoning...

Overview

Two-file Rust change in the test runner plus new integration tests. Execution::step (the async-completion path) now calls evaluate_timeout on the just-completed entry before advancing the sequence, matching what step_sequence_one already does on the synchronous-return path. ExecutionEntry::evaluate_timeout gains a callback_completed: bool so the ...WithDoneCallback result variants are only produced when the timer fires while the callback is genuinely still waiting. Four new subprocess fixtures in test-test.test.ts cover sequential/concurrent × sleepSync/busy-loop/spawnSync and both done()-late shapes.

Security risks

None. No untrusted input parsing, no auth/crypto, no FFI surface change. The unsafe blocks added follow the file's established raw-pointer-reborrow pattern (NonNull into execution.sequences, dereferenced at point of use, dead before the next reborrow).

Level of scrutiny

High — Execution::step is on the hot path of every bun test run and decides pass/fail verdicts. The change is small (~15 Rust lines) and mirrors existing logic, but the interaction surface is wide: retry/repeat (a blocked test now correctly counts as failed and consumes a retry), test.failing (a failing test that both throws and exceeds its deadline now reports timeout rather than pass — this already happened on the sync path, so the async path is being made consistent), concurrent-group siblings, and hook vs. test entries. I traced each of these and they look correct, but a maintainer who owns the test runner should confirm the .failing-plus-timeout precedence is the intended one.

Other factors

My previous review's stdout-drain nit was addressed in 61ab141 and the snapshots now include stdout. The github-actions comment-cop fired on the new doc comment for evaluate_timeout; that comment is a 4-line parameter description, not a workaround justification, so I read it as a false positive — but it is technically unresolved. The PR description lists an extensive set of related test files run on the debug build and notes the one-line conflict with two other open PRs touching the same function.

[robobun]

The stdout-drain comment was addressed in 61ab141 (thread resolved). Since then: f9070c0 trims the evaluate_timeout doc comment the comment lint flagged, and the description now states the precedence question the second review raised: a timeout recorded at completion replaces a thrown error or a test.failing pass, which is what the synchronous-return and timer paths already do on main (checked all three shapes against the release binary and this branch). No code changes beyond the comment; CI is running on f9070c0.

[robobun]

CI summary for the first push (build 100002, e463433): 3 test files failed, all in test/cli/install and all on fetches from api.github.com returning 504/4xx (bun-add.test.ts shows the URL in its failure; GHSA-pfwx-36v6-832x.test.ts and migration/complex-workspace.test.ts fail while installing their GitHub dependencies). The same upstream errors took out the bun install step of the cargo miri test job on the latest commit. None of the failures is a timeout verdict, which is the only output this PR changes, and none touches the files it changes; they have been reported separately. The comment-lint thread re-surfaced here was already resolved by f9070c0 and the lint is green on that commit. Build 100009 is running on the current code.

[robobun]

^{Updated 4:12 PM PT - Aug 18th, 2026}

❌ @robobun, your commit 4400766 has 1 failures in Build #100814 (All Failures):

• Failed to create agent for 🪟 x64 - verify-baseline

---

🧪   To try this PR locally:

bunx bun-pr 39440

That installs a local version of the PR into your bun-39440 executable, so you can run:

bun-39440 --bun

[robobun]

Reworked after the review finding on Execution::step: 13fa482 moves the deadline check to the moment the completion arrives (BunTest::on_callback_completed in the promise and done() handlers), leaves step unchanged, and adds a fixture in which a test that returned in time is followed by a sibling blocking for 300ms before the runner drains the queue; that fixture failed on the previous revision and passes now. 2f04881 trims the doc comment the comment lint flagged. The description is updated to describe the current shape, including the one completion-like path intentionally left alone (a test ended by an unhandled rejection) and what the change means for siblings in a concurrent group. All four review threads are resolved; CI is running on 2f04881.

[robobun]

State of the PR: code is unchanged since 2f04881 (d75b4dc is an empty commit to get a fresh CI run). The last review thread (done(err) after an overrun on a sequential test keeps the plain (fail) line) is confirmed and intentionally left to #39112, which reworks that block; the reasoning is in the thread and in the description. Build 100037 failed only on GitHub returning 429/503 to the aarch64 build's vendor fetch and to the test/cli/install tests that download from GitHub; the Rust library built and uploaded on every platform and no failure involves the test runner. Locally, the whole test/js/bun/test directory passes on the debug build with this change (1156 pass, 1 pre-existing color-dependent snapshot failure that also fails on main in the same environment). Build 100047 is running; the CodeRabbit rate-limit notice above needs nothing from this side.

[claude]

I reviewed this revision and found no remaining issues — the earlier findings (the stdout drain, the completion-time-vs-process-time regression, and the sequential done(err) gap deferred to #39112) are all addressed or acknowledged. Because this changes test-verdict semantics in the runner's unsafe-Rust core and intentionally leaves one sibling path to a follow-up PR, a human look is still worthwhile.

What was reviewed:

• handle_callback_completed reuses get_current_and_valid_execution_sequence so stale/mismatched completions are ignored, matching step's guard.
• The callback_completed flag on evaluate_timeout: verified all three call sites pass the right value and that the ...WithDoneCallback variants are treated identically by every consumer (reporter, junit, debugger status mapping).
• The fifth fixture's ordering (queueMicrotask → sibling blocks before the queue drains) does exercise the arrival-vs-processing distinction the earlier revision got wrong.

Extended reasoning...

Overview

The PR adds a deadline check at the moment an async test/hook callback completes (promise then/catch and done()), so a test that blocks synchronously past its timeout after resuming from the event loop is reported as timed out instead of passing. It touches src/runtime/test_runner/Execution.rs (new handle_callback_completed, a callback_completed flag on evaluate_timeout, two call-site updates), src/runtime/test_runner/bun_test.rs (two on_callback_completed hooks and the wrapper), and adds five subprocess fixtures in test/js/bun/test/test-test.test.ts.

Security risks

None. No untrusted input, no auth/crypto/permissions surface. The change reads a timestamp and writes an enum on a runner-owned struct.

Level of scrutiny

High. This is the test runner's verdict machinery — heavy unsafe pointer aliasing over UnsafeCell, and a user-visible pass→fail semantic change. An earlier revision of this PR did introduce a real pass→fail regression (judging at queue-drain time instead of completion time), which I flagged and the author fixed; the new fixture covers it. That history alone argues for a maintainer sign-off on the final shape.

Other factors

• All five prior review threads (mine and the comment lint's) are resolved. The one intentionally-unfixed sibling — sequential done(err) past deadline still printing plain (fail) — is unchanged from main, still a failure, and documented as landing via #39112 which restructures that block.
• Test coverage is good: five fixtures cover sequential, concurrent-with-sibling (busy loop and spawnSync), done-callback (timer and synchronous), and the regression guard for a sibling blocking after a completed test. The author reports 5 consecutive clean debug runs plus the broader test-runner suite.
• CI failures on the last two builds are all test/cli/install fetches from api.github.com returning 429/503 — unrelated to the diff.
• The unsafe in the new function follows the file's existing pattern exactly (get_current_and_valid_execution_sequence returning NonNull into self.sequences, deref at point-of-use with a SAFETY comment), and handle_uncaught_exception immediately below is the same shape.

[robobun]

Rebased onto main (4400766). The only conflict was with #39570, which made RefDataValue Copy and removed the .clone() calls on the two lines where this PR inserts on_callback_completed; the resolution keeps the new calls and main's by-value forms, so the diff against main is otherwise identical to what was reviewed. The empty retrigger commit was dropped in the rebase. The first rebase target (85c8983) did not build because of the bun_css dependency main had lost at that point, which #39586 has since restored, so the branch is rebased onto that instead. On the rebased debug build the five fixtures fail 4/5 with main's runner sources swapped in (the fifth is the one that must pass) and pass 5/5 with the fix, and the related runner suites pass. Build 100814 is running. The CodeRabbit notices above are its own rate limit and need nothing here.

[claude]

I reviewed this PR across several iterations and the current revision addresses everything raised (stdout drain, the completion-time vs. processing-time regression, and the done(err) edge deferred to #39112). No bugs found on this pass. Because it changes user-visible timeout semantics in the test runner — including how concurrent siblings are attributed when one blocks the event loop — a human look at the semantic call would still be worthwhile.

What was reviewed:

• handle_callback_completed reuses get_current_and_valid_execution_sequence, so stale/mismatched completions are filtered the same way step filters them.
• The callback_completed flag on evaluate_timeout only suppresses the ...WithDoneCallback hint variants; junit/debugger/summary consumers already treat those identically to the plain timeout results.
• The fifth fixture guards the pass→fail regression the earlier revision introduced; the 300ms-vs-100ms margins keep the outcomes scheduling-independent.
• bun_test_catch (is_catch=true) still reaches on_callback_completed after on_uncaught_exception, so a rejected promise past deadline gets FailBecauseTimeout (overwriting the Fail/Pass set by handle_uncaught_exception), matching the stated precedence.

Extended reasoning...

Overview

The PR adds Execution::handle_callback_completed (called from the promise-resolution and done() handlers via BunTest::on_callback_completed) so that a test callback's deadline is checked at the moment the callback finishes, not only by the separate timeout timer. It threads a callback_completed: bool through ExecutionEntry::evaluate_timeout so the "before its done callback was called" hint is only produced by the timer path where done() really is still pending. Five subprocess fixtures in test/js/bun/test/test-test.test.ts cover sequential/concurrent × sleepSync/busy-loop/spawnSync, plus the done() variants and the sibling-blocks-after-completion guard.

Security risks

None. This is verdict computation inside the test runner; no untrusted input, no I/O surface changes, no auth/crypto.

Level of scrutiny

Medium-high. The test runner is a critical path for every user of bun test, and this changes pass/fail attribution: tests that previously (incorrectly) passed after synchronously blocking past their deadline will now fail, and in a test.concurrent group a test whose resumption is delayed past its deadline by a sibling's synchronous block will also be reported as timed out. The PR argues this is the honest interpretation (the printed duration already exceeds the timeout, and on main the outcome was scheduling-dependent), which is reasonable, but it is a semantic decision a maintainer should ratify. The Rust side is small and follows the file's existing NonNull/unsafe-deref patterns exactly.

Other factors

• All four prior review threads are resolved. The one remaining known gap (sequential done(err) past deadline still prints plain (fail)) is not a regression, still fails the test, and is documented as covered once #39112 restructures that block.
• CI is reported green on the re-run (build 100047); earlier failures were GitHub 429/503 on unrelated install tests.
• The description notes one-line signature conflicts with open PRs #38153 and #39089 that also touch evaluate_timeout — worth a maintainer being aware for landing order.
• The implementation is careful about the exact concern I raised on the earlier revision: the deadline is now sampled at completion arrival (Timespec::now_force_real_time() inside handle_callback_completed), not at queue-drain time, and the fifth fixture pins that.