inspector: drain queued frontend messages to the debugger thread before exit - #34219
inspector: drain queued frontend messages to the debugger thread before exit#34219hughescr wants to merge 4 commits into
Conversation
…re exit `bun test`'s TestReporter inspector events can be silently, permanently lost when the process exits before they've actually reached the debugger thread: inspector events emitted from the main thread are queued and a task is posted to the detached debugger thread to write them to the frontend socket. When the last tests in a file are synchronous, the main thread can queue their final start/end events, print the summary, and call exit() -- tearing down the debugger thread -- all without yielding, before that thread's posted task runs. This is a Rust port of oven-sh#29307 ("inspector: drain queued frontend messages before exit; fix TestReporter ID collision"), which was closed unmerged only because the Rust migration deleted the Zig files it touched (maintainers explicitly invited a refile against the new Rust code). This PR ports only that PR's drain-before-exit design -- the TestReporter ID-collision fix from the same original PR, and a separate WebSocket-backpressure correctness fix in debugger.ts, are independent bugs and land as their own PRs. Design (BunDebugger.cpp, Debugger.rs, VirtualMachine.rs): A new `totalPendingDebuggerMessages` atomic tracks messages handed off from the main thread to the debugger thread but not yet passed to onMessageFn. `Bun__debugger__drain()`, called from `VirtualMachine::global_exit` before the debugger connections are torn down, reads that counter only as an entry gate ("is any handoff still pending?"); if so it posts a sentinel task to the debugger thread and waits (capped at 250ms) for it to run. Concurrent tasks on that context are FIFO, so once the sentinel runs, every message queued before the call has had write() invoked for it. (It does not spin-wait for the counter to reach zero -- the sentinel's FIFO ordering is the guarantee, not the counter.) The signal is a std::shared_ptr<BinarySemaphore> rather than the original PR's heap-allocated-and-intentionally-leaked semaphore: Bun's leak-sanitizer builds (fba43af) now flag exactly that pattern, so this port avoids it -- whichever side (the waiting thread, or the posted task) finishes last frees it. `Bun__debugger__drain()` also takes a second, shorter bounded wait (150ms) giving the debugger thread's own (still running) event loop extra wall-clock time to flush anything already handed to the socket layer -- e.g. still sitting in a WebSocket's internal send buffer -- out onto the wire before the process tears down. This is useful independent of any JS-level backpressure tracking: uWS's own send buffer keeps draining on the debugger thread's event loop regardless of whether debugger.ts is pacing/retrying correctly, so the wait still helps even without the separate backpressure fix mentioned above. It is gated on a new `hasQueuedAnyDebuggerMessage` flag -- set once, never cleared, the first time any message is ever queued for the debugger thread -- rather than run unconditionally on every debugger-attached exit: if nothing was ever queued, there is provably nothing to flush, so the wait is skipped outright. This flag is deliberately distinct from `totalPendingDebuggerMessages`, which decrements back to zero as soon as write() has been called for a message even though its bytes can still be sitting unflushed below write() -- gating the flush wait on that counter (as an earlier revision of this change did) would give an already-buffered message zero grace whenever there was no pending handoff at the moment of the exit-time check. Both `totalPendingDebuggerMessages` and `hasQueuedAnyDebuggerMessage` are process-global static state (matching the existing `debuggerScriptExecutionContext`), not per-VM or per-connection; see the disclosure comment at their declarations. Both waits are capped so a wedged/starved debugger thread, or a consumer that has stopped reading entirely, cannot block process exit indefinitely. Scope: verified only for graceful main-process exits (CLI run/test/repl commands, `process.exit()`, bake production builds -- i.e. every `global_exit()` call site in this tree). Behavior on a process torn down by a signal or `abort()`, which do not run `global_exit()`, is unverified; no claim is made about those paths. Test (test/cli/inspect/test-reporter.test.ts): Adds a regression test where several NORMALLY-READING (not paused) consumers, connected to `bun test` subprocesses running fast synchronous tests, are spawned in parallel -- parallel spawns supply the CPU contention that starves the debugger thread and makes the race bite reliably; a lone run on an idle machine may pass even pre-fix. Asserts every subprocess delivers found/start/end events for every test, all "pass", with exit code 0. Synchronization is a handshake, not a fixed sleep: each subprocess's socket is only inspected after both `exited` and the socket's own FIN (`close`) have been observed, which on a Unix stream socket is ordered after all data the subprocess wrote -- guaranteeing the snapshot reflects everything sent, not whatever happened to arrive within an arbitrary wait window. (This addresses the review nit `robobun` raised against oven-sh#29307's original test.) Coverage scope: this test uses `--inspect-wait=unix:...`, which routes through debugger.ts's #connectOverSocket()/SocketFramer path, NOT the ws:/ws+unix: server path that uses webSocketWriter/bufferedWriter -- so it exercises the main->debugger-thread drain only and provides ZERO automated coverage of the separate WebSocket-backpressure fix mentioned above (that fix has, and needs, its own test). Verified fails-before/passes-after: 3/3 runs fail under unpatched system Bun 1.3.14 (`USE_SYSTEM_BUN=1`), with observed loss of started.size 1/5, 4/5, and 2/5 across the three runs; 3/3 runs pass against this branch's `bun bd` build. Claude-Session: https://claude.ai/code/session_013DexKQwDASzoRVDFS8Jtyn
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughAdds native debugger-message draining before virtual-machine exit, tracks pending debugger-thread delivery with capped waits, and adds regression coverage for complete TestReporter event delivery across concurrent inspector subprocesses. ChangesInspector message draining
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/jsc/bindings/BunDebugger.cpp`:
- Line 7: Update DebuggerDrainSignal to inherit from WTF::ThreadSafeRefCounted
and replace std::shared_ptr/std::make_shared usage with WTF Ref and adopt/create
helpers. Capture a Ref<DebuggerDrainSignal> in the cross-thread posted task,
then remove the now-unneeded <memory> dependency while preserving the existing
handoff behavior.
- Around line 415-419: Update the post-JSC::call logic in Bun__debugger__drain()
to check for a pending exception before decrementing
totalPendingDebuggerMessages. Only subtract messageCount when the entire batch
was successfully delivered; preserve the pending count on the exception path so
draining continues to wait for undelivered messages.
In `@test/cli/inspect/test-reporter.test.ts`:
- Line 2: Remove the setDefaultTimeout import and the corresponding
setDefaultTimeout(60_000) call from test-reporter.test.ts. Leave the test
definitions and runner-managed timeout behavior unchanged.
- Around line 443-452: In the results loop, move the exitCode assertion after
all reporter-event assertions. Keep the stderr, found, started, and ended checks
unchanged so captured output and delivery state are reported before validating
process success.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 42d54a99-76c6-4cd2-b908-0cbf25fd339f
📒 Files selected for processing (4)
src/jsc/Debugger.rssrc/jsc/VirtualMachine.rssrc/jsc/bindings/BunDebugger.cpptest/cli/inspect/test-reporter.test.ts
Temporary bridge until an official bun release ships the four inspector delivery fixes (oven-sh/bun#34219 #34220 #34221 #34222; build is upstream main 5df04abfe8 + those PRs, hosted at craigs-public-bucket/bun-trial/5df04abf/bun-linux-x64-baseline.zip). Stock 1.3.14 intermittently loses TestReporter events on 2-core hosted runners (oven-sh/bun#34218), which wedges Stryker dry runs and poisons shard reliability. Trial evidence on trial/custom-bun-inspector: 21/21 clean instrumented dry runs with the custom build vs a ~1-in-7 wedge rate on stock, and a full 13-shard baseline bootstrap green at 100.00. Scope: mutation machinery ONLY — - mutation-baseline-bootstrap.yml: new workflow_dispatch input bun_download_url whose DEFAULT is the pinned build (override for trials, empty string falls through to stock setup-bun version resolution per setup-bun's getInput(...) || undefined handling), wired to the shard job's setup-bun step; header note documents the pin and the revert condition. - ci.yml mutate job: same URL hardcoded on its setup-bun step. The mutate job and bootstrap compute the same bun---version-based toolchain fingerprint, so the pinned binary (reports 1.4.0) restores the bootstrap-seeded baseline; leaving mutate on stock latest would fingerprint-mismatch it. - test-linux/test-macos deliberately untouched: stock bun for production fidelity (running/ pins 1.3.14). REVERT when a bun release includes the fixes: set the bootstrap input default back to '' and delete the ci.yml bun-download-url line, then re-seed the baseline (new fingerprint). Claude-Session: https://claude.ai/code/session_013DexKQwDASzoRVDFS8Jtyn
…ng, scaled default timeout, exit-code-last assertions CodeRabbit review feedback on the exit-drain PR (34219-E/F/G/H): - BunDebugger.cpp: DebuggerDrainSignal now derives from WTF::ThreadSafeRefCounted<DebuggerDrainSignal> with a private constructor and static create() (mirrors SharedEnvStore.h in the same directory), replacing the std::shared_ptr the waiting thread and posted task both held a reference through. Drops the now-unused <memory> include. - BunDebugger.cpp: receiveMessagesOnDebuggerThread now wraps the onMessageFn call in a DECLARE_TOP_EXCEPTION_SCOPE (matching WebKitBackend.cpp's HostClient::onData) and only decrements totalPendingDebuggerMessages when the batch delivered without throwing. An exception partway through a batch leaves the whole batch's count pending, since there's no way to know how far onMessageFn got -- the cost is bounded to at most one extra capped drain wait at exit (Bun__debugger__drain's waits are capped at 250ms/150ms regardless), never a stall. A termination exception is left uncleared so it keeps propagating. - test-reporter.test.ts: setDefaultTimeout scales to 120s under ASAN/debug builds (was a flat 60s), since setDefaultTimeout overrides the CI runner's own --timeout and a flat value would undercut the runner's ASAN allowance. - test-reporter.test.ts: in the drain test's per-result assertions, exitCode is now checked last, after the found/started/ended event-count and status assertions, so a delivery-loss failure reports the more diagnostic event-based mismatch rather than being masked by the exit-code check.
…e assertion Reorder the mid-collection TestReporter test so behavioral assertions (the rawFoundIds uniqueness check) surface before the bare exit-code comparison, and gate the exit-code assertion with a stderr-empty check so a failing child's stderr gets printed via the failed expectation instead of being silently swallowed. This matches the exit-code-last ordering adopted on sibling PR oven-sh#34219.
* upstream/main: (57 commits) node:http/https/http2: raise Node v26.3.0 compat to ~94%, sync the upstream suites, and fix the Windows/macOS transport-layer teardown bugs they exposed (oven-sh#32488) expect: fix panic in toBeArrayOfSize/toHaveBeenCalledTimes with length > i32 max (oven-sh#32266) lexer: fix TOKEN_TO_STRING[TColon] showing " =" instead of ":" (oven-sh#34253) Bun.Terminal: write() returns bytes accepted, fire drain on POSIX (oven-sh#34289) test(serve-body-leak): give release-asan the same 60s per-test timeout as debug (oven-sh#34297) worker: mark the context terminating before the final concurrent-queue drain (oven-sh#34278) buffer: wrap negative ucs2 indexOf offset against raw byte length for Buffer needles (oven-sh#34273) fs.promises.watch: yield events with a null prototype (oven-sh#34279) child_process: latch stdin write EPIPE as 'error' + destroy, fail later writes with ERR_STREAM_DESTROYED (oven-sh#34268) Fix asString assertion when passing String objects as signals (oven-sh#34265) Buffer: carry size_t through toString/write so length 2^32 doesn't wrap to 0 (oven-sh#34274) test: use tempDir in log-test.test.ts instead of hardcoded /tmp path (oven-sh#34294) tty: track raw mode per handle instead of per process (oven-sh#33527) test: expect the bumped mimalloc SHA in process.versions Return freed memory to the OS on a background thread instead of the JS thread (oven-sh#34181) Move WTFTimer out of the shared timer heap to fix a cross-thread race (oven-sh#33131) test: update block-scoped enum lowering expectations to let (oven-sh#34287) Error.captureStackTrace: install .stack as non-enumerable (oven-sh#34259) js_parser: treat "async as T" / "async satisfies T" as a cast, not an arrow (oven-sh#34246) js_parser: accept `!`, `#name`, and `export @dec` in standard decorator grammar (oven-sh#34245) ...
* upstream/main: (422 commits) install: drop packages held only by optional-peer resolution slots from bun.lock (oven-sh#35681) Update mimalloc to the upstream dev3 (v3.4.3) sync (oven-sh#36431) compile(pe): ftruncate the Windows --compile output after writing (oven-sh#36430) Strong: back bun_jsc::Strong with StrongRootBlock; free AbortSignal.timeout at wrapper GC (oven-sh#35849) test(harness): replace toRun matcher with async bunRun + toSpawn (oven-sh#36424) test: measure memory via harness rss() instead of process.memoryUsage.rss() (oven-sh#36429) Deflake a few tests no-orphans(windows): allow CREATE_BREAKAWAY_FROM_JOB and set DIE_ON_UNHANDLED_EXCEPTION on the Job (oven-sh#36414) GarbageCollectionController: replace per-tick heap sampler with idle timer only (oven-sh#35356) exe_format(pe): write a valid OptionalHeader.CheckSum for --compile output (oven-sh#36383) FileSink: flush buffered bytes when process.exit() runs in the same tick as write() (oven-sh#36250) test(http): speed up and de-flake serve-async-stream-client-abort.test.ts (oven-sh#35919) test(20144): stop racing child startup against the 1s SIGKILL guard (oven-sh#34166) test(no-orphans): skip fast-exit perl daemon test on macOS (oven-sh#36413) fs: return negative BigIntStats *Ns for pre-epoch timestamps (oven-sh#36187) event_loop: make DeferredTaskQueue::run tolerate re-entrant map mutation (oven-sh#32703) dotenv: stop panicking on nested `${...}` inside `${VAR:-default}` (oven-sh#36199) fetch: make the idle timer an absolute deadline for the response header block (oven-sh#36145) bundler: don't panic on unterminated naming template placeholders (oven-sh#36325) Buffer#indexOf/lastIndexOf: rare-byte SIMD filter with a Two-Way O(n+m) fallback (oven-sh#36420) ... # Conflicts: # src/jsc/bindings/BunDebugger.cpp
|
Housekeeping for reviewers: I've just merged latest Since I have four related-but-independent PRs open (#34219, #34220, #34221, #34222), a quick map so nobody has to reconstruct it: no two of them touch the same source file. The only overlap anywhere is test coverage — this PR, #34220, and #34221 each extend |
|
Thanks for this, and for the clear write-up of the race in #34218. As with #34221 / #36522, CI does not run on fork branches here, so the drain was carried onto a repo branch as #36524 with your commit credited via a #36524 is this fix (pending-message counter, FIFO sentinel, capped wait, disconnect and exception accounting) with one change: the layer (b) 150 ms grace wait is dropped. |
Fixes #34218
What does this PR do?
Fixes a real (reproducible) loss of
bun test'sTestReporterinspectorevents: when a file's last tests are synchronous, the main thread can queue
their final
start/endevents, print the run summary, and fall straightthrough to
exit()-- tearing down the detached debugger thread -- beforethat thread's already-posted delivery task runs. The events were genuinely
produced (the tests really ran and passed) but never reach the frontend,
with no error and no way to detect the loss from the frontend side.
This is a Rust port of #29307 ("inspector: drain queued frontend messages
before exit; fix TestReporter ID collision"), which fixed the same bug
against the old Zig debugger implementation and was closed unmerged only
because the Rust migration deleted the files it touched -- a maintainer
explicitly invited a refile against the new Rust code. This PR ports only
the drain-before-exit fix from that PR. Two other things bundled into the
original PR are not included here and are being submitted as separate,
independent PRs instead: the
TestReporterID-collision fix from the sameoriginal PR, and a WebSocket-backpressure correctness bug in
debugger.tsfound while investigating this (a!!coercion that treatssendText()'s "still buffered, will still be delivered" return value assuccess, so it's never paced or retried) -- neither shares a root cause or
a call path with the exit-time drain fixed here.
Root cause:
Bun__ensureDebugger's connection hands inspector messagesfrom the main (inspected) thread to a separate, detached debugger thread via
sendMessageToDebuggerThread, which posts a task to that thread to actuallycall
onMessageFn(and, transitively, write the message to the frontendsocket). If the main thread reaches process exit before that posted task
runs, the task -- and the messages it was going to deliver -- never run at
all; nothing waited for it.
Fix: a new
totalPendingDebuggerMessagesatomic counts messages handedoff but not yet passed to
onMessageFn.Bun__debugger__drain(), calledfrom
VirtualMachine::global_exitimmediately before the debuggerconnections are torn down, checks that counter only as an entry gate ("is
any handoff still pending?"); if so, it posts a sentinel task to the same
debugger-thread context and waits (capped at 250ms) for the sentinel to run.
Tasks posted to that context run FIFO, so once the sentinel has run, every
message queued before
Bun__debugger__drain()was called has already hadwrite()invoked for it -- the sentinel's ordering is the guarantee, notthe counter reaching zero.
A second, shorter bounded wait (150ms) then gives the debugger thread's own
(still-running) event loop extra wall-clock time to flush anything already
handed to the socket layer -- e.g. still sitting in a WebSocket's own
internal send buffer -- out onto the wire before the process tears down.
This helps independent of the separate backpressure-tracking bug mentioned
above: uWS's send buffer keeps draining on the debugger thread's event loop
regardless of whether
debugger.tsis pacing/retrying writes correctly. Itis gated on a new
hasQueuedAnyDebuggerMessageflag (set once, nevercleared, the first time anything is ever queued for the debugger thread)
rather than run unconditionally on every debugger-attached exit: if nothing
was ever queued, there's provably nothing to flush, so the wait is skipped.
This flag is deliberately distinct from
totalPendingDebuggerMessages,which decrements back to zero as soon as
write()has been called for amessage even though that message's bytes can still be sitting unflushed
below
write()-- gating the flush wait on that counter instead (anearlier revision of this change did exactly that) would give an
already-buffered message zero grace whenever there happened to be no
pending handoff at the moment of the exit-time check.
Both counters are process-global static state (matching the file's existing
debuggerScriptExecutionContext), not per-VM or per-connection -- disclosedin a comment at their declarations.
Both waits are capped, so a wedged/starved debugger thread, or a frontend
that has stopped reading entirely, can never block process exit
indefinitely; on timeout this simply degrades to the pre-fix behavior for
whatever hadn't been delivered yet.
Scope: verified only for graceful main-process exits -- every
global_exit()call site in this tree is a normal CLI/runtime exit path(
run/test/replcommands,process.exit(), bake production builds).Behavior on a process torn down by a signal or
abort(), neither of whichruns
global_exit(), is unverified by this change and no claim is madeabout those paths.
How did you verify your code works?
Added a regression test in
test/cli/inspect/test-reporter.test.ts("flushes pending inspector messages to the frontend before process exit")
that spawns several
bun test --inspect-wait=unix:...subprocesses inparallel -- each running a handful of trivial synchronous tests -- with a
normally-reading (not paused) inspector client attached to each. Parallel
spawns supply the CPU contention needed to actually starve the debugger
thread and make the race bite reliably; a lone run on an idle machine can
pass even pre-fix. The test asserts every subprocess's client received
found,start, andendevents for every one of its tests, allpass,with exit code 0 -- zero trailing loss at any of the three reporting
stages.
Synchronization is an observable handshake, not a fixed sleep: each
subprocess's inspector socket is only inspected after both the subprocess's
exitedpromise and the socket's own FIN (close) have fired. On a Unixstream socket, FIN is ordered after all data the peer wrote, so this
guarantees the snapshot reflects everything the subprocess actually sent --
process exit alone would not, since the kernel can still be delivering
already-queued bytes after the process is gone. (This preemptively
addresses the same fixed-sleep review nit
robobunraised against#29307's original test.)
Coverage scope: this test uses
--inspect-wait=unix:..., which routesthrough
debugger.ts's#connectOverSocket()/SocketFramerpath, not thews:/ws+unix:server path that useswebSocketWriter/bufferedWriter.It therefore exercises only the main-thread-to-debugger-thread drain fixed
in this PR, and provides no coverage of the separate WebSocket-backpressure
bug mentioned above (which will ship with, and be verified by, its own PR
and test).
Fails before, passes after (per this repo's root
CLAUDE.md, line 106:"Verify your test fails with
USE_SYSTEM_BUN=1 bun test <file>and passeswith
bun bd test <file>. Your test is NOT VALID if it passes withUSE_SYSTEM_BUN=1."):Also ran the full
test/cli/inspect/sweep (bun bd test test/cli/inspect/)against this branch: 38 pass / 4 pre-existing todo / 0 fail across all 4
files in that directory.
Typechecked
src/js(cd src/js && bun x tsc --noEmit) and diffed theoutput against an unmodified-
mainbaseline: identical error set (only aworktree-path string differs in one pre-existing, unrelated error) --
expected, since this PR makes no changes to
src/js/internal/debugger.tsor any other TypeScript source.
https://claude.ai/code/session_013DexKQwDASzoRVDFS8Jtyn