valkey: skip JS error construction in on_close when the worker VM has stopped - #36837
valkey: skip JS error construction in on_close when the worker VM has stopped#36837robobun wants to merge 1 commit into
Conversation
worker.terminate() while a Worker-owned Bun.RedisClient has commands in
flight aborts assert-enabled builds: WebWorker::shutdown()'s socket-group
drain fires the valkey socket on_close, which builds a coded JS Error for
every in-flight/queued command via valkey_error_to_js -> Bun::createError.
With the TerminationException still pending and hasTerminationRequest()
already cleared for process.on('exit'), the first coded error in that
worker lazily initialises nodeErrorCache under a DeferTermination scope
and trips ASSERT(vm.hasTerminationRequest()) in
VMTraps::deferTerminationSlow (whole-process SIGABRT).
Guard ValkeyClient::on_close() and JSValkeyClient::on_valkey_close() on
script_execution_status(): when the VM is not Running, drop the pending
queues without building JS errors and release the socket keep-alive ref
without rejecting the connection promise or calling the onclose handler.
Same pattern as the Bun.listen/serve socket handler guards added in
#34414 / #36579.
|
Warning Review limit reached
Next review available in: 14 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
Comment |
There was a problem hiding this comment.
No issues found, but this touches worker-shutdown ref-counting on the valkey client and I'd like a human to confirm the socket keep-alive ref balance and that update_poll_ref() (still run by the _defer scopeguard on the new early return in on_valkey_close) is safe on a stopped VM.
What was reviewed:
- Guard matches the established
script_execution_status() != Runningpattern fromsocket_body.rs(#36579) andipc.rs. - Stopped-VM queue drain mirrors
shutdown(None)'s finalizing branch —JSPromiseStronghandles release via Drop, no leak. on_valkey_close()still runsScopedRef::adoptbefore the early return, soconnect()'s socket keep-alive ref is released on both fail and reconnect paths.- Test is hermetic (inline RESP3 responder, no real redis), debug-only, and follows the neighboring tests' shape exactly.
Extended reasoning...
Overview
Fixes a whole-process SIGABRT when worker.terminate() lands while a Worker-owned Bun.RedisClient has commands in flight. The worker shutdown's socket-group drain reaches ValkeyClient::on_close(), which was building coded JS errors (via valkey_error_to_js → nodeErrorCache lazy init → DeferTermination) with a TerminationException pending and hasTerminationRequest() already cleared, tripping ASSERT(vm.hasTerminationRequest()) in VMTraps::deferTerminationSlow.
Two guards are added: one in ValkeyClient::on_close() (valkey.rs) that drops both queues without building JS errors when script_execution_status() != Running, then routes to on_valkey_close(); and one in JSValkeyClient::on_valkey_close() (js_valkey.rs) that early-returns after adopting the socket keep-alive ref, skipping the connection-promise reject and onclose callback. A debug-only regression test with an inline RESP3 responder is added to worker-terminate-lifetime.test.ts.
Security risks
None. This is a worker-shutdown crash fix; no user input parsing, auth, or trust-boundary changes.
Level of scrutiny
Medium-high. The change is small (~30 LOC) and follows the exact pattern already used for Bun.listen/Bun.connect socket handlers in socket_body.rs, but it sits at the intersection of worker VM termination, JSC's sticky TerminationException, and intrusive ref-counting on JSValkeyClient. The socket keep-alive ref taken by connect() must be released exactly once by either on_valkey_close() or on_valkey_reconnect(); the new stopped-VM branch routes through on_valkey_close() which adopts it before the early return, so the balance holds. The queue-drain-without-JS mirrors shutdown(None)'s finalizing branch (JSPromiseStrong releases via Drop). The one thing I did not exhaustively verify is that update_poll_ref() — which still runs via the pre-existing _defer scopeguard on the new early return — is fully safe on a stopped VM (it reads JSMap::size() and may call JsRef::upgrade()); this is pre-existing behavior since the outer SocketHandler::on_close _defer already calls it, but worth a maintainer glance.
Other factors
- PR description includes a full lldb backtrace and explicit before/after verification (aborts iteration 1 unpatched → 20/20 clean patched).
- Test alternates
autoReconnect: true/falseto exercise both the fail and reconnect branches ofon_close, is hermetic, and matches the conventions of the four neighboringterminate()-during-X regression tests in the same file. test.skipIf(!isDebug)is appropriate since release WebKit compiles the assert out, matching the sibling tests.- No prior human reviews on the PR.
|
On the two points raised: Socket keep-alive ref balance.
|
What
worker.terminate()while a Worker-ownedBun.RedisClienthas commands in flight aborts assert-enabled builds:5/5 whole-process SIGABRT within the first 10 iterations on release-asan main @ 074656d. Release builds compile the assert out and silently continue building JS errors / rejecting promises on a terminated VM.
Repro
No redis server required; inline RESP3 responder:
Cause
WebWorker::shutdown()clearshasTerminationRequest()(soprocess.on('exit')can run) and then drains every socket group viaRareData::close_all_socket_groups. That reaches the valkey socket'son_close, which for every in-flight / queued command callsvalkey_error_to_js->ErrorCode::fmt->Bun__createErrorWithCode->Bun::createError->globalObject->nodeErrorCache(). With the TerminationException still pending buthasTerminationRequest()now false, theLazyPropertyinit'sDeferTerminationscope tripsASSERT(vm.hasTerminationRequest())inVMTraps::deferTerminationSlow.lldb backtrace of the aborting worker thread:
Same socket-group-drain entry that #34414 / #36579 guarded for
Bun.listen/Bun.connectsocket handlers; valkey'son_closewas unguarded.Fix
Guard
ValkeyClient::on_close()onscript_execution_status(): when the VM is notRunning, drop the in-flight and offline queues without building JS errors (theirJSPromiseStronghandles release via Drop), then route throughon_valkey_close()which now also short-circuits the JS work (connection-promise reject,onclosecallback) after adoptingconnect()'s socket keep-alive ref. Mirrors theBun.listenhandler guards added in #36579.Verification
New test in
test/js/web/workers/worker-terminate-lifetime.test.tsruns the inline-responder repro (alternatingautoReconnect: false/ default so both the fail and reconnect branches ofon_closeare exercised) for 20 iterations. Debug-only (test.skipIf(!isDebug)) since release WebKit compiles the assert out.ASSERTION FAILED: vm.hasTerminationRequest().valkey-gc.test.tsand the rest ofworker-terminate-lifetime.test.tspass unchanged.no test proof · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/js/web/workers/worker-terminate-lifetime.test.ts