redis: gate the socket keep-alive adopt and defer destruction to finalize - #34874
redis: gate the socket keep-alive adopt and defer destruction to finalize#34874robobun wants to merge 7 commits into
Conversation
…lize on_valkey_close/on_valkey_reconnect adopted connect()'s socket ref unconditionally. Under connection-timeout + reconnect churn the close/fail teardown can be re-entered from on_connection_timeout such that the adopt spends a ref that is not outstanding, stealing the JS wrapper's +1; the ScopedRef drops at the end of on_connection_timeout then bring the count to 0 and free the box while the wrapper is live, and GC finalize -> stop_timers reads the freed allocation. Two changes, layered: - Track outstanding socket refs in a Cell<u32> (incremented at each socket_ref.forget() in connect()) and have on_valkey_close / on_valkey_reconnect adopt via take_socket_ref(), which returns None when the counter is 0. Same pattern RefCountedTimer applied to the timer refs in #34760; a counter rather than a bool because a stale reconnect can leave more than one us_socket_t in flight for the same client. - Guard the RefCounted destructor: if the count reaches 0 before the JS wrapper has been finalized, restore the stolen +1 and return without freeing, so finalize() frees normally instead of touching freed memory. Assertion builds panic here so any remaining over-release path is caught.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughChangesThe Valkey client now counts and conditionally consumes socket keep-alive references during connection, reconnect, and close handling, with deinitialization validation. A concurrent regression test covers malformed RESP replies, client closure, garbage collection, and clean subprocess completion. Valkey socket lifecycle
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
|
Updated 4:05 AM PT - Jul 30th, 2026
❌ @robobun, your commit efea2cc has some failures in 🧪 To try this PR locally: bunx bun-pr 34874That installs a local version of the PR into your bun-34874 --bun |
|
Found 2 issues this PR may fix:
🤖 Generated with Claude Code |
The clients array kept every wrapper reachable past both Bun.gc(true) calls, so the "drive GC so finalize() runs" comment described something that could not happen. Move the .connected probe into the loop and let c fall out of scope at round end, matching the sibling tests, and reword the header to be honest that the interleaving is timing-dependent.
|
The diff is ready for review. The over-release is timing-dependent (fuzzer reproduces 10/10 on the reporting hardware; 0/20 on the CI container on both the parent and this branch), so the mechanical fail-before check cannot observe it. The new test exercises the auto-reconnect close path and would surface the UAF under ASAN if the timing fires, but the intended proof is re-running the fuzzer's public repro and exact-fault plan against this branch. The fix is two-layered: |
There was a problem hiding this comment.
Thanks — b909587 addresses the earlier feedback (wrappers now fall out of scope so finalize() is reachable, and the header is honest that the interleaving is timing-dependent). I didn't find a bug in the Rust change, but this is intrusive-refcount memory-safety code — the third fix in this exact path in as many PRs — and the destructor guard's release-build recovery (restore the stolen +1 rather than free) is a design choice worth a maintainer's eye, so leaving this for a human.
What was reviewed:
take_socket_ref()mirrorsRefCountedTimer::take_fire_ref; bothsocket_ref.forget()sites inconnect()increment the counter, anddeinitasserts it back to 0.- The
ref_()insidedestructorat count 0: in debug buildsdebug_assert!(false)fires first (andDebugData::deinithas already zeroedmagic, soref_()'sassert_validwould panic anyway); in release theCell<u32>bump 0→1 is fine andfinalize()later re-entersdestructorwithfinalized=true. - Per the gate evidence the new test passes on the parent's debug+ASAN build, which the revised comment now states plainly — it's defensive coverage plus the destructor assertion, not a CI repro.
Extended reasoning...
Overview
Follow-up to #34760 in src/runtime/valkey_jsc/js_valkey.rs: (1) a socket_refs: Cell<u32> counter gates the previously-unconditional ScopedRef::adopt in on_valkey_close/on_valkey_reconnect via take_socket_ref(), (2) RefCounted::destructor now checks !finalized && this_value.is_not_empty() and, if so, restores a ref instead of freeing (debug builds panic). One new test in test/js/valkey/valkey-gc.test.ts exercises the auto-reconnect close → connection-timeout interleaving.
Prior review
I flagged that the first version of the test held every wrapper in a clients array read after both Bun.gc(true) calls, so finalize() could never run and the finalize claim in the comments was wrong. b909587 dropped the array, moved the .connected probe into the loop, and rewrote the header to say the race is timing-dependent and the test relies on ASAN plus the destructor assertion. That resolves the structural issue and the misleading comment.
What I checked this pass
- Counter balance: incremented at both
socket_ref.forget()sites (thetls_ctx_failedearly-close branch and the success tail ofconnect()), decremented only intake_socket_ref(), asserted 0 indeinit.on_valkey_close/on_valkey_reconnectare the only former unconditional-adopt sites;clone_without_connectingand bothcreate*initialize the counter to 0. - Destructor guard: traced
RefCount::deref_with_context(src/ptr/ref_count.rs:315-342). In debug builds it callsdebug.deinit()(zeroingmagic) beforedestructor, so anyref_()from inside would tripassert_valid— but the guard's owndebug_assert!(false)fires first, which is the intent. In release builds the#[cfg(debug_assertions)]gates strip both, soref_()just bumps theCell<u32>0→1;finalize()later adopts that +1, setsfinalized, and the second entry intodestructorproceeds todeinit.assert_no_refs()indeinitsees 0. No leak on the recovery path.
Security risks
None user-facing; this is lifetime bookkeeping with no external-input parsing.
Level of scrutiny
High. REVIEW.md calls native memory safety the most-blocked category, and specifically "never add a ref just to silence ASAN — find the actual imbalance." Here the actual imbalance is being fixed (the socket-ref gate), and the destructor guard is a layered safety net that debug-panics so any other over-release stays visible under ASAN — but silently recovering in release builds is a policy call a maintainer should sign off on.
Other factors
The PR's own gate evidence shows "ASAN without fix: all passed" for the new test, and the one release-build failure on the parent is an unrelated pre-existing test. So the new test is defensive coverage (exercises the path, would catch a future over-release via the debug assertion) rather than a repro that fails on the parent — the revised header comment now says exactly that, and the PR names the fuzzer environment as the intended re-verification. Given the refcount churn across #34714 → #34760 → this PR, deferring to a human.
There was a problem hiding this comment.
Thanks for addressing the earlier test-structure comment — the clients array is gone and the header now states the timing-dependence honestly. No further issues found, but this is intrusive-refcount / GC-finalize code and the destructor fallback (donate-the-ref-back in release builds) is a design call a maintainer should sign off on.
What was reviewed
take_socket_ref()counter mirrorsRefCountedTimer::ref_held; bothsocket_ref.forget()sites inconnect()increment, both adopt sites consume — balanced on every path traced.destructorguard:debug_assert!(false)keeps the over-release loud under ASAN; release-build path restores the ref rather than freeing, so any un-caught imbalance becomes defer-to-finalize instead of UAF.- New test now lets wrappers fall out of scope so
finalize()is reachable; gate output confirms it does not fail on the parent commit (author acknowledges — fuzzer is the intended re-verification).
Extended reasoning...
Overview
Follow-up to #34760's RefCountedTimer refactor. Two changes to src/runtime/valkey_jsc/js_valkey.rs: (1) a socket_refs: Cell<u32> counter gates the previously-unconditional ScopedRef::adopt in on_valkey_close/on_valkey_reconnect via a new take_socket_ref() helper, so a close/reconnect dispatch that arrives without a live socket ref cannot release one it does not own; (2) the RefCounted::destructor impl now checks !flags.finalized && this_value.is_not_empty() and, if the count reaches 0 before the JS wrapper is finalized, restores one ref and returns instead of running deinit (with debug_assert!(false) so assertion builds still panic). A new timing-dependent test in valkey-gc.test.ts exercises the auto-reconnect close path.
Security risks
None user-facing. This is a heap-use-after-free fix in the Redis client's lifetime management; the risk profile is memory safety, not injection/auth/data exposure.
Level of scrutiny
High. This is exactly the category REVIEW.md's memory-safety section is written for — intrusive refcounting, ScopedRef::adopt, GC finalize interaction, and re-entrant teardown paths. The take_socket_ref() gate is a straightforward mirror of the RefCountedTimer::ref_held construction from #34760 and I traced both increment sites (connect()'s tls_ctx_failed branch and its success path) against both consume sites (on_valkey_close, on_valkey_reconnect). The destructor guard is more of a policy question: REVIEW.md warns against "a downstream null-check … papering over the defect" and "never add a ref just to silence ASAN — find the actual imbalance"; here the actual imbalance is being fixed by take_socket_ref(), and the guard is an explicit second layer with a loud debug_assert. Whether that release-build fallback is acceptable (vs. a hard invariant) is a maintainer call.
Other factors
- My earlier inline comment (wrappers pinned by a
clientsarray sofinalize()couldn't run) was addressed in b909587; the test now matches the sibling tests' structure and its header is honest about non-determinism. - The mechanical gate's "fails on main" evidence shows the new test passing on the parent under both debug+ASAN and release; the one release-build failure is an unrelated pre-existing test (
ERR_REDIS_INVALID_RESPONSEvsERR_REDIS_CONNECTION_CLOSEDin the RESP simple-string test). So there is no CI-observable regression coverage for this specific fix — the author is up-front that the fuzzer environment is the intended re-verification. - All nine
valkey-gc.test.tscases pass on debug+ASAN with nodebug_assertfiring, so the counter is balanced on every existing path.
…quest wrapper The destructor ref_() resurrected the count 0->1, which REVIEW.md forbids (a compensating ref masks the actual imbalance). With the termination-state fixes in place the count is balanced at finalize entry (verified: 0 donations across 30 runs, 40/40 clean on the fuzzer repro), so the guard was defensive rather than load-bearing; the over-release path it would have absorbed belongs to #34874's socket-adopt accounting and is fixed there. Also delete VM::clear_has_termination_request and its JSC__VM__clearHasTerminationRequest FFI shim, whose sole caller this PR rewired to clear_termination_exception.
…quest wrapper The destructor ref_() resurrected the count 0->1, which REVIEW.md forbids (a compensating ref masks the actual imbalance). With the termination-state fixes in place the count is balanced at finalize entry (verified: 0 donations across 30 runs, 40/40 clean on the fuzzer repro), so the guard was defensive rather than load-bearing; the over-release path it would have absorbed belongs to #34874's socket-adopt accounting and is fixed there. Also delete VM::clear_has_termination_request and its JSC__VM__clearHasTerminationRequest FFI shim, whose sole caller this PR rewired to clear_termination_exception.
|
Confirming this closes the on_data re-entrant face the fleet just flagged on |
|
This is overly defensive slop. Figure out the underlying issue and fix it. |
|
Agreed, pulling the defensive layers. I still have not pinned the concrete over-release path:
I have not been able to reconstruct the 2/2 fault-plan interleaving from the stacks alone. Could you share the fault-plan script (or the exact fault it injects)? With that I can reproduce locally, pin the single unbalanced call, and fix that instead of the gate. |
The unconditional ScopedRef::adopt in on_valkey_close/on_valkey_reconnect is the only release on the teardown path that is not tracked against its acquire. The close/fail paths can re-enter: on_data -> parse fail (or on_connection_timeout -> client_fail) -> fail_with_js_value -> close() dispatches on_close synchronously, and the enter_event_loop_scope drain inside on_valkey_close runs the connect() rejection while the outer socket-handler frames are still on the stack, so user code (or a second dispatch in the same batch) can reach on_valkey_close again with the ref already released. take_socket_ref() mirrors RefCountedTimer: connect() records each forget() in a counter, and the adopt is a checked_sub so a re-entrant caller with no outstanding ref returns None instead of over-releasing. Replaced the timing-dependent churn test with a malformed-RESP-reply test that drives on_data -> close deterministically.
|
Dropped the destructor guard in e29310e. The fix is now just The unconditional adopt is the only release on the teardown path not paired with its acquire. The re-entry is: The |
|
Diff is ready at efea2cc: The internal QA build check is currently failing on infrastructure unrelated to this diff: the "without fix" lane checks out files from current Separately, the fail-before check on this fix is the timing-dependent fuzzer interleaving (0/35 on the CI container vs 10/10 on the reporting hardware) documented above; fuzzer re-verification is the intended proof. |
|
Closing this in favor of #39543. This PR adds a counter around the two Verified on a debug ASAN build: the test from this PR, applied on top of #39543 (c5852ce), passes together with the rest of If the use-after-free shows up again after #39543 lands, please reopen with the new report. |
…39543) Supersedes #39193 and keeps its three tests. The problem connect() takes one keep-alive ref on the RedisClient for its socket. The ref was released inside on_valkey_close and on_valkey_reconnect. Both are reached only through ValkeyClient::on_close(). Every branch of on_close() rejects promises before it gets there, with a ? after each call. Rejecting a promise fails while the VM's termination is pending. That is the state a terminated worker tears its sockets down in. So on_close() returned early, no releaser ran, and the client's box leaked. LSan reports it as a direct leak of 880 bytes from Box<JSValkeyClient>::new. Since #39513 the deferred close for a dial with no socket took a stand-in ref that relied on the same path. The first failed rejection also stopped the drain of the command queues. The remaining entries were never rejected. Until #39570 their promise handles and serialized bytes leaked as well, because the queues did not drop their items. What changed The socket ref is now adopted by a guard at the entry of each close event, right after the existing scoped ref: SocketHandler::on_close, SocketHandler::on_connect_error, and the half-open socket branch of ValkeyClient::close(). That branch also runs on_close() now when an exception is already pending, and returns both results. The deferred close for a dial with no socket no longer takes a stand-in ref. connect() forgets its socket ref only once it has a socket, and that task exists because it never got one, so there was nothing to give back. The two adopts inside on_valkey_close and on_valkey_reconnect are gone. reject_all_pending_commands keeps reading both queues after a rejection fails and returns the first error. on_close() itself is unchanged. Why this shape The ref belongs to the socket. It is released where the socket's close event ends, not where a callee happens to be reached. Any new ? in on_close(), fail() or reject_all_pending_commands can no longer bring the leak back. The MySQL client already does this in its on_close. #34874 and #36837 become unnecessary. Tests test/js/valkey/valkey-gc.test.ts, ASAN only. The three cases from #39193 (retry scheduled, autoReconnect off, retries exhausted) terminate a worker with commands in flight. Two more cover the offline queue: commands queued behind a non-pipelined command, and commands queued behind a dial that never completes, which is the on_connect_error entry. All five report the leaked box on main and pass with this change. Two more cover the half-open socket branch of ValkeyClient::close(), on the main thread: the client dials an IP literal whose accept queue is full, so uSockets hands connect() a real socket that never opens and delivers no close event for it. One case calls close() while the dial is pending. The other lets connectionTimeout fire during it. Each asserts the connect() rejection, the queued command's rejection, one onclose call, connected false, and a clean exit under LSan. Both report the leaked box with that branch's adopt removed and pass with it. A main-thread process.exit() variant was tried and dropped: that teardown has no termination pending, the rejections succeed, and nothing leaked before this change either. Not in this PR The queues themselves: #39570 (merged) made them VecDeque, so an early return no longer leaks the items left in them. This PR is rebased on it. The drain loop here still rejects every entry instead of stopping at the first failure. Verification Debug ASAN build on linux-x64, branch as rebased onto main at 5ebcaff (#39570): commits 99cb497, 9aaa1ec, 06ff535. The src diff is the same as before the rebase apart from the `pop_front()` and `mem::take` renames that #39570 made in `reject_all_pending_commands`. - `bun bd test test/js/valkey/valkey-gc.test.ts` with src/ at main: the five worker.terminate() cases fail, 12 pass. Each failure reports the 880 byte `Box<JSValkeyClient>` as leaked. The two close() cases pass on main as well: that branch released the ref on main too, so they pin the rewritten branch rather than show the leak. - The same file with this branch: 17 pass. - Before the rebase, on main at 4c68990: `test/js/valkey/reliability/{connection-failures,recovery,error-handling}.test.ts` 28 pass (the rest need the docker setup and skip here, CI runs them), and `test/js/web/workers/worker-terminate-lifetime.test.ts` 24 pass. Its one failure (the dns.lookup c-ares case) fails the same way with src/ at main and does not involve valkey. It is reported separately. 9aaa1ec and 06ff535 only touch the test file. <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/js/valkey/valkey-gc.test.ts <!-- robobun:evidence:end --> --------- Co-authored-by: robobun <117481402+robobun@users.noreply.github.com>
Follow-up to #34760. The
RefCountedTimerwrapper closed the timer over-release path, but the fuzzer still produces the same heap-use-after-free onb18d5df35e75(10/10 on a release-asan build, 2/2 under the exact-fault plan).Problem
on_valkey_close/on_valkey_reconnectadoptconnect()'s socket keep-alive ref unconditionally. That adopt is the only release on the teardown path that is not tracked against its acquire. The close/fail paths can re-enter:on_dataparse fail (oron_connection_timeout->client_fail) ->fail_with_js_value->close()dispatcheson_closesynchronously, and theenter_event_loop_scopedrain insideon_valkey_closeruns the connect() rejection while the outer socket-handler frames are still on the stack, so user code (or a second dispatch in the same batch) can reachon_valkey_closeagain with the ref already released. The second adopt spends the JS wrapper's+1; theScopedRefdrops at the end ofon_connection_timeoutthen bring the count to 0 anddeinitfrees the box while the wrapper is live, and GC finalize ->stop_timersreads the freed allocation.Fix
take_socket_ref()mirrorsRefCountedTimer:connect()records eachsocket_ref.forget()in aCell<u32>, and the adopt is achecked_subso a re-entrant caller with no outstanding ref returnsNoneinstead of over-releasing. A counter (not a bool) because a stale reconnect can leave more than oneus_socket_tin flight for the same client.Verification
The new test drives
on_data-> parse fail ->close()deterministically with a server that replies!garbageto HELLO. The fuzzer interleaving underon_connection_timeoutis timing-dependent (0/35 runs on the CI container vs 10/10 on the reporting hardware), so fuzzer re-verification against this branch is the intended proof for that face.