Skip to content

fix(sentinel): do not close a connection still referenced by mConn/rConn - #1015

Merged
rueian merged 1 commit into
redis:mainfrom
FZambia:fix/sentinel-switchtarget-closes-live-conn
Jul 28, 2026
Merged

fix(sentinel): do not close a connection still referenced by mConn/rConn#1015
rueian merged 1 commit into
redis:mainfrom
FZambia:fix/sentinel-switchtarget-closes-live-conn

Conversation

@FZambia

@FZambia FZambia commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

On a failover _switchTarget could close the live mConn/rConn and leave the client pointing at a closed mux. This closes only connections the call itself dialed.

Impact

  • After an ordinary failover the client loses all connectivity to a healthy Redis: every command fails with ErrClosing without even attempting to dial, and the client holds no usable connection to any node.
  • Recovery depends on refreshRetry, which switchTargetRetry spawns on failure. That normally re-dials and repairs mConn — so the usual shape of this is a total outage lasting as long as it takes Sentinel's view to settle, not a permanent one.
  • It becomes permanent when it coincides with a concurrent failing refresh: the deduplicated refresh() hands the joining refreshRetry nil, refreshRetry treats that as success and exits, and nothing is left to replace the closed mConn. That is the singleflight bug fixed in fix(singleflight): give waiting callers the real error of the run they waited on #1014, which is why the two belong together.

Root cause

_switchTarget reuses the live mConn/rConn as target when Sentinel reports the address the client already holds. On the ROLE failure paths it then called target.Close() and returned without swapping, so mConn/rConn referenced a closed mux and nothing replaced it (the swap is only reached on success). This is reachable in normal operation: a just-demoted master answers ROLE with slaveerrNotMaster → the live connection is closed out from under every caller.

Fix

Close only connections this call dialed. A reused connection is still referenced by mConn/rConn and must outlive the failure; the caller retries the refresh and replaces it once a real master is found.

Note the trade-off: keeping a demoted node as mConn means writes fail -READONLY until refresh replaces it. That is strictly better than ErrClosing on every command including reads, and it is transient either way.

Tests

  • TestSwitchTargetDoesNotCloseReusedConn — fails against the previous code (closes the connection still referenced by mConn), passes with the fix.

Note

High Risk
Touches Sentinel connection lifecycle during failover; the prior behavior caused prolonged total client outage (ErrClosing on every command). The fix is narrowly scoped but sits on critical Redis connectivity paths.

Overview
Fixes a Sentinel failover bug where _switchTarget could close the live mConn/rConn when ROLE validation failed but the target was the reused connection for an address the client already held (e.g. demoted master returns slaveerrNotMaster).

The change tracks whether target is reused vs newly dialed and uses closeIfOwned on ROLE/command errors and role mismatches so only connections created in this call are closed; reused connections stay open until a successful swap or a later refresh replaces them.

Adds TestSwitchTargetDoesNotCloseReusedConn to lock in that mConn must not be closed on that failure path.

Reviewed by Cursor Bugbot for commit e2e751b. Bugbot is set up for automated code reviews on this repo. Configure here.

_switchTarget reuses the live mConn/rConn as target when Sentinel reports the
address the client already holds. On the ROLE failure paths it then called
target.Close() and returned without swapping, leaving mConn/rConn pointing at a
closed mux. Every subsequent command fails with ErrClosing without attempting
to dial, and nothing replaces it because the swap is only reached on success.

Reachable in ordinary operation: a just-demoted master answers ROLE with
'slave' -> errNotMaster -> the live connection is closed out from under every
caller. Observed as minutes-long total outages against a healthy Redis, with
the client holding zero TCP connections to any node.

Close only connections this call dialled.
@FZambia

FZambia commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

@rueian hi! To give you some background for the series of pull requests - those are things I noticed when I was trying to reproduce an issue with PUB/SUB stopped working in Sentinel setup on GKE (using Redis 8.8.0 via Bitnami redis chart 27.0.15).

So mostly all the issues I am targeting were caught when I added some chaos into the setup. Clean graceful failover worked correctly, but under some conditions, like killing a pod that manages both Redis and Sentinel instances (Bitnami chart default) – several issues happened.

I also plan to open a couple more PRs that fix the failover issues – but they are better to land on top of the initial three.

@rueian
rueian merged commit be61c31 into redis:main Jul 28, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants