-
Notifications
You must be signed in to change notification settings - Fork 252
fix(rueidisaside): prevent canceled cache fills from leaking locks #1010
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Makarechi
wants to merge
6
commits into
redis:main
Choose a base branch
from
Makarechi:fix/rueidisaside-canceled-acquire
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+585
−105
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
8ba7676
fix(rueidisaside): clean up lock after canceled acquisition
Makarechi c9aa469
refactor(rueidisaside): remove redundant context checks
Makarechi 51940d3
fix(rueidisaside): guard canceled cache fills
Makarechi 1c592a6
fix(rueidisaside): order canceled fills on one pipeline
Makarechi 81fe858
fix(rueidisaside): keep flights across client id changes
Makarechi 1854a98
fix(rueidisaside): start flights before keepalive
Makarechi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| package rueidisaside | ||
|
|
||
| import ( | ||
| "context" | ||
| "strconv" | ||
| "time" | ||
|
|
||
| "github.com/redis/rueidis" | ||
| ) | ||
|
|
||
| type flight struct { | ||
| done chan struct{} | ||
| } | ||
|
|
||
| func (c *Client) beginFlight(key string) (f *flight, leader bool) { | ||
| c.mu.Lock() | ||
| defer c.mu.Unlock() | ||
|
|
||
| if f = c.flights[key]; f != nil { | ||
| return f, false | ||
| } | ||
| f = &flight{done: make(chan struct{})} | ||
| c.flights[key] = f | ||
| return f, true | ||
| } | ||
|
|
||
| func (c *Client) finishFlight(key string, f *flight) { | ||
| c.mu.Lock() | ||
| defer c.mu.Unlock() | ||
|
|
||
| if c.flights[key] == f { | ||
| delete(c.flights, key) | ||
| close(f.done) | ||
| } | ||
| } | ||
|
|
||
| func (c *Client) populate( | ||
| ctx context.Context, | ||
| ttl time.Duration, | ||
| key string, | ||
| fn func(ctx context.Context, key string) (val string, err error), | ||
| f *flight, | ||
| ) (val string, err error) { | ||
| defer c.finishFlight(key, f) | ||
|
|
||
| id, err := c.keepalive() | ||
| if err != nil { | ||
| return "", err | ||
| } | ||
|
|
||
| cleanup := true | ||
| defer func() { | ||
| if cleanup { | ||
| delkey.Exec(context.Background(), c.client, []string{key}, []string{id}) | ||
| } | ||
| }() | ||
|
|
||
| if c.useLuaLock { | ||
| val, err = acquireLock.Exec(ctx, c.client, []string{key}, []string{id, strconv.FormatInt(ttl.Milliseconds(), 10)}).ToString() | ||
| } else { | ||
| val, err = c.client.Do(ctx, c.client.B().Set().Key(key).Value(id).Nx().Get().Px(ttl).Build()).ToString() | ||
| } | ||
| if err == nil { | ||
| cleanup = false | ||
| return val, nil | ||
| } | ||
| if !rueidis.IsRedisNil(err) { | ||
| return val, err | ||
| } | ||
|
|
||
| ctx = context.WithValue(ctx, ttlKey, &ttl) | ||
| if val, err = fn(ctx, key); err == nil { | ||
| err = setkey.Exec(ctx, c.client, []string{key}, []string{id, val, strconv.FormatInt(ttl.Milliseconds(), 10)}).Error() | ||
| } | ||
| if err == nil { | ||
| cleanup = false | ||
| } | ||
| return val, err | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even if the client id changed here, why can't we just wait for the flight to finish?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we jump to retry, we will have a busy retry loop until the existing flight acquires the lock, which is not that good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point — we can wait for the existing per-key flight here. Fixed in
81fe858.The flight map is now keyed strictly by cache key, and
beginFlightlooks up an active flight before validating the client ID. Active flights are no longer reset on invalidation, so a caller that observed an old or new client ID joins the same in-process flow and waits with its own context instead of repeatedly jumping toretry.The client-ID check remains only when no flight exists, which prevents a stale caller from becoming a new leader. In that narrow case one retry is still needed to obtain the current ID.
I also updated the Redis 7 and legacy disconnect tests to verify that an ID change does not start a second loader while the old flight is active, and added a unit regression for joining a flight across generations.
go test,go test -race, andgo vetpass.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follow-up in
1854a98: I removed the remaining stale-ID/no-flight retry as well. The per-key flight is now created beforekeepalive, so the whole local miss flow starts with strict key-based singleflight. The leader resolves the current client ID inside that flight, while every same-key follower waits with its own context. This removes the retry branch entirely and still keeps different keys independent.A new regression test blocks client-marker creation and verifies that a follower neither creates another marker nor runs another loader. Full tests, race tests, and
go vetpass.