Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,54 @@ func main() {
}
```

### Fencing tokens

Set `Options.Fence` to mint a **fencing token** with the lock: a strictly
increasing value, incremented atomically on each new acquisition and returned by
`Lock.FenceToken`. Stamp every write to the protected resource with the token and
have the resource reject any write carrying an older token. A lock holder that
pauses (GC, scheduling) long enough to lose the lock without noticing is then
fenced out — its writes carry a stale token and are refused. This is the
mitigation described in Martin Kleppmann's
[How to do distributed locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html).

```go
func fence() {
client := redis.NewClient(&redis.Options{Network: "tcp", Addr: "127.0.0.1:6379"})
defer client.Close()

locker := redislock.New(client)

ctx := context.Background()

// Obtain a lock with a fencing token.
lock, err := locker.Obtain(ctx, "my-key", time.Second, &redislock.Options{Fence: true})
if err != nil {
log.Fatalln(err)
}
defer lock.Release(ctx)

// FenceToken is 0 when the lock was obtained without Options.Fence. Stamp
// writes to the protected resource with the token; reject older ones.
if token := lock.FenceToken(); token != 0 {
fmt.Printf("fenced write with token %d\n", token)
}
}
```

A few notes:

- Enforcement is the resource's job: `redislock` mints and returns the token, but
the check (`reject if incoming < highest applied`) must happen atomically at the
resource, which is the only place the comparison and the write can be made one
operation.
- The token is stored at `<key>:fence` (the first key, for multi-key locks) and
persists across release so it keeps increasing; it is never reset by `Release`.
- The token is only as monotonic as the underlying Redis. On a single instance it
is strict; on a Sentinel/Cluster failover that loses the `INCR`, it can regress.
For strict cross-failover monotonicity, source the token from a linearizable
store instead.

### External watchdog

`redislock` deliberately does not bundle a built-in watchdog goroutine: refresh
Expand Down
28 changes: 28 additions & 0 deletions README.md.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,34 @@ import (
func main() {{ "Example" | code }}
```

### Fencing tokens

Set `Options.Fence` to mint a **fencing token** with the lock: a strictly
increasing value, incremented atomically on each new acquisition and returned by
`Lock.FenceToken`. Stamp every write to the protected resource with the token and
have the resource reject any write carrying an older token. A lock holder that
pauses (GC, scheduling) long enough to lose the lock without noticing is then
fenced out — its writes carry a stale token and are refused. This is the
mitigation described in Martin Kleppmann's
[How to do distributed locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html).

```go
func fence() {{ "ExampleClient_Obtain_fence" | code }}
```

A few notes:

- Enforcement is the resource's job: `redislock` mints and returns the token, but
the check (`reject if incoming < highest applied`) must happen atomically at the
resource, which is the only place the comparison and the write can be made one
operation.
- The token is stored at `<key>:fence` (the first key, for multi-key locks) and
persists across release so it keeps increasing; it is never reset by `Release`.
- The token is only as monotonic as the underlying Redis. On a single instance it
is strict; on a Sentinel/Cluster failover that loses the `INCR`, it can regress.
For strict cross-failover monotonicity, source the token from a linearizable
store instead.

### External watchdog

`redislock` deliberately does not bundle a built-in watchdog goroutine: refresh
Expand Down
22 changes: 22 additions & 0 deletions example_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,28 @@ func ExampleClient_Obtain_retry() {
fmt.Println("I have a lock!")
}

func ExampleClient_Obtain_fence() {
client := redis.NewClient(&redis.Options{Network: "tcp", Addr: "127.0.0.1:6379"})
defer client.Close()

locker := redislock.New(client)

ctx := context.Background()

// Obtain a lock with a fencing token.
lock, err := locker.Obtain(ctx, "my-key", time.Second, &redislock.Options{Fence: true})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one little concern left: I don't like the fact that we are automatically generating key names for the fence key, I always prefer to delegate this to the user as there are sometimes non-trivial implications, e.g. redis cluster or other redis implementations where key names matter. I would therefore suggest:

lock, err := locker.Obtain(ctx, "my-key", time.Second, &redislock.Options{FenceKey: "my-key:fence"})

what do you think?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — auto-generating the key name hides exactly the kind of Cluster/key-placement footgun you're describing, so letting the caller own it is the right call. Done: Options.FenceKey is now caller-supplied.

I also took it one step further for Cluster safety — the fence key now goes in KEYS (not ARGV), as the last entry when fencing is on, with an ARGV flag telling the script to treat it specially. That way Redis validates the slot up front and a misplaced fence key fails with CROSSSLOT before the script runs, rather than mid-execution.

if err != nil {
log.Fatalln(err)
}
defer lock.Release(ctx)

// FenceToken is 0 when the lock was obtained without Options.Fence. Stamp
// writes to the protected resource with the token; reject older ones.
if token := lock.FenceToken(); token != 0 {
fmt.Printf("fenced write with token %d\n", token)
}
}

func ExampleLock_Refresh_watchdog() {
client := redis.NewClient(&redis.Options{Network: "tcp", Addr: "127.0.0.1:6379"})
defer client.Close()
Expand Down
21 changes: 19 additions & 2 deletions obtain.lua
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
-- obtain.lua: arguments => [value, tokenLen, ttl]
-- obtain.lua: arguments => [value, tokenLen, ttl, fenceKey]
-- Obtain.lua try to set provided keys's with value and ttl if they do not exists.
-- Keys can be overriden if they already exists and the correct value+tokenLen is provided.
-- When fenceKey is set, returns a fencing token instead of "OK".

local function pexpire(ttl)
-- Update keys ttls.
Expand All @@ -22,6 +23,20 @@ local function canOverrideKeys()
return true
end

local fenceKey = ARGV[4]

-- reply returns the fencing token, advancing it only on a fresh acquisition,
-- or "OK" when fencing is disabled.
local function reply(fresh)
if fenceKey == nil or fenceKey == "" then
return redis.status_reply("OK")
end
if fresh then
return redis.call("incr", fenceKey)
end
return tonumber(redis.call("get", fenceKey) or "0")
end

-- Prepare mset arguments.
local setArgs = {}
for _, key in ipairs(KEYS) do
Expand All @@ -34,7 +49,9 @@ if redis.call("msetnx", unpack(setArgs)) ~= 1 then
return false
end
redis.call("mset", unpack(setArgs))
pexpire(ARGV[3])
return reply(false)
end

pexpire(ARGV[3])
return redis.status_reply("OK")
return reply(true)
51 changes: 41 additions & 10 deletions redislock.go
Original file line number Diff line number Diff line change
Expand Up @@ -81,23 +81,32 @@ func (c *Client) ObtainMulti(ctx context.Context, keys []string, ttl time.Durati
value := token + opt.getMetadata()
ttlVal := strconv.FormatInt(int64(ttl/time.Millisecond), 10)

fence := opt.getFence()
fenceKey := ""
if fence {
fenceKey = keys[0] + ":fence"
}

var fenceToken int64

if err := withRetry(ctx, ttl, opt.getRetryStrategy(), func(ctx context.Context) (bool, error) {
ok, err := c.obtain(ctx, keys, value, len(token), ttlVal)
ok, ft, err := c.obtain(ctx, keys, value, len(token), ttlVal, fenceKey)
if err != nil {
// any non-nil error from obtain is terminal (transient redis
// errors are unlikely to clear within a lock TTL and retrying a
// broken server is futile).
return true, err
}
if ok {
fenceToken = ft
return true, nil
}
// lock is held by someone else; retryable.
return false, nil
}); err != nil {
return nil, err
}
return &Lock{Client: c, keys: keys, value: value, tokenLen: len(token)}, nil
return &Lock{Client: c, keys: keys, value: value, tokenLen: len(token), fenceToken: fenceToken}, nil
}

// withRetry runs attempt repeatedly until it signals it is done, the retry
Expand Down Expand Up @@ -160,15 +169,21 @@ func withRetry(ctx context.Context, ttl time.Duration, retry RetryStrategy, atte
}
}

func (c *Client) obtain(ctx context.Context, keys []string, value string, tokenLen int, ttlVal string) (bool, error) {
_, err := luaObtain.Run(ctx, c.client, keys, value, tokenLen, ttlVal).Result()
func (c *Client) obtain(ctx context.Context, keys []string, value string, tokenLen int, ttlVal, fenceKey string) (ok bool, fenceToken int64, err error) {
res, err := luaObtain.Run(ctx, c.client, keys, value, tokenLen, ttlVal, fenceKey).Result()
if err != nil {
if errors.Is(err, redis.Nil) {
return false, nil
return false, 0, nil
}
return false, err
return false, 0, err
}

// With fencing the reply is the integer token; otherwise it is the "OK" status.
if token, isInt := res.(int64); isInt {
return true, token, nil
}
return true, nil

return true, 0, nil
}

func (c *Client) randomToken() (string, error) {
Expand All @@ -190,9 +205,10 @@ func (c *Client) randomToken() (string, error) {
// Lock represents an obtained, distributed lock.
type Lock struct {
*Client
keys []string
value string
tokenLen int
keys []string
value string
tokenLen int
fenceToken int64
}

// Obtain is a short-cut for New(...).Obtain(...).
Expand Down Expand Up @@ -221,6 +237,12 @@ func (l *Lock) Token() string {
return l.value[:l.tokenLen]
}

// FenceToken returns the lock's fencing token, or 0 if it was obtained without
// Options.Fence. Tokens start at 1, so 0 always means unfenced.
func (l *Lock) FenceToken() int64 {
return l.fenceToken
}

// Metadata returns the metadata of the lock.
func (l *Lock) Metadata() string {
return l.value[l.tokenLen:]
Expand Down Expand Up @@ -297,6 +319,11 @@ type Options struct {
// Token is a unique value that is used to identify the lock. By default, a random tokens are generated. Use this
// option to provide a custom token instead.
Token string

// Fence enables a fencing token, minted on each new acquisition and
// returned by Lock.FenceToken. Stored at "<key>:fence".
// Default: disabled.
Fence bool
}

func (o *Options) getMetadata() string {
Expand All @@ -313,6 +340,10 @@ func (o *Options) getToken() string {
return ""
}

func (o *Options) getFence() bool {
return o != nil && o.Fence
}

func (o *Options) getRetryStrategy() RetryStrategy {
if o != nil && o.RetryStrategy != nil {
return o.RetryStrategy
Expand Down
67 changes: 67 additions & 0 deletions redislock_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,73 @@ func TestObtain_custom_token(t *testing.T) {
}
}

func TestObtain_fence(t *testing.T) {
rc := redisConnect(t)
lockKey := rc.lockKey()
rc.keys = append(rc.keys, lockKey+":fence")

// no token without the Fence option
plain := quickObtain(t, time.Hour)
defer plain.Release(t.Context())
if got := plain.FenceToken(); got != 0 {
t.Fatalf("expected no fence token without Options.Fence, got %v", got)
}
if err := plain.Release(t.Context()); err != nil {
t.Fatal(err)
}

// first fenced obtain
lock1, err := Obtain(t.Context(), rc, lockKey, time.Hour, &Options{Fence: true})
if err != nil {
t.Fatal(err)
}
tok1 := lock1.FenceToken()
if exp, got := int64(1), tok1; exp != got {
t.Fatalf("expected first fence token %v, got %v", exp, got)
}

if err := lock1.Release(t.Context()); err != nil {
t.Fatal(err)
}

// next obtain must mint a greater token
lock2, err := Obtain(t.Context(), rc, lockKey, time.Hour, &Options{Fence: true})
if err != nil {
t.Fatal(err)
}
defer lock2.Release(t.Context())

if tok2 := lock2.FenceToken(); tok2 <= tok1 {
t.Fatalf("expected fence token > %v, got %v", tok1, tok2)
}
}

func TestObtain_fence_reentrant(t *testing.T) {
rc := redisConnect(t)
lockKey := rc.lockKey()
rc.keys = append(rc.keys, lockKey+":fence")

// obtain with a known token
lock1, err := Obtain(t.Context(), rc, lockKey, time.Hour, &Options{Token: "foo", Fence: true})
if err != nil {
t.Fatal(err)
}
defer lock1.Release(t.Context())

tok1 := lock1.FenceToken()

// re-obtain (override) must not advance the token
lock2, err := Obtain(t.Context(), rc, lockKey, time.Hour, &Options{Token: "foo", Fence: true})
if err != nil {
t.Fatal(err)
}
defer lock2.Release(t.Context())

if exp, got := tok1, lock2.FenceToken(); exp != got {
t.Fatalf("re-entrant override must not mint a new token: expected %v, got %v", exp, got)
}
}

func TestObtain_retry_success(t *testing.T) {
rc := redisConnect(t)
lockKey := rc.lockKey()
Expand Down