Repository navigation
Make fair task gc conditional on rangeid (#12283) - #12427
Merged
Merged
Conversation
Add ConditionRangeID to CompleteTasksLessThanRequest and make CompleteTasksLessThan conditional on unchanged rangeid, just like CreateTasks. When fair task reader gc's tasks, it's not protected by write fencing like adding tasks. So there's a small race window: a node advances its ack level to C, but it's not persisted yet. Another node takes over and loads the previous ack level, A. It writes a task above A but below C, call it B. The first node now deletes below C, deleting B. Note that this only happens with fair queues because fifo queues always write above the rangeid. - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) - [x] validated in tla+ model (not in repo yet)
Contributor
rkannan82
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add ConditionRangeID to CompleteTasksLessThanRequest and make CompleteTasksLessThan conditional on unchanged rangeid, just like CreateTasks.
When fair task reader gc's tasks, it's not protected by write fencing like adding tasks. So there's a small race window: a node advances its ack level to C, but it's not persisted yet. Another node takes over and loads the previous ack level, A. It writes a task above A but below C, call it B. The first node now deletes below C, deleting B. Note that this only happens with fair queues because fifo queues always write above the rangeid.
Note
High Risk
Changes fair-queue persistence GC and ownership fencing; a bug could delete another owner's tasks or leave stale nodes running without unloading on conflict.
Overview
Fair-queue task GC can no longer delete persisted tasks unless the store still shows the same range ID as the current owner, closing a race where a stale node could delete tasks written by a new owner after range steal.
Persistence:
CompleteTasksLessThanRequestgains optionalConditionRangeID(fair/v2 only). Cassandra v2 batches the delete with a CAS range-id check; SQL v2 runs delete inside a transaction after locking the queue row. Classic/v1 paths reject non-zeroConditionRangeID.Matching:
CompleteFairTasksLessThanpasses the in-memoryrangeID, holdsdb.lockduring the conditional LWT, treatsConditionFailedErroras lost ownership (not a generic store error), and the fair task reader callssignalIfFatalon GC failure so the partition unloads. Tests cover conditional delete and tighten range-stealing / backlog assertions.Reviewed by Cursor Bugbot for commit c658860. Bugbot is set up for automated code reviews on this repo. Configure here.