Skip to content

Make fair task gc conditional on rangeid (#12283) - #12427

Merged
carlydf merged 1 commit into
release/v1.30.xfrom
fair-task-reader-rangeid-bugfix-v1.30
Oct 7, 2026
Merged

carlydf merged 1 commit into
release/v1.30.xfrom
fair-task-reader-rangeid-bugfix-v1.30

Conversation

@carlydf

@carlydf carlydf commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Add ConditionRangeID to CompleteTasksLessThanRequest and make CompleteTasksLessThan conditional on unchanged rangeid, just like CreateTasks.

When fair task reader gc's tasks, it's not protected by write fencing like adding tasks. So there's a small race window: a node advances its ack level to C, but it's not persisted yet. Another node takes over and loads the previous ack level, A. It writes a task above A but below C, call it B. The first node now deletes below C, deleting B. Note that this only happens with fair queues because fifo queues always write above the rangeid.

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)
  • validated in tla+ model (not in repo yet)

Note

High Risk
Changes fair-queue persistence GC and ownership fencing; a bug could delete another owner's tasks or leave stale nodes running without unloading on conflict.

Overview
Fair-queue task GC can no longer delete persisted tasks unless the store still shows the same range ID as the current owner, closing a race where a stale node could delete tasks written by a new owner after range steal.

Persistence: CompleteTasksLessThanRequest gains optional ConditionRangeID (fair/v2 only). Cassandra v2 batches the delete with a CAS range-id check; SQL v2 runs delete inside a transaction after locking the queue row. Classic/v1 paths reject non-zero ConditionRangeID.

Matching: CompleteFairTasksLessThan passes the in-memory rangeID, holds db.lock during the conditional LWT, treats ConditionFailedError as lost ownership (not a generic store error), and the fair task reader calls signalIfFatal on GC failure so the partition unloads. Tests cover conditional delete and tighten range-stealing / backlog assertions.

Reviewed by Cursor Bugbot for commit c658860. Bugbot is set up for automated code reviews on this repo. Configure here.

Add ConditionRangeID to CompleteTasksLessThanRequest and make
CompleteTasksLessThan conditional on unchanged rangeid, just like
CreateTasks.

When fair task reader gc's tasks, it's not protected by write fencing
like adding tasks. So there's a small race window: a node advances its
ack level to C, but it's not persisted yet. Another node takes over and
loads the previous ack level, A. It writes a task above A but below C,
call it B. The first node now deletes below C, deleting B. Note that
this only happens with fair queues because fifo queues always write
above the rangeid.

- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
- [x] validated in tla+ model (not in repo yet)
@carlydf
carlydf requested review from a team as code owners October 6, 2026 16:54
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @carlydf's task in 58s —— View job


Reviewed 10 files, 0 findings.

@carlydf
carlydf merged commit 2af91fb into release/v1.30.x Oct 7, 2026
107 of 114 checks passed
@carlydf
carlydf deleted the fair-task-reader-rangeid-bugfix-v1.30 branch October 7, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants