keyvaluestorage: sort IterateValues by id so Iterate groups whole keys (SYN-116) - #753
Merged
Merged
Conversation
Storage.Iterate builds per-key callback groups from consecutive runs, but IterateValues scanned unsorted (insertion order), so one key's rows split across groups whenever peers' row batches interleaved — and the split is stable across restarts. Consumers treating a callback as "all rows of this key" silently misbehaved (SYN-116; froze read-marker republish in the SDK, SYN-97). Sorting by id (key+"-"+peerId) makes the grouping group by key, same as IteratePrefix. The diff-loading scans stay unsorted — ldiff.Set is order-independent. Claude-Session: https://claude.ai/code/session_01WCH11UrasWuzpKpgHRepow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Storage.Iteratebuilds its per-key callback groups from consecutive rows, butinnerstorage.IterateValuesscanned with no sort — SQLite full-scans in rowid (insertion) order, so one key's rows split across multiple callbacks whenever peers' row batches interleave, and the split is stable across restarts (upsert preserves rowids). Any consumer treating a callback as "all rows of this key" silently misbehaves; in the SDK this froze read-marker republish (SYN-97 — same 16 keys re-broadcast on every boot, forever).Fix:
IterateValuessorts byid(key+"-"+peerId), the same orderingIteratePrefixalready uses, so the consecutive-run grouping actually groups by key. The diff-loading scans inNew/initstay unsorted —ldiff.Setis order-independent.Regression test inserts per-peer batches (16 keys × 3 peers, one key's rows maximally non-adjacent in insertion order) and asserts the scan is id-ordered and every key's rows form one consecutive run; it fails on the previous unsorted scan.
Measured cost of the sorted scan (any-store v0.4.7): 10k rows 2.0ms → 3.2ms, 50k rows 9.9ms → 18.5ms per full iteration.
Tracked as SYN-116.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WCH11UrasWuzpKpgHRepow