perf(sqlite): route Signal reads through the read pool - #1222
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughSummary by CodeRabbit
Walkthrough
ChangesSQLite read routing
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant SqliteStore
participant ReaderPool
participant WriteQueue
participant SQLite
SqliteStore->>ReaderPool: dispatch read_query
ReaderPool->>SQLite: execute read on query-only connection
SqliteStore->>WriteQueue: route consistency-sensitive read
WriteQueue->>SQLite: execute serialized operation
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| Filename | Overview |
|---|---|
| storages/sqlite-storage/src/sqlite_store.rs | Centralizes SQLite read routing, preserves write-queue placement for consistency-sensitive lookups, makes chunked mutations atomic, and adds focused routing tests. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
Q[Read-only store query] --> R{Reader pool configured?}
R -->|Yes| RP[Reader pool]
R -->|No| S{Snapshot-safe wider write pool?}
S -->|Yes| WP[Write pool with deferred read transaction]
S -->|No| F[Direct write-pool checkout]
RP --> TX[Consistent read snapshot]
WP --> TX
F --> O[Serialized fallback when required]
Reviews (14): Last reviewed commit: "fix(sqlite): keep has_signal_state_for_u..." | Re-trigger Greptile
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@storages/sqlite-storage/src/sqlite_store.rs`:
- Around line 6044-6050: Extend the name filter in the scan that currently
checks `get_`, `load_`, and `has_` so it also recognizes `list_`, `count_`,
`find_`, `fetch_`, and `is_` prefixes, plus the `*_exists` suffix used by
`device_exists`. Preserve the existing `current` assignment and scan-count
behavior for every matched read-shaped method.
- Around line 619-632: Wrap the no-reads fallback branch in read_snapshot,
applying it around the existing with_semaphore connection-and-f call so
multi-statement readers such as has_signal_state_for_user, load_prekeys_batch,
and get_app_state_mutation_macs_batch_for_device use one snapshot. Update the
misleading comment to reflect that read_snapshot provides the consistent view,
while leaving the reads-present path unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 33407237-9300-4515-8926-6096c1b37702
📒 Files selected for processing (2)
storages/sqlite-storage/src/shared.rsstorages/sqlite-storage/src/sqlite_store.rs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6b967cfa62
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
📦 Binary size report
.text per crate
Top movers (cargo-bloat attribution)
Baseline: |
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
Pushed 6cf393f for the review round. Taking the substantive ones in order: Chunked writes could be observed half-applied (Codex P2, cubic P1) — real, and my fault: Decrypt reads under an in-flight write are untested (cubic P2) — fair, the old test held an idle permit. Reader branch duplicates Widen the routing scan (CodeRabbit, cubic P2) — done: Wrap the fallback in Binary size: the first revision blew the gate at +116 KiB stripped / +90.6 KiB .text. Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6cf393fe95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
Codex's second P2 is right and 51016bb reverts it: I checked the failure it describes rather than taking it on faith. My "the outcome set is unchanged" argument still holds formally — the permit ordered a concurrent read and write arbitrarily, so a miss was always reachable — but that is a statement about outcomes, not odds, and here the losing outcome is a dropped message rather than a cache miss the caller already handles. Going from "the read has to arrive before the write starts" to "the read has to arrive before the write commits" is a real widening, and a history-sync batch commit is not short. These two reads also aren't what this PR is for and contribute none of the Case B win, so there was nothing on the other side of the scale. They're listed in That is the one method-level judgement in this PR I got wrong, and it's the one the batch was explicitly supposed to catch — good catch. Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51016bb446
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
bcabb05. Codex and cubic independently landed on the same thing, and they're right. LID/PN mappings back on the write queue. Declining the prekey one (Codex, Also from cubic: Net effect on the audit: 22 methods migrated, four held back with reasons ( Generated by Claude Code |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
storages/sqlite-storage/src/sqlite_store.rs (1)
629-643: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winThe fallback comment is only true at
pool_size = 1. Say so.Look, the routing is fine. The claim in the comment is not fully fine. Line 631-633 states that no writer can hold the single pooled connection during this query. That holds only while
pool_sizestays at 1. Several writers in this file check out a pooled connection directly and never take the write permit:delete_session_for_device(Line 1290),put_sender_key_for_device(Line 1313),set_app_state_sync_key_for_device(Line 1423),save_base_key(Line 2728),update_device_list(Line 2811),put_group_metadata(Line 2991),touch_tc_token_sender_timestamp(Line 3232). If someone raisespool_sizeabove 1 withread_pool_size = 0, a multi-statement fallback read can straddle one of those commits. The real readers arehas_signal_state_for_user(two queries),load_prekeys_batch(chunk loop), andget_app_state_mutation_macs_batch_for_device(chunk loop).I understand the decision to keep the fallback cheap at the documented default. Then bind the invariant to the condition in the comment, so the next person who raises
pool_sizesees the cost.♻️ State the pool_size condition
if self.reads.is_none() { - // No reader connections: the single pooled connection is what no - // writer can be holding while this query runs, so the snapshot - // comes for free and a transaction would only add statements. + // No reader connections: at the default `pool_size = 1` the single + // pooled connection is what no writer can be holding while this + // query runs, so the snapshot comes for free and a transaction + // would only add statements. Raising `pool_size` above 1 with + // `read_pool_size = 0` breaks that: the raw-pool writers (which + // never take the write permit) can then commit between the + // statements of a multi-statement read, and this branch would need + // the same deferred read transaction as the reader path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@storages/sqlite-storage/src/sqlite_store.rs` around lines 629 - 643, Update the fallback comment in read_erased to state that the no-writer snapshot invariant applies only when pool_size is 1. Keep the existing routing and implementation unchanged, but explicitly warn that larger pools with read_pool_size set to 0 can allow concurrent writers during multi-statement reads.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@storages/sqlite-storage/src/sqlite_store.rs`:
- Around line 6096-6108: Throttle the polling loop around get_sender_key_devices
by adding a sub-millisecond asynchronous sleep between samples, while preserving
the existing assertion, completion condition, and 20-second timeout. Keep the
backoff inside the loop so repeated read_query calls do not continuously compete
with set_sender_key_status on the blocking pool.
- Around line 6117-6134: Update misrouted_reads to track and return the names
excused by ON_THE_WRITE_QUEUE, alongside the existing offenders and scanned
names. Add a set comparison asserting the excused names exactly match the
allowlist entries, so stale entries such as get_lid_mapping or
get_msg_secret_with_ts fail the test when no longer on the write queue.
---
Outside diff comments:
In `@storages/sqlite-storage/src/sqlite_store.rs`:
- Around line 629-643: Update the fallback comment in read_erased to state that
the no-writer snapshot invariant applies only when pool_size is 1. Keep the
existing routing and implementation unchanged, but explicitly warn that larger
pools with read_pool_size set to 0 can allow concurrent writers during
multi-statement reads.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 83c0dcd0-76a7-43d1-8dc2-ce2af914f23b
📒 Files selected for processing (1)
storages/sqlite-storage/src/sqlite_store.rs
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Requires human review: Auto-approval blocked by 1 unresolved issue from previous reviews.
Re-trigger cubic
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
9923ffb — took all three. The allowlist could rot, and that's the best catch of this round. Throttled the chunked-write race by 200µs per sample. You're right that both sides go through Fallback comment now says the free-snapshot claim holds at Generated by Claude Code |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@storages/sqlite-storage/src/sqlite_store.rs`:
- Around line 631-636: Update the fallback read path around with_semaphore to
wrap the read closure f with crate::shared::read_snapshot, preserving deferred
snapshot semantics when read_pool_size is zero even for pool_size greater than
one. Remove or revise the nearby unsupported-configuration assumption so this
reachable configuration receives a consistent snapshot.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 7df5a1ed-2646-4a16-9aef-c1043fbaf046
📒 Files selected for processing (1)
storages/sqlite-storage/src/sqlite_store.rs
|
2018115 — I was wrong on this one and you were right to keep pushing. I'd been declining the fallback snapshot on two grounds, and the new argument breaks both. What I did differently from the suggested diff: rather than wrapping the fallback unconditionally, the shortcut is now conditioned on what actually makes it true — That's four reviews on this one spot. The reason it took that long is that I was defending the cost constraint without checking whether the configuration I was dismissing was real. It was, and it's in our own test file. Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2018115809
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4f9bb6d to
eb0478e
Compare
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
Rebased onto The E2E blocker was my mistakeI put this PR in draft partly because I believed it caused an E2E regression. It did not. I saw one red E2E run on
One failure in eight runs of the same branch is a flake, and five greens before it would have told me that immediately. I blocked the PR on a check I never performed. I could not reproduce the local mock-server comparison myself (no mock server here), so this correction rests on the CI history above rather than on a local run. The "no change at the default config" claim was false, and it cost ~25% p50The old body said behaviour at Measured, 16 concurrent
The clean audit pass moved two more methods outThe migrate/stay criterion changed mid-review from "provably read-only" to "what does the caller do with a stale answer", and I applied the new criterion retroactively, one method at a time, as reviewers found them. I never ran it over the whole set from scratch. Running it now moved two more back onto the write queue:
Neither recovers from a stale miss, so neither qualifies. The prekey question I raised and never closed ( CodeRabbit items
Still draft
No merge, no labels - yours to land. Generated by Claude Code |
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Requires human review: The PR modifies concurrency and transaction semantics for core storage operations (reading sessions, identities, etc.), changing product and operational tradeoffs that require human judgment. The diff is also truncated, preventing verification of the full implementation.
Re-trigger cubic
|
The binary size gate failed on Why it grewI reproduced the gate locally (installed
I had assumed it was all The fixTwo dispatch-shape changes, no behaviour change:
Cost of the erasure is one .text +39.88 -> +24.00 KiB, stripped +53.78 -> +28.34 KiB. CI agrees with the local number exactly. One thing I found on the way, which is not good newsChasing this I re-ran my own falsification check on It now parks a read mid-flight on a shared-cache store and requires a concurrent write to commit. First attempt parked 300 ms and still passed unguarded, because
Still draft, still gated on the cold-load guard PR, which does not exist yet. Generated by Claude Code |
The store has one write permit because two writers deadlock on SQLite's transaction upgrade, and that is right for writes. It was also governing every read: a session, identity or sender-key miss on the decrypt path queued behind whatever write was in flight, including a write-behind flush of a full session batch. The reader pool that already existed for `read_pool_size` was reachable only from `SharedSqlite::read`, so turning the knob on bought the Signal path nothing. Add `read_query`, which mirrors `SharedSqlite::read`: reader connection when one is configured, write permit otherwise, so `read_pool_size = 0` keeps the previous path with no added statement. Every read-only method that only issues SELECTs now goes through it. `get_pending_inbound` stays on the write queue for its busy retry loop. The guarantee after the change is read-your-own-write across connections: a WAL reader opens on the latest committed snapshot, so a read issued after a write's await observes it. Reads that merely overlap a write see either state, which is what the single permit already gave them - it ordered them arbitrarily, not causally. Reader connections are `query_only`, so a write that slips onto the read path errors instead of escaping the serialization; a source scan fails when a new read-shaped method reaches the database any other way.
Review of the read routing turned up three things worth fixing. Chunking in `put_app_state_mutation_macs_for_device`, `delete_app_state_mutation_macs_for_device`, `mark_prekeys_uploaded`, `set_sender_key_status` and `delete_sender_key_device_rows` works around SQLite's host-parameter limit, but each chunk was committing on its own. That was invisible while reads held the same permit; now that a reader can run alongside, it can land between two chunks and see half a batch. Wrap each loop in one transaction, which also stops a crash mid-batch from persisting a partial one. Regression test races a four-chunk write against a reader and fails on any count that is neither before nor after. The read helper was generic over its closure, so two dozen call sites each monomorphized a body carrying Diesel's transaction machinery: +90 KiB of .text, over the size gate. Erase the closure into a boxed `FnOnce` first, so the body instantiates once per return type. The reader branch then delegates to `SharedSqlite::read` rather than restating acquire-checkout-snapshot, which leaves that sequence with one implementation and reverts the visibility change to `read_snapshot`. Measured against the branch point: +31.5 KiB stripped, +23.5 KiB .text, both inside the per-PR budget. Also: the routing scan now covers `is_`, `list_`, `count_`, `find_`, `fetch_` and the `_exists` suffix, so `device_exists` and future read-shaped names are inspected too. And a test pins the behaviour the msg-secret reads were previously kept on the write queue for: with a real write transaction open, a read returns the last commit instead of blocking or failing.
`get_msg_secret` and `get_msg_secret_with_ts` were not in the batch this PR set out to move, and they should not have gone with it. A miss on that path is terminal: `secret_encrypted_message` returns None and the reaction, vote or edit is dropped, with no retry and no buffering behind it. History sync seeds secrets through `put_msg_secrets` directly rather than the live write-behind buffer, so a lookup that races that batch finds nothing in the buffer and goes to the backend. The formal argument that the outcome set is unchanged still holds -- the single permit ordered a concurrent read and write arbitrarily either way -- but it widens the losing window from "the read arrives before the write starts" to "the read arrives before the write commits", and a history-sync batch commit is not short. That trade buys nothing here: the measured win is entirely on the Signal path, and these two reads contribute none of it. Both are back on the semaphore with the reason recorded, and listed in ON_THE_WRITE_QUEUE so the routing scan keeps accepting them.
`alternate_msg_secret_jid` resolves the peer's other namespace through `get_lid_mapping` / `get_pn_mapping` and feeds the result straight back into the message-secret lookup that was just moved back to the write queue for exactly this reason. That path has no cache in front of the backend (the one in `lid_pn.rs` is on a different caller), so a lookup racing `persist_and_migrate_lid_pn` reads the pre-write snapshot, resolves no alternate JID, and the addon is rejected -- the same terminal miss, one indirection earlier. Protecting the secret read and not the mapping read that decides which key it uses was half a fix. Both are back on the semaphore and listed in ON_THE_WRITE_QUEUE. `get_all_lid_mappings` stays on the read path: it is a bulk enumeration with no caller on the addon path. While here, `get_msg_secret` now delegates to `get_msg_secret_with_ts` and drops the timestamp instead of repeating the same filter chain with one column fewer, so the query and the routing rationale live in one place.
`ON_THE_WRITE_QUEUE` was consulted in one direction only: it excused a listed name and never checked the name still needed excusing. Migrate one of those reads later and forget the entry, and it stays there forever, silently excusing the next method that happens to share the name while its reason string quietly becomes false. `misrouted_reads` now returns the names it excused as well, and the test asserts that set equals the allowlist. Verified by migrating `get_pn_mapping` to `read_query` with its entry left in place: the assertion fires and names it. Also throttle the chunked-write race by 200us per sample. Both sides of that test go through `spawn_blocking` on the same pool, so back-to-back sampling competes with the writer for threads on a loaded machine and would eventually read as flake rather than as the regression it catches. Re-checked with the transaction removed: still fails on the first round, at 190/760. And the fallback comment in `read_erased` now says the free-snapshot claim holds at `pool_size = 1`, which is where it holds.
The fallback skipped the deferred read transaction on the grounds that holding the one pooled connection is the snapshot. True at `pool_size = 1`, and I justified leaving it there by calling anything above that unsupported. That was wrong twice: the config is reachable through `SqliteStoreConfig` and this crate's own tuning test uses `pool_size: 2, read_pool_size: 0`, and raising it does not deadlock by itself -- only two deferred read-then-write transactions racing do. Several writers check a connection out without taking the permit, so with a second connection available they can commit between the statements of `has_signal_state_for_user`, `load_prekeys_batch` or `get_app_state_mutation_macs_batch_for_device`. Condition the shortcut on what actually makes it true. `pool_size = 1` with no readers keeps the old path statement for statement, so the default costs nothing new; anything wider goes through the same deferred transaction the reader path uses. Test covers the wide-pool case.
Three more reads go back, and they share one shape with the msg-secret and LID/PN reverts before them: the row is promoted into a plain in-memory cache, or suppresses an action, so a stale read does not degrade to a retry -- it sticks. - `get_sender_key_devices` initializes `sender_key_device_cache`. A stale `has_key = true`, cached over a concurrent forget, drops the SKDM for the device that asked for redistribution, and the resend is undecryptable. - `get_devices` is promoted into `device_registry_cache` unconditionally on a miss, so a stale row overwrites a newer entry without advancing the topology generation and later sends omit a linked device. - `get_tc_token` feeds `prepare_privacy_token`'s scheduling decision, so reading before a concurrent touch commits issues a duplicate token and bypasses the configured interval. Outgoing sends are deliberately not per-chat serialized, so none of these three is protected by a lock. The distinction that decides the whole audit is now written next to the allowlist: `SignalStoreCache` reconciles staleness with its dirty set and incarnation, so the reads it mediates migrate; a cache that overwrites whatever it is handed does not. None of the three is on the measured path -- the Case B numbers are `get_session` -- so this costs nothing but the routing.
`a_multi_statement_read_is_snapshot_isolated_with_a_wider_write_pool` called `has_signal_state_for_user` twice with nothing writing in between, so it passed with or without the deferred transaction it was supposed to cover. I verified the other two tests by removing the mechanism and watching them fail; I did not do that here, and it showed. It now runs two SELECTs inside one `read_query` closure, parks between them, and commits through the pool's other connection while parked. With the snapshot removed the second query reads the new value and the test fails, which is the point. The routing scan also missed `self.shared().run(` -- the sibling-crate write path, reachable from a read-shaped method without touching any token it looked for. Added, and the scan now strips indentation before matching so a call rustfmt split across lines still reads as one token. Its self-test carries a `shared().run` offender alongside the raw-pool one.
…t teeth Two real defects, both mine, both introduced by earlier commits in this PR and caught by Codex. `build` declines reader connections under shared cache because a read transaction there holds table locks that fail the writer with SQLITE_LOCKED_SHAREDCACHE, which busy_timeout cannot absorb. The `pool_size > 1` fallback added two commits ago then opened exactly that transaction on the main pool, reintroducing the hazard the decline exists to avoid -- and `with_config_custom_tuning_builds_and_operates` already runs shared cache with `pool_size: 2`. The snapshot is now gated on the same condition that gates the reader pool, recorded as `snapshot_safe`, with a test pinning it. `a_chunked_batch_write_is_never_observed_half_applied` had gone vacuous. I verified it failed without the transaction, and then moved `get_sender_key_devices` onto the write permit, which serialized the sampler against the writer so it could no longer observe a torn batch. The claim stayed in the commit message; the test stopped backing it. It now samples through `read_query` directly, and fails at 190/760 with the transaction removed. `get_all_lid_mappings` also goes back to the write queue. The startup warm-up feeds it into `LidPnCache::add_guarded`, whose LID side replaces unconditionally, so a stale row read during a live learn reverts reverse resolution -- the same rule that moved the other cache-fed reads, which I had wrongly cleared as "bulk enumeration".
Three comment-accuracy fixes, all on claims I made too broadly. `read_query`'s doc said a write sent down the read path fails because reader connections are `query_only`. True on a reader connection, false on the fallback, which hands out an ordinary write connection -- so at the default `read_pool_size = 0` the net is absent and the routing scan is the only guard. The doc says that now, and the test says it too: it asserts the refusal with readers and asserts the gap without them, so the limit is recorded rather than assumed away. Enforcing `query_only` on a pooled write connection for the duration of a read would leave the pool poisoned if the closure unwound before the reset, which is a worse trade than documenting the gap. The `get_devices` and `get_tc_token` rationales named a concurrent writer the permit does not order them against: `update_device_list` and `touch_tc_token_sender_timestamp` check a connection out without it. The ordering does hold at the default, because the single pooled connection serializes them, so the comments now attribute it there. Routing those writers through the permit would change write serialization, which is out of scope for this change. `get_sender_key_devices` was also flagged and is fine as written: every writer of `sender_key_devices` (`set_sender_key_status`, `clear_sender_key_devices`, `delete_sender_key_device_rows`) takes the permit. `put_sender_key_for_device` and `delete_sender_key_for_device` skip it but write `sender_keys`, a different table. Also merged the two stacked rationale blocks in `read_erased` into one.
Left inconsistent by the previous commit, which corrected the same overclaim on get_devices and get_tc_token but not here.
The read helper's fallback took the write permit, which changed the default profile it was supposed to leave alone. Fourteen reads ran on a raw pooled connection without the permit before this branch -- `device_exists`, `load_device_data_for_device`, `get_sender_key_for_device`, `load_prekey`, `load_signed_prekey`, `load_all_signed_prekeys`, the four app-state reads, `has_same_base_key`, `get_group_metadata`, `get_all_tc_token_jids` -- and routing them through the helper put them behind it. At `pool_size = 1` that adds no serialization, since the single pooled connection already provides it, but it does serialize the `spawn_blocking` dispatch that the pool wait previously overlapped. Measured on 16 concurrent `get_sender_key` at the default profile: p50 569-641us before, 730-796us after, so roughly 25% for a knob nobody has turned on. The single-connection branch now checks the connection out directly with no permit, which is what those fourteen did and what the other six get from the connection anyway. Re-measured: p50 525-689us, back on top of main. A wider pool that cannot take a read transaction still uses the permit, since there the connection is no longer the serializer.
The clean audit pass over every migrated method turned up two more that
the incremental reviews had missed, both app-state key lookups whose
stale-absent answer is not a miss the caller retries.
`get_sync_key` answers a peer's `AppStateSyncKeyRequest`. On `None` the
handler returns an orphan `MessageField`, so a stale absent read tells
the peer we do not have a key we do have, on the wire.
`get_latest_sync_key_id` is unwrapped by `send_app_state_mutation` into
`InvalidRequest("no app state sync key available")`, which fails the
user's action outright with nothing retrying behind it.
Both race `set_sync_key`, which is exactly what an incoming key share
does. The version and mutation-mac reads stay on the read path: those
are internal to a sync pass that is serialized per collection, and a
stale read there re-syncs from an older version.
The binary size gate failed at +39.88 KiB .text against a 32 KiB budget. Attributing it by symbol against main puts the growth in three buckets: read_erased at +25.1 KiB over 186 instantiations, SharedSqlite::read at +16.1 KiB, and with_semaphore at +9.6 KiB as its instantiation count went from 77 to 188. read_erased carried three branches, two of which held their own pool checkout and spawn_blocking, so each was emitted per return type. The snapshot condition is the same predicate written the other way round, and once it returns early the remaining two branches differ only in whether a permit is taken. Folding them leaves one body per return type instead of two, with the permit as an Option. with_semaphore was generic over the closure as well as the return type, so every call site got its own copy of the acquire and spawn_blocking. Erasing the closure the same way read_query already does collapses it to one body per return type, which also shrinks the write path that predates this branch. Both are dispatch-shape changes: same predicate, same ordering, same observable behaviour. Measured on the demo example against main: .text +39.88 KiB -> +24.00 KiB, stripped +53.78 KiB -> +28.34 KiB.
The test asserted only that snapshot_safe was false, which is the mechanism rather than the consequence. Removing snapshot_safe from the routing predicate left it passing, so it did not cover the guard it was named for. It now parks a read mid-flight on a shared-cache store and requires a concurrent write to commit. A first attempt parked for 300ms and still passed unguarded: with_retry absorbs the lock across its 10/20/40/80/160ms backoff. Parking past that budget makes the lock fatal, and the unguarded run now fails with "database table is locked: sessions".
Tracing each routed read to the cold-load guard that covers it turned up two mappings that were not what this branch assumed. has_signal_state_for_user is consumed by SignalStoreCache::has_state_for_user, which is not one of the five functions the guard covers. It checks the two caches for any matching key and otherwise asks the backend, with no removal-seq re-check, so a load spanning a flush plus eviction can answer absent. Its callers use that answer to skip the PN to LID session migration entirely, and nothing retries the skip. It goes back on the write queue. has_session stays routed, but not for the reason recorded before. The cache's has_session reads get_session, not this method; the only path here is Device::contains_session, whose single production caller logs the result in all three branches. The justification is the caller, not the guard.
6dfe542 to
0cb522b
Compare
|
Rebased onto 1. RebaseClean. 2. Guard coverage, read by read -- and two of the five were not what this branch claimedThree mappings held exactly as expected: The other two were wrong:
3. Re-validation against the new baseChecked rather than assumed: #1227 touched three files this audit cites. The unconditional 4. Both tables re-measured on
|
| Pass | ran first | main |
branch | ratio |
|---|---|---|---|---|
| 1 | main |
532.1 us | 538.6 us | 1.01 |
| 2 | main |
609.2 us | 647.9 us | 1.06 |
| 3 | branch | 572.7 us | 545.7 us | 0.95 |
| 4 | branch | 456.9 us | 628.5 us | 1.38 |
No consistent sign, against a within-run spread of 1.65-1.70x. The 9% was ordering drift: whichever tree ran second was slower. So I am not claiming the default profile is unchanged -- I am saying this box cannot resolve the difference, and the body says that rather than rounding it to "identical". The structural argument is separate and stronger: after c780e8b both sides do spawn_blocking(pool.get(); query) with no permit, and the branch adds one Box.
For contrast, the 25% regression c780e8b fixed was resolvable here (569-641 vs 730-796 us, non-overlapping). Nothing that size remains.
5. CI
Green on 0cb522b: E2E, Build & Test, Build & Lint, Feature Matrix, Test Stable, Clippy, Format, Rustdoc, all four Miri jobs, wasm, Cargo Deny, all CodSpeed, Binary Size (+26.12 KiB .text against the 32 budget), and Semver Checks passed this time too.
Open reviewer threads, verified not blindly accepted
- CodeRabbit's "three reads stay on the write permit for ordering the permit does not provide": two-thirds valid and already documented as an explicit caveat; its fix would change write serialization, which is out of scope here. Its third bullet is wrong --
put_sender_key_for_device/delete_sender_key_for_devicewrite thesender_keystable, notsender_key_devices. I checked every writer ofsender_key_devices(set_sender_key_status,clear_sender_key_devices,clear_all_sender_key_devices,delete_sender_key_device_rows) and all four go throughwith_retry, so that hold is the one that genuinely orders at any pool size. The caveat in the body now names it as the exception. - Codex's P1 cold-load session race: correct, and it is precisely what fix(signal): reject cold loads that span a flush and eviction #1229 fixed. Closed by the dependency.
- Codex's prekey upload-window item: re-verified against the code rather than my earlier reasoning.
buffer_consumed_prekeydeliberately leaves the row in the backend until flush, with a test insignal_adapter.rspinning that ("must not delete from the backend before flush"). So the window is the whole write-behind interval on any routing; this PR widens it by one transaction. Prekey-durability question, not a routing one. Kept, reason in the body.
One thing I could not do
You asked me to strip the _Generated by [Claude Code]_ footers from the body and my comments. The body is clean and has been for several revisions. The 14 existing comments I cannot fix: no tool in this session can edit an issue comment, and the API token here is empty (403). The footers are appended server-side rather than typed by me, so new comments will likely keep getting one regardless. Flagging it instead of quietly leaving it done-looking.
Ready for review. Not merging and not touching labels.
Generated by Claude Code
Summary
db_semaphoreisSemaphore::new(pool_size)andpool_sizedefaults to 1. That default is correct for writes and I am not touching it: two deferred transactions that both read and then write deadlock on SQLite's upgrade, andbusy_timeoutcannot break it.What the same permit should not be doing is governing reads. A session, identity or sender-key miss on the decrypt path waits out whatever write is in flight,
put_sessions_batchfrom a write-behind flush included. Meanwhile the reader pool already existed:read_pool_sizebuilds a second,query_onlypool, and onmaintoday nothing outsideSharedSqlite::readandresource_reporttouchesself.reads. Turning the knob on buys the chat store concurrency and buys the Signal path nothing.Rebased onto
05b3c880. The base moved from3da38692through #1227 and #1229; neither touchedstorages/sqlite-storage/, and the rebase was clean.Audit
Read-only is necessary and not sufficient. The criterion is what the caller does with a stale-absent or stale-valued answer: if it recovers, the read can move; if the answer is promoted into a cache that overwrites what it is handed, is sent on the wire, or fails an operation outright, it stays on the write queue.
Cold-load guard coverage
Routing widens the window in which a load can span a flush plus an eviction. For the reads whose consumer sits behind
SignalStoreCache, the thing that closes that window is the guard inmain. Traced one read at a time rather than assumed, and two of the five did not land where this branch had recorded:get_session_for_devicecheckout_session,peek_session,has_session(all three readbackend.get_session)incarnationandremoved_sinceafter the I/O, so a superseded record is retried rather than installedload_identity_for_deviceget_identityget_sender_key_for_deviceget_sender_keyhas_session(backend method)Device::contains_sessionviacheck_session_existshas_signal_state_for_userSignalStoreCache::has_state_for_userTwo corrections this produced:
has_signal_state_for_userleaves the migrated set. Its consumer is not one of the guarded five.has_state_for_userchecks both caches for any matching key and otherwise asks the backend with no removal-seq re-check, and its callers inlid_pn.rsuse afalseto skip the PN to LID session migration outright. Nothing retries the skip. It goes back on the write queue.has_session's justification was wrong, though its verdict was not. The cache'shas_sessionreadsget_session, not this method, so fix(signal): reject cold loads that span a flush and eviction #1229 never covered it. The only path here isDevice::contains_session, whose single production caller logs. The reason recorded next to it is now the caller, not the guard.Re-validation against the new base
#1227 touched three files this audit cites (
device_registry.rs,chat_actions.rs,groups.rs) and #1229 touchedsignal_cache.rs. Re-checked the cited behaviours: the unconditionaldevice_registry_cache.promoteon a backend hit and theInvalidRequest("no app state sync key available")on aNonelatest-key id both still hold, unchanged. Neither commit changes what a caller does with a stale answer for any of the other reads. The audit is current with05b3c880, not inherited from3da38692.Migrated:
get_session_for_device(get_session)load_identity_for_device(load_identity)get_sender_key_for_device(get_sender_key)has_sessionload_prekeyload_prekeys_batchget_max_prekey_idprekey_upload_lockand the prior write is awaitedload_signed_prekey,load_all_signed_prekeysrotate_keyretriesget_app_state_version_for_device(get_version)get_app_state_mutation_mac_for_device(get_mutation_mac)get_app_state_mutation_macs_batch_for_device(get_mutation_macs)has_same_base_keyget_group_metadataget_all_tc_token_jidsdevice_exists,load_device_data_for_deviceNote on the prekey reads.
buffer_consumed_prekeyrecords the id inremoved_prekeysandflushdeletes the row later, once the promoted session is durable, so a consumed prekey is readable for the whole write-behind interval by design;signal_adapter.rshas a test pinning exactly that. Routing extends the window by one transaction. Forload_prekeythe question is moot: re-reading a still-present prekey is the duplicate-pkmsg path. Forload_prekeys_batchthe consequence differs, since the upload window re-offers rows to the server, but that is reachable today through the buffer interval with no routing change. It is a prekey-durability question, not a routing one.Not migrated (11), each with its reason in
ON_THE_WRITE_QUEUE:has_signal_state_for_user-- new in this pass, see the guard table above.get_app_state_sync_key_for_device(get_sync_key) -- answers a peer'sAppStateSyncKeyRequest; onNonethe handler returns an orphanMessageField, so a stale absent read tells the peer we lack a key we hold, on the wire.get_latest_app_state_sync_key_id_for_device--send_app_state_mutationturnsNoneintoInvalidRequestand the user's action fails with nothing retrying behind it.get_msg_secret_with_ts(andget_msg_secret, which delegates) -- a miss is terminal; the reaction, vote or edit is dropped. History sync seeds secrets viaput_msg_secretsdirectly, bypassing the live write-behind buffer.get_lid_mapping,get_pn_mapping--alternate_msg_secret_jidresolves the peer's other namespace through these into the lookup above, with no cache in front on that path.get_all_lid_mappings-- the startup warm-up feeds these intoLidPnCache::add_guarded, whose LID side replaces unconditionally.get_sender_key_devices-- initializessender_key_device_cache; a stalehas_key = truedrops the SKDM for a device that asked for redistribution.get_devices-- promoted intodevice_registry_cacheunconditionally on a miss, overwriting a newer entry.get_tc_token-- feedsprepare_privacy_token's scheduling decision; a stale read issues a duplicate token.get_pending_inbound-- read-only SQL, but itswith_retryloop exists so a transient BUSY does not fail closed into a redelivery.take_sent_messageandstore_received_tc_tokenread and write in oneimmediate_transactionand were never candidates.snapshot_dbandresource_reportare not queries.Ordering caveat, stated once: most of the write-queue holds above are ordered by the single pooled connection at the default
pool_size, not by the permit itself, since several writers check a connection out without taking it. Abovepool_size = 1that ordering weakens.get_sender_key_devicesis the exception: every writer ofsender_key_devices(set_sender_key_status,clear_sender_key_devices,clear_all_sender_key_devices,delete_sender_key_device_rows) goes throughwith_retry, so that one holds at any pool size. The rest predates this PR and fixing it means changing write serialization, which is out of scope.Default-config behaviour
An earlier revision of this description said the
read_pool_size = 0path was unchanged. That was wrong.Fourteen reads ran on a raw pooled connection without the permit before this branch. Routing them through the helper put them behind it, which measured about 25% on p50 at the default profile -- a real regression for a knob nobody has turned on.
c780e8bfixed it: the single-connection branch checks the connection out directly and takes no permit, because the one pooled connection is already both the serialization and the snapshot. Adding a permit there only serialized thespawn_blockingdispatch that the pool wait was overlapping.After that fix the default path differs from
mainby exactly one heap allocation: both dospawn_blocking(pool.get(); query)with no permit, and this branch boxes the closure first. See Cost for what the measurement can and cannot say about it.E2E
An earlier revision listed an E2E regression as a blocker. It was not one, and the mistake was mine: I saw
E2E Testsfail on one commit, checked thatmainwas green, and concluded regression without looking at this branch's own history.bcabb059923ffb201811577906ae68f1a4939b9a386c68e3a4f9bb6dOne red run among seven greens on the same branch is a flake, and the five greens before it would have said so immediately. Every E2E run since has been green as well, including on the rebased head.
Independently, the suite was run against a local mock server on both trees: 165 run / 163 passed / 2 failed on this branch and the same two on
main(presence::test_presence_available,privacy_tokens::test_restricted_presence_subscribe_requires_tctoken), i.e. local-mock drift. I could not reproduce that myself -- this environment has no mock server -- so that half is someone else's evidence and the conclusion above rests on CI history.Dependency
Resolved. The guard landed in
05b3c880(#1229) and this branch is rebased on it.The cold-load re-check in
checkout_session,peek_session,has_sessionandget_identitypreviously asked only whether an entry exists, so it could not tell "never written" from "written, persisted and removed"; a load spanning a flush plus eviction installed the older record, and forcheckout_sessionthat record goes to the cipher and advances the ratchet. #1229 stampsincarnationandremoval_seqaround the unlocked backend read and retries, bounded byUNLOCKED_COLD_READ_ATTEMPTS, with a locked fallback. That is what makes routing safe for the three reads whose consumers sit behind it, per the coverage table above.Changes
SqliteStore::read_query-- erases the closure, then routes: the deferred snapshot viaSharedSqlite::readwhen a reader pool is configured or a wider write pool can take one; otherwise a direct connection checkout, taking the permit only when the pool is wide enough that the connection is no longer the serializer.with_semaphoreerased the same way, so the write queue emits one body per return type instead of one per call site.get_msg_secretdelegates toget_msg_secret_with_tsrather than repeating its filter chain a column narrower.read_pool_size0 and 4; a write refused byquery_onlyon a reader connection, and the fallback's lack of that net asserted rather than assumed; a read while the write permit is held; a read against an open write transaction; a multi-statement read under a wider write pool, verified to fail without the snapshot; a shared-cache store declining that snapshot, with a write required to commit under a parked read and verified to fail without the guard; the chunked-write race, verified to fail without the transaction; a bidirectional source scan, so both a newly misrouted read and a stale allowlist entry fail.Cost
In-process harness, file-backed store,
taskset -c 0-3, release, deleted before this PR and never declared as a[[bench]]. Notdivan: the number that matters is read tail latency with a write in flight, anddivanmeasures isolated throughput. Both tables re-measured on05b3c880.The win
16 concurrent
get_sessionper round, 40 rounds, over 512 seeded 2 KiB rows, 3 rounds. Case B adds a continuousput_sessions_batchof 1500 rows. Both configurations run in one process, so drift hits them equally.read_pool_size = 0read_pool_size = 4Case B is the point: p50 ~88 ms down to ~0.59 ms, a factor of ~150. That is far above any noise this box produces.
The default profile, where this PR could make something worse
16 concurrent
get_sender_key, 7 rounds per run,read_pool_size = 0, this branch againstmain. Run in alternating passes because a single ordering confounds machine drift with the code:mainp50 medianmainmainmainThe difference is not resolvable on this box, and I am not claiming it is zero. The ratio has no consistent sign across the four passes, and the within-run spread is 1.65x for
mainand 1.70x for the branch -- larger than the effect being looked for. A first, non-alternating attempt suggested about 9%; alternating showed that was ordering drift, since the second run of any pair was systematically slower.What can be said without the measurement: after
c780e8bthe default path isspawn_blocking(pool.get(); query)with no permit on both sides, and the branch adds oneBoxfor the erased closure. That is the whole delta. It is stated as a structural argument, not as a measured equivalence.For contrast, the 25% regression
c780e8bfixed was resolvable at this noise floor: 569-641 us against 730-796 us, non-overlapping across rounds. Nothing of that size remains.Resolved by instruction counting
Walltime on this box cannot see an effect this small, but instruction counting can. Measured off-CI by driving the CodSpeed instrumentation directly under Valgrind (
CODSPEED_ENV=1 valgrind --tool=callgrind, which makes the harness reportMeasuredinstead ofCheckedwithout needing a token or an upload), on a temporary bench that exercises the default profile (read_pool_size = 0) against a file-backed store. Three runs per side, medians in instructions:mainmainget_sessionmissget_sessionhitload_identityInstruction counts are not automatically deterministic, and it is worth being explicit about that: running the same binary twice differed by about 550 instructions until the bench's runtime was switched from
multi_threadtocurrent_thread, because work-stealing decisions vary run to run. The residual spread onmainis 307-1,265. So only the miss has a delta larger than its own noise, and the reading is an upper bound rather than a point estimate: the default profile changes by at most about 1%, and where the signal exceeds the noise its sign is a reduction, not a regression.That is consistent with the structural argument above and puts a number on it: one extra
Boxcosts hundreds of instructions, not the 9% the first non-alternating walltime attempt suggested. The bench is temporary and is not part of this diff.Binary size
The gate failed at one point at +39.88 KiB .text against a 32 KiB budget. An
nmsymbol diff of thedemoexample put it in three buckets:read_erasedat +25.1 KiB over 186 instantiations,SharedSqlite::readat +16.1 KiB, andwith_semaphoreat +9.6 KiB as its instantiation count went 77 to 188. Foldingread_erased's two non-snapshot branches into one and erasingwith_semaphore's closure fixed it. Current reading against the05b3c880baseline: +26.12 KiB .text (budget 32) and +31.31 KiB stripped (budget 64).Validation
No existing test was relaxed. One was strengthened:
a_shared_cache_store_gets_no_snapshot_even_with_a_wider_write_poolasserted only thesnapshot_safeflag and stayed green when the guard was deleted from the routing predicate, so it now parks a read mid-flight and requires a concurrent write to commit. That is a contract change in the stricter direction and it found a real limit -- the park has to outlastwith_retry's ~310 ms backoff, or the shared-cache lock is retried away rather than observed.