test(signal): add deterministic durability chaos coverage

[jlucaso1]

Why

Signal counter leases are security-critical, but the existing tests mostly cover one interleaving at a time. We need repeatable coverage for failures that only appear when cancellation, cache lifecycle, failed storage, tombstones, and recovery interact.

What changed

• Added a deterministic DM + group state machine with replayable seeds. It exercises failed pre-wire writes, covered sends, cancellation, checkout/flush overlap, clean eviction, lossy clears, crash reloads, tombstones, out-of-order delivery, receiver sender-key loss, and retry/SKDM recovery.
• Published DM and group messages are checked against the exact derived cipher key and IV, so the suite detects key-material reuse rather than relying only on logical counters.
• Added a real SQLite restart test. A child persists both leases, advances them without flushing, dies via SIGKILL, and the parent verifies recovery at the reservation ceiling plus exact same-incarnation reloads.
• Added a small smoke matrix to normal CI and a scheduled/manual nightly workflow for the 128 × 256 matrix and SQLite crash test. Manual replay rejects malformed or out-of-range inputs, and workflow actions are commit-pinned with checkout credentials removed before tests.
• Added agent_docs/signal_durability.md with the invariants, publication boundary, lifecycle rules, tuning constraints, and review checklist.

Everything that executes is test-only (cfg(test), dev dependencies, or CI), so this adds no production allocations or hot-path work.

Lease-size check

I also checked the supplied production trace. It contains 12 logical group messages, 23 outbound stanzas after 11 pairwise retry resends, and one sender-key creation. Since group retries use pairwise Signal rather than spending sender-key iterations, the observed sender-key run is consistent with 12 iterations. That supports batching, but the log does not expose counters and is too short to prove that 64 is optimal.

Keeping 64 is still a conservative tradeoff: one crash burns at most 3.2% of the peer forward-jump bound (64 / 2000), while a monotonic chain needs at most one synchronous reservation write per 64 sends. The new doc explicitly requires run-length/restart telemetry before retuning it and calls out cumulative real-crash gaps.

Validation

• cargo fmt --all -- --check
• cargo clippy --all --tests -- -D warnings
• cargo test --workspace --exclude e2e-tests --no-fail-fast
• cargo test -p wacore --no-default-features --lib
• 128 seeds × 256 actions in the ignored nightly state machine
• Exact replay of the matrix-discovered cross-gate fault-injection interleaving
• Rejection checks for malformed seeds and out-of-range matrix sizes
• SQLite SIGKILL/restart test
• actionlint .github/workflows/signal-durability-nightly.yml

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds deterministic signal-cache chaos tests, SQLite subprocess crash-recovery tests, durability documentation, and a configurable nightly GitHub Actions workflow with failure artifacts.

Changes

Signal durability validation

Layer / File(s)	Summary
Deterministic cache durability harness wacore/src/store/signal_cache_durability_chaos.rs, wacore/src/store/signal_cache.rs, wacore/Cargo.toml	Adds randomized DM and group sender-key actions, flush-failure injection, cache reload and clearing scenarios, invariant checks, deterministic replay, and smoke/nightly test entry points.
SQLite process-restart recovery tests/signal_durability_sqlite.rs	Runs signal-state operations in a child process, terminates it before final persistence, reloads SQLite state, and verifies counters, fingerprints, flushing, and cleanup.
Durability documentation and review contract agent_docs/signal_durability.md, AGENTS.md	Documents counter leases, cache gates, publication ordering, deletion and teardown semantics, review checks, and verification procedures.
Nightly durability execution .github/workflows/signal-durability-nightly.yml	Adds scheduled and manual workflow execution for both tests, configurable seeds and steps, toolchain setup, and conditional SQLite artifact upload.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• oxidezap/whatsapp-rust#1022: Changes coalesced flush behavior directly exercised by the durability tests.
• oxidezap/whatsapp-rust#1026: Introduces pre-wire gate and lease behavior validated by the new tests.
• oxidezap/whatsapp-rust#1048: Updates encryption-path pre-wire persistence semantics covered by this suite.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: deterministic durability chaos coverage for Signal state.
Description check	✅ Passed	The description is directly aligned with the changeset and its test-only, CI, and documentation additions.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/signal-durability-chaos

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📦 Binary size report

Metric	main	PR	Δ
bin size (stripped)	10.93 MiB	10.93 MiB	0
bin .text	8.91 MiB	8.91 MiB	0
bin allocated (text+data+bss)	10.93 MiB	10.93 MiB	0
llvm-lines wacore	514,696	514,696	0
llvm-lines wacore copies	17,572	17,572	0
llvm-lines whatsapp-rust lib	777,662	777,662	0
llvm-lines whatsapp-rust lib copies	25,303	25,303	0
deps crates (Cargo.lock)	472	472	0

.text per crate

Crate	main	PR	Δ
.text whatsapp_rust	1.69 MiB	1.69 MiB	0
.text wacore	537.22 KiB	537.22 KiB	0
.text wacore_binary	148.45 KiB	148.45 KiB	0
.text wacore_libsignal	203.49 KiB	203.49 KiB	0
.text wacore_appstate	158.33 KiB	158.33 KiB	0
.text wacore_noise	26.03 KiB	26.03 KiB	0
.text waproto	1.60 MiB	1.60 MiB	0
.text whatsapp_rust_sqlite_storage	513.08 KiB	513.08 KiB	0
.text whatsapp_rust_tokio_transport	43.79 KiB	43.79 KiB	0
.text whatsapp_rust_ureq_http_client	10.47 KiB	10.47 KiB	0
.text std	1.02 MiB	1.02 MiB	0
.text other deps	2.95 MiB	2.95 MiB	0

Baseline: c9081101a (latest main run) · Head: 11265305d · Graphs

[greptile-apps]

Greptile Summary

This PR adds test-only durability coverage for Signal counter leases: a deterministic in-memory chaos state machine (DM + group sends, cancellation, failed flushes, tombstones, crash/clean/lossy reloads) and a real SQLite subprocess crash test that kills the child with SIGKILL and verifies the parent recovers to the reservation ceiling. It also adds a nightly CI workflow and a signal_durability.md developer reference with the invariants and review checklist. No production code changes.

• Chaos harness (signal_cache_durability_chaos.rs): 815-line state machine with 13 action types, replayable seeds via SIGNAL_CHAOS_SEED, per-step 2-second timeouts, and key/IV uniqueness sets for both DM and group paths.
• SQLite crash test (tests/signal_durability_sqlite.rs): spawns the current test binary as a child, verifies SIGKILL termination, then checks that recovery fast-forwards exactly to SENDER_CHAIN_RESERVATION_BATCH and that same-incarnation reloads do not burn additional lease counters.
• Nightly workflow: all action refs pinned to commit SHAs; artifacts uploaded on failure for post-mortem.

Confidence Score: 5/5

All changes are test-only (cfg(test), dev-dependencies, CI); no production code path is touched.

Every changed file is test infrastructure or documentation. The chaos harness and SQLite crash test cover the security-critical invariants they claim, CI action refs are pinned to commit SHAs, and libc/uuid/tokio are all already in dev-dependencies. The two findings are test design observations that narrow but do not break the stated coverage.

wacore/src/store/signal_cache_durability_chaos.rs — the dm_cancel fingerprint gap and the pre-encrypt group fingerprint coupling are both worth a second look before the harness is cited as comprehensive proof against counter reuse.

Important Files Changed

Filename	Overview
wacore/src/store/signal_cache_durability_chaos.rs	New 815-line deterministic chaos harness; main concerns are the burned fingerprint blind-spot in dm_cancel and the pre-encrypt fingerprint capture in group_send
tests/signal_durability_sqlite.rs	SIGKILL crash + SQLite restart test; libc already in dev-deps, recovery assertions correctly anchor to SENDER_CHAIN_RESERVATION_BATCH
.github/workflows/signal-durability-nightly.yml	New nightly workflow; all action refs are pinned to commit SHAs, concurrency group is correct, artifacts preserved on failure
agent_docs/signal_durability.md	New developer doc covering lease invariants, publication ordering, cancellation/tombstone rules, and review checklist — well-aligned with the cache implementation
wacore/src/store/signal_cache.rs	Adds #[path] module inclusion of the chaos test module; no production code changed
wacore/Cargo.toml	Adds tokio time feature to dev-dependencies for tokio::time::timeout used in run_seed
AGENTS.md	Adds reference to new signal_durability.md agent doc alongside existing docs

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[ChaosHarness::new] --> B[initial DM send + group setup]
    B --> C{SplitMix64 action}

    C -->|DmSend| D[checkout_session]
    D --> E[derive fingerprint\nadvance chain]
    E --> F{fail_gate?}
    F -->|Yes| G[inject write failure\ngate stays pending]
    F -->|No| H[flush succeeds\npublish to wire]
    H --> I[published_dm.insert]

    C -->|DmCancel| J[checkout_session\nadvance chain\nNO fingerprint tracked]
    J --> K[commit_dm\ngate may be pending]

    C -->|GroupSend| L[get_sender_key\ncompute fingerprint pre-encrypt]
    L --> M[group_encrypt]
    M --> N{fail_gate?}
    N -->|No| O[published_group.insert\npending_group.push]

    C -->|CleanReload| P[flush + clear_after_flush\nverify counters unchanged]
    C -->|CrashReload| Q[new SignalStoreCache\nnew incarnation\nreads from backend ceil]
    C -->|LossyClear| R[clear_with_incarnation\nstale checkout rejected]
    C -->|CheckoutDuringFlush| S[checkout + flush\ngated session stays gated]
    C -->|RecoverGroup| T[delete receiver key\ndeliver triggers SKDM recovery]

    P --> U[assert_invariants]
    Q --> U
    R --> U
    S --> U
    T --> U
    I --> U
    G --> U
    K --> U
    O --> U

    U --> C

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[ChaosHarness::new] --> B[initial DM send + group setup]
    B --> C{SplitMix64 action}

    C -->|DmSend| D[checkout_session]
    D --> E[derive fingerprint\nadvance chain]
    E --> F{fail_gate?}
    F -->|Yes| G[inject write failure\ngate stays pending]
    F -->|No| H[flush succeeds\npublish to wire]
    H --> I[published_dm.insert]

    C -->|DmCancel| J[checkout_session\nadvance chain\nNO fingerprint tracked]
    J --> K[commit_dm\ngate may be pending]

    C -->|GroupSend| L[get_sender_key\ncompute fingerprint pre-encrypt]
    L --> M[group_encrypt]
    M --> N{fail_gate?}
    N -->|No| O[published_group.insert\npending_group.push]

    C -->|CleanReload| P[flush + clear_after_flush\nverify counters unchanged]
    C -->|CrashReload| Q[new SignalStoreCache\nnew incarnation\nreads from backend ceil]
    C -->|LossyClear| R[clear_with_incarnation\nstale checkout rejected]
    C -->|CheckoutDuringFlush| S[checkout + flush\ngated session stays gated]
    C -->|RecoverGroup| T[delete receiver key\ndeliver triggers SKDM recovery]

    P --> U[assert_invariants]
    Q --> U
    R --> U
    S --> U
    T --> U
    I --> U
    G --> U
    K --> U
    O --> U

    U --> C

Loading

_{Reviews (2): Last reviewed commit: "test(signal): harden durability recovery..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/signal-durability-nightly.yml:
- Line 41: Update the actions/checkout step in the nightly workflow to disable
credential persistence, ensuring the repository token is not available to
subsequent build or test processes.

In `@tests/signal_durability_sqlite.rs`:
- Around line 204-209: Update the async test call sites that invoke
remove_database to run the cleanup via tokio::task::spawn_blocking, matching the
existing child-process pattern. Move the Path argument into the blocking closure
as needed, await the task, and preserve cleanup of the database, WAL, and SHM
files.

In `@wacore/src/store/signal_cache_durability_chaos.rs`:
- Around line 668-684: Update env_u64 and env_usize to reject malformed or
out-of-range environment values instead of defaulting or clamping them. Report
the variable name and supplied value when parsing or validation fails, while
preserving valid hexadecimal seeds, decimal values, and counts within the
supported range.
- Around line 93-110: Make the group recovery path reachable by adding a
deterministic chaos action that invalidates receiver sender-key state or forces
the forward-jump boundary, then handle it in the action execution and related
group-delivery/retry logic. Update the action selection in action, the Action
handling around the referenced group recovery branches, and any queue setup
needed so NoSenderKeyState or too-far-future recovery is exercised without
exceeding the existing queue constraints.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8f957733-5fd8-4fe2-b527-d95ab6c18358

📥 Commits

Reviewing files that changed from the base of the PR and between c908110 and ab74fdb.

📒 Files selected for processing (7)

• .github/workflows/signal-durability-nightly.yml
• AGENTS.md
• agent_docs/signal_durability.md
• tests/signal_durability_sqlite.rs
• wacore/Cargo.toml
• wacore/src/store/signal_cache.rs
• wacore/src/store/signal_cache_durability_chaos.rs