Skip to content

fix(cluster): reject a truncated v3 gossip header instead of panicking the process - #495

Merged
TinDang97 merged 1 commit into
mainfrom
fix/gossip-v3-header-oob
Aug 14, 2026
Merged

fix(cluster): reject a truncated v3 gossip header instead of panicking the process#495
TinDang97 merged 1 commit into
mainfrom
fix/gossip-v3-header-oob

Conversation

@TinDang97

@TinDang97 TinDang97 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

The defect

Gossip wire v3 (#493) appended a 40-byte sender_master_id to the header. The length guard at the top of deserialize_gossip was deliberately left at the smaller v2 header size (2130) so a genuine v2 peer would still parse — but the v3 branch below it then read data[HEADER_SIZE_V2..HEADER_SIZE] (data[2130..2170]) unconditionally.

Any frame declaring version ≥ 3 with a length in 2130..2170 indexes past the end of the slice.

Why this is a remote DoS, not a bad parse

bus.rs reads a peer-supplied length (capped at 64 KiB — 2130 passes), reads exactly that many bytes, and hands them straight to deserialize_gossip. Measured against a cluster-enabled server built from the merged ac2b036d, sending one unauthenticated 2130-byte frame to the bus port:

thread 'cluster-ctl' panicked at src/cluster/gossip.rs:278:47:
range end index 2170 out of range for slice of length 2130
FATAL: thread 'cluster-ctl' panicked; aborting the whole process rather than
serving with a dead shard or a dead cluster control plane

The process is gone; the next PING is connection-refused. Any cluster-enabled node can be killed by a single unauthenticated frame.

The fix

A v3 sender always writes the full 40 bytes, so a short v3 header is malformed and is now rejected with an error rather than zero-filled — fail-closed. The loose guard that the v2 back-compat path depends on is untouched.

Evidence

  • Red: test_deserialize_rejects_v3_header_truncated_inside_master_id walks every truncation point in 2130..2170; without the fix it panics at gossip.rs:278:47.
  • Green: same test passes; re-verified end to end — the identical attack frame now leaves the server alive and answering PING, zero panics in the log.
  • Lib suite 4636 passed / 0 failed; cluster_client_bootstrap 20/20; fmt --check clean; clippy --all-targets -D warnings clean on both feature sets.

Why the fuzzer missed it

gossip_deser was already correct and already green on both #486 and #493. It had simply not synthesised a valid 4-byte magic ("Redi") together with that exact 40-byte length window inside the 15-minute PR budget. Seeds for the panicking shape (2130 / 2140 / 2169) and for the legitimate v2-exact frame are added to fuzz/corpus/gossip_deser, so the window is covered from the first iteration.

Found by an adversarial review of the merged cluster work.

Refs #493

Summary by CodeRabbit

  • Bug Fixes

    • Fixed handling of truncated version 3 cluster gossip messages.
    • Malformed messages are now safely rejected instead of potentially causing a process panic.
    • Added regression coverage for truncated message headers.
  • Documentation

    • Added a changelog entry describing the fix.

…g the process

Gossip wire v3 (#493) appended a 40-byte `sender_master_id` to the header. The
length guard at the top of `deserialize_gossip` was deliberately left at the
SMALLER v2 header size (2130) so a genuine v2 peer would still parse — but the
v3 branch below it then read `data[HEADER_SIZE_V2..HEADER_SIZE]`
(`data[2130..2170]`) unconditionally. Any frame declaring version 3 or above
with a length in `2130..2170` indexed past the end of the slice.

That is reachable from the network, not just from a fuzzer. `bus.rs` reads a
peer-supplied length (capped at 64 KiB, which 2130 passes), reads exactly that
many bytes, and hands them straight to `deserialize_gossip`. Measured against a
cluster-enabled server built from ac2b036: one unauthenticated 2130-byte frame
to the bus port produces

    thread 'cluster-ctl' panicked at src/cluster/gossip.rs:278:47:
    range end index 2170 out of range for slice of length 2130
    FATAL: thread 'cluster-ctl' panicked; aborting the whole process

and the server is gone — subsequent PING is connection-refused. So the impact is
a full remote denial of service on any cluster-enabled node, not a dropped
connection.

A v3 sender always writes the full 40 bytes, so a short v3 header is malformed
and is now rejected with an error rather than zero-filled — fail-closed, and it
leaves the v2 back-compat path (which is what the loose guard exists for)
untouched.

Red/green: `test_deserialize_rejects_v3_header_truncated_inside_master_id` walks
every truncation point in `2130..2170` and panics at 278:47 without the fix.
Re-verified end to end after the fix: the same frame now leaves the server alive
and answering PING, with zero panics in the log.

The `gossip_deser` fuzz target was already correct and already green on both
#486 and #493 — it simply had not synthesised a valid 4-byte magic together with
that exact 40-byte length window inside its 15-minute PR budget. Seeds for the
panicking shape and for the legitimate v2-exact frame are added to
`fuzz/corpus/gossip_deser` so the window is covered from the first iteration.

Found by an adversarial review of the merged cluster work.

Refs #493
author: Tin Dang
@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@TinDang97 TinDang97 added the ci-full Run the full integration-test matrix on this PR label Aug 14, 2026
@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c48b5d84-d50b-41aa-ac37-00c6827ec913

📥 Commits

Reviewing files that changed from the base of the PR and between ac2b036 and 33d8899.

📒 Files selected for processing (6)
  • CHANGELOG.md
  • fuzz/corpus/gossip_deser/v2_header_exact
  • fuzz/corpus/gossip_deser/v3_truncated_master_id_2130
  • fuzz/corpus/gossip_deser/v3_truncated_master_id_2140
  • fuzz/corpus/gossip_deser/v3_truncated_master_id_2169
  • src/cluster/gossip.rs

📝 Walkthrough

Walkthrough

The gossip deserializer now rejects truncated version-3 headers before reading sender_master_id. A regression test covers every truncation point, and the changelog documents the fix.

Changes

Gossip header safety

Layer / File(s) Summary
Header validation and regression coverage
src/cluster/gossip.rs, CHANGELOG.md
The deserializer returns an error when a v3-or-newer header ends inside sender_master_id. The regression test checks every truncation point. The changelog documents the fix and fuzz coverage.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 33d88

The change rejects truncated v3 gossip headers instead of allowing a process crash, and no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: pilotspacex-byte

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main fix: rejecting truncated v3 gossip headers instead of allowing a process panic.
Description check ✅ Passed The description thoroughly explains the defect, fix, impact, tests, and validation, but does not use the template headings or include explicit performance and notes sections.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/gossip-v3-header-oob

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@TinDang97 TinDang97 added the ci-fuzz Run the fuzz-pr CI job (15 min/target) on this PR label Aug 14, 2026
@TinDang97
TinDang97 merged commit 5430cf6 into main Aug 14, 2026
44 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-full Run the full integration-test matrix on this PR ci-fuzz Run the fuzz-pr CI job (15 min/target) on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant