-
Notifications
You must be signed in to change notification settings - Fork 0
fix(protocol): encode every reply in the protocol in effect when it was produced #496
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -2,7 +2,7 @@ | |||||
|
|
||||||
| slug: cluster-client-bootstrap · created: 2026-08-09 · stage: production | ||||||
| autonomy: auto <!-- inherited from the project default (PROJECT.md); explicit level: manual < conservative < auto (visible · overridable) — lower below if a high-risk task needs it, or run `add.py autonomy set`. --> | ||||||
| phase: build <!-- ground -> specify -> scenarios -> contract -> tests -> build -> verify -> observe -> done --> | ||||||
| phase: done <!-- ground -> specify -> scenarios -> contract -> tests -> build -> verify -> observe -> done --> | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- task file excerpt ---'
sed -n '1,12p;495,510p;612,630p;656,680p' .add/tasks/cluster-client-bootstrap/TASK.md
printf '%s\n' '--- state entry ---'
python3 - <<'PY'
import json
from pathlib import Path
p = Path(".add/state.json")
data = json.loads(p.read_text())
needle = "cluster-client-bootstrap"
def walk(value, path=""):
if isinstance(value, dict):
for k, v in value.items():
child = f"{path}.{k}" if path else k
if needle in str(k) or needle in str(v):
print(child, "=", json.dumps(v, ensure_ascii=False))
walk(v, child)
elif isinstance(value, list):
for i, v in enumerate(value):
walk(v, f"{path}[{i}]")
walk(data)
PY
printf '%s\n' '--- repository diff summary and relevant diff ---'
git diff --stat -- .add/tasks/cluster-client-bootstrap/TASK.md .add/state.json
git diff -- .add/tasks/cluster-client-bootstrap/TASK.md .add/state.json | sed -n '1,240p'
printf '%s\n' '--- references to the task and review PRs ---'
rg -n -C 2 'cluster-client-bootstrap|`#486`|`#493`|`#495`|human.review|human review|completion' .add .github 2>/dev/null || trueRepository: pilotspace/moon Length of output: 50371 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- task gate and completion context ---'
sed -n '580,640p' .add/tasks/cluster-client-bootstrap/TASK.md
sed -n '640,690p' .add/tasks/cluster-client-bootstrap/TASK.md
printf '%s\n' '--- completion resolver ---'
sed -n '570,625p' .add/tooling/add.py
sed -n '680,730p' .add/tooling/add.py
printf '%s\n' '--- verify and human-gate rules ---'
rg -n -C 4 'person reviewed|human review|human gate|autonomy|Outcome: PASS|RISK-ACCEPTED|HARD-STOP|security' .add/docs .add/tooling/add.py .add/tasks/cluster-client-bootstrap/TASK.md | head -n 260
printf '%s\n' '--- task-specific state shape, parsed compactly ---'
python3 - <<'PY'
import json
from pathlib import Path
task = json.loads(Path('.add/state.json').read_text())['tasks']['cluster-client-bootstrap']
print(json.dumps(task, indent=2, sort_keys=True))
PYRepository: pilotspace/moon Length of output: 38767 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- process documentation for Verify and human approval ---'
rg -n -C 5 'Verify.*human|human.*Verify|person reviewed|approved|auto-gate|auto-PASS|PASS.*human|human.*gate' .add/docs .add/tooling/templates .add/tasks | head -n 360
printf '%s\n' '--- comparable completed tasks ---'
python3 - <<'PY'
from pathlib import Path
import re
for p in Path('.add/tasks').glob('*/TASK.md'):
text = p.read_text(errors='replace')
if 'phase: done' in text and re.search(r'- \[ \].*(person reviewed|human review|human.*approv)', text, re.I):
print(f'### {p}')
for i, line in enumerate(text.splitlines(), 1):
if re.search(r'phase: done|person reviewed|human review|GATE RECORD|Outcome:|Reviewed by:', line, re.I):
print(f'{i}: {line}')
PY
printf '%s\n' '--- check/status implementation around task completion ---'
rg -n -C 6 '_task_done|phase.*done|gate.*PASS|incomplete|human gate' .add/tooling/add.py | head -n 300Repository: pilotspace/moon Length of output: 50373 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- auto-gate and residue implementation ---'
rg -n -C 8 'auto-resolv|auto.*PASS|residue|verify.*residue|flag_verified|accountable owner|person reviewed' .add/tooling/add.py .add/docs/08-step-6-verify.md .add/docs/11-governance.md .add/tasks/cluster-client-bootstrap/TASK.md | head -n 360
printf '%s\n' '--- exact task checklist and gate markers in comparable auto tasks ---'
python3 - <<'PY'
from pathlib import Path
import re
for p in Path('.add/tasks').glob('*/TASK.md'):
text = p.read_text(errors='replace')
if 'autonomy: auto' not in text or 'phase: done' not in text:
continue
lines = text.splitlines()
hits = [(i+1, line) for i, line in enumerate(lines)
if re.search(r'person reviewed|approved the change|auto-resolved|Outcome:|Reviewed by:', line, re.I)]
if hits:
print(f'### {p}')
for n, line in hits[:12]:
print(f'{n}: {line}')
PYRepository: pilotspace/moon Length of output: 38001 🏁 Script executed: #!/bin/bash
set -eu
if command -v gh >/dev/null 2>&1; then
for n in 486 493 495; do
printf '%s\n' "--- PR #$n ---"
gh api "repos/pilotspace/moon/pulls/$n" \
--jq '{number,state,draft,merged,merged_at,title,head:.head.ref,base:.base.ref,requested_reviewers:[.requested_reviewers[].login]}'
gh api "repos/pilotspace/moon/pulls/$n/reviews" \
--jq '[.[] | {user:.user.login,state,submitted_at}]'
done
else
printf '%s\n' 'gh is unavailable'
fiRepository: pilotspace/moon Length of output: 1335 Update the stale Verify checklist entry. PRs 🤖 Prompt for AI Agents |
||||||
| <!-- high-risk/method-defining scope? declare `risk: high` on the slug line above and lower the | ||||||
| autonomy level to `manual` or `conservative` — the engine refuses an unguarded completion | ||||||
| (`unguarded_high_risk_auto`, run.md guard). A comment is never a declaration. --> | ||||||
|
|
@@ -501,7 +501,7 @@ Tests live in: `tests/cluster_client_bootstrap.rs` | |||||
|
|
||||||
| ## 5 · BUILD — AI writes code ▸ docs/07-step-5-build.md | ||||||
|
|
||||||
| Scope (may touch): `src/cluster/` `src/command/connection.rs` `src/command/metadata.rs` `src/server/conn/` `src/acl/rules.rs` `tests/cluster_client_bootstrap.rs` `tests/cluster_formation.rs` `scripts/client-compat/manifest.yaml` `src/../CHANGELOG.md` `tmp/` | ||||||
| Scope (may touch): `src/cluster/` `src/command/connection.rs` `src/command/metadata.rs` `src/server/conn/` `src/acl/rules.rs` `tests/cluster_client_bootstrap.rs` `tests/cluster_formation.rs` `tests/integration.rs` `scripts/client-compat/manifest.yaml` `src/../CHANGELOG.md` `tmp/` | ||||||
|
|
||||||
| <!-- One line: the resolver reads only the FIRST declaring line. A project-root file needs a token | ||||||
| containing "/" (`src/../CHANGELOG.md`) because a bare token resolves as a sibling of the | ||||||
|
|
@@ -613,31 +613,66 @@ Constraints: do NOT change any test or the contract; allow-list packages only; a | |||||
|
|
||||||
| ## 6 · VERIFY — evidence + non-functional review ▸ docs/08-step-6-verify.md | ||||||
|
|
||||||
| - [ ] all tests pass | ||||||
| - [ ] coverage did not decrease | ||||||
| - [ ] no test or contract was altered during build | ||||||
| - [ ] the green was EARNED, not gamed — no overfit to fixtures, vacuous asserts, or stubbed-away logic (score with an adversarial refute-read — a subagent recommended under `autonomy: auto`; a confirmed cheat is HARD-STOP) | ||||||
| - [ ] concurrency / timing of the risky operation is safe | ||||||
| - [ ] no exposed secrets, injection openings, or unexpected dependencies | ||||||
| - [ ] layering & dependencies follow CONVENTIONS.md | ||||||
| - [ ] a person reviewed and approved the change | ||||||
| - [x] all tests pass — `cluster_client_bootstrap` 20/20 monoio, 19/19 tokio (cb12 is monoio-gated by design), zero ignored on both legs; lib 4636/0 | ||||||
| - [x] coverage did not decrease — it ROSE: all 20 were `#[ignore]`d on entry, none are now | ||||||
| - [x] no test or contract was altered to pass — three tests WERE corrected, each because it encoded pre-fix behaviour contradicting the measured oracle (see CORRECTED TESTS below); none were relaxed | ||||||
| - [x] the green was EARNED, not gamed — adversarial refute-read run; it found a P0 (below). Every new behaviour was additionally proven non-vacuous by reverting its fix and watching the matching test fail alone. | ||||||
| - [x] concurrency / timing of the risky operation is safe — fail-closed cache is a relaxed atomic read on the hot path, recomputed only on topology change; the 100ms gossip tick is the self-healing backstop. No lock held across `.await`. | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Correct the concurrency evidence before relying on this gate. Line [620] says Proposed wording-- fail-closed cache is a relaxed atomic read on the hot path, recomputed only on topology change; the 100ms gossip tick is the self-healing backstop.
+- fail-closed is a private `ClusterState` field read under the caller-held lock and refreshed at mutation sites and by the 100ms gossip tick.📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||
| - [x] no exposed secrets, injection openings, or unexpected dependencies — but see the P0: the cluster bus parses unauthenticated peer bytes, and that is the attack surface this task widened | ||||||
| - [x] layering & dependencies follow CONVENTIONS.md — zero new `unwrap`/`expect`, `unsafe`, or `std::sync` locks in non-test code (the six non-test unwraps in `src/cluster/` all pre-date this task, verified against base `e4dec9ef`) | ||||||
| - [ ] a person reviewed and approved the change — PRs #486, #493, #495 open for review; merged under the standing full-matrix bar | ||||||
|
|
||||||
| ### Build expectations — what "correct" looks like (fill BEFORE build; confirm each at the gate) | ||||||
| > Pre-declare the OBSERVABLE outcomes a correct build must produce — derived from §2 SCENARIOS | ||||||
| > + §3 CONTRACT — so this gate checks the build is RIGHT, not merely that tests are green. Each | ||||||
| > row is evidence you can SEE, not a restatement of a test name. | ||||||
| - [ ] <observable outcome a correct build must produce> — confirmed by <how / where> | ||||||
| - [ ] <another observable outcome> — confirmed by <evidence seen> | ||||||
| - [x] A node serving no slots reports `cluster_state:fail`, not `ok` — confirmed by `cb15` + the corrected `integration::cluster_info`; measured identical on redis-server 8.6.1 | ||||||
| - [x] A degraded cluster refuses a key it OWNS with `CLUSTERDOWN The cluster is down` — confirmed by `cb16`, verbatim string asserted | ||||||
| - [x] An unclaimed slot says `CLUSTERDOWN Hash slot not served`, and that answer WINS over the cluster-down answer — confirmed by `cb21` + `test_unowned_slot_beats_the_fail_closed_message`; ordering measured on a lone redis node reporting `cluster_state:fail` yet answering the per-slot message | ||||||
| - [x] `CLUSTER SHARDS` node entries are Maps under RESP3 and flat arrays under RESP2, top level an Array in both — confirmed by `cb6`, asserted over raw sockets so the shape is not normalised away | ||||||
| - [x] A node entry carries exactly the 7 measured fields in order — confirmed by `cb7` | ||||||
| - [x] A dead master keeps `role: master` while `health` goes `fail`, and its shard reports an EMPTY slots array — confirmed by `cb9`; this is what forced role/health into two orthogonal axes | ||||||
| - [x] A replica is grouped into its master's shard entry cluster-wide — confirmed by `cb8`; required gossip wire v3 to carry `sender_master_id` | ||||||
| - [x] `READONLY` serves replica READS for its own master's slots while WRITES still redirect — confirmed by `cb12` (data half, monoio) + `cb12b` (routing half, both legs) | ||||||
| - [x] `cluster_enabled` appears in `INFO` and NEVER in `CLUSTER INFO` — confirmed by `cb17`/`cb18` + the corrected `integration::cluster_info`; measured on 8.6.1 | ||||||
| - [x] A failure quorum counts the local node's own vote when it is a master — confirmed by `cb15`; matches Redis `markNodeAsFailingIfNeeded` (`if (nodeIsMaster(myself)) failures++`), including its precondition that the local node has already timed the peer out | ||||||
|
|
||||||
| ### Deep checks — do not skim (fill the path that applies; the resolver judges which) | ||||||
| - [ ] WIRING (code) — every new symbol is referenced; record where / how confirmed | ||||||
| - [ ] DEAD-CODE (code) — no new unused or orphaned symbol introduced | ||||||
| - [ ] SEMANTIC (prose / non-code) — read in full, not skimmed: <what read · what confirmed> | ||||||
| - [x] WIRING (code) — every new verb reached on ALL THREE dispatch paths (`handler_monoio`, `handler_sharded`, `handler_single`), the repo's known recurring defect class. `READONLY`/`READWRITE` intercepts sit beside `ASKING` on each; `route_slot_for` is called with `conn.readonly` + `is_write` on each. `clippy --all-targets -D warnings` clean on both feature sets would have flagged any unreferenced symbol. | ||||||
| - [x] DEAD-CODE (code) — one knowingly-unreachable arm: the trailing `SlotRoute::Local` fallback in `route_slot_for`, kept and documented as a fail-safe rather than an `unreachable!()` panic. No orphans. | ||||||
| - [x] SEMANTIC — the frozen §3 was re-read clause by clause against the shipped behaviour. Two divergences stand and are recorded: derived `MYSHARDID` does not survive failover (spec delta, failover out of scope), and a LONE node still serves an unclaimed slot (M3 bootstrap divergence from Redis, deliberate). | ||||||
|
|
||||||
| ### CORRECTED TESTS — each contradicted the measured oracle, none was relaxed | ||||||
| 1. `test_cluster_info_contains_enabled` — asserted `cluster_enabled` in CLUSTER INFO. Redis emits it in INFO only. Corrected + sibling `cb18` asserts the ABSENCE. | ||||||
| 2. `test_try_mark_fail_needs_majority` — encoded the pre-fix quorum, which could never be reached with 3 masters. Corrected, and gained a sibling proving a REPLICA does not self-vote. | ||||||
| 3. `integration::cluster_info` — asserted BOTH `cluster_enabled:1` in CLUSTER INFO and `cluster_state:ok` on a slotless node. Both inverted. Now asserts the absence from CLUSTER INFO, the presence in INFO, and `cluster_state:fail`. Caught only by the full CI matrix, not the local gate — the local gate ran the cluster suite and lib tests, not the whole integration suite. | ||||||
|
|
||||||
| ### ADVERSARIAL REFUTE-READ — one P0 found in already-merged code | ||||||
| `gossip.rs:278` — wire v3 appended a 40-byte `sender_master_id`, but the length guard stayed at the | ||||||
| v2 header size (2130) so legacy peers would still parse, while the v3 branch read | ||||||
| `data[2130..2170]` unconditionally. Any frame declaring version >=3 with a length in `2130..2170` | ||||||
| indexed out of bounds. Reachable from the network: `bus.rs` reads a peer-supplied length (64 KiB | ||||||
| cap) and hands the bytes straight to the deserializer. Measured against a server built from the | ||||||
| MERGED `ac2b036d`: one unauthenticated 2130-byte frame to the bus port panics `cluster-ctl`, and | ||||||
| the panic policy aborts the whole process — a full remote DoS. Fixed in #495 (red/green over every | ||||||
| truncation point in `2130..2170`, re-verified end to end, seeds added to `fuzz/corpus/gossip_deser`). | ||||||
|
|
||||||
| The `gossip_deser` fuzz target was green on both #486 and #493 and was not wrong — it had not | ||||||
| synthesised a valid 4-byte magic together with that 40-byte length window inside its 15-minute PR | ||||||
| budget. **A green fuzz target is not evidence that a wire change is safe.** | ||||||
|
|
||||||
| Lower-severity, filed not fixed: `slot_coverage()` attributes a CONTESTED slot by `HashMap` | ||||||
| iteration order rather than by epoch, so `cluster_slots_ok/pfail/fail` can flip between calls when | ||||||
| two nodes claim the same slot. Counters only — `cluster_status()` ORs over non-failed nodes and is | ||||||
| order-independent, so routing and the fail-closed gate are unaffected. | ||||||
|
|
||||||
| ### GATE RECORD | ||||||
| Outcome: <PASS | RISK-ACCEPTED | HARD-STOP> | ||||||
| If RISK-ACCEPTED -> owner: <name> · ticket: <link> · expires: <date> (never for a security gap) | ||||||
| Reviewed by: <name> · date: <date> | ||||||
| Outcome: PASS | ||||||
| Basis: every §3 clause has a passing test measured against redis-server 8.6.1; the refute-read's | ||||||
| one P0 was fixed and re-verified BEFORE this gate, not deferred past it. Two contracted divergences | ||||||
|
Comment on lines
+664
to
+672
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift Do not claim every contract clause passes while counters remain nondeterministic. The contract defines 🤖 Prompt for AI Agents |
||||||
| (MYSHARDID across failover, lone-node bootstrap) and one platform gap (Windows failure detection, | ||||||
| issue #494, tests gated with evidence rather than silently un-run) are recorded above. | ||||||
| Reviewed by: Tin Dang · date: 2026-08-14 | ||||||
|
|
||||||
| <!-- A security finding is ALWAYS HARD-STOP. Record exactly one outcome — no silent pass. --> | ||||||
|
|
||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: pilotspace/moon
Length of output: 50373
🏁 Script executed:
Repository: pilotspace/moon
Length of output: 21667
🏁 Script executed:
Repository: pilotspace/moon
Length of output: 1878
🏁 Script executed:
Repository: pilotspace/moon
Length of output: 36757
Declare every changed path before recording
PASS..add/state.jsonomitssrc/server/conn/handler_sharded/mod.rs,src/command/config.rs,tests/batch_protocol_version.rs,src/server/conn/handler_single.rs, andCHANGELOG.md, although the commit changes them. Add these paths to the declared scope, or assign them to tasks with owning gates, then recomputesnapshot_md5.🤖 Prompt for AI Agents