fix(security): ACL gate before privileged intercepts, fail-closed unknown users, real CLIENT KILL USER (c10k cluster B) - #430
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PR Summary by QodoFix ACL bypasses: gate privileged intercepts, fail-closed users, real kill-by-user
AI Description
Diagram
High-Level Assessment
Files changed (14)
|
Code Review by Qodo
1.
|
…430 review) `ACL DELUSER` disconnects the sessions a deleted user still holds (B3). It called `kill_clients(filter, None)` — and `None` means "no self-kill exemption", so when the caller deleted its OWN account, `kill_clients` `shutdown(2)`d the caller's socket while its own `:1` was still in flight. The user saw a connection error instead of a reply. `kill_clients` already carries `self_id` for exactly this: for the matching entry it sets `kill_flag` but skips the fd shutdown, leaving the teardown to the handler loop's next `is_killed()` check. That is reply-then-disconnect, which is what a client expects and what `CLIENT KILL` of oneself already did. `handle_acl` had no way to know who was calling, so the caller's `client_id` is now threaded in (one new parameter; the three production call sites pass their connection's id, the unit tests pass 0). Test: `self_deluser_replies_before_disconnect_{single,multi}_shard`. RED with `None` restored: `a self-DELUSER must answer before disconnecting, got: ""` at shards=1 AND shards=4 — an empty reply, which is the defect exactly. Noted while tracing this, NOT fixed here: `handler_single` never registers connections with the client registry at all, so `CLIENT LIST`, `CLIENT KILL` and this disconnect are all inert on that path. It is the embedded / single-shard tokio handler and the gap predates this branch. Reported by qodo-code-review on #430. Gates: fmt; clippy -D warnings on default and runtime-tokio,jemalloc; lib acl 84; acl_user_revocation 6/6 on both runtimes. author: Tin Dang
7421a0d to
42b7e36
Compare
…ny unknown users; track post-AUTH identity (c10k B1/B2/B3)
Three ACL/auth defects from the c10k hardening review, all in the same
blast radius: an authenticated account with minimal grants could take the
server over, and revoking that account could not stop it.
B1 — privileged intercepts ran ABOVE the ACL gate. Both connection
handlers intercept EVAL/EVALSHA/SCRIPT, ACL and CLUSTER before the
permission check, and each intercept `continue`s on a match, so the gate
below never ran. The comment attached to that gate claimed the exact
invariant the code above it violated ("must run before any
command-specific handlers ... so that low-privilege users cannot reach
admin commands"), which is presumably why it survived so long.
Verified end-to-end before the fix: a user holding nothing but
`~app:* +get` is correctly refused a plain SET, and then succeeds at
ACL SETUSER evil on nopass ~* +@ALL -> +OK, persisted
CONFIG SET maxmemory 999999999 -> +OK, persisted
EVAL "return 'pwned'" 0 -> "pwned"
SCRIPT LOAD / REPLICAOF / BGSAVE -> all accepted
The gate now sits above every privileged intercept in both handlers, with
AUTH and HELLO lifted above IT — Redis marks both NO_AUTH, and gating
them would strand a restricted client with no way to re-authenticate and
break the RESP3 handshake. Existing consistency suites authenticate as
`default` (unrestricted), which takes the `acl_skip_allowed` fast path,
so they were blind to all of this.
B2 — unknown users failed OPEN. check_command_permission,
check_key_permission and check_channel_permission each began with
`self.users.get(username)?`, and None is their "allowed" answer. Nothing
closes a live session when its account disappears, so `ACL DELUSER alice`
or an `ACL LOAD` that drops her silently PROMOTED every connection alice
still had open to full `~* +@all`. Unknown users are denied now. The
matching guard is `ensure_default_user`: `default` is inserted (from
requirepass, exactly as the no-aclfile path does) after any load that
could omit it, so fail-closed cannot brick a server whose ACL file never
mentions `default`.
B3 — CLIENT KILL USER was inert. `client_registry::register` captures the
username once, at accept time, when it is always `default`; every
AUTH/HELLO success updated only the connection-local copy. So CLIENT LIST
reported `user=default` for every session and `CLIENT KILL USER alice`
returned 0 — the primary incident-response lever for a compromised
credential, useless. All 13 assignment sites across the three handlers
now go through one `ConnectionState::adopt_user`, which publishes to the
registry as well as refreshing the ACL cache. With identity tracked, ACL
DELUSER also disconnects the sessions the deleted user still holds, as
Redis does.
Tests (red proven by reverting src/ and re-running, both shard configs):
- tests/acl_privileged_intercepts.rs — 9 escalation vectors must be
NOPERM, with a GET positive control, no-side-effect checks on the
denied ACL SETUSER / CONFIG SET, and AUTH/HELLO still exempt.
RED: `ACL SETUSER evil ...` returned +OK at shards=1 and shards=4.
- tests/acl_user_revocation.rs — CLIENT LIST reports the post-AUTH user,
CLIENT KILL USER tears the session down and leaves others alone, ACL
DELUSER disconnects.
RED: CLIENT LIST showed `user=default`; DELUSER left bob connected.
- src/acl/table.rs unit tests for unknown-user denial and
ensure_default_user (creates only when missing, honours requirepass).
Gates: fmt; clippy -D warnings on default and runtime-tokio,jemalloc
(macOS + Linux VM); lib 4469 monoio / 3631 tokio; the conn/auth
integration suites (maxclients_reject_parity, parked_idle_parity,
idle_downshift_parity, tls_idle_downshift_parity,
client_tracking_invalidation) green on both runtimes.
Refs: tmp/C10K-HARDENING-REVIEW.md cluster B.
author: Tin Dang
…igured (c10k B4) `MOON_URING=1` (tokio, Linux) arms an experimental io_uring bridge that binds a SECOND SO_REUSEPORT listener on the server's own port and serves commands from its own accept path — one with no auth gate, no ACL check and no client registry. On a server that HAS configured authentication the kernel then load-balances new connections between a listener that enforces auth and one that does not, so roughly half of them skip authentication entirely. The bridge's documented limitation named maxclients and CLIENT LIST/KILL. It did not name this. The bridge now refuses to arm when requirepass or aclfile is set, logging at error level, and the shard stays on the stable pure-tokio path (which does enforce auth). Refusing is the fail-closed answer and matches how every other failure in this block degrades; killing the process over an experimental opt-in would be worse. The rule is a free function so it is unit-testable off Linux and under both runtimes. Test: src/shard/event_loop.rs — no auth allows the bridge, requirepass and aclfile each refuse it. Refs: tmp/C10K-HARDENING-REVIEW.md B4. author: Tin Dang
`Stream::task_park_safe` (vendored monoio-rustls) decides whether an idle TLS connection may be task-parked on raw-fd readability. It sampled `session.wants_write()` BEFORE calling `process_new_packets()` — but processing a buffered record can itself queue outbound bytes and still return `Ok`, so the veto read a state from before those bytes existed, passed, and parked the connection owing the peer a reply that nothing would flush. The reachable case in rustls 0.23 is the TLS 1.2 renegotiation rebuff: a post-handshake ClientHello makes rustls queue a `NoRenegotiation` warning alert into `sendable_tls` and return Ok. moon builds rustls with `tls12` enabled, so a 1.2 client reaches it. Correcting the review's own claim: the TLS 1.3 KeyUpdate(update_requested) case it named is NOT reachable. rustls parks its KeyUpdate reply in `queued_key_update_message` and only moves it into `sendable_tls` on the next outbound record, so `wants_write()` reads false under either ordering. That is a finding, not an oversight, and the new test pins it: if a future rustls starts queueing eagerly the test fails loudly and the KeyUpdate deadlock becomes live — at which point the reordered veto is already what handles it. Test: tests/tls_park_keyupdate.rs — drives a real in-memory rustls 1.3 session pair (no live server, no monoio), asserts the deferral and that the deferred reply does ride out with the next record. Refs: tmp/C10K-HARDENING-REVIEW.md B5. author: Tin Dang
…one (Windows CI) `CLIENT KILL` does two things: it sets a cooperative `kill_flag` AND it `shutdown(2)`s the fd so a connection parked in `read().await` is torn down at once. Only the first is portable — `force_close_fd` is `#[cfg(unix)]`, so on Windows the session dies when the handler next looks at it, i.e. on the client's next command. `is_closed()` asserted on a bare `read`, which is the unix-only half. It passed on macOS and Linux and failed all four `acl_user_revocation` tests on Windows. It now sends a PING first, exercising the same teardown a real client sees on every platform; on unix the socket is already shut down so the poke fails immediately and costs nothing. Deadline raised 2s -> 5s and a killed connection is re-poked while it drains any reply that was already queued. Worth recording why this reached CI at all: `Check (Windows)`, `Check (macOS)` and `Check (console feature)` are SKIPPED on pull requests — they run only on push-to-main or `workflow_dispatch`. Every PR in this stack shows "green" without ever having been built on Windows. This failure surfaced only because the stack's PRs target non-main bases (so `pull_request: branches: [main]` never fired at all) and the CI had to be dispatched by hand. Gates: acl_user_revocation 4/4 on both runtimes. author: Tin Dang
…430 review) `ACL DELUSER` disconnects the sessions a deleted user still holds (B3). It called `kill_clients(filter, None)` — and `None` means "no self-kill exemption", so when the caller deleted its OWN account, `kill_clients` `shutdown(2)`d the caller's socket while its own `:1` was still in flight. The user saw a connection error instead of a reply. `kill_clients` already carries `self_id` for exactly this: for the matching entry it sets `kill_flag` but skips the fd shutdown, leaving the teardown to the handler loop's next `is_killed()` check. That is reply-then-disconnect, which is what a client expects and what `CLIENT KILL` of oneself already did. `handle_acl` had no way to know who was calling, so the caller's `client_id` is now threaded in (one new parameter; the three production call sites pass their connection's id, the unit tests pass 0). Test: `self_deluser_replies_before_disconnect_{single,multi}_shard`. RED with `None` restored: `a self-DELUSER must answer before disconnecting, got: ""` at shards=1 AND shards=4 — an empty reply, which is the defect exactly. Noted while tracing this, NOT fixed here: `handler_single` never registers connections with the client registry at all, so `CLIENT LIST`, `CLIENT KILL` and this disconnect are all inert on that path. It is the embedded / single-shard tokio handler and the gap predates this branch. Reported by qodo-code-review on #430. Gates: fmt; clippy -D warnings on default and runtime-tokio,jemalloc; lib acl 84; acl_user_revocation 6/6 on both runtimes. author: Tin Dang
b53a654 to
5374628
Compare
Cluster B of the c10k hardening review (
tmp/C10K-HARDENING-REVIEW.md). Four security fixes and one ordering fix.B1 — privileged commands ran BEFORE the ACL gate 🔴
Both handlers intercept
EVAL/EVALSHA/SCRIPT,ACLandCLUSTERabove the ACL permission check, and each interceptcontinues on a match — so the gate below it never ran. The comment attached to that gate claimed the exact invariant the code above it violated:Verified exploitable on
mainbefore the fix. A user holding nothing but~app:* +getis correctly refused a plainSET, and then:mainACL SETUSER evil on nopass ~* +@all+OK— and it persistedCONFIG SET maxmemory 999999999+OK— and it persistedEVAL "return 'pwned'" 0pwned— arbitrary LuaSCRIPT LOAD/REPLICAOF no one/BGSAVEAny authenticated account escalates to full admin. The gate now sits above every privileged intercept in both handlers, with
AUTHandHELLOlifted above it — Redis marks bothNO_AUTH, and gating them would strand a restricted client with no way to re-authenticate and break the RESP3 handshake.Why this survived: the consistency suites authenticate as
default, which is unrestricted and takes theacl_skip_allowed()fast path. They are structurally blind to ACL ordering.B2 — unknown ACL users failed OPEN 🟠
check_command_permission,check_key_permissionandcheck_channel_permissioneach began withself.users.get(username)?— andNoneis their allowed answer. Nothing closes a live session when its account disappears, soACL DELUSER alice(or anACL LOADthat drops her) silently promoted every connection alice still held to full~* +@all.Unknown users are denied now. The matching guard is
ensure_default_user:defaultis inserted fromrequirepass(exactly as the no-aclfile path does) after any load that could omit it, so fail-closed cannot brick a server whose ACL file never mentionsdefault.B3 —
CLIENT KILL USERwas inert 🟡client_registry::registercaptures the username once, at accept time, when it is alwaysdefault; AUTH/HELLO updated only the connection-local copy.CLIENT LISTreporteduser=defaultfor every session andCLIENT KILL USER alicereturned0— the primary incident-response lever for a compromised credential, useless.All 13 assignment sites across the three handlers now go through one
ConnectionState::adopt_user, which publishes to the registry as well as refreshing the ACL cache. With identity tracked,ACL DELUSERalso disconnects the sessions the deleted user still holds, as Redis does.B4 — the io_uring bridge served commands with no auth 🔴
MOON_URING=1(tokio, Linux) binds a secondSO_REUSEPORTlistener on the server's own port and serves from its own accept path — no auth gate, no ACL check, no client registry. On a server that has configured authentication the kernel load-balances new connections between a listener that enforces auth and one that does not. The documented limitation named maxclients andCLIENT LIST/KILL; it did not name this.The bridge now refuses to arm (at
error!) whenrequirepass/aclfileis set and the shard stays on the pure-tokio path. Refusing is the fail-closed answer and matches how every other failure in that block degrades; killing the process over an experimental opt-in would be worse.B5 — TLS park veto sampled
wants_write()too early 🟡, and the review's stated cause was wrongStream::task_park_safereadwants_write()beforeprocess_new_packets(). Processing a buffered record can queue outbound bytes and still returnOk, so the veto read a pre-processing state, passed, and parked the connection owing the peer a reply.The review named TLS 1.3
KeyUpdate(update_requested)as the trigger. It isn't reachable — rustls parks its KeyUpdate reply inqueued_key_update_messageand only moves it intosendable_tlson the next outbound record, sowants_write()reads false under either ordering. I found this while writing the red test: the premise assertion failed.The reachable case is the TLS 1.2 renegotiation rebuff (
common_state.rs: a post-handshake ClientHello →send_warning_alert(NoRenegotiation)→Ok), and moon builds rustls withtls12enabled. That one cannot be driven without forging encrypted records, so it has no e2e test. What ships instead is a canary:tests/tls_park_keyupdate.rspins rustls's deferral on a real in-memory 1.3 session pair, so if a future rustls starts queueing eagerly the test fails loudly and the KeyUpdate case becomes live — at which point the reordered veto already handles it.Verification
RED proven by reverting
src/and re-running, at both shard configs:ACL SETUSER evil …+OK+OKCLIENT LISTuser=defaultuser=defaultACL DELUSERTests added
tests/acl_privileged_intercepts.rs— 9 escalation vectors must beNOPERM, with aGETpositive control, no-side-effect checks on the deniedACL SETUSER/CONFIG SET, andAUTH/HELLOstill exempt.tests/acl_user_revocation.rs—CLIENT LISTreports the post-AUTH user;CLIENT KILL USERtears the session down and leaves others alone;ACL DELUSERdisconnects.tests/tls_park_keyupdate.rs— the B5 canary.ensure_default_user(creates only when missing, honours requirepass),uring_bridge_allowed.Gates — fmt; clippy
-D warningson default andruntime-tokio,jemalloc(macOS + Linux VM); lib 4452 monoio / 3617 tokio; the conn/auth integration suites (maxclients_reject_parity,parked_idle_parity,idle_downshift_parity,tls_idle_downshift_parity,client_tracking_invalidation, plus the stack's ownblocking_peer_eof/blocking_ghost_waiter) green on both runtimes, re-run after the rebase onto #428.Broad regression —
scripts/test-consistency.shon the Linux VM: 209/209 PASS.scripts/test-commands.sh --skip-benchshows 13 failures — byte-identical onorigin/mainbuilt and run the same way (bash mangling null bytes in the KNN$qparam); pre-existing, not from this branch.Notes
.planningsubmodule pointer intentionally untouched.