clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast - #4074
Conversation
|
Thank you very much for the contribution, I really like the idea behind it. I do see though that it is marked as a Draft - is it still work in progress, or has it reached to its final state? Let us know when it is ready to review. |
|
Hi, |
2cf5b36 to
34f1042
Compare
|
Now it's ready for review, thanks! |
f93b82d to
4348580
Compare
|
Follow-up commit Since this PR is already open for review, I added this as a distinct follow-up commit rather than squashing it into the main one, so the incremental change is easy to review and the existing review isn't disrupted by a force-push of the main commit. It enforces consistency between
Hybrid environments are unaffected. |
82e123e to
b7a173e
Compare
|
Config API for controller-managed clusters is now the per-cluster Same The controller-managed ids and the Verified locally (the build links wolfSSL, so the controller runs without a node): the new syntax loads, a managed id with no controller config aborts, a controller config for an unmanaged id aborts, and matching config starts. Docs (admin guide, tests appendix, README) and the PR description are updated to this form. |
|
Follow-up Previously the clusterer-side integration (the
So clusterer_controller can be completely omitted — a build without it produces the stock upstream clusterer module: no Verified with |
Crypto is libsodium-only (see ce8e805)Worth stressing for reviewers: as of ce8e805 the module's cryptography settled on libsodium, and the earlier wolfSSL / OpenSSL-based paths were dropped entirely — there is no TLS-library fallback anymore. clusterer_controller now uses XChaCha20-Poly1305 + Argon2id for the shared-secret / at-rest key material and a Noise_NNpsk0 (Curve25519 / ChaCha20-Poly1305 / SHA-256) handshake for the join, all on libsodium primitives. Why the switch:
libsodium is therefore a hard build dependency of the module now; there is no |
Why this PR also touches
|
aad3a24 to
5dc976d
Compare
Follow-up already implemented: a consumer messaging API over the controller's encrypted plane(Updated — this comment originally claimed a script could migrate from clusterer's generic messaging "by renaming the call". That is true of the function names and false of the delivery guarantees; the correction and what was done about it are at the end.) A heads-up on where this module goes next — the work is implemented and tested on a development branch, and will be proposed once this PR lands, since it extends the packet layer introduced here. The controller's encrypted UDP plane (XChaCha20-Poly1305 under the rotating session key, Noise_NNpsk0 join) is currently private to the module. The follow-up opens it to consumers as a small API, at two levels. Module tier, bound via
Payloads up to 1300 bytes, channel names up to 31 chars; sends are IPC-marshalled to the controller worker so any process can send. Consumer packets carry a cleartext magic so the pre-decrypt rate limiter classifies them against their own budget instead of the deliberately tight join budget — without that separation, the join limiter silently dropped one consumer packet in ten under load. Script tier — the same three functions and two events clusterer offers, plus a list send: Addressing a subset is not the same as filtering one
A broadcast is the opposite case and gets the opposite treatment. There are no unaddressed nodes, so it stays one multicast; sending it as N unicasts would cost roughly twice the packets (2(N−1) against one multicast plus N−1 acknowledgements) for no benefit, which matters at the 256 nodes this module is designed for. Delivery guarantees, opt-inDefault is best-effort, unordered, at-most-once. A reliable broadcast stays a multicast and repairs by unicast to the nodes that did not answer. Repair rather than rebroadcast is the important part: a duplicate that asked to be acknowledged is acknowledged again, so resending the multicast to fix two missing nodes would have every other member answer a second time. The subtle case is a lost acknowledgement, not a lost message. The sender resends identical bytes with the same sequence number and the receiver's replay check correctly drops the duplicate — so without more, a receiver whose ACK was lost swallows every retransmission in silence while the sender spends its budget and concludes failure. Hence the re-ACK: the replay check does the deduplication, the re-ACK closes the loop, and delivery stays at most once. The correctionThe original text said a script moves between clusterer's messaging and this one "by renaming the call". The surface is deliberately identical, but the guarantees are not: Two related fixes came out of the same review. Consumer traffic now has its own sequence space: it shared one monotonic counter per sender with the control plane, so on any multipath network a reordered consumer packet could make a One implementation note for reviewers: raising a scriptable event from the controller worker requires the process to be declared with |
The complete script, MI and event surface — including what the follow-up adds(Updated: adds the list send, the opt-in reliable flag, the per-cluster consumer settings, and a note on how counts are returned.) Everything this module exposes today, plus what the consumer messaging follow-up adds, so the script- and operator-facing surface can be reviewed as a whole. Script functionsCurrent:
Added by the follow-up — deliberately clusterer's generic-message surface, so the shapes are familiar (the delivery guarantees differ; see the previous comment):
On return values. The count comes back in an output variable and the function itself is simply true or false — which is why the call goes inside the What the count is, and is not. It is how many nodes the message was sent to, which is known immediately. How many acknowledged it cannot be known at that point — the answers arrive afterwards — so for a reliable send that result currently goes to the log as it completes, reported against the configured budget: Surfacing that back to the script as a completion event is the remaining piece of this work. (An earlier revision of this comment documented padding past unused arguments — Events
Clusterer's exact parameter set, so existing event routes port unchanged. The module reserves the Pseudo-variables
MI commands
Module parametersExisting: Added for the consumer plane, each a global default that a cluster may override in its own
The per-cluster form follows what Module API (for other modules)Bound with A hardening change worth flagging to reviewersGiving consumer traffic its own rate budget raised what a single source address can push through to the cipher from 20 packets a second to 1000 — the right budget for a real peer, a poor one for a stranger, since the packets fail to decrypt but the node still performs the failed decryption, and the sender needs to know nothing but the port and two cleartext magic bytes. Consumer packets are therefore now required to come from an address the cluster already knows. That is safe to require because consumer traffic only ever passes between joined members — control traffic, where a stranger legitimately appears (a It does not fix address spoofing and does not claim to — a forged source copying a member's address still reaches the limiter. What it removes is the far easier attack of pointing a flood at the port from anywhere. Verified on a three-node cluster: 399,000 forged consumer packets in three seconds from a non-member address, after which membership was intact, cross-node traffic still worked, and the rate limiter had not logged once — the packets never reached it. |
|
Pushed a small follow-up fix:
|
Hi @razvancrainea Razvan, Best regards, |
|
Found and fixed a real bug while testing this module's actual multi-node discovery for the first time in production (every deployment so far had been single-node, where peer discovery never needed to cross the wire).
On two production hosts with the Checked three other multi-node clusters already running elsewhere - same Fix: bind the join to Pushed to this branch: 0b3af68 |
Groundwork for clusterer_controller (added later in this series): an API the
controller binds to, a way to declare which clusters it manages, and the hooks
that let a controller-managed cluster take its identity at runtime instead of
from the config.
Declaring a managed cluster uses a per-cluster 'cluster_options' modparam - the
same "key=value, key=value" idiom as my_node_info:
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
cluster_id is required; use_controller is a 0/1 flag defaulting to 0 (native),
and only use_controller=1 pre-creates the controller-managed stub, which never
touches the DB and is guarded against hijacking a native cluster of the same id.
Native clusters need no cluster_options line at all, and every other clusterer
setting (db_mode, ping_*, my_node_id, sharing_tag, ...) stays a global modparam.
The interim 'use_controller' / 'cluster_id' int modparams (never released) are
kept registered only to fail with a migration hint.
The managed-id set is exported through the ctrl binds (managed_count /
managed_ids) so the controller can cross-check it pre-fork: a managed id with no
matching 'cluster' config has no BIN socket or crypto params, and a 'cluster'
config for an unmanaged id has nothing legitimate to drive. Either mismatch
aborts at startup naming the offending id, rather than half-forming a cluster.
All of it is compiled only under CLUSTERER_CTRL_SUPPORT:
- the clusterer_ctrl API (clusterer_ctrl.c) and the cluster_options modparam
plus the load_clusterer_ctrl_binds export;
- the controller stub pre-create loop, the child_init guard, the shm
current_id mirror, the on-demand stub, and the shtag_managed /
controller_managed logic;
- the per-cluster identity and hybrid-db_mode accessors (cluster_self_id,
cl_db_mode, GET_CURRENT_ID, use_controller), which get #else fallbacks to
the stock globals (current_id / db_mode / 0) so their call sites compile to
the exact upstream object code with no per-site #ifdef. add_node_info's
internal self_id parameter is gated the same way.
The top-level Makefile exports CLUSTERER_CTRL_SUPPORT=1 iff clusterer_controller
is in the configured build - derived from the include/exclude lists rather than
the current 'modules' subset, so it is stable across 'make all' and a
single-module rebuild - and clusterer/Makefile turns that into the -D.
The point of the gate: a build without clusterer_controller produces the stock
clusterer module. No cluster_options parameter (rejected as unknown), no
behavioural change, no added exports. Verified with
'unifdef -UCLUSTERER_CTRL_SUPPORT' against the base - no semantic difference
from upstream. Enabling the controller rebuilds clusterer with the hooks; the
two are a matched pair.
tm_init_cluster() read this node cluster id once, at mod_init, and baked it into the ";cid=<id>" Via parameter (and into a cached tm_node_id used to decide whether an anycast reply is ours). That assumes the id is known and stable by mod_init, which holds for a statically configured clusterer but not for a controller-managed one: there the id is assigned at runtime, after the node joins, and can change on re-election. So mod_init froze the still-unassigned id (-1, printed as its unsigned form 18446744073709551615) into every outgoing Via, and every node compared incoming cids against -1 - t_anycast_replicate() could never route a reply to the owning node. Read the id live instead: pre-build only the fixed ";<param>=" prefix at init, and have tm_via_cid() append the current get_my_id() per request (returning no parameter while the node still has no id); compare incoming cids against the live get_my_id() too. cl_get_my_id() already returns the runtime id, so this needs nothing from the caller and self-corrects across re-elections.
A control plane for the clusterer module: nodes discover each other over
encrypted UDP multicast, elect a master, and are assigned their cluster identity
at runtime, so an HA cluster needs no per-node id in the config and no database
behind it. Native, controller-managed and hybrid topologies coexist - a cluster
is controller-managed only if the clusterer declares it so via cluster_options.
Crypto is libsodium-only, no fallback: XChaCha20-Poly1305 for the payload AEAD,
Argon2id as the bootstrap KDF, and Noise_NNpsk0 for the join handshake, with
X25519 / HKDF-SHA256 / RNG underneath. libsodium is linked dynamically (distro
static libs are usually non-PIC), so target hosts need the runtime package. The
module is added to exclude_modules in Makefile.conf.template, like the other
modules with external-lib dependencies - enable it via include_modules - and
libsodium-dev joins the CI apt requirements.
Membership and liveness:
- master election, with the invariant "MASTER_ALIVE keepalive armed <=> I am
the elected master", which is what stops a node demoted purely by election
from continuing to broadcast and flapping between two masters;
- master-mediated ALIVE, so liveness costs O(N) messages per round rather than
every node pinging every other (O(N^2));
- a membership digest carried in MASTER_ALIVE, with a RESYNC repair path when
a node's view diverges from the master's;
- the join handshake is ACKed and retransmitted under a bounded budget, and
both the KEY_GRANT and the join-time state snapshot are unicast to the
joining node instead of broadcast to everyone.
Several controller clusters can share one BIN socket - each has its own
multicast group, shared-port unicast is routed by cluster_id, and multicast is
used as the fallback when two clusters collide on a port.
Hardening: the join path is flood-DoS rate-limited and its inputs are validated
and bounds-checked (including the Noise msg2 decrypt output and the zero-length
cipherstate passthrough). Decrypt failures are classified rather than logged
uniformly - a bootstrap-key failure means a wrong password, a foreign cluster or
tampering and warns, while a transient session-key mismatch during a rekey or a
split-brain heal is expected and stays at DBG. The split-brain defer budget
resets on a fresh higher-IP JOIN_REQ, so a lower-IP node waits for a live
higher-IP peer instead of self-promoting, bounded by CL_CTR_JOIN_DEFER_HARDMAX.
Multicast is sent and joined on this node's own interface rather than
INADDR_ANY, which is what makes discovery work on a multi-homed host where the
default route does not carry the cluster network.
Validated end-to-end on a 3-node cluster: controller / native / hybrid modes,
master failover and election stability (staggered and simultaneous starts both
converge to a single stable master), multiple controller clusters on one BIN
socket, the MI surface and its error paths, buffer overrun/underrun fuzzing,
wrong-password rejection and config mismatch. Ships with the admin guide, the HA
test appendix, the generated README and a join-rejection test.
Restructured into 3 logical commits (28 → 3) — no code changeThe old history was 28 commits of my own iteration, which made this look like a
Ordering is dependency-first: the clusterer API lands before the module that binds The tree is byte-for-byte identical to the previous head Commits referenced in the comments aboveOlder comments in this thread cite specific hashes. They now map as:
Every other commit in the old history touched only Happy to split it further, or to break out the |
dd0f515 to
22cb916
Compare
The controller already owns an authenticated, encrypted, rate-limited UDP
plane between the nodes of a cluster, plus the membership that says who is on
it. Any other module wanting to exchange a message between nodes had to build
that transport again from scratch. This exports it instead.
A consumer registers a named channel pre-fork and sends on it:
clctr_api_t clctr;
if (load_clctr_api(&clctr) < 0) ... /* mod_init */
clctr.register_channel(&ch, my_cb); /* mod_init - PRE-FORK */
clctr.send_mcast(cluster_id, &ch, &payload, 0);
clctr.send_ucast(cluster_id, node_id, &ch, &payload, 0);
and inherits the XChaCha20-Poly1305 group session key and its rotation, the
per-packet receive gauntlet (magic gate, cluster_id filter, size bound,
per-source rate limiting) and the membership view, with no transport code of
its own.
The delivery contract is the part worth reading, because it is what a consumer
gets wrong:
- the receive callback runs in the CONTROLLER's worker process for that
cluster, not in the process that called send. A consumer needing to wake a
different process brings its own mechanism (shm + eventfd, ipc_send_rpc).
Callbacks run on the cluster's receive path, so they must stay short.
- sends are marshalled to that same worker over IPC, which is what keeps
ordering per node and leaves the anti-replay sequence space single-writer.
- CLCTR_SEND_TO_SELF dispatches locally rather than listening for our own
packet, and a unicast to our own node id degenerates to exactly that, with
nothing on the wire.
- src_node_id is 0 when the sender had not been assigned an id yet.
CLCTR_SEND_RELIABLE asks for acknowledgement and resend while unacknowledged.
It is opt-in per send rather than per channel on purpose: it costs one ACK per
recipient, so a reliable broadcast turns a single packet into N-1 packets back.
Most consumer traffic is better served by being idempotent and retried by its
own logic.
Payload sizing has a deliberate split: CLCTR_MAX_PAYLOAD is a compile-time
lower bound for consumers that size local buffers statically, while the real
runtime limit is cc_max_payload, derived from the interface MTU at mod_init and
larger on jumbo-frame links.
Ships with tests for the consumer sequence, channel filtering, reliable
broadcast, and the script-facing messaging surface.
cl_ctr_handle_member_list() refreshed last_seen for every IP in the list, via cl_ctr_upsert_peer_locked(). A MEMBER_LIST is a membership announcement: the master lists a node until it prunes it, whether that node is reachable or not. Treating it as evidence of liveness makes every receiver believe it has just heard from peers it has never heard from. That is what made a dead member immortal. cl_ctr_alive_bitmap() is built from last_seen, so a backup holding refreshed timestamps for a node that is down asserts to the whole cluster that it is up the moment that backup becomes master - and from then on nobody can age it out. It survives its own funeral. Seen in production: 10.22.20.241 stayed in cluster 243 for about a day, across a master change, after being rolled back to a build with no controller at all. Nothing was listening on its BIN port, and the two surviving nodes spent ~3,500 failed connects a day each trying to reach it - roughly 10,668 of one node's 10,801 daily ERROR lines, which buried every other error on the box. The member-list path now inserts peers it did not know about and leaves last_seen alone for peers it already tracks. Liveness continues to come only from direct evidence: a packet sent BY the peer, or the master's alive bitmap, which the master derives from packets it received itself. A newly learned peer is still seeded with last_seen = now, so it gets one purge window to prove itself instead of being dropped on the next tick. This terminates rather than oscillating: once the master prunes a dead node it stops listing it, and every receiver ages it out one window later.
Brought up to date: consumer messaging API + a liveness fix (3 → 5 commits)Two commits on top of the restructure above, both already running in production
|
A second controller process on the same host takes the same ip:port without
complaint, because the socket needs SO_REUSEADDR and INADDR_ANY to receive
multicast at all. The kernel then treats the two traffic types differently:
a multicast datagram is delivered to EVERY co-bound socket, but a unicast
is delivered to exactly ONE, chosen without reference to the destination
address. Measured on 5.4, 200/200 trials, the winner being the most
recently bound socket; SO_REUSEPORT does not help, it only changes which
socket wins.
Every 1:1 leg therefore lands in the wrong process. The join KEY_GRANT is
unicast, so the joining node never authenticates and dies with
cannot authenticate ... (wrong password, or a foreign cluster on
cluster_id N). Shutting down.
which sends the operator hunting a credential problem that does not exist.
Reversing the start order moves the victim to the other instance, which is
how the mechanism was confirmed: it follows bind order, not address.
This does not make that topology work - one controller instance per host
per port is what is supported, and production is unaffected because a node
runs one. It makes the diagnosis available:
- cl_ctr_setup_socket() probes the port WITHOUT SO_REUSEADDR before
taking it. That bind fails EADDRINUSE precisely when another process
already holds it, and succeeds when the port is free; a probe WITH
SO_REUSEADDR would succeed either way, which is why the real bind
cannot tell. One warning naming the actual constraint. Advisory, never
fatal - a restart can briefly race the outgoing process, and a spurious
warning is cheaper than refusing to start.
- cl_ctr_maybe_forward()'s "no local cluster for this packet" drop was
LM_DBG. L_DBG is 4 and deployments run 3, so a whole class of
"the cluster does not converge" had no trace at any log level anyone
uses. It is now LM_WARN, rate-limited to one per 30s carrying the
suppressed count - a misdelivering peer can produce one per packet, and
a warning that fires at line rate is its own outage.
The in-process case is unchanged: several clusters in ONE process sharing a
port are still recovered by cl_ctr_maybe_forward() through the shared
cluster array, and that path never reaches the new warning.
678bc4c7ed stopped MEMBER_LIST from refreshing last_seen, on the grounds that a membership announcement says who BELONGS to the cluster, not who is reachable. NODE_ASSIGN carries exactly the same kind of claim and was still calling cl_ctr_upsert_peer_locked() on the IP in its payload, so the bug survived the fix. This half was worse, because it is a SELF-loop rather than a peer-to-peer echo. IP_MULTICAST_LOOP means the master receives its own NODE_ASSIGN - the handler's own doc comment says 'all nodes (including master via loopback) apply the assignment'. So every roster announcement refreshed last_seen for every node named in it, including a dead one, and since the master is the node that runs cl_ctr_prune_stale(), the corpse kept itself alive. Measured on the 3-node staging RGS cluster before this commit: a member whose controller plane was isolated with iptables, and which was confirmed silent by tcpdump on the master, was STILL a member on both survivors after four minutes against a 30 s purge deadline. Behaviour was identical with and without 678bc4c7ed, which is what showed the earlier fix was only half of it. Liveness still has two sound sources and both are untouched: the master learns it from the unicast ALIVE a settled non-master sends it, and backups learn it from the master's MASTER_ALIVE bitmap, which cl_ctr_apply_alive_bitmap() uses to set last_seen. A node the master no longer believes in is simply absent from that bitmap, so receivers age it out instead of being told to keep it. cl_ctr_rejoin_superior_master() was audited at the same time and deliberately left alone: it upserts sender_ip from a beacon actually sent by that node, which is direct evidence. (cherry picked from commit 9bd7d101bb45707324699d8fa68ce64088a6ce87)
New commit: NODE_ASSIGN is membership, not livenessThis completes the fix started in "a member list says who belongs, not who is alive". That earlier commit stopped This half was worse, because it is a self-loop rather than a peer-to-peer echo. Measured, on a 3-node clusterA member's control plane was isolated with iptables so it died silently —
The backup pruning one window after the master is the intended behaviour described in Also checked: promoting the backup mid-test (by isolating the master) does not resurrect the dead node in the new master's view. That was the failure loop this whole thing came from. Why this does not starve live peersA settled non-master unicasts its
|
Removing a node from the topology did not touch the transport. clusterer sends with msg_send(send_sock, proto, &node->addr) and the core keeps the TCP connection in its own table keyed by destination, so nothing in clusterer_ctrl_remove_node() - delete_neighbour, remove_node_list, CLUSTER_NODE_DOWN, report_node_state - ever closed the socket. It survived until the TCP layer timed it out or the peer closed first. That is not academic. A node can be dead to the control plane and still perfectly reachable on BIN: hung, half-open, or partitioned on one plane only. Measured on a 3-node staging cluster, with the victim's control plane isolated and BIN deliberately left reachable: clusterer_controller purged the node at its 30 s deadline and both survivors still held ESTABLISHED connections to it, in both directions. The close lives here rather than in clusterer_controller on purpose. The BIN transport belongs to clusterer, and the controller drives it through this API instead of reaching into the core's TCP layer itself. It is exposed as close_node_conn() for the case where only the transport should be dropped; remove_node() now does it for you. Two ordering constraints, both load-bearing: - the url is copied out BEFORE remove_node_list(), which frees the node; - the close happens AFTER cl_list_lock is released and after the capability event_cb callbacks have run, because a callback may still send a final BIN packet to the departing node and pulling the socket first would only turn that into an error. get_node_by_id() walks node_list, which does not contain current_node, so this can never close our own listener. (cherry picked from commit d1d12a5025df4481d939fad5bfaff06307566026)
…at happened Two defects in the previous commit, both found by measuring on a live cluster rather than by reading the code back. 1. It closed at most ONE connection. There are normally two per peer - the one we dialled out to its BIN port and the one it dialled in to ours - and both are reachable by the same address, because the core registers an alias for the peer's advertised port on an accepted connection as well. That aliasing is what lets an inbound connection be reused for outbound sends. tcp_close_connection() closes exactly one per call and flags it F_CONN_FORCE_CLOSED, which makes the next lookup skip it, so the fix is to loop until it reports nothing left. That terminates by construction: each pass removes one connection from the candidate set. The cap is paranoia against a future core change that stops setting the flag. 2. The logging announced intent, not outcome. It printed 'dropping BIN connection' before calling, and only said anything afterwards on a negative return - but tcp_close_connection() returns 1 for closed and 0 for nothing found, so the interesting case was silent and the log could not distinguish 'closed it' from 'there was nothing there'. On the first live test that made a no-op look like a success. It now reports the count it actually closed, and an outright failure is an error rather than a debug line. (cherry picked from commit e5838d3b8036fd5d5e7da51ac7b40a9980d85300)
The master's MEMBER_LIST and NODE_ASSIGN are multicast, and IP_MULTICAST_LOOP delivers them back to the master itself. Both handlers ran the learn path on the payload IPs, so the master re-inserted every node its own announcement named - including a peer it had just purged. The re-seeded copy kept the roster naming the dead node, every receiver re-learned it one window later, and the purge could never converge. Observed live on a 3-node staging cluster: 'new peer <ip>' every 35 s, in lockstep with the purge cycle, for a node whose control plane was provably blocked the whole time. An announcement we authored was built FROM this table; it cannot teach us anything. Receivers other than the author still learn membership from these packets exactly as before - that is the legitimate propagation path, and a node the master stops listing now ages out everywhere one window later, which restores the termination argument cl_ctr_learn_peer_locked() was written around. This also explains why a resurrected peer escaped removal forever: a MEMBER_LIST entry carries only IP + is_master, so the re-learned copy had node_id 0, and cl_ctr_prune_stale() only propagates removal to clusterer for node_id > 0. 'removed node_id' fired exactly once per incident and never again. (cherry picked from commit bbfc93cf57742fc89f11d531299d326f4dac3376)
Membership authority: in a controller-managed cluster the only way in is the controller's JOIN handshake, after which the controller calls clctl.add_node(). Wire self-discovery is the zero-config mode's mechanism and must not apply - but cl_db_mode() deliberately reads 0 for controller-managed clusters (so the controller's runtime add/remove works), and that same predicate gated the learn paths, which put controller-managed clusters on exactly the self-discovery behaviour they must not have. The consequence, measured on a 3-node staging cluster: a node the controller had expelled talked its way straight back in within one ping interval - PING from unknown -> UNKNOWN_ID -> NODE_DESCRIPTION -> add_node() - and the BIN connections that had just been closed were re-dialled (2 established grew to 4). A node dead to the control plane but alive on BIN is exactly the hung / one-plane-partitioned failure the purge exists for. New cl_ctr_owns_membership() (0 in non-controller builds, so default builds are unchanged) now gates all three wire-learn sites: - handle_full_top_update's two unknown-node learns, alongside cl_db_mode; - handle_internal_msg_unknown's NODE_DESCRIPTION add_node - which also no longer gossips the stranger's description onward via flood_message. The UNKNOWN_ID reply to a stranger's ping is kept: telling the node we do not know it is what prompts its controller to re-join properly. The ignore is logged at INFO, rate-limited to one line per 30 s, because a live expelled node re-announces on every ping cycle. (cherry picked from commit f2903b18b17fb0fd32698e081e913a9cda73158d)
Split-brain resolution had two merge paths that behaved differently. A master that learned of a superior via MASTER_BEACON went through cl_ctr_rejoin_superior_master() - demote, adopt the master, and send a JOIN_REQ. A master that learned of it via MASTER_ALIVE merely yielded: it recorded the winner and stopped asserting mastership, and that was all. Yielding alone is not a merge. The winner learned the yielding node only from that packet's sender-upsert - a peer entry with node_id 0 - and the ONLY place a node_id is ever assigned is handle_join_req(). Without a JOIN_REQ the yielded node sits in the winner's table as id 0 forever: it is never named in a NODE_ASSIGN, so it is never added to clusterer on any node, and its membership digest can never match the master's MASTER_ALIVE - which turns into a RESYNC-per-second livelock as the master keeps 're-broadcasting full state' that structurally cannot contain the missing node. Observed live on the 3-node staging cluster after a partition heal: the controller admitted the returning node as a member everywhere, but the master held it at node_id=0, clusterer never learned it, and RESYNC fired once a second indefinitely. This had been masked before the membership-authority change: clusterer's wire self-discovery quietly re-added the node at the BIN level, hiding the controller-level livelock. The yield path now calls cl_ctr_rejoin_superior_master() - identical to the beacon merge - which demotes, arms the dead watchdog, and sends the JOIN_REQ (join_pending-guarded, so an exchange already in flight is not stomped). The function takes the discovery vector as a string so the merge logs say which path found the superior. (cherry picked from commit 5084afa9219feb361a215312263de4a7b96b5864)
Five new commits: membership authority, end to endThese grew out of one operational question — "when the controller expels a dead member, why is its BIN/TCP connection still open?" — and each fix exposed the next layer. All verified on a real 3-node cluster, with the member's control plane isolated by firewall while its BIN plane was deliberately left reachable (the hung / one-plane-partitioned case the purge exists for). 1. 2. 3. 4. 5. Verified, same harness both directionsExpulsion: purge fires at the deadline, clusterer drops the node in the same tick, established BIN connections decay to 0 and stay 0 — no echo, no resurrection. Re-admission (isolation lifted): within 25 s — yield |
2 participants
clusterer_controller — self-forming HA coordination for OpenSIPS
clusterer_controllersits on top of theclusterermodule and removes thestatic
my_node_info/neighbor_node_infowiring: nodes sharing a multicastgroup, a
cluster_idand a password self-organise — they discover eachother, elect a deterministic master (highest IP, sticky), assign
clusterernode-ids, hold sharing-tags on exactly one node, and fail over automatically —
with an encrypted control plane and zero per-node topology config.
This revision lands the unicast / scalability series (steady-state and
join cost cut from O(N²) → O(N), with anti-entropy repair and reliable
handshakes) and a module-wide
cc_→cl_ctr_rename so every symbolmatches the public
cl_ctr_*MI/pvar/function surface.Control plane at a glance
Encrypted UDP on the cluster's multicast group (default
239.0.10.x:3333).Only a small cleartext header is visible on the wire —
magic (2B)+cluster_id (2B)+nonce— everything else is AEAD-sealed(XChaCha20-Poly1305 + Argon2id, key agreement via Noise_NNpsk0).
Two key tiers, told apart by the magic byte:
0xCC01JOIN_REQ,KEY_GRANT,ACK,JOIN_REJECT,MASTER_BEACON0xCC00MASTER_ALIVE(+liveness bitmap +membership digest),ALIVE,NODE_ASSIGN,MEMBER_LIST,RESYNC,GOODBYE,KEY_HANDOFFLegend for the diagrams below:
-->>unicast (1:1),-)multicast (group).1. Cluster formation (cold start)
Nodes start together, discover over multicast, and converge on the highest-IP
node as master with no split brain — lower-IP nodes defer self-promotion while
a higher-IP peer is still joining.
sequenceDiagram autonumber participant N1 as .191 participant N2 as .192 participant N3 as .193 highest IP Note over N1,N3: simultaneous cold start, no master yet N1-)N3: JOIN_REQ (mcast, Noise msg1) N2-)N3: JOIN_REQ (mcast, Noise msg1) N3-)N1: JOIN_REQ (mcast, Noise msg1) Note over N1,N2: higher-IP peer still joining -> defer self-promotion Note over N3: highest IP -> self-promote, mint session key N3-->>N2: KEY_GRANT (unicast, Noise msg2 + salt) N2-->>N3: ACK N3-->>N1: KEY_GRANT (unicast) N1-->>N3: ACK N3-->>N2: NODE_ASSIGN + MEMBER_LIST (unicast) N3-->>N1: NODE_ASSIGN + MEMBER_LIST (unicast) Note over N1,N3: converged - .193 master, .192 backup, .191 member2. Member join into a running cluster
The join is unicast to the joiner: only the newcomer needs the full peer
list, so the master sends it 1:1 instead of multicasting N packets that every
member would decrypt and discard. Only the newcomer's own assignment is
multicast, so existing members learn it. A lost
KEY_GRANTis recovered byACK + retransmit (§5), a lost snapshot by the
JOIN_REQretry.sequenceDiagram autonumber participant J as .191 joiner participant E as .192 member participant M as .193 master J-)M: JOIN_REQ (mcast, Noise msg1 + BIN socket) M-->>J: KEY_GRANT (unicast, session key) J-->>M: ACK (unicast) M-)E: NODE_ASSIGN newcomer .191=id3 (mcast -> all learn it) M-->>J: NODE_ASSIGN .193=id1, .192=id2 (unicast, peers) M-->>J: MEMBER_LIST 3 members (unicast, snapshot) J-)M: ALIVE (joiner now participates) Note over J,M: per join the group sees ONE packet, not N3. Steady-state liveness — master-mediated (O(N²) → O(N))
Previously every node multicast an
ALIVEeveryquery_timeand every node ranan election off every one — O(N²) packets and cluster-wide decrypts. Now a settled
member unicasts its ALIVE to the master; the master folds all peers' liveness
into a per-node-id bitmap on its
MASTER_ALIVE, and members trust that bitmapto keep their election windows populated.
sequenceDiagram autonumber participant A as .191 member participant B as .192 backup participant M as .193 master A-->>M: ALIVE (unicast, ~query_time) B-->>M: ALIVE (unicast, ~query_time) M-)A: MASTER_ALIVE (mcast, ~1/s) + liveness bitmap + membership digest M-)B: MASTER_ALIVE (mcast) + bitmap + digest Note over A,B: refresh last_seen from the master's bitmap - no peer-to-peer ALIVE4. Anti-entropy — membership digest → RESYNC
Every
MASTER_ALIVEcarries a digest = active-peer count + order-independentXOR of
FNV-1a(node_id, ip). A member whose computed digest differs has missed aNODE_ASSIGN(whole peer, or just a node-id) and pulls a rate-limited RESYNC;the master coalesces a burst into one re-broadcast. This is the multicast
counterpart to the 1:1 ACK path.
sequenceDiagram autonumber participant Mem as .192 member participant M as .193 master M-)Mem: MASTER_ALIVE + digest = count + XOR hash Note over Mem: local digest != master's, missed a NODE_ASSIGN Mem-->>M: RESYNC (unicast, rate-limited) M-)Mem: NODE_ASSIGN (all peers) + MEMBER_LIST (re-broadcast) Note over Mem: digests match again5. Reliable handshake — ACK + bounded retransmit
The 1:1 handshake rides unreliable UDP. A lost
KEY_GRANTused to cost the joinerits whole join window; now the master retransmits until ACKed (bounded), so a
single loss heals in milliseconds.
sequenceDiagram autonumber participant J as .191 joiner participant M as .193 master J-)M: JOIN_REQ M-->>J: KEY_GRANT (unicast) x lost Note over M: no ACK within retransmit timeout M-->>J: KEY_GRANT (retransmit) J-->>M: ACK Note over J,M: recovered in ms, not a full join window6. Graceful leave (GOODBYE)
A departing node multicasts
GOODBYE. Survivors re-elect locally off the sameevent — no MEMBER_LIST re-broadcast (that O(N), fragmenting packet was dropped from
the failover path);
MASTER_ALIVEre-asserts and the digest reconciles stragglers.sequenceDiagram autonumber participant D as .191 leaving participant B as .192 backup participant M as .193 master D-)M: GOODBYE (mcast) D-)B: GOODBYE (mcast) Note over B,M: local re-election off the same GOODBYE - no re-broadcast Note over M: still master, 2 members - no role change M-)B: MASTER_ALIVE + digest (count now 2)7. Master failover (crash)
The master goes silent; members detect the missed
MASTER_ALIVEkeepalives and runthe same deterministic election locally — the backup (highest-IP survivor)
promotes and asserts via
MASTER_ALIVE.sequenceDiagram autonumber participant Me as .191 member participant B as .192 backup participant M as .193 master Note over M: master crashes x Note over Me,B: MASTER_ALIVE keepalives stop -> timeout Note over Me,B: identical local re-election -> highest-IP survivor wins Note over B: .192 promotes to master B-)Me: MASTER_ALIVE (new master asserts) + digest Me-->>B: ALIVE (unicast to new master) Note over Me,B: converged - .192 master, .191 member8. Graceful master handoff (KEY_HANDOFF)
On a planned master shutdown the outgoing master hands the session key to its
successor sealed to the successor's long-lived X25519 pubkey (
crypto_box_seal,learned from ALIVE) — so the successor takes over without a rejoin.
sequenceDiagram autonumber participant S as .192 successor participant M as .193 master leaving Note over M: planned shutdown M-->>S: KEY_HANDOFF (unicast, session key sealed to S's pubkey) M-)S: GOODBYE (mcast) Note over S: already holds the key -> promotes with no rejoin S-)S: MASTER_ALIVE (asserts as master)9. Wrong-password rejection
A node with the wrong password cannot produce a bootstrap-valid
JOIN_REQ. Afterrepeated failures from one source the master emits an authenticated
JOIN_REJECT;independently, a genuinely-wrong-password joiner self-terminates after its defer
budget rather than forming a lone split-brain master.
sequenceDiagram autonumber participant R as .200 wrong password participant M as .193 master R-)M: JOIN_REQ (bootstrap AEAD fails to authenticate) Note over M: repeated bootstrap-decrypt failures from .200 M-->>R: JOIN_REJECT (unicast, GCM-authenticated) Note over R: in NODE_NEW state -> exit(-1), no split brain10. Split-brain merge (MASTER_BEACON)
If a partition heals and two masters exist, each periodically emits a
bootstrap-keyed
MASTER_BEACON(readable across different session keys) carryingits
member_count. The inferior master (smaller count, IP tiebreak) yields andrejoins the superior one.
sequenceDiagram autonumber participant A as .191 master, 1 member participant B as .193 master, 3 members Note over A,B: partition heals - two masters B-)A: MASTER_BEACON (bootstrap-keyed, member_count=3) A-)B: MASTER_BEACON (member_count=1) Note over A: inferior (fewer members) -> demote A-)B: JOIN_REQ (rejoin the superior master) B-->>A: KEY_GRANT + NODE_ASSIGN + MEMBER_LIST (unicast) Note over A,B: single cluster, one master11. Shared-port unicast demux (multi-cluster on one node)
When several controller clusters on one node share a multicast port (distinct
groups), the kernel demuxes multicast by group but unicast by port alone — a 1:1
reply can land on the wrong sibling's socket. The receiver recovers it: on a
cluster_idmismatch it forwards the still-encrypted datagram to the correctlocal cluster's worker via
ipc_send_rpc(), which decrypts it with its own key.sequenceDiagram autonumber participant P as peer (cluster 2) participant W1 as worker cluster 1 participant W2 as worker cluster 2 P-->>W1: unicast reply for cluster 2 (arrives on wrong socket) Note over W1: cleartext cluster_id=2 != 1 W1->>W2: ipc_send_rpc(still-encrypted datagram) Note over W2: decrypt with cluster-2 key, process normally Note over W1,W2: forwarded once, never re-forwarded - no loopComplexity impact
Module parameters
clusterid=(required),multicast=A.B.C.D:PORT(required),password=(optional, overrides the globalpassword),bin_socket=bin:IP:PORT(optional, required only when multiple clusters are defined),manage_shtags=0|1(optional, overrides the globalmanage_shtags). Repeat for multiple simultaneous clusters.my_ipinterface.interfacemy_ipis set.query_time5passwordcluster)3eCrEt*5629(change in production)manage_shtagscluster)1$shtag()changes on managed clusters). When 0, sharing tags behave exactly as stockclustererand are left to scripts/MI.master_stickinesscluster)11(sticky): a live master keeps its role when a higher-IP node joins — only the backup slot changes.0: pure highest-IP election — a higher-IP node takes over as soon as it appears.on_config_mismatchreject|warn|adoptrejectmanage_shtags/master_stickiness/query_timediffer from the running cluster's:rejectrefuses the join (JOIN_REJECT),warnlogs once and allows it,adoptmakes the joiner take the cluster's values.Compatibility & deployment
RESYNCtype, ALIVE/MASTER_ALIVElayout, unicast routing). All nodes of a cluster must run this build — old and
new cannot interoperate; deploy with a coordinated cutover. A node that doesn't
understand the digest simply advertises none and is never asked to resync.
cc_→cl_ctr_/CC_→CL_CTR_rename (functions, types,macros); the KDF label and bootstrap salt were renamed too, which changes the
derived keys — another reason for the coordinated cutover. Wire magic bytes
(
0xCC…) are unchanged.Testing
Validated on netns rigs and a live 6-node test cluster (2 controller clusters):
cold start → deterministic single master; single / chained / rapid-double failover;
higher-IP join → sticky backup; two clusters sharing port 3333 → both converge with
misdelivered unicast correctly re-routed; wrong-password node rejected without split
brain; staged stop→rejoin captured on the wire confirming the unicast join path.