clustering.md

Clustering Sable

Sable clusters use one writable primary and one or more read-only replicas. Every member answers DNS from its own local runtime, so losing the primary does not stop name resolution. Configuration, zone, authorization, integration, and secret changes stop until an administrator performs a manual promotion.

Sable does not claim quorum-based or partition-safe automatic failover. A recovery promotion is an operator decision, and the former primary must be offline before it is made.

Before enrollment

Prepare every node with:

• independent configuration, database, and cluster-state directories;
• a unique, stable node name;
• an absolute HTTPS advertise_url reachable by every other member;
• an HTTPS certificate whose names cover that advertised URL;
• one or more DNS service addresses that clients and operators can recognize;
• synchronized system time. Cluster heartbeats outside a 30-second clock window are rejected.

Do not share a SQLite database or /var/lib/sable between nodes. Replication moves control-plane state; local databases, caches, logs, certificates, and runtime files remain independent.

The cluster onboarding wizard can use an externally managed certificate, request one through ACME, import a PEM key pair, or generate a private self-signed identity. Enrollment pins the cluster and member certificate authorities or an existing self-signed server certificate, so private trust is supported without disabling TLS verification. A fresh self-signed installation defaults to Sable Private CA; existing ACME and configured certificate setups retain their source selection. Cluster trust does not make a certificate trusted by browsers automatically.

Create the primary

1. Open Cluster → Initialize Primary.
2. Configure the node name, cluster-state directory, advertised HTTPS URL, and certificate source. Initialization and reinitialization start at the Node step, even when saved settings exist. On Save & Continue, an existing private CA with Sable Private CA selected triggers Replace cluster CA?. Cancel saves nothing. Replace CA & Continue generates a new CA and certificate in a separate directory, retaining the previous files for recovery. This may break existing tokens and member trust; replicas may need to re-enroll. Choose Restart & Continue when prompted. It stops Sable cleanly; the native install's service manager restarts it and the browser resumes the final step. Manually launched processes need a supervisor or a manual relaunch.
3. Choose a stable cluster domain and review this node's prefilled DNS service addresses. Explicit listener addresses and nonstandard ports are preserved; wildcard listeners use matching local interface IPs. Edit the list to the addresses clients should actually use. The cluster domain identifies the deployment; it does not replace the member's advertised HTTPS URL.
4. Initialize the cluster and confirm that the local role is Primary.

Only the primary accepts cluster-scoped Settings changes, zone and policy mutations, user/role/token changes, integration changes, and RFC 2136 updates.

Enroll a replica

1. On the primary, create an enrollment token from the Cluster page. Tokens expire after 15 minutes, are stored only as SHA-256 digests, and are consumed by the first successful join.
2. On the new server, open Cluster → Join Existing Cluster and configure its node identity, advertised HTTPS URL, certificate source, and cluster-state directory. Restart if requested.
3. Enter the primary's advertised HTTPS URL, the enrollment token bundle, and the replica's DNS service addresses.
4. Wait until both nodes report Online and In sync at the same applied and current generation before relying on the replica.

When private trust is available, the enrollment bundle starts with sable-enroll-v1. and includes the primary trust anchor. A short token carries no private trust and requires the primary certificate to be otherwise trusted. The primary records the joining member's identity and certificate authority, and later synchronization uses those pinned authorities for member HTTPS connections.

Retry a failed first enrollment

For an empty cluster whose first replica could not join:

1. Update both nodes to 1.3.4 or newer. Preserve an application backup first.
2. On the primary, Delete Cluster returns it to standalone mode and preserves DNS data. Do not use this procedure to dissolve a cluster with working replicas.
3. Open Initialize Primary, review Node and HTTPS settings, and select Sable Private CA. If a CA already exists, explicitly confirm replacement on Save & Continue. Previous CA and key files remain in their original directory; the configuration points to a new replacement-* subdirectory.
4. Click Restart & Continue, finish initialization, then create a new token.
5. Configure the replica's own HTTPS endpoint, complete any requested restart, and join with the primary URL and complete new token. Verify both nodes are Online and In sync.

A long token confirms that trust was bundled, not that it matches the certificate currently served. If verification still fails, compare the served certificate with the token's trust anchor rather than repeatedly regenerating certificates. Certificate names must cover the advertised URL: a stable IP is fine, and a DNS name must resolve from every member.

This onboarding flow is not an in-place CA migration for an initialized cluster. Do not simply replace its private certificate with an unrelated CA or ACME certificate in Settings: existing members retain their pinned trust. Retained files are recovery material, not an automatic rollback or trust migration.

What replication carries

Each generation is a signed, content-addressed application snapshot written to the replica before its durable manifest advances.

Replicated state includes:

• authoritative zones, records, DNSSEC policy, and encrypted signing keys;
• resolver, cache, TSIG, blocking, and query-log runtime settings;
• devices ([[clients]]): their names, types, and blocking rule sets;
• Dynamic DNS settings and external provider credentials;
• UniFi settings and controller credentials;
• OpenID Connect settings, client secret, linked identities, and role mappings;
• alert settings, every destination with its URL, keys, and header values, the key browser pushes are signed with, and the browsers that turned alerts on;
• Insights settings: what each kind of finding does and the limits that make one;
• passkey availability (security.passkeys_disabled);
• users, roles, permission grants, password hashes, passkey public credentials, API-token hashes, and token revocations.

Node-local state includes:

• web, DNS, DoT, DoH, and DoQ listeners and certificate configuration;
• database paths, security bootstrap, cluster identity, and the advertised URL;
• the OpenID Connect callback override;
• update checks, installed binaries, and the updates.pre_release preference;
• browser sessions, audit history, token-use timestamps, query/server logs, dashboard history, and response caches;
• each node's record of which alerts it has sent.

Replicas validate and activate a complete candidate before recording the new generation. A rejected candidate leaves the previous runtime and manifest active.

Passkeys work across trusted node addresses under the same registrable domain. Credentials replicate, but browser sessions and in-progress sign-in challenges remain node-local. See Sign in with passkeys for naming and failover examples.

Monitoring and node queries

The Cluster page refreshes role, connectivity, applied/current generation, generation lag, last contact, and last successful synchronization. It also links to each advertised console.

The DNS client offers enrolled nodes as presets. A node preset queries /dns-query on that member's advertised HTTPS endpoint and reuses the certificate authority pinned during enrollment. If a preset fails while the console link works, confirm that DoH is enabled on the advertised HTTPS listener and that the URL hostname appears on the member certificate.

Prometheus exposes aggregate and per-node cluster gauges. The operations guide includes the metric names and routine health checks.

Alerts in a cluster

A cluster sends each alert once. Set up alerts on the primary in Settings → Alerts, and turn on browser alerts there too. Every replica gets the same settings, destinations, secrets, and browsers with the next generation, so any node can send after a handoff. The node that leads sends almost everything:

• Node down when a node has not checked in with the lead for five minutes, and Node back up when it checks in again. A node that is back within five minutes, such as one restarting for an update, alerts nothing. After the lead restarts or takes over, each node gets five minutes to check in before it can count as down.
• Rolling update started, finished, failed, or stopped. A failed rollout names the release and, when one node was to blame, that node. Each stays news for a day, so a lead that restarts to finish a rollout still sends it.
• Whatever each replica sees in itself, such as a failed backup. Replicas hand these to the lead in their heartbeats, once a minute or as soon as they change. If a replica stops reporting, its alerts stop counting as news after three minutes.
• Domain watches. Each replica reads its own query log for watched names and hands the lookups to the lead in the same heartbeat. The lead weighs them with its own, so a device that asks two nodes alerts once, naming every node that saw it.

The lead cannot report that it stopped answering, so replicas watch it. When the lead has been out of reach for five minutes, the replica with the lowest node ID sends Lead node not answering, using its own copy of the destinations, and Lead node back up when the lead answers again. A planned handoff that gives the cluster a new lead within five minutes sends nothing.

Problems, such as a node or the lead down or a failed rollout, go out at high priority. The cluster switch in Settings → Alerts turns all of these off together; a replica's own alerts follow their own types.

Each node keeps its own record of which alerts it has sent, so the first round after a replica becomes the lead only takes stock of what is news: the old lead already sent it.

A replica hands its alerts only to a primary that says it takes them, and a snapshot from an older primary leaves a newer replica's alert and Insights settings as they are, so alerts keep working while a rolling update runs mixed versions.

Devices in a cluster

Each node's Insights reads its own query log, but the addresses Insights ties to hardware come from the lead. The lead reads its host's neighbor table and the UniFi controller, and hands replicas what it learned: everything from the last two weeks when a replica first synchronizes and again every 12 hours, and the last 15 minutes' sightings once a minute. A replica in a container, or on a network where it cannot see client hardware addresses, then names devices and follows their rotating IPv6 addresses the way the lead does. A replica on an older release ignores what it is handed.

Going the other way, each replica tells the lead every five minutes which client addresses asked it something lately. That lets the lead's Insights tell a device that only uses one node from a device that doesn't use Sable at all. The lead holds that finding back until every replica has reported, so a replica on an older release, which never reports, keeps it from appearing.

Planned primary handoff

Use a planned handoff for maintenance when both nodes are healthy:

1. Confirm the target replica is online, fully synchronized, and running the intended version.
2. From the current primary's Cluster page, choose Promote to Primary on the target replica.
3. Wait for both nodes to show the new primary assignment and matching current generation.
4. Perform maintenance on the former primary, now a replica.

Sable keeps the recently demoted primary's synchronization endpoint available so every member can converge on the handoff without an out-of-band push.

Recovery promotion

Use recovery promotion only when the primary is unavailable:

1. Establish that the former primary is offline or isolated from clients and the management network. Do not rely on an unverified network partition.
2. On the replica being recovered, choose Promote This Replica and confirm the warning.
3. Verify DNS, the writable console, and the new primary assignment before admitting control-plane writes.
4. Do not return the former primary to service until it can reach the promoted node. Start it under observation and verify that it converges as a replica before directing clients or administrators back to it.

Sable does not merge independent writes from two primaries. If both nodes were writable during a partition, stop and preserve both data directories before choosing the authoritative state.

Remove, leave, or delete

• Remove Replica runs on the primary. It revokes that member and removes it from future generations; it does not erase files on the removed server.
• Leave Cluster runs on a replica. It stops synchronization and keeps the node's current DNS data as a standalone deployment. Remove the stale member from the primary separately.
• Delete Cluster runs on the primary. The primary becomes standalone and enrolled replicas leave when they next synchronize.

Preserve a backup before dissolving a cluster whose state may be needed later.

Rolling updates

Use Cluster → Rolling Updates on the primary to apply a reviewed release to all nodes. Sable updates replicas one at a time, verifies their running version and synchronization, then updates the primary last. See the update guide for requirements, failure handling, and manual upgrades of nodes that do not yet support this feature.

Manual rollout for older nodes or image-managed deployments

For a two-node deployment:

1. Create and export a backup from the primary.
2. Update the replica to an exact version.
3. Verify its DNS transports, console, and synchronization at the current generation.
4. Update the primary to the same version.
5. Confirm both member versions and run a node-specific DoH query against each.

Use sudo sable update --version VERSION for a pinned rollout. Cross-version compatibility is not yet a published long-term contract, so keep a mixed-version window short and do not combine an upgrade with a promotion unless recovery requires it.

For installations with web updates enabled, each node's About page can check, install, and restart that node, including replicas. Its Include pre-releases setting is saved locally and is not overwritten by replication.

Backup and recovery

A whole-deployment backup includes the local member's manifest, trust anchor, enrollment state, and content-addressed cluster snapshots. Back up the primary for the authoritative control-plane state.

A replica restore is normally overwritten by the next primary generation, so the console refuses it. Rebuild a replica by restoring the required node-local material and enrolling it again, or simply enroll a fresh node. When restoring a cluster that no longer exists, restore one node, start it in isolation, and use Delete Cluster to return it to standalone operation before building new membership.

See backup and restore for archive contents, passphrase handling, database overrides, and fresh-host recovery.

Troubleshooting synchronization

Symptom	Checks
Restart required	Complete the controlled restart before initializing or joining; the saved node identity is not active yet.
Token rejected	Create a new token. The previous token may have expired, been consumed, or been entered against the wrong primary URL.
Node offline	Test the advertised HTTPS URL from the other node, verify its certificate names and issuer, check both clocks, then inspect journalctl -u sable.
Generation lag grows	Inspect the replica log for validation, disk, vault, or activation errors. The prior generation remains active until the whole candidate succeeds.
Writes are refused	Confirm the local role. Cluster-scoped and RFC 2136 writes belong on the primary; follow the primary link from the Cluster page.
DoH preset fails	Confirm /dns-query is enabled on the advertised HTTPS listener and that the advertised hostname matches the pinned member certificate.
Former primary reappears after recovery	Keep it away from clients until it reaches the promoted node and reports the replica role at the current generation.

For the persistence and trust model, see the architecture. For configuration fields and replicated boundaries, see the configuration guide.