Repository navigation
Run server-zd hot standbys, takeovers, and step downs - #1168
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
longquanzheng
force-pushed
the
zd-hot-standby-server
branch
from
November 1, 2026 20:09
4d9f633 to
da11bd8
Compare
LeaseConfig gains held_lease_poll_limit, which replaces the fixed one-second limit on reads of a lease another process holds, and an optional acquisition delay provider. The waiter asks the provider once for each released or expired lease version and reads the lease again after the wait, so a hot standby can take the lease before a lagging one. The conditional lease write still decides who acquires. read_node_lease reads the lease without competing for it. LeaseConfig moves next to its acquisition methods, which keeps lease.rs under the file size limit.
NodeMetrics gains the hot standby state, lag in batches and seconds, a recent-error gauge, takeover counts by standby mode, and a takeover duration histogram. The server records them from the standby status. The dex.HotStandby gRPC health name reports SERVING only while the process is a hot standby within its lag threshold. Readiness names are unchanged, so rollouts can wait for a hot standby without routing to it.
standby.mode defaults to cold, the behavior of earlier releases. hot follows the log while the process waits for the lease. The section also sets the held lease poll interval, the step-down hold-off, the delay of a standby that is not hot, and the lag threshold. hot rejects lease.failIfHeld and dev accepts only cold.
…step down The server now loops: wait for the lease, serve, and on an AdminService step down release the lease, stand by, and compete again after the hold-off. A lost lease, fencing, or another failure still exits the process. Each serving period owns its engine, registry, dispatch, Stream Store, search projection, and Attribute Sync, and stops them before the lease release. The process keeps the gRPC listener, metrics, health, the lease watchdog, and the object store clients. In standby.mode hot, a waiting process runs HotStandby, publishes its state, lag, and recent error as metrics and as the dex.HotStandby health name, and takes over from its warm store. A standby that is not hot waits laggingTakeoverDelay before it writes an available lease. An error before the lease keeps the standby following and retries with backoff. AdminService is registered behind admin authentication and outside the startup gate, so a standby answers describe and step down calls with FAILED_PRECONDITION. A step down checks its node epoch precondition, keeps the old gRPC instance until its answer is sent, then retires that instance with GOAWAY. The answer names the successor, this process taking the lease back, or an unknown holder. An unconfirmed release disarms the lease deadline and stands by until the lease expires.
The failover harness learns hot standby slots, the dex.HotStandby health name, node role and hot standby state metrics, AdminService calls, and the node's log seq range. Tests cover the hot standby failure matrix: a killed, terminated, or frozen active; a standby killed while following, while bootstrapping, or during its takeover; both processes killed; object storage faults at the standby; three processes with one bootstrapping; a standby that log cleanup passed; and late writes of a stalled active. Step down tests cover a handover to a hot standby, taking the lease back without a standby, a step down racing a crash, a behind standby, ten switchovers under load, describe, the node epoch precondition, and calls without an admin credential.
… load A takeover error that returns the standby keeps it following and retries; a stopped standby reports the inactive state. The harness gives the first active a 6 s lease, which rides out host stalls when many tests run at once. The lease reader retries a failed read. The EpochStart count is only a lower bound for serving periods when log cleanup keeps the whole log. The router closes relayed connections after about 2 s of unanswered probes, and the hot takeover bound includes the lagging delay that a loaded host can trigger.
The failover tests start at most four node groups at once, at least two, a quarter of the host's CPUs in between. More groups starve each other's lease renewals. The hot standby metric families join the other closed-label families in fixed_labels.rs, so the metrics crate keeps ten source files. The caller spawns and owns the task that retires a gRPC instance. The engine stop budget also covers a cancelled takeover's lease release.
The server-zd operations page gains a Hot standby section between Lease and Step down, in English and Simplified Chinese. It covers the standby configuration, takeover order, health names and probes, and a two-replica StatefulSet with an OnDelete update. The metrics table lists the standby families. The sentences that said server-zd does not serve AdminService yet are removed from the operations page, the CLI reference, and the server-zd README, which also documents the standby section and metrics.
The serve loop checks a step down's StepDownPrecondition, node ID and node epoch, with is_satisfied_by under the same guard that starts the step down, and refuses a mismatch with PreconditionMismatch. The retiring gRPC instance waits 250 ms after the answer before GOAWAY. The P9 hot standby test follows dexcli's default flow, describe and then a step down that expects that node lease, and checks that a retry with the old precondition leaves the new active unchanged and that a step down for another node ID changes nothing.
The behind standby test slows only the log reads, so the standby's lease reads and writes fit in the step down hold-off. A failing scenario keeps its directory until the node group has dumped the process logs.
The process lifecycle and serving node start futures are large enough to overflow a 2 MiB test thread stack in the e2e tests. Boxing them keeps the awaiting futures small.
Reads checked only whether the registry halted, and the lease was checked only at command commit. A process resumed after a freeze could therefore answer FlowService reads from stale state after another process took the node over, in hot and cold mode alike. The request gate now holds the lease validity deadline. A FlowService or WorkerTaskService call answers UNAVAILABLE "the node lease is not valid" when the lease is not valid as the call arrives or as its response goes out. Health checks of the readiness names answer NOT_SERVING while the lease is not valid, whatever the reported status; dex.HotStandby and Watch streams are unchanged. P3 now freezes the active in hot and cold mode, lets the standby take over and acknowledge a new Flow, and requires every read and health check queued at the old active to be refused. Without the gate, the reads return NotFound and the health checks SERVING.
…andby reopens the store A standby phase now runs its own gRPC instance, which closes each connection after 3 s with a 1 s grace. A client connected to a standby therefore reconnects instead of receiving UNAVAILABLE forever. At each takeover the process starts a serving instance without an age limit and retires the standby instance like a stepped-down one. A step down now drains the public calls of the retiring instance, until their response bodies end, within the drain budget, and waits up to 5 s for the period's local store to close before a standby or a take-back reopens it. Before honoring a step down, the serve loop checks again whether the engine stopped or the registry halted, and then answers Internal and exits instead. Finished retirements are pruned. The server node's budget tests move to their own file.
The engine now reports when a node's local store has closed, including a store that a hot standby's restore replaced. The serve loop takes the waiter before the engine stops and waits on it, instead of tracking its own weak handle to the store.
A client whose first connection lands on a standby while another process is active now reaches the active once the standby connection ages out, keeps its connection to the serving instance, and reaches the successor soon after a switchover. Removing the standby connection age or the serving instance swap fails the test. A step down while readers keep summary and search calls in flight at the active leaves it a hot standby that never retried its bootstrap. In local runs the reads end before the engine stops, so this test does not fail when the drain and the store close wait are removed.
The lease check now tells release handovers from expiry handovers and measures a release handover from the first sample that showed the lease released, not the last. Hot scenarios bound every release handover by 800 ms, P9 without a standby requires the take-back within 1.5 s after the hold-off, a behind standby within 3 s, and P7 every step down within 600 ms. With the hot lease poll set back to 1 s, P7 fails. P2 bounds the time from SIGTERM to the hot standby serving. P10 now holds the third process's log reads until it reports behind and requires it to log a deferral of at least 1 s. Taking the hot branch of the acquisition delay for every standby fails P10. Hot takeovers must serve the hot standby's store and replay at most 64 batches, read from the process log, because the engine reports the store source only to tests. Every standby must send no object storage write from the time it is confirmed a standby until the next induced failure or the end of the test, counted from s3mock's per-operation counters.
…t hold-offs only During the step down hold-off, the read that names a successor now races cancellation and times out after 1 s, so a slow object store never delays a shutdown. A failed hot standby logs its failure once instead of every 500 ms. stepDownHoldOff must be positive, and must exceed the lagging delay plus the held lease poll only in hot mode, where both apply. When log cleanup is on, the failover harness checks that the serving periods never outnumber the highest lease epoch instead of skipping the check.
…lerts The operations page, in English and Simplified Chinese, lists AdminService among the served services, drops the sentence that the admin audience setting has no effect yet, and makes the step down hold-off rule hot-only. Its hot standby section now says that a standby closes client connections after about 3 seconds, that a process whose lease is no longer valid refuses public calls and readiness at once, recommends a 10-second lease for standby deployments and relates it to Attribute Store write transactions, and suggests alerts for a missing, failed, restarting, or lagging standby. The server-zd README describes the same behavior and the step down sequence.
The hot late writes test freezes the active during a held lease renewal and requires 412 for late lease and manifest writes, like the cold test. Step downs record a possible lease release for the lease check.
With PostgreSQL or ParadeDB visibility, the active projects Flows, steps down to a hot standby, and the new active's SearchOutbox consumer then projects writes to Flows started before and after the step down. Each serving period builds its own watermarks, mode rebuilder, and projection start from its engine, and stops its projection before the engine stops. The test runs where DEX_ZD_POSTGRES_URL or DEX_ZD_PARADEDB_URL is set, as in the PostgreSQL and ParadeDB CI jobs.
Health statuses are shared by every gRPC instance of the process, so a standby phase instance answered SERVING once the process served, while its FlowService and WorkerTaskService still refused calls until it retired. A client that waited for readiness on that connection then got UNAVAILABLE "the node is starting". Readiness names now answer NOT_SERVING on a standby phase instance, so a client reconnects to the serving instance first.
The lease history samples every 100 ms, so the first sample that shows a release can trail the release, and a take-back right after the hold-off then measures slightly under it. Linux CI measured 4.998 s. The window now starts 250 ms before the hold-off.
free_loopback_address bound port 0 and released it, so an outgoing connection could take the port before the server bound it, and the server exited with "Address already in use". This happened in Linux CI and locally once the failover tests ran many processes. Ports now come from 20000 to 32000, below the ephemeral ranges of Linux and macOS, from a random start and a per-binary counter, each checked free by a bind.
dexzd_takeovers_total gains a store_source label, hot_standby or restored, from the engine's report whether the node serves its hot standby's store. dexzd_takeover_duration_seconds is now a histogram family by the same label. The hot takeover tests require the slot to report a takeover from the hot standby's store. The metrics tables in the operations page, in both locales, and the README describe the label.
A step down and a shutdown now close the Stream Store first, so waiting ReadStream calls end at once. Only in-flight RPC tasks finish before the release. Other open public calls no longer delay it; the lease gate refuses their answers after the release. After a step down the process waits for those calls and for the local store to close before it reopens the store, and keeps answering admin calls and the pending step down meanwhile. On SIGTERM the gRPC server drains after the release, within the same 4 s budget. Tests: P7 keeps a waiting ReadStream at each active and bounds every step down; P2 and the in-flight step down test hold a hard-delete read across the handover with a key-prefix delay on the active's endpoint.
The lease gate checked a PollWorkerTask call only when it arrived and at its response head, so an open stream could still receive Tasks built from stale state after the lease lapsed. WorkerDispatch now takes the node's lease validity deadline and checks it before it picks a Worker for each Task; a dispatch then ends as ServerClosing. The operations guide now names the exceptions to the gate: other frames of an open PollWorkerTask stream and gRPC health Watch streams.
The SQL step down test now steps the second active down again, so the first process serves a second period with its own SQL search wiring. Its writes, and the earlier periods' Flows, must reach the SQL projection. CI runs it against PostgreSQL and ParadeDB.
A standby-phase gRPC instance retires 250 ms after a takeover and ages its connections after 3 s, so a step down that reached it just after the takeover lost its answer when the serving process stepped down through it. Such instances now forward step downs with a mark, and a serving process refuses them as NotActive at once; the caller retries on a serving connection. The new serve test reproduces the race by stepping down on an old standby connection as soon as it describes the new active. StartupService also takes its drain token with the service clone, before its checks, so a drain that starts after a call was accepted waits for it.
worker_registry.rs went over the 1000-line limit of rule R1. The lease constructor now takes an optional deadline and new_with_metrics passes None, so the private builder is gone.
The store_source label doubles the takeover counter and duration series, so the one-Flow scrape now has 163 series; raise its bound to 175.
Move the details of the new tests and the stop order into short inline comments, so each new comment block stays under 20 words. The operations guide now says that a step down reaching a standby connection right after a takeover answers FAILED_PRECONDITION and should be run again.
longquanzheng
force-pushed
the
zd-hot-standby-server
branch
from
November 1, 2026 22:54
bdbe925 to
54a8988
Compare
This was referenced Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #1166, now merged. It also merges #1167's engine branch until #1167 merges.
Summary
standbysection:mode:cold(the default) orhot;heldLeasePollInterval;stepDownHoldOff;laggingTakeoverDelay;behindLagThreshold.LeaseConfig.held_lease_poll_limit,LeaseAcquisitionDelay).DescribeActiveNode, andStepDownActiveNodewith aStepDownPrecondition.FAILED_PRECONDITION.dexzd_hot_standby_state, the lag gauges,dexzd_takeovers_total{standby_mode}, anddexzd_takeover_duration_seconds, and a newdex.HotStandbyhealth name.Why
This is the server half of the hot standby:
Nothing in the server used them yet. A takeover still restored a snapshot and replayed the log, and AdminService answered
UNIMPLEMENTED.This PR wires them into the server process. A hot standby now takes over within about one lease poll after the lease becomes available: 100–200 ms in local runs. An operator can switch over with
dexcli admin step-down.Fixes for cold standbys too
Compatibility
standby.modedefaults tocold, which keeps the earlier behavior.standbysection, so upgrade the binary first.Testing
cargo fmt --all --check, Clippy with--all-targets -D warnings,make rust-rules-check,make copyright-check, andcd docs && npm run check.cargo test -p dexzd-server --test serve_process standby_failover::five times locally: 27 of 27 passed each time.serve_processbinary twice locally: 64 passed, 1 ignored.