Repository navigation
Test hot standby recovery from failures and interruptions - #1167
Merged
Merged
Conversation
…dary A dropped standby, and one whose stop timed out and aborted its driver, now report Stopped. A takeover rechecks the followed cluster after the handover, when a standby that had not bootstrapped before the lease has. Unit tests pin that a cancelled handover keeps the driver and that a timed-out stop fences before it aborts. An engine test pins that a store whose positions end exactly at a closed stream's recorded end is reused. HotStandbyTakeoverError no longer repeats its cause, and its docs say when a released store may still be open.
Engine tests cover a standby that follows through seeded read failures, timeouts, and latency spikes without bootstrapping again; that keeps retrying through a ten-minute storage outage, while following or while it bootstraps, and recovers once storage heals; that restarts with a growing delay at a corrupt batch until it is repaired; and that restores the newest snapshot when retention deletes the one it was reading. Takeovers serve a standby's store while it is behind and across incarnation changes it has not seen.
A HotStandbyTakeovers profile of the engine simulation runs a hot standby beside the serving process. Most crashes of the serving process end in a takeover from it; takeovers crash at random points and standby processes crash or lose object storage. Every check requires that the standby never writes object storage and never fails, each takeover compares the serving store with the never-crashed twin, and after the calm tail the standby must follow the log again.
A start given an existing store must serve from it unless log cleanup passed the store's applied positions or a partition applied nothing. The simulation reports any other discard as store-discarded-without-gap and counts the explained ones.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
A following standby reports read_error once three log reads in a row fail, so an unreachable object storage is not mistaken for an idle node. A takeover whose standby bootstrapped into another cluster while the start waited for the lease fails with InvalidConfig instead of opening the store cold. Stopped and Failed are terminal, so a drop keeps the failure reason and an aborted driver cannot overwrite them. A test hook passes the commands a standby applies to a caller's sink, and restored comments keep their original wording.
The committed-log shadow now also records the frames a hot standby applies, so a warm takeover after a crash before the serving process applied a durable batch no longer differs from the twin. Standbys are checked for any object storage write their own process sends. The profile changes node incarnations, so standbys follow across segment boundaries, and the store-discard invariant uses the manifest each start read, covers takeovers, and accounts for snapshots. The profile joins the determinism test with a pinned digest and the nightly workflow, and its eight failing seeds run as regression cases.
A hot standby whose first bootstrap finds its store directory locked, as by a lingering handle of a node that just stepped down, now retries with the restart backoff instead of failing, and its last_error says that another process or a lingering handle holds the lock. LocalStoreCloseWaiter, from EngineNode::local_store_close_waiter and HotStandby::local_store_close_waiter, waits until no handle keeps the store open, so a caller can reopen the directory after a step down.
Opening a RocksDB store whose lock another process holds renames that process's info LOG before it fails. A hot standby now probes the directory's LOCK with the same fcntl lock RocksDB takes, without opening the store: a lock another process holds, such as the serving node's, fails it as a configuration error, and one a lingering handle of this process holds is retried. dexzd-storage tracks the directories this process opened, so the probe never touches a lock it holds itself. A test-only counter reports following periods that crossed into the next incarnation's stream.
At every twin comparison, each batch in the log that carries records of a partition, up to that partition's applied position, must have had its frame finished by some process. A standby that skipped a batch the crashed serving process never applied is now caught although the twin lacks it too. Standbys share their takeover's incarnation, the profile requires standbys that crossed a segment boundary while following, and the README describes the discard rule, the new invariant, and the profile's redeploys and write checks.
The last handle's drop closes a RocksDB store after its count reached zero, so a waiter that saw no handle could still race the lock's release. LocalStoreCloseWaiter now also waits until this process no longer has the store's directory open, which a new standby on that directory needs.
A store directory reached through a symbolic link or a dotted path now names the same registry entry, so a probe never takes and releases a lock this process holds through another path. Registering creates the directory first, so it can be canonicalized before the store opens. The probe closes its LOCK file before it releases the registry, and new tests pin that a probe leaves this process's lock to a second process, that a store unregisters only after its database closed, and that the close waiter waits while the directory is still open.
A hot standby can apply frames the serving process has not applied yet, so a check then compared the serving store with a twin that was ahead of it. Such checks now skip the comparison and count the skip, and the end of a seed requires the serving store to have caught up. The profile requires comparisons right after a warm takeover, and seed 1470103794 runs as a regression case.
EngineNode::is_serving_hot_standby_store is no longer test-only, so a server can label its takeover metrics and logs with whether a takeover served the standby's warm store or opened the configured store as an ordinary start does.
RocksDB's own lock table compares only path text, so a store opened again through a symbolic link succeeded, and closing it released the first store's lock to other processes. The registry now refuses a second registration of one canonical directory, and creates only the last level of a missing directory, as RocksDB did, so a mistyped path still fails.
The calm tail ends with one more twin comparison that only a pending hard-delete upload may skip. The hot standby profile allows at most one percent of comparisons skipped behind a standby and requires most warm takeovers to be compared right away, and the engine README names EngineNode::is_serving_hot_standby_store.
…ands wait_until_caught_up returned once the follower had read the last batch, but applier lanes apply read batches later on blocking threads. When the follower read every batch after a snapshot restore in one pass, the test could count applied commands before any lane ran. The helper now also waits until every partition of the standby store is applied through the last batch, and the snapshot test prints the standby status on failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stopped.HotStandbyTakeoverErrorno longer prints its message twice.tests/engine/hot_standby_faults.rs):Failed, backs off to its cap, catches up when storage heals, and then takes over correctly.HotStandbyTakeovers(dexzd-simulation). The fault schedule:Failed.Why
The hot standby (#1162) and the warm takeover (#1164) must recover from unusual failure and interruption cases, not only from clean ones. That includes:
These tests make that behavior repeatable. They fail when the recovery logic regresses. For example, if a transient error were classified as a configuration error, or if the backoff stopped growing, these tests would fail. The re-review of #1164 also asked for a few small fixes and tests, which are included here.
Testing
>to>=.cargo fmt --all --check, Clippy for the engine and simulation crates,cargo check --workspace --all-targets --locked,make rust-rules-check, andmake copyright-checkpass.Known limits:
(incarnation, end_seq)boundary; the engine test covers it.