[longquanzheng]

Summary

• A step down ends the calls still running. Right after the lease release, every FlowService and WorkerTaskService call whose handler is still running ends with UNAVAILABLE ("the server process stopped serving during the call; retry it").
  • Their service clones, and so their local store handles, drop at once.
  • Calls that finish before the release keep their answers.
  • PollWorkerTask response streams that already started stay open until their task ends or the old instance retires. They hold no store handle, and the stopped dispatch hands out no task.
  • AdminService calls, including the step down itself, keep running.
  • After a call, the gate also refuses the answer of a period that ended, so a later period's lease never lets it through.
• The stepped-down process waits for the old store before it continues. Before it reopens the store or competes for the lease again, it waits up to 4 s for public calls and up to 60 s for the old store to close. Both waits are cancellable.
  • If a stuck task still holds the store after that, the process exits 1 without taking the lease.
  • The step down then answers Internal, unless another process already took over.
• Only the newest gRPC instance accepts new connections. All instances share one listener. Each accepts through a gate that the next instance closes before it starts.
  • An instance that stopped accepting keeps its open connections. A connection it accepted while it stopped is handed to its successor (InstanceListener, HandedOverConnections), so no new connection meets a GOAWAY before its first call.
  • A serving instance has its own public service gates and serves from its start. A standby or startup instance never serves public calls.
  • When a serving period starts, the replaced standby instance stops accepting, answers public calls on its open connections with a reconnect refusal, and sends GOAWAY at once instead of after 250 ms.
  • A stepped-down serving instance stops accepting without GOAWAY. It retires 250 ms after the step down answer goes out, so that answer still reaches its caller.
  • Together these remove the "the node is starting" refusal that a call could get after readiness reported SERVING.
• Docs. The ActiveNodeAdministration contract, the server-zd README, and the operations guide (English and Simplified Chinese) describe the new behavior.
• Shutdown and failure stops are unchanged.

Why

#1174: after a step down, a process that took its own lease back could exit with Internal instead of serving.

• An open public call kept a clone of the old period's service alive, and so its RocksDB store.
• The lifecycle waited 4 s and then 5 s, and continued anyway.
• A cold take-back then opened the store, and the per-process directory registry refused it (WouldBlock).

Reproduced with a 15 s hard-delete Attribute read held across the step down, and no other standby:

• Cold: the step down answered Internal 9.1 s after the call, and the process exited.
• Hot: the process held the lease for about 3.9 s while it could not serve. The take-back then re-armed the shared lease validity deadline, so the old period's read answered OK even though that period no longer held the lease.

This PR's first CI run also exposed a startup race from #1168. After the swap, the standby-phase instance kept accepting connections for 250 ms, but its public gates were never activated. A server integration test got "the node is starting" 223 ms after readiness reported SERVING.

A first fix shared the gates between instances. It let a retiring instance cut Worker streams, and a later fix that served public calls only on serving instances made a health Check on a just-accepted connection fail with Internal "h2 protocol error ... GoAway": tonic's accept loop does not prioritize its stop signal, so a replaced instance could still accept a connection and close it at once. The accept gate fixes that.

Design decisions

• Calls end after the release, not when the stop starts, so the commit drain still finishes in-flight work.
• 60 s is twice the 30 s object storage request timeout. A handle held longer is stuck, and only a restart clears it, so the process exits rather than following forever.
• No new ServerError variant. The exit uses ServingTaskFailed.
• A test-util-only engine API. The engine gains LocalStoreCloseWaiter::for_directory_for_test, enabled through dexzd-server's dev-dependencies.
• Hand over, not close. A connection accepted while an instance stops accepting goes to its successor through an unbounded channel. Closing it would fail the client's first call.

Known gap, not changed here

• Already on main: right after a take-back, connections still open on the old stepped-down serving instance can see readiness SERVING for up to about 250 ms while their public calls answer "stepped down". Tracked in server-zd: stepped-down instance reports readiness SERVING after a take-back for up to 250 ms #1210; smaller review tidy-ups are in server-zd: tidy-ups from the #1182 review (swap order, stale gate comments, comment length) #1211.

Testing

• standby_failover::step_down::{cold,hot}_take_back_ends_a_read_held_across_the_step_down
  • Before the fix, the cold test failed with Internal/WouldBlock, and the hot test's held read returned OK.
  • After the fix, the held read ends with the new UNAVAILABLE 92–173 ms after the step down, and the process takes the lease back after the 5 s hold-off.
  • Reverting the whole fix fails both tests. Reverting only the cancellation fails both on the message check.
• step_down_keeps_the_answer_of_a_call_that_ends_before_the_release: a transactional InvokeRPC that its Worker answers during the commit drain keeps its OK answer. Moving the call cancellation to the start of the stop fails this test.
• startup_calls
  • no_public_call_answers_starting_once_the_process_serves and no_public_call_answers_starting_once_a_take_back_serves send GetServerInfo over new and kept connections across a start or a take-back. No call sent after the first served call may answer "the node is starting", and every call on a connection opened after that point must be served.
  • a_worker_stream_kept_across_the_start_stays_open: a Worker stream is accepted only on an instance where it stays open.
• public_api_authentication::flow_and_worker_calls_need_valid_credentials, which hit the GoAway failure, passed 20/20 locally.
• Unit tests in node_lifecycle::local_store_close_tests, with paused time and a store that stays open:
  • The wait reports the store still open after 64 s, and cancellation ends the wait early.
  • A store still open answers a pending step down with Internal and ends the process with an error.
  • Treating "still open" as "continue anyway" fails a test, and so does a 5 s budget.
• Changed tests:
  • step_down_with_calls_in_flight_leaves_a_hot_standby now proves through the period-end message that its read was still open at the release.
  • A standby connections test retries a describe that races the replaced standby instance's GOAWAY, and records a serving period that may be too short for the harness router to see (NodeGroup::note_serving_period), so the harness EpochStart count allows it.
• Local runs on the final head: standby_failover three rounds 39/39, startup_calls 5/5. Earlier heads also ran the full serve_process suite and e2e 46/46. cargo fmt --check, make clippy, make rust-rules-check, make copyright-check, and cd docs && npm run check.
• CI is green on the final head, including the ParadeDB, PostgreSQL, and external S3 tests.

Fixes #1174

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!