server-zd: stepped-down instance reports readiness SERVING after a take-back for up to 250 ms #1210

Observed

Found in review of #1182 (head 68bbcf3); the same sequence exists on main before #1182.

After a step down that ends in a take-back by the same process, connections still open on the old stepped-down serving instance can see readiness SERVING for up to about 250 ms plus a round trip, while FlowService and WorkerTaskService calls on those connections answer "this server process stepped down".

Sequence:

1. The step down answer is sent only when the outcome is known; for a take-back that is when the take-back starts serving (node_lifecycle.rs, resolve_start).
2. The old serving instance retires through GrpcServer::retire_after, which sleeps ANSWER_TRANSPORT_GRACE (250 ms) after the answer before it sends GOAWAY (server_node/grpc_server.rs).
3. Meanwhile the new serving period calls set_serving() on the shared health reporter (server_node.rs, serve_until_stopped).
4. The old instance's health server was built with serves_public_calls = true (settings.health.server(phase == GrpcInstancePhase::Serving)), and the lease validity deadline was re-armed by the take-back, so LeaseCheckedHealth::check (dexzd-api/src/health_status.rs) lets SERVING through on the readiness names "", dex.FlowService, dex.WorkerTaskService.

Readiness and the public gates therefore disagree on that instance for the window. A prober or client that reuses such a connection sees SERVING and then a stepped-down refusal.

Where to look

• server-zd/crates/dexzd-server/src/server_node/grpc_server.rs: GrpcServer::start, deactivate_public_services, retire_after.
• server-zd/crates/dexzd-api/src/health_status.rs: LeaseCheckedHealth.
• server-zd/crates/dexzd-server/src/server_node/node_lifecycle.rs: the step down path.

Suggested fix (hypothesis, not validated)

Give each instance's health server a per-instance flag that deactivate_public_services sets, and have LeaseCheckedHealth::check answer NOT_SERVING on readiness names once it is set; or make serves_public_calls dynamically switchable.

Acceptance

• After deactivate_public_services on an instance, a unary health Check on a readiness name over a connection to that instance answers NOT_SERVING, including after a take-back re-arms the lease deadline.
• A process test in server-zd/crates/dexzd-server/tests/serve_process/standby_failover/step_down.rs (or startup_calls.rs) keeps a connection to the active process across a step down plus take-back, and checks that no readiness Check on it answers SERVING while a public call on the same connection answers stepped down.