feat(proxyd): consensus-gated /readyz and idle graceful shutdown

[artmakh]

Description

Two related changes that together let Kubernetes drive proxyd's startup and shutdown cleanly without dropping requests on either end.

1. Split /healthz and add /readyz

/healthz is unchanged — it returns 503 only when the server is draining, suitable for liveness probes (won't restart the pod on transient upstream/consensus flaps).

A new /readyz endpoint is added for readiness probes. It returns 503 when:

• the server is draining or shutting down, or
• any consensus-aware backend group has an empty consensus group / Latest == 0.

Otherwise it returns 200. Backend groups without consensus enabled are skipped, so /readyz behaves identically to /healthz for non-consensus deployments and is safe to wire unconditionally.

2. Idle-based graceful shutdown mode

Adds an opt-in graceful shutdown mode that swaps the existing "drain everything immediately" behaviour for an idle-based one. Behind a config flag; default behaviour is unchanged.

When enabled (graceful_shutdown_idle = true), on shutdown signal proxyd will:

1. Flip /readyz to 503 (so Kubernetes removes the pod from Service endpoints).
2. Keep /healthz returning 200 (so the kubelet does not restart the pod mid-shutdown).
3. Continue serving RPC and WebSocket requests normally — no 503 injection on / or /{authorization}.
4. Wait until no non-healthcheck request has been received for graceful_shutdown_idle_seconds (default 10).
5. Only then close the listeners and exit.

This lets in-flight clients finish naturally instead of seeing a wall of 503s the moment a pod starts terminating, while still giving k8s a clean signal to stop sending new traffic. Existing /{authorization} paths bump the idle timer the same way unauthenticated / requests do — both go through the same HandleRPC/HandleWS entry point.

New config (in [server]):

graceful_shutdown_idle = true
graceful_shutdown_idle_seconds = 10  # default if omitted

Suggested k8s probe wiring

livenessProbe:
  httpGet: { path: /healthz, port: <rpc_port> }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: <rpc_port> }
  periodSeconds: 2
  failureThreshold: 2

Implementation notes

• New HandleReadyz in proxyd/server.go: iterates BackendGroups and 503s if any consensus-aware group has len(GetConsensusGroup()) == 0 or GetLatestBlockNumber() == 0. Also 503s on isDraining or isShuttingDown.
• New Server fields: gracefulShutdownIdle bool, gracefulShutdownIdleDuration time.Duration, isShuttingDown atomic.Bool, lastRequestNanos atomic.Int64.
• HandleRPC and HandleWS bump lastRequestNanos at handler entry, before any auth/rate-limit work, so retries still keep the pod alive (matches the "is anyone still using me" semantics).
• Drain() branches on the flag: legacy path is untouched (sets isDraining, sleeps graceful_shutdown_seconds); idle path sets isShuttingDown only, ticks every 1s, and returns when time.Since(lastRequest) >= idleDuration.
• WebSocket caveat: the idle timer is bumped on connection upgrade, not per-message. A long-lived subscription with no new connects after SIGTERM will not extend the idle window. The legacy Drain() already closes WS abruptly, so this is not a regression — calling it out so reviewers are aware.

Tests

Added TestHandleReadyz in proxyd/server_test.go: a table-driven unit test covering the gate logic — drain, no backend groups, group without consensus enabled, empty consensus group, Latest == 0, ready group, and the multi-group case where any unready group fails the gate. Constructs *ConsensusPoller directly with an in-memory tracker, no real backends or goroutines required.

The new idle-shutdown path is not covered by a dedicated unit test — it is intrinsically time-dependent (ticker + time.Since against lastRequestNanos), and a faithful test would either need a fake clock injected into Drain() or a real-time wait, neither of which feels worth it for the size of the change. Manually verified end-to-end against a HA deployment (Redis-backed consensus tracker + Redis cache, multiple replicas):

• /readyz returns 503 during cold start and flips to 200 once the first consensus cycle completes; /healthz stays 200 throughout.
• /readyz returns 200 on a deployment with no consensus-aware backend groups.
• Killing a backend so the consensus group empties flips /readyz back to 503 while /healthz remains 200.
• With graceful_shutdown_idle = true and no traffic, Drain() returns after ~idle_seconds from start.
• Sending a request every second keeps Drain() blocked; stopping traffic causes it to exit ~idle_seconds later.
• During the idle wait, /readyz returns 503 while /healthz and / keep returning 200.
• With the flag off, shutdown behaviour is byte-identical to before (sleep for graceful_shutdown_seconds, then shut down).

Happy to add a fake-clock unit test for Drain() if reviewers want one.

Additional context

Two motivating problems, both observed on rollouts of a HA proxyd deployment (Redis-backed consensus tracker + Redis cache):

1. Cold-start error storm. proxyd starts accepting RPC traffic before backend probes have completed (probe_success_threshold defaults to 2 → ~8s minimum), before the consensus poller has run its first cycle (so GetConsensusGroup() is empty and tracker.Latest == 0), and before the HA leader has been elected (stateHeartbeat() skips lock acquisition while local state is invalid). During this window the cache is also cold, so every request becomes a miss that cascades into BackendGroup.Forward() and either returns ErrNoBackends or, in CL mode, ErrCLConsensusSyncNotReady (-32025). The result is a noisy spike of 5xx responses on every cold start. A k8s readiness probe is the standard fix: a 503 on /readyz keeps the pod out of the Service endpoints until consensus is ready, so the cold-start error storm never reaches clients. We deliberately did not overload /healthz for this — consensus state is allowed to flap during normal operation, and failing liveness on a transient consensus break would cause k8s to kill an otherwise-healthy proxyd.

2. Shutdown error storm. As soon as Drain() is called today, every RPC request gets a 503 for the full graceful_shutdown_seconds window — even though those requests could have been served correctly. K8s already handles "stop sending new traffic" via the readiness probe (now that /readyz exists); we don't need proxyd to also reject in-flight requests to achieve a clean rollout. The new mode delegates the "stop traffic" job to k8s and uses the idle window purely as a "is anyone still using me" check before exit.

The two changes are intentionally bundled because the idle-shutdown mode is only safe with the new /readyz endpoint: without a separate readiness signal, k8s would have no way to know it should stop sending traffic to a pod that is still happily serving on /.