network: detect a wedged discv5 socket (operator health check + boot-node /healthz) - #2982
network: detect a wedged discv5 socket (operator health check + boot-node /healthz)#2982iurii-ssv wants to merge 3 commits into
Conversation
Greptile SummaryThe PR adds socket-read tracking so operator and boot-node health checks can detect wedged discv5 listeners.
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains established. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| network/discovery/timed_conn.go | Adds an atomic timestamping wrapper around successful discv5 UDP reads. |
| network/discovery/dv5_service.go | Routes the post-fork listener through TimedConn and exposes discovery staleness. |
| network/p2p/p2p.go | Extends operator health reporting to reject stale discovery sockets. |
| utils/boot_node/health.go | Implements boot-node health evaluation from routing-table occupancy and socket-read freshness. |
| utils/boot_node/node.go | Wires TimedConn into the boot-node listener and serves the new health endpoint. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
UDP["discv5 UDP socket"] --> TC["TimedConn records successful reads"]
TC --> OP["Operator DiscoveryStale"]
OP --> PH["p2p Healthy"]
PH --> OW["Operator watchdog restart"]
TC --> BH["Boot-node health check"]
RT["Routing table state"] --> BH
BH --> HZ["/healthz"]
HZ --> BL["Boot-node liveness restart"]
Reviews (2): Last reviewed commit: "utils/boot_node: add /healthz tied to di..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
7bc0d99 to
c9d0cc1
Compare
|
@greptile pls re-review |
There was a problem hiding this comment.
Sound, well-reasoned design: the socket-read timestamp is a genuinely cause-agnostic wedge signal, it is wired to the only listener that actually drains the socket, and greptile's 'quiet socket' P1 is properly fixed by the populated-table gate (verified against geth's 3s revalidation interval, not just the comment thread). The one thing worth changing before merge is the operator path, which cannot distinguish 'wedged' from 'never received any UDP' and will crash-loop a node that restarts while its bootnodes are unreachable; the rest are minor robustness/observability/test-wiring points. [verdict: with_fixes]
c9d0cc1 to
b9f0d08
Compare
b9f0d08 to
a16e6da
Compare
| // loads seed nodes from the persistent enode DB straight into the table, so | ||
| // AllNodes() can be >0 before the socket has been read once — and StaleFor stays | ||
| // disarmed until that first read. Those seeds then fail revalidation and age out, | ||
| // the table empties, and the empty-table grace trips instead. A runtime wedge |
There was a problem hiding this comment.
I fear the fallback described here doesn't hold under a dispatch-loop wedge.
Table entries are only deleted in tableRevalidation.handleResponse, and revalidation itself goes through the wedged dispatch goroutine (initCall blocks on t.callCh, which that goroutine drains), so the seeds never age out, AllNodes() stays >0, lastNonEmpty keeps refreshing, and /healthz stays green indefinitely.
The dispatch-wedge case is still caught (a prior read has armed StaleFor), but a stall where the socket never yields a read (kernel/conntrack-level) lands exactly in populated-table + disarmed-socket = healthy forever. Worth correcting the comment at minimum, or considering a self-probe: write a junk packet to the socket's own LocalAddr() and require LastRead() to advance, which covers it deterministically and independently of peers.
| // A wedged discv5 socket leaves discovery silently dead while bootstrap keeps | ||
| // looping; surface it so the hprobe watchdog restarts the node. n.disc is nil | ||
| // in tests and briefly at startup, hence the guard. | ||
| if n.disc != nil && n.disc.DiscoveryStale(discoveryStaleGrace) { |
There was a problem hiding this comment.
Could this lead to a restart loop on a healthy node that just has no inbound UDP?
After a single successful read ever, 3 minutes of inbound silence fails Healthy: e.g. inbound UDP blocked upstream (firewall/NAT/conntrack change) while the node keeps operating fine over its established libp2p TCP peers; the restart drops every peer connection and can't fix the network.
A total blackhole self-limits (the arm resets on restart), but a partial one the odd scan packet or NAT-refreshed reply arriving each boot re-arms every cycle and loops indefinitely.
Maybe stamp writes in TimedConn too and only declare a wedge when reads AND writes are both stale: all v5 sends funnel through the same dispatch goroutine, so a real wedge freezes both, while an inbound blackhole leaves writes flowing (nursery bootnodes are re-pinged every refresh).
| func (h *bootNodeHealth) check() error { | ||
| now := h.now() | ||
| if len(h.lister.AllNodes()) > 0 { | ||
| h.lastNonEmpty.Store(now.UnixNano()) |
There was a problem hiding this comment.
Seems that the empty-table clock is driven by probe arrivals rather than table state: if probing pauses (kubelet restart, or the livenessProbe manifest lands in a later rollout than the image), the first resumed check compares now against a stale observation and can fail with no effective grace.
Restarting a healthy pod. Conversely, a table flapping in and out of empty faster than the probe period resets the clock every time, so the grace never accumulates. Might be worth sampling the table from a small owned ticker and having check() read only the sampled state.
| return false | ||
| } | ||
| age, ok := dvs.socketConn.ReadStaleness() | ||
| recordDiscoveryReadStaleness(dvs.ctx, int64(age.Seconds())) |
There was a problem hiding this comment.
Should we distinguish the never-read state in the gauge?
Right now "read 200ms ago" and "never read since boot" both record 0 and never-read is precisely the state the health check deliberately ignores, so it's the one you'd want visible to a human for alerting.
A sentinel (-1) or a companion read-ever metric would keep it distinguishable. Also, since this only records when Healthy() reaches the staleness check, the series goes quiet whenever isReady/discoveryFailed short-circuit earlier: i.e. exactly when discovery is broken.
| // shows up as a last-read timestamp that stops advancing. StaleFor turns that | ||
| // into a liveness signal, used by both the operator and boot nodes. | ||
| // | ||
| // Only ReadFromUDPAddrPort is overridden; writes, Close and LocalAddr fall |
There was a problem hiding this comment.
Might be worth stating the scope here: only the post-fork listener reads through TimedConn, the pre-fork listener reads SharedUDPConn's buffer, so a pre-fork dispatch stall moves neither health signal.
The known blocking cause is gone post-#2980, but as written the doc reads as if the wedge signal covers discovery generally.
| func (h *bootNodeHealth) handler() http.HandlerFunc { | ||
| return func(w http.ResponseWriter, _ *http.Request) { | ||
| if err := h.check(); err != nil { | ||
| http.Error(w, err.Error(), http.StatusServiceUnavailable) |
There was a problem hiding this comment.
Do we need /healthz unauthenticated on the public ENR-advertised port?
Every request takes the discv5 table mutex via AllNodes() and allocates the full node slice, there's no method filtering, and the error body discloses internal state.
Maybe memoize check() for ~1s, reject non-GET/HEAD with 405, and return a fixed body with the reason logged instead — or serve it on a pod-internal listener.
ovidiu-ssv-labs
left a comment
ovidiu-ssv-labs
left a comment
There was a problem hiding this comment.
🔁 Re-review vs c9d0cc1: 2 fixed, 1 partial, 0 still open, 3 new. All three prior findings were addressed — the never-read arming guard, the ListenV5 wiring test, and the staleness gauge + warn log are all in and the packages build and test green. Two things remain: the operator's arming latch is permanent, so momosh-ssv's restart-loop case (blocked inbound UDP after a prior read) is genuinely still open; and the boot node's empty-table branch ignores the socket signal, turning a discv5 protocol-ID/config mismatch into a permanent CrashLoopBackOff that also destroys the /p2p debug surface. The core design — a cause-agnostic socket-read timestamp wired to the only listener that actually drains the socket — is sound and well documented. [verdict: with_fixes]
| // A wedged discv5 socket leaves discovery silently dead while bootstrap keeps | ||
| // looping; surface it so the hprobe watchdog restarts the node. n.disc is nil | ||
| // in tests and briefly at startup, hence the guard. | ||
| if n.disc != nil && n.disc.DiscoveryStale(discoveryStaleGrace) { |
There was a problem hiding this comment.
Finding 1 · [IMPORTANT] [still open] The operator's arming latch is permanent, so lost inbound UDP still produces a self-sustaining restart cycle
Status: independently verified as still open. This confirms momosh-ssv's question on this line, with the mechanism traced end to end.
Mechanism. read is a one-way latch with no reset path:
// network/discovery/timed_conn.go:50-57
func (c *TimedConn) ReadFromUDPAddrPort(b []byte) (n int, addr netip.AddrPort, err error) {
n, addr, err = c.UDPConn.ReadFromUDPAddrPort(b)
if err == nil {
c.lastReadUnixNano.Store(c.nowFn().UnixNano())
c.read.Store(true) // set once, never cleared for the process lifetime
}
return n, addr, err
}Once set, ReadStaleness() returns ok=true forever, so Healthy() at p2p.go:568 fails on any 3-minute inbound-UDP drought, regardless of cause. startHealthProber treats a failed round as terminal — it returns an error, the node exits non-zero, and the orchestrator restarts it.
Why the fix for prior finding 1 does not cover this. The never-read guard only exonerates a process that has never read a packet. A node running normally for hours has read=true. If inbound UDP then stops — upstream firewall/security-group change, NAT/conntrack rotation killing reply flows, an ISP-level UDP filter — the node keeps operating perfectly over its established libp2p TCP peer set, but exits within ~5 minutes.
Why it self-sustains rather than self-limiting. The stamp is taken at the socket read, before any decode: geth's UDPv5.readLoop reads every datagram on the port and only then attempts decode. So a port-scan probe, a stray discv4 packet, or a single NAT-refreshed reply is enough to flip read to true. Anything short of a perfect blackhole re-arms every boot, settling into a ~5-minute exit cycle.
Why it matters. Each cycle tears down every libp2p connection, forces full re-handshake/re-subscription, and risks missed attestation/sync-committee duties — direct penalty exposure on a validator node. The restart cannot fix an upstream UDP filter, so the loop is unbounded.
On the suggested fix in the thread (stamp writes). Does not discriminate — in the #2980 wedge, discv5 keeps sending (dispatch retries) even though reads are stale; in the blocked-inbound-UDP case, discv5 also keeps sending. Writes advancing with reads stale is the signature of both.
What does discriminate: not the socket — the impact. A genuine discv5 wedge progressively costs the node peers (churn with no replacement); an upstream UDP filter on a node with a warm, healthy TCP peer set does not.
Suggested fix: Require corroborating evidence that the node is actually impaired before restarting:
if n.disc != nil && n.disc.DiscoveryStale(discoveryStaleGrace) {
peers := len(n.host.Network().Peers())
if peers < n.cfg.MinPeers {
n.logger.Warn("discv5 socket wedged and peer set degraded", ...)
return fmt.Errorf("discv5 socket not drained for >%s and only %d peers (discovery wedged)", discoveryStaleGrace, peers)
}
n.logger.Error("discv5 socket not drained, but peer set still healthy — not restarting", zap.Duration("grace", discoveryStaleGrace), zap.Int("peers", peers))
}Alternative: raise discoveryStaleGrace substantially (discv5 revalidates at a 3s PingInterval, so even 30 minutes is a 600x margin) and cap self-inflicted restarts. Either way, add a test pinning that a permanent external UDP fault does not restart-loop.
| } | ||
| return nil | ||
| } | ||
| if emptyFor := now.Sub(time.Unix(0, h.lastNonEmpty.Load())); emptyFor > h.emptyTableGrace { |
There was a problem hiding this comment.
Finding 2 · [IMPORTANT - debatable] Boot node: the empty-table branch ignores the socket signal, so a protocol-ID/config mismatch becomes a permanent CrashLoopBackOff
Mechanism. check() splits on table population; the empty-table branch never consults h.socket:
// utils/boot_node/health.go:74-87
if len(h.lister.AllNodes()) > 0 {
h.lastNonEmpty.Store(now.UnixNano())
if h.socket.StaleFor(h.readStaleGrace) { ... } // socket consulted
return nil
}
if emptyFor := now.Sub(...); emptyFor > h.emptyTableGrace {
return fmt.Errorf("discv5 routing table empty for >%s", h.emptyTableGrace) // socket ignored
}So AllNodes() == 0 for 10 minutes fails closed even when the socket is demonstrably being drained.
Concrete break. The boot node's table is populated exclusively by inbound traffic (no Bootnodes configured), and geth only adds an inbound peer after a completed handshake; a V5ProtocolID mismatch (wrong Network option, or a protocol-ID rollout landing on the boot node ahead of the fleet) makes every packet decode-fail before reaching the table. The socket is read normally (TimedConn is fresh), but AllNodes() stays 0 forever. /healthz then 503s every 10 minutes and the livenessProbe restarts the pod indefinitely — a restart cannot fix a config mismatch.
Why this is worse than the pre-PR silent failure. The boot node's only diagnostic surface is /p2p on the same HTTP server. Under CrashLoopBackOff that endpoint is unreachable for most of the cycle, so the operator is left with a flapping pod and no way to observe why the table is empty. A fresh deployment whose ENR hasn't been distributed yet hits the same flap, and CrashLoopBackOff backoff (up to 5 min) means it may be down at the moment the first cold-bootstrapping node arrives — the exact failure #2979 exists to prevent.
Ruled out: 'populated table full of dead peers during a network outage' does NOT false-positive — geth's tableRevalidation drains failed peers well inside the 3-minute read grace. Only the empty-table branch has this gap.
Suggested fix: Fold the socket signal into the empty-table branch — fail closed only when BOTH the table is empty AND the socket is undrained:
if emptyFor := now.Sub(time.Unix(0, h.lastNonEmpty.Load())); emptyFor > h.emptyTableGrace {
if age, everRead := h.socket.ReadStaleness(); everRead && age <= h.readStaleGrace {
h.logger.Error("discv5 routing table empty while socket is being drained — check DiscoveryProtocolID / network config", zap.Duration("empty_for", emptyFor), zap.Duration("read_age", age))
return nil
}
return fmt.Errorf("discv5 routing table empty for >%s and socket undrained", h.emptyTableGrace)
}Needs socketDrainState widened to expose ReadStaleness() (time.Duration, bool) — *TimedConn already has it. Add two test cases: empty table + fresh socket past the empty grace (healthy, with error log), and empty table + never-read past the empty grace (503).
3 participants
Problem
A discv5 socket can wedge — stay bound but stop being drained — leaving discovery silently dead while the process still looks healthy. #2979 documents a boot node that sat like this for ~20 days undetected. Neither node type surfaces it today: the operator's
p2pNetwork.Healthy()only checks a discovery-bootstrap flag (a runtime wedge never trips it, since the bootstrap loop keeps running and just yields nothing), and the boot node's HTTP handler always returns200.#2980 removes one cause of the wedge on the operator (the blocking
Unhandledsend); this PR adds the missing detection, so a wedge from any cause becomes a self-correcting restart.Approach
One shared, cause-agnostic signal — "how long since the discv5 socket was last read" — with actuation scoped per node type.
TimedConn(network/discovery): wraps the socket and stamps the last successful read;StaleFor(d)is the wedge signal. Both node types wrap the conn they hand toListenV5.DiscoveryStalefeedsp2pNetwork.Healthy(), so the existinghprobewatchdog restarts the node on a wedge. No routing-table check here — a stale-but-populated table would mask it on an operator./healthzreturns non-200 when the routing table has been empty past a cold-start grace, or — while the table is populated — the socket has gone unread (discv5 revalidates its peers, so a populated-but-unread socket is a wedge; an empty/quiet table produces no such traffic and is judged only by the grace). Empty-table is the boot node's definitional health (the 0-vs-72/102 datapoint in boot-node: discv5 can wedge silently — no health signal tied to discovery actually working #2979).Grace values: 3 min read-staleness (both node types); 10 min empty-table cold-start (boot node only).
Notes
fix/discv5-unhandled-wedge), because the operator change wraps the socket ininitDiscV5Listener, which network/discovery: stop undecodable packets from wedging discv5 #2980 rewrites. Retarget tostageonce network/discovery: stop undecodable packets from wedging discv5 #2980 merges./debug/pprof, and a secondsepoliaboot node.livenessProbeon/healthzis added by the downstream PRs below.Tests
TimedConn: seeded-not-stale, stale boundary (injected clock, no real sleeps), read-stamps, errored-read-doesn't-stamp.DiscoveryStale;TestP2PNetwork_Healthygainsdiscovery wedged/ready with live discoverycases — the existing nil-disccases stay green, proving the guard./healthzacross populated+fresh, populated+wedged, empty-within-grace (incl. the quiet-socket regression), and empty-past-grace.Closes #2979. Full discovery + p2p + boot_node suites pass, discovery also under
-race.Downstream infra PRs for the boot-node side of this (what makes the boot node self-healing)
livenessProbeto theboot-nodechart (inert by default)./healthz).Merge order: #2980 → this → build image → ssvlabs/charts#183 → ssvlabs/gitops-production#881.