Derive liveness timeouts from measured RTT, attempt ICE restart before teardown, and retry lookups at the originator

[zkjoie]

Motivation

Every timeout in the maintenance path is a constant:

• Liveness: probe after PEER_LIVENESS_IDLE_MS = 15 s idle, evict after PEER_LIVENESS_TIMEOUT_MS = 45 s unanswered (crates/core/src/swarm/transport/liveness.rs). A silently failed peer (laptop sleep, network switch, tab frozen) is therefore removed 60–75 s after it went quiet, while a peer on a slow mobile link can be falsely suspected if one probe round trip approaches the budget.
• ICE Disconnected is given a 30 s grace (DISCONNECTED_CONNECTION_GRACE_MS, crates/core/src/dht/stabilization.rs) and then closed; no restartIce() is attempted in either backend, so a link WebRTC could have repaired costs a fresh SDP round trip routed through the DHT.
• Lookups (FindSuccessorSend) have no originator-side timeout or retry; a lookup lost at a dying hop simply fails, and convergence waits for the next periodic pass.

The measurement layer records six counters and no latency at all (ReliabilityEvidence, crates/measure/src/reliability.rs), so nothing adaptive is possible today.

Rhea, Geels, Roscoe & Kubiatowicz, "Handling Churn in a DHT" (USENIX 2004) evaluated exactly this design space under emulated churn: fixed timeouts were the worst option; per-neighbour TCP-style estimates (RTO = SRTT + 4·RTTVAR, Jacobson 1988 / RFC 6298) performed best, with virtual-coordinate estimates close behind; and they warn that false suspicion of failure creates a positive feedback loop of recovery traffic — which is the failure mode Rings' sync-storm gate was built to prevent, so the fix must lower false suspicion, not raise it. The same paper shows recursive routing (which Rings uses) tolerates loss better than iterative routing only when the originator retries on timeout.

Proposal

1. Measure RTT. Time the liveness probe (ProbeRequest → first authenticated reply) and every report-carrying request the node originates (finger and connect lookups, topology queries) per peer; keep SRTT/RTTVAR in the measurement ledger next to the existing counters. Pure state, updated at the effect boundary.
2. Derive the liveness deadline per peer: timeout = clamp(SRTT + 4·RTTVAR, 2 s, 45 s); keep 45 s only for peers with no sample. Keep the idle threshold as is.
3. ICE restart before teardown. On Disconnected, call restartIce() (native webrtc and web_sys) once at the start of the grace window and only then fall back to closing at 30 s. This keeps the DHT edge and skips the signalled re-handshake.
4. Originator-side lookup retry. A routed request that carries a report handler gets a deadline derived from the first hop's RTT and the hop budget; on expiry the originator re-issues once through a different first hop (next closest preceding finger). Bound finger-table convergence by proved ranges #770 already gives finger lookups a token and a single-flight rule, so the retry composes with it.
5. Do not shorten the false-suspicion side. Any per-peer timeout below the fixed value must be justified by that peer's own samples; unknown peers keep the conservative constant.

Acceptance

• Ledger exposes per-peer srtt_ms/rttvar_ms; the inspect RPC shows them.
• A peer with SRTT 50 ms that goes silent is evicted in a few seconds; a peer with SRTT 2 s is not evicted by one slow reply.
• Disconnected → Connected recovery via ICE restart is covered by a controlled-transport test; the DHT edge is never removed during a successful restart.
• A lookup whose next hop dies mid-route completes via retry within one derived deadline; the sync-storm gate's false-disconnect and repair-amplification assertions still hold.

Related

Rhea et al. §4 (timeouts), §5 (recursive vs iterative routing); RFC 6298 (RTO computation); #770 (finger lookup tokens).

Issue family (browser-churn stability, 2026-09-14)

#773 churn simulator · #774 adaptive successor list and stabilization period · #775 age-ranked peer cache and re-join · #776 RTT-derived timeouts, ICE restart, lookup retry · #777 inbox replication and Leave · #778 stability-weighted storage · #779 browser lifecycle events. Finger convergence is #768 / #770; native seed redial is #763.