feat(metrics): add gossip arrival-time histograms and position counters - #566
Conversation
Network health had plenty of block-production timing but nothing about reception timing, so "are votes arriving late, or not arriving at all?" could only be answered out-of-band with tooling/event-monitor's collector-side clock. Adds three histograms recording the absolute distance between a gossip message's arrival and the start of the interval it was due in, plus three counters splitting arrivals into before/inside/after that interval. The interval is what counts as inside, not the slot: an attestation landing in its own slot's aggregation interval missed the production interval it was due in, so it reads as after. Aggregates anchor to the most recent aggregation-interval boundary instead of their own data.slot, since a stale-group catch-up aggregate legitimately carries an older slot and would otherwise fill the histogram with large values that are not a health problem. That bounds their delay to one slot and makes before unreachable, so the aggregate counter does not export that series. Blocks are sampled only when received on gossip; req/resp sync backfill delivers them many slots after they were due and would swamp the histogram. Threading a BlockSource through new_block to tell the two apart also fixes ChainEvent::BlockGossip, which events.rs already documents as gossip-only yet until now also fired for sync-fetched blocks.
Devnet verification4-node all-ethlambda local devnet on this branch, one aggregator, All six metrics populateMean delay per node, at slot 43:
The Invariants checked atomically per scrapeOn every node, for every kind:
Label seeding behaves as designedBefore any gossip arrived, all series already existed at zero, and the aggregation counter exports only two: There is no Bucket distribution is sane
Resolution sits where it is needed: the sub-interval range 0.05-0.4 separates most of the mass, the 0.8 bound marks the Caveat on what this run did not exercise
Test suiteFull suite green: 223 workspace unit tests (5 new), plus 122 forkchoice / 74 stf / 119 ssz / 3 signature spec tests and 8 |
…trics An aggregator never receives its own aggregate back over gossip, so it reported an empty aggregate arrival profile: the one node whose aggregation timing matters most was the one node with no data. Observes locally produced aggregates in the same series from Handler<AggregateProduced>, which already exists for exactly this asymmetry (it emits ChainEvent::Aggregate for the same reason). The timestamp is taken at handler entry but observed after the stale-session guard, so a late worker's discarded output does not inflate the count. Sharing one series with received aggregates rather than splitting by a source label costs little in practice. Delivery of a local aggregate is held to the interval-2 boundary upstream, so it lands near zero unless proving overran the interval, and a late aggregate is late for every node at once. Measured on a 4-node devnet the two populations are near-identical: of the sole aggregator's 24 own aggregates 67% fell in the lowest bucket, against 70% of a receiver's 23 gossip-received ones, with a matching overrun tail on both. Both are dominated by production time, not propagation.
Follow-up: an aggregator now counts its own aggregatesAn aggregator never receives its own aggregate back over gossip, so Own and received aggregates share one unlabelled series. I initially expected that to be a meaningful compromise, on the reasoning that a local aggregate's delivery is held to the interval-2 boundary and so measures proving overrun, whereas a received one measures propagation. A second devnet run showed that expectation was wrong, so the docs and code comment now say what is actually observed.
Near-identical, in both mode and tail. The reason is that a late aggregate is late for every node simultaneously, and on a single host propagation is sub-50 ms, so both populations are dominated by production time rather than propagation. Splitting them by a The mechanism half of the prediction did hold: two thirds land in the lowest bucket, consistent with boundary-pinned delivery, with a tail where proving overran the interval. So the operator guidance is to read a rising tail as aggregation cost and cross-check Devnet caveats worth recordingThe first two restart attempts produced numbers I would have misread as real:
Only the third run, with fresh genesis and pre-created data dirs, gave a chain healthy enough to measure.
|
🤖 Kimi Code ReviewOverall Assessment: Solid PR adding well-designed gossip arrival metrics with correct timing logic and proper separation between gossip and sync sources. No critical issues found. Minor Suggestions:
Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Codex Code Review
Other than that, the I couldn’t run a full Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
Greptile SummaryThe PR adds gossip arrival-delay histograms and interval-relative position counters, while distinguishing gossip blocks from synchronization responses.
Confidence Score: 5/5The PR appears safe to merge; no concrete blocking or independently actionable non-blocking defects were identified. The source discriminator is propagated through both gossip and synchronization call paths, and the new metric calculations and documented interval semantics are internally consistent.
|
| Filename | Overview |
|---|---|
| crates/blockchain/src/lib.rs | Records message arrival times at handler entry, gates block gossip events and metrics by source, and includes locally produced aggregates in the aggregate timing series. |
| crates/blockchain/src/metrics.rs | Defines the new histograms, counters, interval geometry, position classification, and startup label seeding. |
| crates/net/api/src/lib.rs | Adds the BlockSource discriminator to the typed block-delivery protocol. |
| crates/net/p2p/src/gossipsub/handler.rs | Marks decoded gossipsub blocks as gossip-originated when forwarding them to the blockchain actor. |
| crates/net/p2p/src/req_resp/handlers.rs | Marks blocks returned by root and range synchronization requests as sync-originated. |
| docs/metrics.md | Documents the new custom metrics, interval-relative semantics, aggregation anchor, and graphing guidance. |
Sequence Diagram
sequenceDiagram
participant Peer
participant P2P as P2P actor
participant BC as Blockchain actor
participant Metrics as Prometheus metrics
participant Import as Validation/import
Peer->>P2P: Gossip message
P2P->>P2P: Decompress and SSZ-decode
P2P->>BC: NewBlock(..., Gossip) / attestation / aggregate
BC->>BC: Capture handler-entry time
BC->>Metrics: Observe delay and position
BC->>Import: Validate and process message
Peer->>P2P: Req/resp sync block
P2P->>BC: NewBlock(..., Sync)
BC->>Import: Process without gossip metric/event
Reviews (1): Last reviewed commit: "Merge branch 'main' into feat/gossip-arr..." | Re-trigger Greptile
#566 added `lean_gossip_*_arrival_delay_seconds` and `lean_gossip_*_arrival_total`, which answer "are votes arriving late, or not arriving at all?" from inside each node. Nothing graphed them, so answering it still meant an out-of-band event-monitor run against a collector clock. Adds a "Gossip Arrival Timing" row to `client-dashboard.json`: a 3x4 grid, one column per message kind (block / attestation / aggregate), one row per view. - delay p99/p50 per node, with a dashed line at one interval (0.8s): above it the typical message misses the interval it was due in - delay distribution heatmap, which exposes the bimodal profile that percentiles average away - arrival position stacked by `position`, the only view that recovers the sign the absolute-value histogram discards (rising `before` is clock skew, rising `after` is propagation or CPU). The aggregate panel omits `before`, unreachable by construction since aggregates anchor to the latest aggregation-interval boundary rather than their own data slot - on-time fraction per node, so one late node separates from a fleet-wide drop All 15 queries were run against the central Prometheus before committing. Also adds a receiver-side timing block to the node-health checklist, whose item 5 covered only the node's own duties, and corrects where dashboards get deployed: the JSONs live in the host dir bind-mounted at `/var/lib/grafana/dashboards`, not in `<GRAFANA_PROV_DIR>/dashboards`, which holds only the provider yaml. A JSON dropped in the provisioning tree is silently ignored, which reads as a working copy that never appears. Recorded as `GRAFANA_DASHBOARDS_DIR`.
2 participants
Motivation
We expose plenty of production timing (
lean_block_building_time_seconds,lean_pq_sig_*) but nothing about reception timing. When finality degrades the first question is usually "are votes arriving late, or not arriving at all?", and until now that could only be answered out-of-band withtooling/event-monitor, which stamps arrivals on a collector clock over SSE. These metrics answer it from inside each node, on the node's own clock, with no external collector.Not in the leanMetrics spec yet, so they live under
## Custom Metrics (non-leanMetrics)indocs/metrics.md. Upstreaming the rows is a follow-up.Metrics
lean_gossip_block_arrival_delay_secondslean_gossip_attestation_arrival_delay_secondslean_gossip_aggregation_arrival_delay_secondslean_gossip_block_arrival_totalposition=before,inside,afterlean_gossip_attestation_arrival_totalposition=before,inside,afterlean_gossip_aggregation_arrival_totalposition=inside,afterBuckets, shared:
0.05, 0.1, 0.2, 0.4, 0.8, 1.2, 1.6, 2.4, 4, 8, 16, interval-aligned (0.8 = one interval, 4 = one slot).Semantics
Arrival is stamped as the first statement of each network-message handler, before verification. Both metrics derive from one signed delta; the histogram observes its absolute value, the counter classifies it.
block.slotdata.slot[0, 4s)The interval, not the slot, is what counts as
inside. An attestation that lands in its own slot's aggregation interval missed the production interval it was due in, so it readsafter.insideis half-open: a delta of exactly one interval isafter.Why aggregates anchor differently.
aggregation.rsshows an aggregate published at interval 2 of slot N can legitimately carrydata.slot < N(stale-group catch-up). Anchoring those todata.slotwould fill the histogram with large values that are not a health problem. Anchoring to the latest boundary bounds the delay to one slot and makesbeforestructurally unreachable, so the aggregate counter does not export that series; a sweep unit test guards the invariant instead.Known trade-off. Because the histogram observes the absolute delta, an arrival 300 ms early shares a bucket with one 300 ms late. The counter's
positionlabel is what separates them, and it now does so at interval rather than slot resolution.Drive-by fix:
ChainEvent::BlockGossipwas firing for sync blocksnew_blockis shared by the gossip topic and req/resp. Telling them apart needs aBlockSourcediscriminator, which also fixes a pre-existing mislabel:events.rsdocumentsBlockGossipas "A block seen on gossip, before import", yet it fired for req/resp sync-fetched blocks too. Now gated.tooling/event-monitorsees fewer, more accurateblock_gossipevents; noCONTRACT.mdchange needed.Sync arrivals are excluded from the block metrics entirely, since backfill delivers blocks many slots after they were due and would swamp the histogram.
Graphing
Four panels, following the conventions in
leanMetrics/dashboards/; natural home is beside the existinglean_gossip_*_size_bytespanels. Dashboard JSON belongs in that repo, so it is not in this PR.histogram_quantile(0.5/0.95/0.99, sum by (le, job) (rate(..._bucket[$__rate_interval])))plus avector(0.8)threshold line (same tricklean_tick_interval_duration_secondsuses). Above the line, the typical message misses its interval.sum by (le) (rate(..._bucket[$__rate_interval])). The in-Prometheus analogue of event-monitor's beeswarm; exposes a bimodal profile that percentiles average away.rate(...{position="inside"}) / rate(...), unitpercentunit.sum by (position) (rate(..._total[$__rate_interval])), stacked. The only panel that recovers the sign the histogram discards: risingbeforemeans clock skew, risingaftermeans propagation or CPU.Reading caveats: percentiles between bucket bounds are interpolated, not measured; a
+Infp99 is a real answer (>1% over 16 s late), not a plotting glitch; and aggregate p99 is not comparable to the other two on one axis since it is bounded below 4 s by construction.Design notes
SlotPosition, the geometry helpers and the entry points all live inmetrics.rs;SlotIntervalgained one method,to_ms_since_genesis(slot), the inverse of the existingfrom_ms_since_genesis. That keeps interval numbering encapsulated (no publicindex(), no explicit discriminants) and lets a round-trip test pin the two inverse matches together.genesis_timeis read per message fromstore.config()rather than cached; it is aMetadataread served from the RocksDB block cache.lean_attestations_invalid_total.Testing
metrics.rscovering the geometry helpers against pre-computed values at every interval boundary (including the half-openinsideedge, the aggregate wrap, and a pre-genesis arrival), plus theSlotIntervalround-trip inlib.rs.cargo clippy --workspace --all-targets -- -D warningsclean.