diff --git a/.gitignore b/.gitignore index e23fba0..8418ee7 100644 --- a/.gitignore +++ b/.gitignore @@ -104,3 +104,4 @@ docs/build/ !docs/*.md !.claude/skills/**/*.md *log*.txt +!CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9a5e9b6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,54 @@ +# TART — JAM Telemetry Backend + +Rust/Axum backend that ingests telemetry from JAM blockchain nodes (binary TCP, JIP-3), +stores in TimescaleDB, and serves via REST API + WebSocket. Grafana dashboards visualize everything. +Target: 3M events/s from 1024 nodes — performance-critical design decisions are intentional. + +API: `localhost:8080` (local dev only — the real stack runs remotely at `https://jamtoaster.network/api`; a stale local instance can look healthy while serving old data) +TEST-DB: `postgres://tart:tart_password@localhost:5432/tart_test` +DB: `postgres://tart:tart_password@127.0.0.1:5432/tart_telemetry` + +## Key paths + +- `src/grafana.rs` + `src/grafana_store.rs` — Grafana endpoints (active development) +- `src/api.rs` — REST/WS routes, `src/store.rs` + `src/batch_writer.rs` — legacy endpoints, don't touch unless asked +- `src/server.rs` — TCP ingestion, `src/enricher.rs` — cross-event correlation +- `grafana/provisioning/dashboards/` — dashboard JSONs (Infinity plugin, uid: `jamtart-api`) +- `docs/grafana-guide.md` — dashboard recipes and shared query types +- `docs/endpoint-doc-conventions.md` — rules for endpoint/schema doc comments (they become the public OpenAPI docs) +- `/api/docs/openapi.json` (served by the backend) — endpoint source of truth: every endpoint self-describes with its feeding events and the question it answers + +## Grafana dashboards + +All panels use the Infinity plugin to HTTP GET `localhost:8080/api/grafana/*` endpoints. + +- **Curl the endpoint first** to see response shape before writing/editing panel config. +- The `tart-backend` skill has the live backend URLs and a debugging endpoint index. +- **Edit panels surgically** with the Edit tool — don't rewrite entire files or create Python scripts. +- Grafana variables: `$node`, `$core`, `$service`, `$interval` +- Time macros: `${__from:date:iso}`, `${__to:date:iso}` +- Read `docs/grafana-guide.md` for dashboard recipes; endpoint specs live in the OpenAPI doc. + +## Key architecture assumptions + +- **Dual-write ingestion:** every event is written to both `ingested_raw_events` (1h retention, browsing store with hot columns) and in-memory DashMap counters that flush every 5s to 14 per-group count tables (e.g. `status_counts`, `assurance_counts`, `segment_counts`). All 115 event types go through both paths. +- **Aggregate hierarchy:** no continuous aggregates over raw events — the count tables are the single aggregation source, with `_1m`/`_1h` continuous aggregates on top. UNION views (`all_event_stats_30s/1m/1h`, `all_core_stats_1m`) combine the 14 groups. Grafana endpoints auto-select tier based on time range. Migrations are a squashed 7-file baseline (2026-08); never drop a continuous aggregate created by an earlier migration (scheduler deadlock on fresh DBs). +- **`ingested_raw_events`** has 1h retention — queries against it (via the `events` view alias) only return the last hour of data. `store.rs` endpoints that query `events` without time bounds effectively get ≤1h of data. +- **Enricher** (`src/enricher.rs`): per-node stateful correlation. WorkPackageReceived is the source event — core, service_ids, submission_id propagate to ~30 downstream events via ID chains. Enriched fields are DB-only (not on WS broadcast path). +- **Hot columns** on `ingested_raw_events`: `slot`, `core`, `submission_id` — populated at ingestion, avoid JSONB queries. +- **Separate tables:** `event_services` (service×event junction with gas), `node_stats` (extracted Status fields), `wp_tracking`, `slot_convergence` — all written at ingestion time. + +## External references (sibling checkouts of this repo) +- Telemetry events: `../polkajam/crates/jam-std-common/src/telemetry.rs` +- JIP-3 spec: `../JIPs/JIP-3.md` (public: https://github.com/polkadot-fellows/JIPs/blob/main/JIP-3.md) +- polkajam implementation (node): `../polkajam/` +- Infinity plugin source (when stuck): `../grafana-infinity-datasource/` + +## Testing + +Run `./run-tests.sh` — it sets up the test DB and runs all tests serially. It's slow, so run it once and read the full output. Don't re-run it repeatedly to grep/filter/count. + +## Working style + +- **Plan before coding.** Explain the problem and proposed approach before editing files; use plan mode to prepare the plan. +- **Research before guessing.** Read docs, curl endpoints — don't trial-and-error. diff --git a/docs/endpoint-doc-conventions.md b/docs/endpoint-doc-conventions.md new file mode 100644 index 0000000..6003acb --- /dev/null +++ b/docs/endpoint-doc-conventions.md @@ -0,0 +1,100 @@ +# Endpoint documentation conventions + +The OpenAPI spec (swagger page, `/api/docs/openapi.json`) is generated from `utoipa` +doc comments in the source: + +- `src/grafana.rs` — `///` on handlers (first line = summary, rest = description) and + `description = "..."` strings inside `#[utoipa::path(...)]`, plus query-param structs. +- `src/grafana_types.rs`, `src/onchain_types.rs` — `///` on `ToSchema` structs and fields. + +Everything in those doc comments is public API documentation. Write it for a consumer +of the API, at the JAM/JIP-3 level of abstraction — never for a reader of tart's code. +(Established by [issue #21](https://github.com/paritytech/jamtart/issues/21).) + +## The four rules + +1. **JIP-3 level of abstraction.** Describe endpoints in protocol terms — nodes, + validators, cores, services, slots, work packages, guarantees, assurances, shards, + segments, preimages, blocks — never in terms of how tart stores or aggregates data. +2. **`EventName(ID)` nomenclature.** Reference telemetry events as + `WorkPackageReceived(94)`, `AssuranceReceived(131)`, etc. Canonical name↔ID table: + `src/event_type_meta.rs`. On-chain endpoints reference no events — that data comes + from the chain's state, so say so instead of inventing an event. +3. **No database internals.** No table names, SQL, column names, aggregate/rollup + names, TimescaleDB/Postgres/JSONB/hypertable mentions, or storage types (i16/i32). +4. **State the question answered.** Every endpoint description ends with + `Answers: .` + +## Handler doc template + +```rust +/// +/// +/// <1–3 sentences: what each row represents, which events feed it as +/// EventName(ID), and semantics the consumer needs: windows, thresholds, +/// what "failed" or "converged" means.> +/// +/// Answers: . +``` + +Before/after (from the #21 rewrite): + +```rust +// BAD — documents the SQL +/// Work package pipeline bottleneck analysis with percentile timings. +/// +/// Queries `wp_tracking` table using `percentile_cont(0.5)` and +/// `percentile_cont(0.95)` on the inter-stage timestamp deltas ... + +// GOOD — documents the meaning +/// Where time goes inside the guarantor work-package pipeline. +/// +/// Median and 95th-percentile durations of each stage a work package passes +/// through on its guarantors: authorize (WorkPackageReceived(94) → +/// Authorized(95)), refine (→ Refined(101)), ... +/// +/// Answers: which pipeline stage dominates work-package latency, and how +/// often do work packages fail outright? +``` + +Response descriptions (`responses(description = ...)`) get one sentence about the +response shape ("Array of per-core rows, ascending by slot"), including mode switches +("with `interval`: one row per time bucket instead"). Schema struct docs say what one +instance represents; field docs give units and semantics ("milliseconds from X to Y"). + +## Translation table + +| Instead of | Say | +|---|---| +| `wp_tracking` table | "per-work-package pipeline tracking" | +| `event_stats_1m` / "continuous aggregate" | "pre-aggregated counts (30 s / 1 min / 1 h resolution, auto-selected from the range)" — mention resolution only when it affects the consumer | +| `ingested_raw_events` | "recent raw events (retained ~1 hour)" — retention IS consumer-relevant, keep it | +| `guarantee_convergence` etc. | describe the measurement: "per-report guarantee propagation" | +| `SUM FILTER`, `COUNT(*)`, `time_bucket(...)` | drop; describe the result, not the computation | +| "JSONB payload" | "full event payload" | +| "stored as signed i32 in PostgreSQL" | drop | + +Not leaks — keep them: + +- Event **group** names (`status`, `blocks`, `wp_pipeline`, `assurances`, ...) — public + API vocabulary returned by `/event-types` and accepted by `event_types` parameters, + even where they coincide with table names. Make the context explicit: "the + `wp_pipeline` event group". +- Protocol constants (5-slot availability window, 6 s slots), percentiles, and + measurement precision ("percentiles are approximate, histogram-based"). +- Consumer-facing formats: hex service IDs, accepted input formats, Grafana `{a,b}` + multi-select syntax, pagination and caps, sort orders. +- Consumer-visible behavior phrased as behavior: retention windows, aggregation + resolution, sampling cadence — never as a property of a named table. + +## Special case + +`GET /api/grafana/db-stats` is an operational endpoint about the collector's own +storage — naming TimescaleDB, hypertables and compression there is its subject +matter, not a leak. Its docs must open by saying it reports tart's internal storage +state, not JAM protocol data. + +## Golden rule + +Docs describe what the code **does**, not what it was meant to do. If they disagree, +fix the code or document the actual behavior — never document the intent.