Repository navigation
Oracle trust engine: adapter migration, durable evidence, consensus-gated settlement, temporal smoothing - #476
Merged
OxDev-max merged 5 commits intoOct 23, 2026
Conversation
…ce adapter registry Introduces the OracleSourceAdapter interface and OracleSourceRegistry, and migrates the three production sources onto them. OracleService no longer contains bespoke fetch-and-parse blocks; every live value now passes through one ingestion pipeline: quarantine gate -> fetch (timeout) -> schema check -> canonicalize (finite, plausible bounds, fixed precision) -> freshness -> evidence - CoinGeckoAdapter: usdc.usd.price, xlm.usd.change_24h_pct (uses last_updated_at as the upstream timestamp) - HorizonAdapter: stellar.ledger.latest (chain context for MarketCrash) - DefiLlamaAdapter: protocol.tvl.change_24h_pct, with per-request slug - Each response keeps HTTP evidence: URL, status, SHA-256 of the body, size and a short excerpt - Repeated failures, or one malformed/out-of-bounds response, quarantine a source per metric, with a half-open probe after the cooldown - Per-source outcomes and quarantines go to Prometheus via MetricsService (refract_oracle_source_observations_total, refract_oracle_source_quarantines_total) - OracleReading gains `sources` and `observations` so readings carry their evidence. Degraded fail-safe behaviour is unchanged - OracleService gains refreshPrices(), which OracleScheduler already calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… outages Oracle readings used to be written to Postgres fire-and-forget. If the database was down they were silently dropped, which left holes in the audit trail that justifies payouts. A blocking write would have stalled polling and settlement instead. Neither is acceptable. OracleEvidenceBuffer sits between the oracle and Postgres: - Every record gets a strictly increasing seq when it is accepted, then is appended to a local fsync'd NDJSON write-ahead spool (EvidenceSpool). append() resolves once the record is on disk, whatever the state of Postgres, so polling never waits on the database. - A background flusher drains the spool to Postgres in seq order (head-of-line, batched) with exponential backoff while the DB is unreachable. The spool survives restarts and is replayed on boot. Inserts are idempotent on seq, so a crash between commit and ack is harmless. - Torn final lines from a crash mid-write are discarded on recovery and the file is rewritten so new appends start on a clean line. - If even the local write fails (disk full or size cap reached), the seqs are not reused. The lost run is written as an explicit oracle_evidence_gaps row as soon as writes succeed again, status() reports complete=false until then, and append() rejects with EvidenceNotDurableError so callers that need proof can fail closed. - Status is exported as Prometheus gauges: pending, lost and complete. Schema (src/db/migrations/004_oracle_evidence_sequence.sql): adds seq, kind, payload and ingested_at to oracle_events, a unique (seq, recorded_at) index, and the oracle_evidence_gaps table. It also drops chk_oracle_value_nonneg, which rejected every negative MarketCrash or TVL change reading and would have wedged the ordered replay. OracleEventRepository is now the idempotent store the buffer flushes to. OracleScheduler and ClaimService record through the buffer. The spool directory (ORACLE_EVIDENCE_SPOOL_DIR) gets a persistent volume in docker-compose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ClaimService used to call OracleService.check*() and pay out whenever that single reading crossed a threshold. That kept the single-source-of-truth failure the threat model flags: corroboration, quorum and degradation had no effect on the code that moves money. ConsensusService (src/oracle/consensus) now produces a ConsensusDecision per coverage type. It observes the trigger metric from every registered live source and evaluates: - quorum: enough usable live sources (ORACLE_CONSENSUS_MIN_SOURCES), otherwise "insufficient" - agreement: max-min spread within the trigger's maxSpread, otherwise "conflicted" - confidence: source coverage x agreement must reach ORACLE_CONSENSUS_MIN_CONFIDENCE - degradation: a registered source failed, was stale or was quarantined. Reported as "degraded" but still actionable if quorum holds - evidence: every decision is appended to the evidence trail before it is returned. If the append fails the decision is "unrecorded" and not actionable The value is the median of the usable inputs. Mock-tier sources never count toward quorum. Decisions are cached briefly (default 30s) so one scan does not refetch per policy. ClaimService now depends only on ConsensusService. Payout requires a SettlementAuthorization, which authorizeSettlement() issues only for an actionable, triggered, recorded decision of the matching coverage type. Each settled ClaimResult records decisionId (coverage type @ evidence seq) so a payout can be traced back to its observations and HTTP evidence. Behaviour change: LiquidationShield and FlightDelay have no live source, so their decisions are always "insufficient" and they no longer settle automatically on random mock values. The default quorum is 1 because each production metric has one live source today. Raise it once a second source is registered. claim.service.spec.ts is rewritten for the gated flow (the previous spec did not compile). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cross-source corroboration protects against one wrong source, but not against one bad sample from a good source: a transient spike, a stale or partially deployed CDN cache, or a crafted response aimed at nudging a threshold. Because consensus used single point-in-time observations, one print could flip a StablecoinDepeg (or, once it has a live source, LiquidationShield) condition. TemporalSmoother keeps a rolling window per (source, metric, subject): - A source contributes the median of its window to consensus, not its latest print. A single spike cannot move it. A sustained move comes through once it is most of the window. - A source counts toward quorum only once it has ORACLE_SMOOTHING_MIN_SAMPLES distinct samples spanning at least ORACLE_SMOOTHING_MIN_SPAN_MS. Until then it is reported as a "warming up" failure, so a fresh or newly recovered source cannot settle on its first reading. - Samples are keyed by upstream observedAt. A cache replaying the same response, even with a different body, adds no sample and cannot overwrite the first value, so it cannot fabricate history. - Old samples fall out of the window (ORACLE_SMOOTHING_WINDOW_MS) and each window is capped (ORACLE_SMOOTHING_MAX_SAMPLES). OracleSourceRegistry gains subscribe(), so every successful observation (scheduler polls and consensus rounds alike) feeds the windows. Freshness is unchanged: a source still has to deliver a fresh observation this round to count at all. Decision inputs now carry rawValue, samples and spanMs alongside the smoothed value, so the evidence trail shows both what the source said and what consensus used. Defaults: 15 min window, 3 samples over at least 2 min. With a 60s poll, a source becomes actionable about 2-3 min after boot or recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@zenasben5 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
…ngine # Conflicts: # docker-compose.yml # src/claim/claim.service.ts # src/metrics/metrics.service.ts # src/oracle/consensus/consensus.service.ts # src/oracle/oracle-event.repository.ts # src/oracle/oracle-reading.ts # src/oracle/oracle.module.ts # src/oracle/oracle.scheduler.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes the oracle trust engine cover the money path. It has four commits, one per issue, and each builds on the one before:
feat(oracle): route CoinGecko, Horizon and DeFiLlama through the source adapter registryfeat(oracle): durable, ordered evidence buffer that survives Postgres outagesfeat(claim): make consensus decisions the sole input to settlementfeat(oracle): smooth each source over time before cross-source consensusReviewing commit by commit is recommended. Each commit message explains its design in more detail.
1. Source adapters: one ingestion path
OracleSourceAdapterinterface andOracleSourceRegistry(src/oracle/adapters/).CoinGeckoAdapter(USDC price, XLM 24h change),HorizonAdapter(latest ledger, used as chain context) andDefiLlamaAdapter(protocol TVL 24h change) replace the inline axios blocks.OracleServicehas no fetch logic of its own any more.quarantine gate → fetch (timeout) → schema check → canonicalize (finite, plausible bounds, fixed precision) → freshness (per-metric max age, future-timestamp rejection) → evidence.
refract_oracle_source_observations_total{source,metric,outcome}andrefract_oracle_source_quarantines_total.OracleReadingnow carriessourcesandobservations. The degraded fail-safe behaviour is unchanged.2. Evidence buffer: complete and ordered through DB outages
seq, andON CONFLICT (seq, recorded_at) DO NOTHINGmakes crash replays harmless.oracle_evidence_gapsonce writes recover, andappend()throwsEvidenceNotDurableErrorso that settlement fails closed (see commit 3).refract_oracle_evidence_pending,refract_oracle_evidence_lost_total,refract_oracle_evidence_complete.src/db/migrations/004_oracle_evidence_sequence.sqladdsseq,kind,payload,ingested_at, the unique index and the gaps table.chk_oracle_value_nonneg. That check rejected every negative MarketCrash or TVL reading, which are exactly the readings that matter, and it would have wedged the ordered replay.oracle-evidencevolume, and the Dockerfile creates the spool directory withnodeownership.3. Consensus-gated settlement
ConsensusServiceproduces aConsensusDecisionper coverage type:insufficientmaxSpread: elseconflicteddegradedis still actionable if quorum holdsunrecordedClaimServicenow depends only onConsensusService, not onOracleService. The payout path accepts only aSettlementAuthorization, andauthorizeSettlement()issues one only for an actionable, triggered, recorded decision of the matching coverage type.ClaimResultcarriesdecisionId(<CoverageType>@<evidence seq>), which links the payout to its observations and HTTP evidence.4. Temporal smoothing
TemporalSmootherkeeps a rolling window per (source, metric, subject), fed from every registry observation (scheduler polls and consensus rounds).MIN_SAMPLESdistinct samples spanning at leastMIN_SPAN_MS. Until then it reports "warming up".observedAt, so a cache replaying the same response adds nothing and cannot overwrite the first value.rawValue, smoothedvalue,samplesandspanMs.Configuration (all optional, with safe defaults)
ORACLE_QUARANTINE_AFTER_FAILURES/ORACLE_QUARANTINE_MS3/300000ORACLE_EVIDENCE_SPOOL_DIR/ORACLE_EVIDENCE_SPOOL_MAX_BYTES.data/oracle-evidence/256 MiBORACLE_CONSENSUS_MIN_SOURCES1(each metric has one live source today; raise to 2 when a second source lands)ORACLE_CONSENSUS_TARGET_SOURCES/ORACLE_CONSENSUS_MIN_CONFIDENCE2/0.5ORACLE_CONSENSUS_DECISION_TTL_MS30000ORACLE_SMOOTHING_WINDOW_MS/_MIN_SAMPLES/_MIN_SPAN_MS/_MAX_SAMPLES15 min/3/2 min/60STELLAR_HORIZON_URL(the name docker-compose uses) is honoured, withHORIZON_URLas a fallback.Testing
OracleService, spool, buffer, repository, consensus and smoother, plus 11 for the consensus-gatedClaimService. All pass, and each commit passes on its own.Pre-existing repo breakage (not fixed here)
maindoes not currently build or run tests, independent of this PR:package.jsonis empty.app.module.tshas merge debris and imports modules that don't exist.claim.module.ts,claim-settlement.service.ts,oracle-publisher.service.ts,oracle.controller.ts,db.module.tsandoracle.scheduler.spec.ts.src/parameters/is missing.How this PR was verified despite that:
tsc --strictand tested with jest in an isolated harness.ClaimServicewas typechecked against a stub of the emptyClaimSettlementService, because the real file has no exports.oracle.service.tsandoracle.module.ts, which were also empty.OracleModulenow exportsConsensusServiceandOracleEvidenceBuffer, so a restoredClaimModuleonly needs to importOracleModule, as it did before.Deployment notes
src/db/migrations/004_oracle_evidence_sequence.sql.ORACLE_EVIDENCE_SPOOL_DIRin every deployed environment.Closes #227
Closes #228
Closes #229
Closes #230