From bfb3281f1a49fd3210dcaabcebbe5def6c9c2556 Mon Sep 17 00:00:00 2001 From: ummarig <262570923+ummarig@users.noreply.github.com> Date: Fri, 25 Sep 2026 15:46:59 +0100 Subject: [PATCH] docs(testnet): add health monitoring and nonce recovery Closes #627 Closes #626 Closes #625 Closes #619 --- testnet/monitoring/balance-thresholds.md | 179 ++++++++++++ testnet/monitoring/event-watchlist.md | 233 +++++++++++++++ testnet/monitoring/uptime-check-spec.md | 349 +++++++++++++++++++++++ testnet/runbooks/nonce-desync.md | 273 ++++++++++++++++++ 4 files changed, 1034 insertions(+) create mode 100644 testnet/monitoring/balance-thresholds.md create mode 100644 testnet/monitoring/event-watchlist.md create mode 100644 testnet/monitoring/uptime-check-spec.md create mode 100644 testnet/runbooks/nonce-desync.md diff --git a/testnet/monitoring/balance-thresholds.md b/testnet/monitoring/balance-thresholds.md new file mode 100644 index 0000000..9d35871 --- /dev/null +++ b/testnet/monitoring/balance-thresholds.md @@ -0,0 +1,179 @@ +# Testnet Balance Thresholds for Long-Lived Accounts + +## Status + +> **Proposed operating policy.** No automated balance check or Friendbot +> top-up exists in this repository. The known-account registry currently has +> only placeholders, so these values are starting guardrails rather than +> measurements from existing accounts. + +This policy applies to long-lived Stellar Testnet `G...` identities used to +deploy, initialize, record payments, submit sweeps, or demonstrate the +contracts. A `C...` contract address is not the Friendbot funding target; fund +the public identity that pays the transaction. + +## Units + +- Native XLM uses stroops. +- `1 XLM = 10,000,000` stroops. +- Store and compare balances as integer stroops, not floating-point values. +- SEP-41 payment amounts use each asset's own base units. + +The `EphemeralAccount` source constant `1,000,000,000` stroops (100 XLM) is +internal reserve bookkeeping, not the minimum balance for a fee-payer account. +Do not copy it into this policy. + +## Starting thresholds + +| State | Balance | Stroops | Action | +|---|---:|---:|---| +| Hard floor | 5 XLM | `50,000,000` | Stop scheduled writes and demos; re-fund first | +| Re-fund trigger | 10 XLM | `100,000,000` | Request one Friendbot top-up before the next run | +| Working target | 50 XLM | `500,000,000` | Restore toward this target after a top-up | + +Decision logic: + +```text +balance < 50,000,000 + -> hard_floor_breach + +50,000,000 <= balance < 100,000,000 + -> refund_required + +balance >= 100,000,000 + -> usable for routine low-volume work + +balance < 500,000,000 + -> schedule restoration toward the working target +``` + +These are conservative operational values, not Stellar protocol minimums. A +read-only identity may use a lower, explicitly approved threshold. Identities +that run batches, relayers, or demonstrations should use the 50 XLM target. + +## When to check + +Check the public fee-payer balance: + +- immediately before a demonstration or integration run; +- hourly while the identity is actively used; +- daily for a dormant but registered fixture; +- after any insufficient-funds or unexpectedly high-fee report; and +- before a scheduled funding request. + +A balance check is read-only and must not submit a transaction. + +## Friendbot procedure + +Friendbot is an operator-triggered funding source. The monitor may recommend a +re-fund, but it must not loop faucet requests automatically. + +### 1. Identify and read the account + +Use only the public `G...` address recorded in +[`../registry/known-test-accounts.md`](../registry/known-test-accounts.md) or +another approved deployment configuration. Confirm the network is Stellar +Testnet and record: + +- public account address; +- UTC check time; +- balance in stroops; and +- reason for the check. + +Use Horizon, Soroban RPC account data, or an approved balance-query command. +Do not place a secret key in the command or logs. + +### 2. Request one top-up when required + +The documented endpoint is: + +```text +https://friendbot.stellar.org/?addr= +``` + +An operator may instead use the installed Stellar CLI's supported funding +command. Confirm the exact command syntax with the CLI help before use; do not +copy a funding command from a different CLI major version. + +Only request a top-up when the balance is below the 10 XLM trigger or when the +hard floor requires restoration. Do not retry in a tight loop because the +faucet may be rate-limited by account, IP, or network policy. + +### 3. Verify the result + +After the request: + +1. Read the public account balance again. +2. Require an increased balance or record the faucet response and + `no_balance_change`. +3. Confirm the identity can still act as the intended fee payer. +4. Update the registry's last-checked and last-refunded fields. +5. Restore toward the 50 XLM working target before a presentation. + +A successful HTTP response is not proof that the balance increased. + +## Registry fields + +Each populated long-lived account row should eventually include: + +| Field | Meaning | +|---|---| +| Public `G...` account | identity that pays fees | +| Role | deployer, recorder, relayer, watcher, demo operator, recipient | +| Contract/fixture IDs | account or contracts used by the role | +| Threshold policy | approved XLM/stroop values | +| Last balance check | UTC timestamp and balance | +| Last Friendbot attempt | UTC timestamp and result, without credentials | +| Next review | earliest recalibration date | + +A contract ID, recorded payment amount, and fee-payer balance are different +values. Never use `Payment.amount` or `SweepCompleted.amount` as a native XLM +account balance. + +## Safety rules + +- Never automate Friendbot in a loop. +- Never include a secret key, seed phrase, or auth signature in a URL or log. +- Never use a `C...` contract ID as the Friendbot address. +- Do not silently replace a low-balance identity; update every dependent + deployment reference. +- Do not spend or transfer a known demo identity's funds as part of checking + its balance. +- A testnet reset can remove an account. Record `account_not_found` as a + deployment/baseline failure, not as a reason to reuse old credentials. +- Keep enough balance for the next scheduled activity, not only the pending + transaction. + +## Recalibration + +Review the thresholds after at least seven days of representative activity or +30 funded transactions. Recalculate using: + +- median and high-percentile transaction/resource fees; +- frequency of each role; +- number of retries and failed submissions; +- testnet resets or redeployments; and +- the cost of one emergency top-up before a demonstration. + +Raise thresholds when observed burn invalidates the working target. Do not +lower the hard floor merely because a particular run happened to fit. + +## Limitations + +The repository has no populated known-account list or measured fee-spend +history. Soroban resource fees, rent, account entries, and network conditions +can vary, so a positive balance does not guarantee that a later complex call +will have enough resources. + +The current `EphemeralAccount` internal reserve and standalone +`ReserveContract` configuration are separate protocol values. They must not be +treated as demo fee-payer thresholds without a separate design decision. + +## Related documents + +- [`../registry/known-test-accounts.md`](../registry/known-test-accounts.md) +- [`../docs/network-config.md`](../docs/network-config.md) +- [`../config/soroban-cli-identity-setup.md`](../config/soroban-cli-identity-setup.md) +- [`../docs/getting-started.md`](../docs/getting-started.md) +- [`uptime-check-spec.md`](uptime-check-spec.md) +- [`event-watchlist.md`](event-watchlist.md) diff --git a/testnet/monitoring/event-watchlist.md b/testnet/monitoring/event-watchlist.md new file mode 100644 index 0000000..88ff7ab --- /dev/null +++ b/testnet/monitoring/event-watchlist.md @@ -0,0 +1,233 @@ +# Testnet Event Watchlist + +## Status + +> **Proposed event reference and absence policy.** No event collector or +> absence alert is active in this repository, and no honest positive event-rate +> baseline has been measured. + +This document identifies the current source event symbols, the conditions under +which their absence is meaningful, and the evidence required before an alert. + +## Source-of-truth rules + +The current source files are: + +- `contracts/ephemeral_account/src/events.rs` +- `contracts/sweep_controller/src/lib.rs` +- `contracts/reserve_contract/src/events.rs` + +The deployed testnet WASM may differ. Record the contract ID, code/WASM hash, +and deployed interface before enabling a baseline. A method or payload mismatch +is a version signal, not automatically a network outage. + +A testnet reset can remove contracts and historical events. Segment every +baseline by reset epoch. + +## Event encoding + +Soroban contract events contain an event-name symbol in `topics[0]`, optional +indexed values, and a contract-type payload in data. Indexers must filter by +both contract ID and `topics[0]`. + +The Rust struct name is not always the on-chain symbol. For example, +`PaymentReceived` is published as `payment`, and `SweepExecutedMulti` as +`swept_mul`. + +## EphemeralAccount events + +| Topic `0` | Payload | Emitted by | Absence meaning | +|---|---|---|---| +| `created` | `AccountCreated { creator, expiry_ledger }` | successful `initialize` | Meaningful after a known initialization should have committed | +| `payment` | `PaymentReceived { amount, asset }` | first successful `record_payment` | Meaningful after a known first payment record | +| `multi_pay` | `MultiPaymentReceived { asset, amount }` | later distinct-asset `record_payment` | Meaningful after a known later record; not every account emits it | +| `swept_mul` | `SweepExecutedMulti { destination, payments }` | account-side `sweep` or `sweep_claim` | Meaningful after a known sweep/claim operation | +| `expired` | `AccountExpired` | successful `expire` or `recover` | Meaningful after a known terminal recovery action; not automatic at expiry ledger | +| `reserve` | `ReserveReclaimed` | reserve reclaim bookkeeping | Meaningful after a known sweep/expiry/reclaim action | + +Important payload rules: + +- Current source names the `AccountExpired` aggregate field + `amount_returned`; older documents may call it `total_amount`. +- `Payment.amount` and `PaymentReceived.amount` use the asset's base units. +- `Payment.timestamp` is the ledger timestamp captured by `record_payment`. +- `expiry_ledger` and `ReserveReclaimed.sweep_id` are ledger values, not wall + clock timestamps. +- A `payment`/`multi_pay` event proves metadata was recorded, not that a real + asset transfer occurred. +- A `swept_mul` event proves the account-side state path ran, not by itself that + the destination balance changed. + +## SweepController events + +| Topic `0` | Payload | Emitted by | Absence meaning | +|---|---|---|---| +| `sweep` | `SweepCompleted { ephemeral_account, destination, amount }` | current `execute_sweep` or `claim` | Meaningful after a known controller operation | +| `dest_auth` | `DestinationAuthorized { destination }` | initialization with a locked destination | Expected once for a known locked-controller initialization | +| `dest_upd` | `DestinationUpdated { old_destination, new_destination }` | successful authorized destination update | Meaningful after a known destination update | + +`SweepCompleted.amount` is a raw sum of recorded amounts. It can combine +different asset base units and must not be presented as one monetary total. + +A `sweep` event is not a balance receipt. Verify final transaction status and +per-asset destination balance when the metric means funds received. + +## ReserveContract events + +| Topic `0` | Payload | Emitted by | Absence meaning | +|---|---|---|---| +| `init` | `ContractInitialized { admin }` | successful initialization | Expected once for a known initialized contract | +| `reserve` | `BaseReserveUpdated { old_value, new_value, admin }` | successful base-reserve change | Meaningful after a known configuration update | + +The current source publishes `init`, not `initialized`, and `reserve`, not +`base_reserve_updated`. Some existing registry prose predates those symbols. +Use the deployed interface as the final authority. + +`ReserveContract` is not currently wired into `EphemeralAccount` by the +checked-in deployment tooling. Its events are optional for the core profile. + +## AccountFactory events + +The current `AccountFactory` source emits no direct events. A successfully +initialized child account emits its own `created` event. Do not alert on the +absence of a factory-specific event unless a future deployed WASM explicitly +adds one. + +## Current minimum baseline + +There is no defensible positive frequency baseline today because: + +- the known-account registry contains placeholders; +- no synthetic workload is active; +- no historical event cursor is committed; +- deployment records disagree; and +- testnet resets can erase history. + +The only honest standing minimum for new events in an arbitrary idle window +is: + +```text +0 new events per event type +``` + +This does not mean an idle deployment is unhealthy. Event absence is meaningful +only when a specific committed or expected operation should have produced it. + +## Conditional expectations + +| Event | Expected baseline | Alert precondition | +|---|---|---| +| `created` | one per registered initialized child account | known successful initialization and cursor covers its ledger | +| `payment` | one for a fixture recorded for its first asset | known first `record_payment` succeeded | +| `multi_pay` | one per known later distinct asset | later record was expected | +| `swept_mul` | one for each successful account-side sweep/claim | operation and account are known | +| `expired` | one for each explicit expiry/recovery action | terminal action was expected; time eligibility alone is insufficient | +| ephemeral `reserve` | one per known reclaim attempt | reclaim was expected | +| `sweep` | one per known controller completion | controller operation and account are known | +| `dest_auth` | one historical event for a locked controller | initialization with a destination is known | +| `dest_upd` | one per authorized destination update | update is known | +| Reserve `init` | one historical event when initialized | initialization is known | +| Reserve `reserve` | one per base-reserve change | configuration change is known | +| AccountFactory event | not applicable | never alert with the current source | + +A fixture must not be required to emit both `payment` and `multi_pay`. Declare +which operation is expected from its purpose and state. + +## Absence-alert policy + +Do not alert merely because: + +- no event appeared in the last five minutes; +- no sweep was attempted; +- no payment was recorded; +- no account reached an explicit `expire`/`recover` action; +- no destination update was requested; +- no reclaim was attempted; or +- an optional contract is not configured. + +An event-overdue alert is valid only when: + +1. the triggering operation was expected to succeed; +2. the transaction/ledger range is known; +3. the contract ID and deployed interface are current; +4. the collector cursor covers the range and is not lagging; and +5. the event symbol is appropriate for that operation. + +A practical initial indexing deadline is two collector poll cycles, with a +five-minute maximum. It must be tuned to the actual collector and is not a +contract SLA. + +## State versus event failures + +- RPC timeout, contract-not-found, or a non-decoding read: investigate RPC, + deployment, and interface state first. +- State changed successfully but the event is absent: investigate the event + cursor, decoder, retention, and deployed ABI. +- Event exists but state is inconsistent: investigate the deployed version, + transaction correlation, or state projection. +- All old IDs disappear after a ledger decrease: declare the baseline stale and + require a new reset epoch and deployment verification. + +Do not infer failure solely from a missing event. + +## Collector record + +A normalized event should retain at least: + +```json +{ + "network": "testnet", + "reset_epoch": "", + "contract_id": "", + "topic_0": "payment", + "ledger": 0, + "transaction_hash": "", + "event_index": 0, + "data": {} +} +``` + +Deduplicate by reset epoch, ledger, transaction hash, contract ID, and event +index. Preserve raw XDR for incident analysis according to a documented +retention policy, but never store credentials or unrelated auth payloads. + +## Units + +| Field | Unit | +|---|---| +| `expiry_ledger`, `sweep_id` | ledger sequence | +| `Payment.timestamp` | ledger timestamp seconds | +| payment/sweep amounts | asset base units (`i128`) | +| `AccountExpired.amount_returned` | raw aggregate of recorded amounts; not cross-asset money | +| reserve amounts | stroops | + +Native XLM uses 10,000,000 stroops per XLM. Other SEP-41 assets can use +different base units. Never add heterogeneous amounts and label the result a +balance. + +## Activation checklist + +- [ ] Reconcile and verify contract IDs after the latest reset. +- [ ] Record deployed interfaces and current code identity. +- [ ] Configure an explicit contract/topic allowlist. +- [ ] Persist and test the ledger cursor and deduplication. +- [ ] Populate known account/operation expectations. +- [ ] Define pending-transaction and pending-expiry handling. +- [ ] Test missing-event, delayed-indexer, and RPC-outage cases. +- [ ] Calibrate positive frequency only after a declared workload. +- [ ] Approve owners and non-production notification routes. + +Until then, the status is: + +```text +testnet_event_watchlist: not_configured +``` + +## Related documents + +- [`../registry/event-topics.md`](../registry/event-topics.md) +- [`../registry/known-test-accounts.md`](../registry/known-test-accounts.md) +- [`../registry/contract-status.md`](../registry/contract-status.md) +- [`../config/nonce-tracking.md`](../config/nonce-tracking.md) +- [`../runbooks/failed-sweep-signature.md`](../runbooks/failed-sweep-signature.md) +- [`uptime-check-spec.md`](uptime-check-spec.md) diff --git a/testnet/monitoring/uptime-check-spec.md b/testnet/monitoring/uptime-check-spec.md new file mode 100644 index 0000000..a1e3150 --- /dev/null +++ b/testnet/monitoring/uptime-check-spec.md @@ -0,0 +1,349 @@ +# Testnet Uptime-Check Specification + +## Status + +> **Specification only.** This document defines a future read-only health +> check. The repository does not contain a scheduler, monitor, alert +> integration, or uptime history. + +A future check should prove that configured contracts resolve and return +internally consistent state. It must not be confused with a synthetic +create-pay-sweep transaction or an event-absence policy. + +## Questions answered by the check + +Each cycle should answer: + +1. Does the configured contract ID resolve on Stellar Testnet? +2. Does the deployed contract expose the expected read interface? +3. Does the returned state decode and satisfy basic invariants? +4. Is a known account fixture in the lifecycle state recorded for it? +5. Is the Soroban ledger advancing and the collector current? + +The check does not prove that a token transfer, signature, relayer, payment +watcher, or sweep authorization is correct. + +## Source and deployment preconditions + +The proposed interface is based on the checked-in contract source. Before a +monitor is enabled: + +- reconcile `testnet/registry/contract-status.md` with deployment artifacts; +- verify every contract ID through a read-only call; +- record the deployed WASM/code identity and interface; +- populate at least one known initialized `EphemeralAccount` fixture in + [`../registry/known-test-accounts.md`](../registry/known-test-accounts.md); +- define which contracts are required versus optional; and +- identify the current Stellar testnet reset epoch. + +A stored ID is configuration, not proof of liveness. A method-not-found or ABI +decode error is an interface/version signal until the deployed identity is +checked. + +## Required configuration + +| Setting | Meaning | +|---|---| +| `NETWORK` | `testnet` | +| `NETWORK_PASSPHRASE` | `Test SDF Network ; September 2015` | +| `RPC_URL` | Primary Soroban RPC endpoint | +| `FALLBACK_RPC_URL` | Optional, independently reviewed fallback | +| `EA_CONTRACT_ID` | Verified controller-linked account or fixture contract | +| `SWEEP_CONTROLLER_ID` | Verified `SweepController` contract | +| `READER_IDENTITY` | Funded identity used only for simulations/reads | +| `EXPECTED_*_WASM_HASH` | Optional code identity for drift detection | +| `POLL_INTERVAL_SECONDS` | Core frequency | +| `REQUEST_TIMEOUT_SECONDS` | Per-probe deadline | + +Do not store secret keys or auth payloads in monitor configuration. A read-only +simulation should not require a transaction submission. + +## Contract profiles + +### Core profile + +- `EphemeralAccount` +- `SweepController` + +Enable the core profile only after both IDs and a known initialized account +fixture are verified. + +### Optional profile + +- `ReserveContract` +- `AccountFactory` + +A missing optional contract is `not_configured`, not automatically a public +network outage. The current registry does not verify either optional contract +as deployed. + +## Read-only transport + +Use Soroban RPC `simulateTransaction` or an SDK equivalent that submits no +transaction. Do not call any state-changing method as an uptime probe, +including: + +- `initialize`, `record_payment`, `sweep`, `sweep_claim`, `expire`, + `recover`, `reclaim_reserve`, or `upgrade` on `EphemeralAccount`; +- `initialize`, `execute_sweep`, `claim`, or + `update_authorized_destination` on `SweepController`; or +- `initialize` and `set_base_reserve` on `ReserveContract`. + +CLI examples below are operator references. A worker must use a confirmed +dry-run or simulation path so a version change cannot accidentally submit a +write. + +## Probe matrix + +| Contract | Method | Arguments | Expected shape | Core frequency | +|---|---|---|---|---| +| `EphemeralAccount` | `get_info` | none | `AccountInfo` or `NotInitialized` | Every cycle | +| `EphemeralAccount` | `get_status` | none | `0..=3` | Every cycle | +| `EphemeralAccount` | `is_expired` | none | Boolean | Every cycle | +| `EphemeralAccount` | `get_payment_count` | none | `0..=10` | Every cycle | +| `SweepController` | `get_nonce` | none | `u64` | Every cycle | +| `SweepController` | `can_sweep` | account address | Boolean | Every cycle when fixture exists | +| `ReserveContract` | `has_base_reserve` | none | Boolean | Optional, every 30 minutes | +| `ReserveContract` | `get_base_reserve` | none | `Option` in stroops | Optional, every 30 minutes | +| `ReserveContract` | `get_admin` | none | `Option
` | Optional, every 30 minutes | + +The current `known-test-accounts.md` contains placeholders. Until a real +initialized fixture is registered, an `EphemeralAccount` result must be +reported as `fixture_missing`, not silently treated as a healthy active +account. + +## Response contracts + +### `EphemeralAccount::get_info` + +A successful initialized response has this logical shape: + +```json +{ + "creator": "
", + "status": 0, + "expiry_ledger": 0, + "recovery_address": "
", + "payment_received": false, + "payment_count": 0, + "payments": [], + "swept_to": null +} +``` + +Required invariants: + +- `status` is `0`, `1`, `2`, or `3`; +- `payment_count` is between 0 and 10; +- `payment_count` equals the number of returned payment records; +- `payment_received` equals `payment_count > 0`; +- payment assets are unique and amounts are positive; +- amounts remain in the asset's base units; and +- `swept_to` is absent or a valid address. + +`get_info` must fail with `NotInitialized` for an uninitialized account. That +error is not an initialized-account health pass. + +### `get_status`, `is_expired`, and payment count + +Current status values are: + +| Value | State | +|---:|---| +| `0` | `Active` | +| `1` | `PaymentReceived` | +| `2` | `Swept` | +| `3` | `Expired` | + +`is_expired` is true when the current ledger is greater than or equal to +`expiry_ledger`. Record the ledger used for the comparison; do not convert the +decision to wall-clock time. + +`get_status` returns `Active` for an uninitialized contract, and +`get_payment_count` returns zero. Cross-check both with `get_info` before +reporting a green fixture. + +### `SweepController::get_nonce` + +The response is an unsigned 64-bit integer. Normalize it as a decimal string so +JavaScript consumers do not lose precision. + +- Zero is a valid newly initialized value. +- A missing storage key also defaults to zero, so a zero response alone does + not prove initialization. +- For the same contract and reset epoch, the value must not decrease. + +### `SweepController::can_sweep` + +A Boolean response is valid in either state. Current source returns `true` only +when the account has a recorded payment, status is `PaymentReceived`, and the +account is not expired. Cross-check a `true` result with the account's +`get_info` and `is_expired` values. + +A `false` result is not automatically an outage: a new, already swept, expired, +or uninitialized fixture can legitimately return false. + +## Optional ReserveContract checks + +For an enabled and initialized optional contract: + +- `get_base_reserve` returns `Some(stroops)` or `None`; +- `has_base_reserve` agrees with the presence of a stored value; +- `get_admin` returns the configured admin when initialized; and +- the value is within the contract's accepted limit. + +`None` or `false` is valid before configuration. Report it as `not_ready`, not +as a failed network call. + +`ReserveContract` reads currently extend its instance TTL. A repeated read +probe can therefore keep the contract active; that is a side effect of the +current source and is not proof of user traffic. + +## Example operator read + +```bash +stellar contract invoke \ + --simulate-only \ + --id \ + --network testnet \ + --source \ + -- \ + get_info + +stellar contract invoke \ + --simulate-only \ + --id \ + --network testnet \ + --source \ + -- \ + get_nonce + +stellar contract invoke \ + --simulate-only \ + --id \ + --network testnet \ + --source \ + -- \ + can_sweep \ + --ephemeral_account +``` + +A monitor should call the equivalent RPC simulation and store structured +results rather than parse human-formatted CLI text. + +## Timing and retry policy + +Initial proposal, subject to a measured baseline: + +| Setting | Value | +|---|---:| +| Core cycle | Every 300 seconds, with 0-30 seconds jitter | +| Optional cycle | Every 1,800 seconds | +| Per-attempt timeout | 10 seconds | +| Retries | Two after the initial attempt | +| Backoff | 1 second, then 3 seconds | +| Maximum core-cycle duration | 30 seconds | +| Consecutive-failure threshold | Three cycles before a low-urgency alert | + +Do not overlap retries for the same contract. A retry is a transport attempt, +not a new state sample, until one result is accepted and recorded. + +Stellar testnet ledgers close roughly every five seconds, but the ledger +sequence returned by the RPC is authoritative. Associate every probe with the +latest ledger observed in that cycle. + +## Result schema + +```json +{ + "check_id": "", + "network": "testnet", + "reset_epoch": "", + "contract_id": "", + "method": "get_info", + "arguments": {}, + "started_at": "", + "completed_at": "", + "duration_ms": 0, + "rpc_ledger": 0, + "outcome": "pass", + "result": {}, + "error": null +} +``` + +Controlled outcomes: + +- `pass` +- `fail` +- `degraded` +- `not_configured` +- `fixture_missing` +- `timeout` +- `rpc_error` +- `contract_not_found` +- `interface_mismatch` +- `decode_error` +- `unexpected_value` +- `nonce_regression` +- `testnet_reset` + +Store `u64` and `i128` values as decimal strings. Do not store credentials, +auth signatures, or unrelated transaction payloads in a probe record. + +## Pass criteria + +A core cycle passes only when: + +1. the network and reset epoch match configuration; +2. the ledger is advancing; +3. all required contract calls return and decode; +4. the known initialized account satisfies the `get_info` invariants; +5. `get_status`, `is_expired`, and payment count agree with that state; +6. `get_nonce` does not regress; and +7. a `can_sweep` response, when requested, is internally consistent. + +Do not report a pass solely because a safe default such as `Active`, zero, or +false was returned. + +## Testnet reset handling + +A lower ledger sequence, missing contract, or changed code identity can indicate +a reset or redeployment. On detection: + +1. stop using the old baseline; +2. create a new reset epoch; +3. quarantine old contract IDs and cursor history; +4. require deployment-record reconciliation; +5. verify new IDs, interfaces, and fixtures; and +6. resume only after preflight passes. + +Never compare a nonce, fixture state, or event rate across reset epochs without +segmenting it. + +## Acceptance criteria for a future implementation + +The implementation is ready when it can demonstrate that it: + +- uses simulation and never submits a state-changing probe; +- records contract ID, method, ledger, latency, decoded result, and error; +- preserves 64-bit and 128-bit numeric precision; +- distinguishes valid defaults from initialization failures; +- handles missing optional contracts and fixtures explicitly; +- detects nonce regression and testnet resets; +- obeys the documented frequencies and bounded timeouts; and +- survives deployment ID and interface changes without a false green state. + +Until those conditions are implemented and exercised, the status is: + +```text +testnet_uptime_check: not_configured +``` + +## Related documents + +- [`../registry/verify-contract-live.md`](../registry/verify-contract-live.md) +- [`../registry/known-test-accounts.md`](../registry/known-test-accounts.md) +- [`../registry/contract-status.md`](../registry/contract-status.md) +- [`balance-thresholds.md`](balance-thresholds.md) +- [`event-watchlist.md`](event-watchlist.md) diff --git a/testnet/runbooks/nonce-desync.md b/testnet/runbooks/nonce-desync.md new file mode 100644 index 0000000..b97a377 --- /dev/null +++ b/testnet/runbooks/nonce-desync.md @@ -0,0 +1,273 @@ +# Runbook: Recover SweepController Nonce Desynchronization + +## Purpose + +Use this runbook when an off-chain signer or relayer's cached +`SweepController` nonce no longer matches the live testnet contract. + +The authoritative value is always the result of: + +```text +SweepController::get_nonce() +``` + +An off-chain counter is a cache and queue coordination aid. It must never be +used to overwrite, guess, or decrement contract state. + +This nonce is distinct from a Stellar account sequence number, an +`expiry_ledger`, and any per-account counter. There is one security nonce per +`SweepController` deployment, shared by every account it services. + +## Signed message and nonce lifecycle + +The current controller signs and verifies this digest: + +```text +SHA256( + destination.to_xdr() + || nonce encoded as an unsigned u64 in 8-byte big-endian order + || sweep_controller_id.to_xdr() +) +``` + +The ephemeral-account address is not included in the current digest. Preserve +the exact destination and controller ID used to create a signature. + +For a committed `execute_sweep` transaction, the controller verifies the +signature and advances its nonce as part of the atomic transaction. A reverted +transaction does not persist a nonce increment. Query the chain after any +uncertain result rather than predicting rollback from a local counter. + +A `sweep` event can be emitted by more than one controller path in current +source. Do not infer the on-chain nonce by counting events; `get_nonce` is the +authority. + +## Symptoms + +Typical indicators include: + +- repeated signature-verification failures after a successful sweep; +- a signature that looks correct until another relayer submits; +- a local nonce lower or higher than the contract value; +- a local increment after a failed or unknown transaction; +- multiple workers signing from the same cached value; +- stale signatures after a testnet reset or controller redeployment; or +- a signature failure caused by a different destination, contract ID, signer + key, or XDR encoding. + +A fresh signature failure does not by itself prove nonce desynchronization. + +## Immediate safety actions + +1. Pause signing and submission for the affected controller. +2. Preserve the local nonce, destination, controller ID, transaction hashes, + UTC time, and approximate ledger. +3. Do not reuse any signature built with a stale or uncertain nonce. +4. Do not retry an in-flight transaction before resolving its result by hash. +5. Serialize all remaining work per controller ID. +6. Do not call a write method to "reset" the contract nonce; no such method is + required or documented. +7. Keep signing seeds, secret keys, and full signatures out of tickets and + public logs. + +## Detection procedure + +### 1. Capture incident context + +Record: + +- exact controller contract ID and network passphrase; +- local/cached nonce used for each failed signature; +- destination used in the invocation; +- transaction hash, if submitted; +- whether the path was `execute_sweep` or another controller method; +- approximate ledger and UTC time; and +- all workers or services that can sign for this controller. + +### 2. Query the authoritative nonce + +Use a confirmed read-only simulation or dry-run invocation: + +```bash +stellar contract invoke \ + --simulate-only \ + --id \ + --network testnet \ + --source \ + -- \ + get_nonce +``` + +Record the result as an unsigned decimal integer. An automated worker should +use the SDK/RPC simulation equivalent and must not submit a transaction. + +A successful `0` is valid, but the storage getter also defaults to zero when +the key is absent. Zero alone does not prove controller initialization. + +If the query fails, verify the RPC, network, exact controller ID, testnet +reset status, and deployed interface. Do not reset the local cache merely +because the query failed. + +### 3. Compare the values + +Let: + +```text +L = local/cached nonce +C = value returned by get_nonce() +``` + +| Comparison | Interpretation | Action | +|---|---|---| +| `L == C` | cache is currently aligned | inspect destination, controller ID, key, XDR, and deployed code if the signature still fails | +| `L < C` | signer is behind the chain | discard signatures made with `L`; fetch `C` and sign again | +| `L > C` | local state is ahead or based on a different deployment/path | pause and reconcile before resetting the cache | +| `C - L > 1` | multiple operations or a tracking bug likely occurred | stop the queue and account for every pending operation | +| `C < previous observed C` | reset, contract change, or bad history | quarantine the baseline; do not reuse the old counter | + +Never modify the on-chain value to match the local cache. + +### 4. Correlate committed activity + +Inspect finalized controller and account activity over the relevant ledger +range. Correlate: + +- controller ID; +- ephemeral-account ID; +- destination; +- transaction hash and ledger; +- final transaction result; and +- controller `sweep` and account `swept_mul` events when available. + +Events support diagnosis but do not replace `get_nonce`. A failed Soroban +transaction is atomic and should not leave a nonce increment, state change, +event, or transfer committed. + +### 5. Verify every signed component + +If `L == C` and a new signature still fails, stop treating it as a nonce +problem. Check, in order: + +1. The destination is byte-for-byte the address used in the signature. +2. The controller ID is the same testnet deployment. +3. The nonce is exactly eight bytes, unsigned big-endian. +4. Destination and controller addresses use the same Soroban `Address::to_xdr` + encoding as the contract. +5. The private key matches the controller's `authorized_signer` public key. +6. No timestamp or extra field was added to the current message format. +7. The signature is exactly 64 bytes and was not truncated or text-mangled. +8. The deployed WASM/interface is the expected version. + +A successful `get_nonce` call does not prove any of those values. + +## Recovery: local nonce is behind the chain + +1. Hold the per-controller signing lock. +2. Read and record `C`. +3. Mark every pending signature built with a nonce below `C` as invalid. +4. Discard those signatures. +5. Read `get_nonce` again immediately before the next signature. +6. Sign the exact destination and controller ID with that fresh value. +7. Submit while holding the lock and wait for a definitive result. +8. Re-read `get_nonce` and record the progression. + +Do not change the nonce bytes inside an already signed message and expect the +old signature to remain valid. + +## Recovery: local nonce is ahead of the chain + +1. Pause all signers and submissions. +2. Query and record `C`. +3. Determine whether the local increment came from a failed/unknown + transaction, a different controller deployment, a nonstandard path, or a + bookkeeping bug. +4. Reconcile every pending transaction by hash. +5. Discard signatures that depend on the invalid local value. +6. Remove the persistent local counter or set the cache to `C` only after the + chain state is understood. +7. Re-read `C` immediately before each new signature. + +If `C` unexpectedly decreased for the same contract ID, treat the deployment +or reset epoch as changed and require fresh verification. + +## Recovery: concurrent signers + +Use one queue or mutex per `SweepController` contract ID: + +1. acquire the controller lock; +2. read `get_nonce`; +3. sign and submit while holding the lock; +4. wait for a definitive transaction result; +5. re-read `get_nonce`; and +6. release the lock. + +Do not allow two workers to sign from the same cached value. The first +successful committed sweep invalidates the other signature. + +A timeout releases no lock automatically until the submitted transaction's +result is resolved. + +## Recovery: fresh signature still fails + +When `L == C`, build a new signature only after verifying the other components +above. If it still fails, classify the incident as destination, controller ID, +signer key, XDR encoding, deployed-version, or host-verification failure—not +as nonce desynchronization. + +Preserve the diagnostic result and do not repeatedly submit different payloads +to guess which field is wrong. + +## Verification and closure + +The incident is closed only when: + +- the controller lock/queue is consistent; +- no stale or unknown transaction remains; +- a new signature is built from a fresh authoritative nonce; +- destination and controller ID are unchanged; +- a committed `execute_sweep` advances the nonce by one, or the failure is + reclassified with direct evidence; and +- the expected account state, controller event, and destination balance are + correlated for a successful sweep. + +For a successful sweep, also query the account's terminal state and relevant +balances. A controller event alone is not sufficient evidence of value +movement. + +## Escalation + +Escalate to the contract/testnet maintainers when: + +- `get_nonce` fails for a previously verified controller; +- the nonce decreases in the same deployment/reset epoch; +- a transaction succeeded but the nonce did not change as expected; +- a reverted transaction appears to have committed a nonce increment; +- the same contract ID resolves to an unexpected code/interface; +- all configured IDs disappear after a suspected testnet reset; or +- the destination or signer identity may be wrong. + +Attach the contract ID, local and on-chain nonces, redacted signature metadata, +transaction hashes, finalized status, relevant events, and deployed code +identity. Never attach signing seeds or secret keys. + +## Prevention checklist + +- [ ] Read `get_nonce` immediately before every signature. +- [ ] Serialize sign-and-submit per controller. +- [ ] Hold the lock through definitive transaction finalization. +- [ ] Reconcile uncertain submissions by hash before retrying. +- [ ] Treat testnet reset/deployment changes as a new nonce epoch. +- [ ] Log the controller ID, nonce used, destination ID, and transaction hash, + but not the secret key or full reusable signature. +- [ ] Alert on nonce regression and unexplained jumps. +- [ ] Test concurrent-worker and unknown-transmission scenarios in the future + signer implementation. + +## Related documents + +- [`../config/nonce-tracking.md`](../config/nonce-tracking.md) +- [`failed-sweep-signature.md`](failed-sweep-signature.md) +- [`../registry/event-topics.md`](../registry/event-topics.md) +- [`../registry/verify-contract-live.md`](../registry/verify-contract-live.md) +- [`../../docs/SIGNATURE_FORMAT.md`](../../docs/SIGNATURE_FORMAT.md) +- [`../../tools/sweep-signer/`](../../tools/sweep-signer/)