diff --git a/testnet/runbooks/README.md b/testnet/runbooks/README.md index 94fef9b..7ad8304 100644 --- a/testnet/runbooks/README.md +++ b/testnet/runbooks/README.md @@ -12,6 +12,10 @@ This directory contains operational runbooks for diagnosing and resolving issues | [`multi-payment-edge-cases.md`](multi-payment-edge-cases.md) | Repeated `record_payment` for same asset | EphemeralAccount | | [`failed-sweep-signature.md`](failed-sweep-signature.md) | `SweepController::execute_sweep` signature rejection | SweepController, EphemeralAccount | | [`stuck-ephemeral-account.md`](stuck-ephemeral-account.md) | EphemeralAccount won't sweep or expire | EphemeralAccount, SweepController | +| [`wrong-network-passphrase.md`](wrong-network-passphrase.md) | Transaction rejected with `tx_bad_auth`; signed for the wrong network | Network layer (any contract) | +| [`insufficient-reserve-errors.md`](insufficient-reserve-errors.md) | Reserve-related failures: network base reserve vs. internal reserve tracking | EphemeralAccount, network layer | +| [`event-not-appearing-in-horizon.md`](event-not-appearing-in-horizon.md) | Successful transaction but expected contract event missing from Horizon | EphemeralAccount, SweepController | +| [`contract-not-found-error.md`](contract-not-found-error.md) | RPC cannot resolve a contract ID (stale ID, wrong network, never deployed) | Any contract | ## Quick Start diff --git a/testnet/runbooks/contract-not-found-error.md b/testnet/runbooks/contract-not-found-error.md new file mode 100644 index 0000000..fda8636 --- /dev/null +++ b/testnet/runbooks/contract-not-found-error.md @@ -0,0 +1,249 @@ +# Runbook: "Contract Not Found" Style RPC Errors + +## Overview + +Every invocation against a Bridgelet contract starts with the RPC resolving a +contract ID to a live ledger entry. When that resolution fails, the failure +happens **before any contract code runs** — there is no `Error(Contract, #N)`, +no event, and nothing to correlate on-chain. The typical shapes: + +- `stellar contract invoke` aborts with an RPC error such as + `Invalid contract ID` or a contract-entry lookup that returns nothing; +- `simulateTransaction` fails with an infrastructure/host error naming the + contract rather than a contract error; +- `getLedgerEntries` for key `contract:` returns a null entry. + +This runbook covers the three real causes, in the order they should be +checked: + +1. **Wrong or stale contract ID** — the ID was valid once (or never was): it + was copied from old artifacts, or the testnet was reset since. +2. **Wrong network configured** — the ID belongs to another network + entirely. +3. **The contract was never deployed** — notably `account_factory`, which has + no testnet deployment, so any remembered or locally-built ID for it will + always fail this way. + +## Symptom + +- Invocation or simulation fails at the RPC layer with a contract-resolution + error; the CLI never prints a contract error code. +- `get_info` on an address that "used to work" fails the same way after a + redeploy or reset. +- Only one particular call fails this way while others against a different ID + succeed — the ID is the variable, not the network or the CLI. + +## Prerequisites + +- The exact contract ID string being used +- Soroban RPC access: `https://soroban-testnet.stellar.org` +- Canonical deployment records: + - `testnet/registry/contract-status.md` + - `testnet/registry/last-verified.json` + - `deployment-artifacts/contract-ids.txt` + - `testnet/docs/network-config.md` +- A funded testnet identity for simulations + +## Diagnosis + +### Step 1: Validate the ID string itself + +A Soroban contract address is a `C...` strkey. A malformed copy fails even +before the RPC is contacted: + +```bash +# The CLI rejects invalid strkeys during parsing +stellar contract invoke \ + --simulate-only \ + --id "$CONTRACT_ID" \ + --network testnet \ + --source "$READER" \ + -- \ + get_info +``` + +An immediate parse/argument error (not an RPC round-trip failure) means the +string is truncated, contains whitespace, or is a different address type. +Note the related mix-ups: + +| Prefix | Type | Invokable as a contract? | +|---|---|---| +| `C...` | Soroban contract address | yes | +| `G...` | classic account | no — this is where Bridgelet ephemeral accounts hold their *classic* XLM balances, not where their code lives | +| `S...` | secret/seed | never — and it should not be in any log | + +### Step 2: Ask the network whether the contract instance exists + +```bash +# Sanity check: the RPC is alive and its ledger is advancing +curl -fsS https://soroban-testnet.stellar.org \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","id":1,"method":"getLatestLedger"}' +``` + +The authoritative existence check is an RPC `getLedgerEntries` call whose key +is the base64 XDR of `ScLedgerKey::Contract` for the ID. A present entry +includes the contract instance (or code); a null entry means nothing is +recorded at that address on this network. The exact key-encoding procedure is +owned by `testnet/tooling/contract-id-lookup-spec.md`. As a quick local +cross-check, `stellar contract ls` lists the contract IDs your CLI already +knows about — useful for spotting a profile mix-up, but not proof of +on-chain existence. + +- **Entry present**: the contract exists. A "not found"-style failure here is + a client/config issue (wrong RPC URL in one of several configured profiles) + — compare `stellar network show` against `testnet/docs/network-config.md`. +- **Entry absent**: the ID resolves to nothing on this network. Continue to + Steps 3–5. + +### Step 3: Distinguish "not found" from "not initialized" + +An existing-but-never-initialized `EphemeralAccount` instance answers +differently, and the difference matters: + +```bash +stellar contract invoke \ + --simulate-only \ + --id "$CONTRACT_ID" \ + --network testnet \ + --source "$READER" \ + -- \ + get_info +``` + +| Result | Meaning | +|---|---| +| `Error(Contract, #2)` — `NotInitialized` | The instance **exists**; `initialize()` was never called. Not this runbook — complete initialization per `testnet/docs/getting-started.md` | +| RPC/host error before any contract code runs | The ID does not resolve — continue | + +### Step 4: Rule out the wrong network + +An ID is only meaningful on the network it was deployed to. A mainnet or +localnet `C...` string is a valid strkey on testnet — it simply has no ledger +entry there, producing exactly the "not found" symptom. + +Check what the client is actually pointed at: + +```bash +stellar network show testnet +echo "$STELLAR_NETWORK_PASSPHRASE" +``` + +Compare against `testnet/docs/network-config.md`: +passphrase `Test SDF Network ; September 2015`, RPC +`https://soroban-testnet.stellar.org`. If the ID came from a mainnet or +localnet environment, the fix is to use the testnet ID — and if you also +observed `tx_bad_auth` on submissions, see +`wrong-network-passphrase.md`; the two failures are the two halves of the same +config mix-up. + +### Step 5: Check whether the ID is stale — resets and copied artifacts + +Testnet is reset periodically. After a reset, **every account and contract ID +is gone** until `scripts/deploy-testnet.sh` is re-run and the IDs are +re-recorded (`testnet/docs/network-config.md`, `testnet/policies/reset-policy.md`). +A perfectly correct ID from before the reset now resolves to nothing. + +Two project-specific staleness traps: + +- **Artifact conflict.** `deployment-artifacts/contract-ids.txt` and + `deployments/testnet.json` can hold IDs from an older deployment that + conflicts with newer records. `testnet/examples/README.md` states the rule + plainly: *a copied ID is not proof of availability* — verify before use. +- **Age.** `testnet/registry/last-verified.json` is a staleness mechanism + (30-day threshold per `testnet/policies/reset-policy.md`), not proof of + liveness. An ID whose "last verified" timestamp is older than the threshold + must be re-verified, not trusted — see + `testnet/tooling/contract-id-lookup-spec.md` for the lookup procedure. + +### Step 6: Confirm the contract was ever deployed at all + +Some contracts have **no testnet deployment by design of the current state**, +and no amount of re-verification will find them: + +| Contract | Testnet status (see `testnet/registry/contract-status.md`) | +|---|---| +| `EphemeralAccount`, `SweepController` | Deployed per `testnet/docs/network-config.md`; verify live before each use | +| `ReserveContract` | Not deployed; also not wired into `EphemeralAccount` (hardcoded internal reserve) | +| `account_factory` (`AccountFactory`) | **Not deployed** — not built by `scripts/build.sh`, not in CI, not in the deploy script | + +So the issue scenario of "calling a contract that was never deployed, e.g. +`account_factory`" is not hypothetical: any `C...` ID for the factory, whether +remembered from a local run or copied from a sibling project, will always +fail with a not-found-style error on testnet. Do not chase it. Instead: + +- for multiple ephemeral accounts, follow the deploy-and-initialize loop in + `testnet/examples/batch-account-creation-mock.md` (the documented stand-in + for the factory); +- if you genuinely need the factory, the path is to build and deploy it + yourself per `testnet/registry/account-factory-status.md`, then record the + new ID in the registry — not to reuse an ID from elsewhere. + +## Common Causes + +| Cause | Clue | Fix | +|---|---|---| +| Malformed ID string | CLI parse error, no RPC round-trip | Re-copy the full `C...` string; strip whitespace | +| Stale ID after testnet reset | Every lookup empty; registry timestamps pre-date the reset | Redeploy via `scripts/deploy-testnet.sh`; update registry | +| ID copied from conflicting artifacts | Works in one environment, not another | Verify live per `testnet/registry/verify-contract-live.md`; prefer verified registry values | +| ID from another network | Valid strkey, no entry on testnet | Use testnet IDs; see Step 4 and `wrong-network-passphrase.md` | +| Never-deployed contract (`account_factory`, `ReserveContract`) | Fails identically for every ID of that contract | Expected; use documented alternatives (Step 6) | +| Wrong RPC URL in a secondary profile | Entry exists per one client, not another | Diff every configured network profile | + +## Resolution + +1. Establish which of the three causes applies via Steps 1–6. +2. For stale/reset IDs: redeploy, then update + `testnet/registry/contract-status.md`, `last-verified.json`, and any + deployment artifacts in the same change, per the registry's Update + Process. +3. For never-deployed contracts: switch to the documented stand-in flow; do + not file the failure as an outage. +4. Re-run the original invocation with `--simulate-only` first; only submit + for real once simulation returns a result from the contract (success or a + contract error code — either proves the ID resolves). + +## Prevention Checklist + +- [ ] Contract IDs are read from a verified registry lookup at startup, not + hardcoded from docs or old transcripts +- [ ] CI validates IDs against a live `getLedgerEntries` before deployment + gates +- [ ] Post-reset checklist includes redeploy → re-verify → update registry → + update consumers +- [ ] Dashboards alert on registry staleness (> 30 days) before integrators + hit not-found errors +- [ ] Environments pin the network profile once and derive the RPC URL, + passphrase, and IDs from it + +## Escalation + +If a **verified-fresh** ID (checked via `testnet/registry/verify-contract-live.md` +within the same session) still fails to resolve, capture the ID, the +`getLatestLedger` output, the `getLedgerEntries` response, and the RPC URL in +use, and escalate per `testnet/runbooks/on-call-testnet.md` — that pattern +points at an RPC-side or deployment-side problem rather than a config error. + +## Related Documentation + +- `testnet/registry/verify-contract-live.md` — the live verification + procedure +- `testnet/registry/contract-status.md` — deployment status of every + contract +- `testnet/registry/account-factory-status.md` — why `account_factory` is + never found on testnet +- `testnet/tooling/contract-id-lookup-spec.md` — authoritative lookup order + and staleness thresholds +- `testnet/docs/network-config.md` — endpoints, passphrase, recorded IDs +- `testnet/examples/batch-account-creation-mock.md` — the factory stand-in +- `testnet/runbooks/wrong-network-passphrase.md` — the signature-side half of + a wrong-network config +- `testnet/policies/reset-policy.md` — what a testnet reset invalidates + +--- + +## Version History + +| Version | Date | Changes | +|---|---|---| +| 1.0 | 2026-10-15 | Initial contract-not-found runbook | diff --git a/testnet/runbooks/event-not-appearing-in-horizon.md b/testnet/runbooks/event-not-appearing-in-horizon.md new file mode 100644 index 0000000..51aa04f --- /dev/null +++ b/testnet/runbooks/event-not-appearing-in-horizon.md @@ -0,0 +1,243 @@ +# Runbook: Contract Event Not Appearing in Horizon + +## Overview + +A transaction is visibly successful — it shows `SUCCESS` in an explorer, or +`getTransaction` returns a final result — yet the contract event you expected +does not appear when you query Horizon's events endpoint (or Soroban RPC's +`getEvents`). This runbook separates the four reasons this happens, in the +order they should be checked: + +1. **Indexing lag** — Horizon ingests contract events asynchronously after + ledger close; the event is not lost, just not indexed yet. +2. **Wrong topic filter** — the event exists, but your filter does not match + the on-chain topic encoding. +3. **Wrong contract queried** — the event was emitted by a different contract + than the one you are watching. +4. **No event was ever emitted** — the transaction succeeded but its success + path does not emit the event you assumed it would. + +Only after those are excluded is an event genuinely "missing". + +## Symptom + +- `GET https://horizon-testnet.stellar.org/events?contract_id=` returns an + empty `data` array (or no entry matching your expected topic). +- An explorer shows the transaction as successful. +- Your indexer/desired workflow is stuck waiting for a terminal event + (`swept_mul`, `sweep`, `expired`, `reserve`, …) that never arrives. + +## Prerequisites + +- The transaction hash of the "successful" transaction +- The contract ID(s) you expect the event from +- Event topic reference: `testnet/registry/event-topics.md` +- Access to both Soroban RPC and Horizon (they are independent pipelines — + see `testnet/runbooks/rpc-outage.md`) + +## Diagnosis + +### Step 1: Prove the transaction really succeeded, and record its ledger + +```bash +curl -fsS https://soroban-testnet.stellar.org \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","id":1,"method":"getTransaction", + "params":{"hash":""}}' | jq '.result.status, .result.ledger' +``` + +Record the ledger sequence. Everything below is anchored to it. If the status +is not `SUCCESS`, stop — you do not have an event-visibility problem, you have +a failed transaction; route to `testnet/examples/error-handling.md`. + +### Step 2: Check indexing lag before anything else + +Horizon serves contract events from its own ingestion pipeline, which trails +the Soroban RPC by at least one ledger close (roughly five seconds on +testnet, longer under load or during ingestion hiccups). + +```bash +# Horizon's most recently ingested ledger +curl -fsS "https://horizon-testnet.stellar.org/ledgers?order=desc&limit=1" | + jq '.records[0].sequence' +``` + +If Horizon's latest ingested ledger is **behind** the transaction's ledger, +wait and re-query with the same parameters. Do not change your filter yet — +changing the filter while lagging converts a timing problem into a mystery. + +Also confirm you are not querying a cursor that points *past* the event: a +`cursor` taken from a later event (or a `start_ledger` newer than the tx +ledger) skips it permanently in forward order. When in doubt, re-query from +the transaction's own ledger with no cursor. + +### Step 3: Verify the topic filter against the canonical encoding + +Topic filters are exact matches against XDR-encoded `ScVal`s, not free-text +searches. The first topic is always an `ScSymbol` carrying the **event +name** — which is frequently *not* the payload struct name: + +| You are looking for | Correct topic[0] | Common wrong guesses | +|---|---|---| +| `AccountCreated` payload | `created` | `AccountCreated`, `account_created` | +| `PaymentReceived` payload | `payment` | `PaymentReceived`, `multi_pay` (that is the *second* payment) | +| `MultiPaymentReceived` payload | `multi_pay` | `MultiPaymentReceived`, `payment` | +| `SweepExecutedMulti` payload | `swept_mul` | `sweep`, `SweepExecutedMulti` | +| `AccountExpired` payload | `expired` | `AccountExpired`, `expire` | +| `ReserveReclaimed` payload | `reserve` | `ReserveReclaimed`, `reserve_reclaimed` | +| `SweepCompleted` payload (controller) | `sweep` | `SweepCompleted`, `swept_mul` | +| `DestinationAuthorized` payload | `dest_auth` | `DestinationAuthorized` | +| `DestinationUpdated` payload | `dest_upd` | `DestinationUpdated`, `dest_update` | + +Symbols are case-sensitive and short — several were truncated by Soroban's +9-character `symbol_short!` limit (`swept_mul`, `dest_upd`, `dest_auth`). The +authoritative table is `testnet/registry/event-topics.md`; if your filter was +built from a Rust struct name, it will never match. + +Correct shapes: + +```bash +# Horizon — one topic +curl -fsS "https://horizon-testnet.stellar.org/events?contract_id=&type=contract&order=asc&start_ledger=" + +# Filtered, via the RPC the CLI uses +stellar contract events \ + --contract-id "$EPHEMERAL_ACCOUNT_ID" \ + --start-ledger "$START_LEDGER" \ + --filter '{"topics": [["swept_mul"],["reserve"]]}' \ + --network testnet +``` + +A filter list like `[["swept_mul"],["reserve"]]` means "topic[0] is +`swept_mul` **or** `reserve`". Nested arrays of length > 1 (e.g. +`[["payment","multi_pay"]]`) constrain topic[0] and topic[1] *together* — a +frequent cause of empty results when the intent was "either event". + +### Step 4: Confirm you are watching the right contract + +Events are scoped to the contract that emitted them, and Bridgelet splits a +single logical flow across two contracts: + +| Event topic | Emitted by | +|---|---| +| `created`, `payment`, `multi_pay`, `swept_mul`, `expired`, `reserve` | the `EphemeralAccount` instance | +| `sweep`, `dest_auth`, `dest_upd` | the `SweepController` deployment | + +The classic miss: watching only the ephemeral account for a `claim()` flow and +waiting for a `sweep` event that was emitted on the **controller** +(`gas-free-claim-walkthrough.md` documents this split). Query both contracts +over the same ledger range and correlate. + +Also verify the contract ID itself — a stale ID after a testnet reset answers +empty for everything. Route to `contract-not-found-error.md` if +`getLedgerEntries` shows no contract instance. + +### Step 5: Verify that this success path emits the event at all + +A `SUCCESS` status proves the transaction executed; it does not prove any +particular event was emitted. Per `testnet/registry/event-topics.md` and the +contract sources: + +| Call | Events actually emitted | +|---|---| +| First `record_payment` on an account | `payment` (not `multi_pay`) | +| Any later `record_payment` | `multi_pay` (not `payment`) | +| `sweep` / `sweep_claim` on the account | `swept_mul`, then `reserve` | +| `execute_sweep` / `claim` on the controller | `sweep` (on the controller only) | +| `expire` / `recover` | `expired`, then `reserve` | +| `update_authorized_destination` | `dest_upd` | +| `initialize` of the controller with a locked destination | `dest_auth` | +| `AccountFactory` operations | **none** — it emits no events; account creation is tracked via each child's `created` event | + +If the table shows no event for your call, nothing is broken — you were +waiting for an event the code never emits. + +### Step 6: Cross-check with RPC `getEvents` and with state + +Horizon and the Soroban RPC are independent consumers. If RPC `getEvents` +returns the event but Horizon does not, the lag or gap is on Horizon's side — +keep polling Horizon; do not re-run the transaction. + +Then confirm the *state change* independently of any event plumbing: + +```bash +stellar contract invoke \ + --simulate-only \ + --id "$EPHEMERAL_ACCOUNT_ID" \ + --network "$NETWORK" \ + --source "$READER" \ + -- \ + get_info +``` + +The internal reserve event can also be read back from storage via +`get_last_reserve_event` and `get_reserve_reclaim_event_count` — useful for +`reserve` events specifically, because those getters are the contract's own +record of what it emitted. + +## Common Causes + +| Cause | Signature | Fix | +|---|---|---| +| Indexing lag | Empty now, populated minutes later with identical query | Wait; poll with same cursor/params | +| Filter uses struct name | Always empty, no matter how long you wait | Use the topic symbols in Step 3 | +| Nested-array filter semantics | Always empty | Use one inner array per alternative | +| Watching the wrong contract | Other contract's query shows it | Query both contracts (Step 4) | +| `start_ledger`/`cursor` past the event | Empty in forward order | Re-query from the tx ledger without cursor | +| RPC `getEvents` retention exceeded | Error or empty for old ranges | Use Horizon for history; RPC retains only a recent window | +| Success path emits nothing | Event never appears anywhere | Step 5 table — expected behavior | +| Testnet reset | Everything before the reset vanishes | Redeploy and re-record IDs (`testnet/policies/reset-policy.md`) | + +## Resolution + +- Lag: no action; the event arrives. Alerting should tolerate minutes of + Horizon lag before paging. +- Wrong filter/contract/range: correct the query per the steps above; no + on-chain action needed. +- Genuinely absent event: do **not** replay the transaction hoping for a + second emission — many state machines forbid the second call + (`AlreadySwept` (7), `DuplicateAsset` (13), …). Verify state via + `get_info` and treat the workflow as complete if state is terminal. +- Suspected true loss (state changed but its event never appears anywhere + after lag, retention, and filter issues are excluded): capture the tx hash, + the queries tried, and both endpoints' results, then escalate. + +## Escalation + +Capture: tx hash, both contract IDs, exact filter JSON, start ledger/cursor, +Horizon latest-ingested ledger, RPC `getEvents` output, and the corresponding +`get_info` state read. Escalate per `testnet/runbooks/on-call-testnet.md`. + +## Prevention Checklist + +- [ ] Topic filters are generated from `testnet/registry/event-topics.md`, + never from Rust type names +- [ ] Indexers subscribe to **both** the account and controller contracts for + every flow +- [ ] Resume logic stores the Horizon `paging_token`, not a timestamp +- [ ] Alerts distinguish "event missing" from "state not terminal" — the + state read is the source of truth +- [ ] Dashboards tolerate indexing lag of several ledger closes + +## Related Documentation + +- `testnet/registry/event-topics.md` — canonical topics and payload structs +- `testnet/examples/monitoring-a-single-account-lifecycle.md` — ordering the + full event sequence for one account +- `testnet/monitoring/event-watchlist.md` — which events to watch for + alerting +- `testnet/examples/reading-contract-storage.md` — independent state reads + when you distrust the event pipeline +- `testnet/runbooks/rpc-outage.md` — endpoint outages masquerading as missing + events +- `testnet/runbooks/contract-not-found-error.md` — stale IDs that answer + empty for every query +- `testnet/policies/reset-policy.md` — why pre-reset history disappears + +--- + +## Version History + +| Version | Date | Changes | +|---|---|---| +| 1.0 | 2026-10-15 | Initial event-not-appearing-in-Horizon runbook | diff --git a/testnet/runbooks/insufficient-reserve-errors.md b/testnet/runbooks/insufficient-reserve-errors.md new file mode 100644 index 0000000..8f51184 --- /dev/null +++ b/testnet/runbooks/insufficient-reserve-errors.md @@ -0,0 +1,242 @@ +# Runbook: Insufficient Reserve Errors + +## Overview + +Two completely different things in this project are called a "reserve", and +conflating them is the single most common misdiagnosis when a testnet +transaction fails near an account's balance boundary: + +1. **Stellar's minimum base reserve** — a network rule enforced by the + protocol. Every funded classic account must hold at least + `base_reserve × (number of ledger entries)`; the current base reserve is + **0.5 XLM**. Violations surface as classic operation result codes before any + Bridgelet contract code runs. +2. **Bridgelet's internal reserve tracking** — bookkeeping inside + `EphemeralAccount`. At `initialize()` the contract records a fixed internal + reserve of `BASE_RESERVE_STROOPS = 1_000_000_000` (1 XLM, defined in + `contracts/ephemeral_account/src/lib.rs`) and later "reclaims" it to the + sweep/expire destination as a tracked transfer with a `reserve` event. This + value is **not enforced by the network**, and it can never cause a network + reserve error. + +This runbook teaches you to recognize which one you are looking at, and how to +resolve each. + +## Symptom + +A transaction fails with a balance-related error, or an operator notices that +an account's on-chain balance does not match what the project's reserve +trackers report. The two families of symptoms: + +**Network-level (real, protocol-enforced):** + +- Classic operation result codes: `op_low_reserve`, `op_underfunded`, + `op_line_full`, or transaction-level `tx_insufficient_balance`. +- Soroban flows failing at submission with `INSUFFICIENT_FEE`-class outcomes + when the source cannot cover resource fees (rent) for a contract instance. + +**Project-internal (bookkeeping only):** + +- `reclaim_reserve()` returning `0`, or a `reserve` event with + `amount: 0` / `fully_reclaimed: true`. +- `Error(Contract, #4)` (`InvalidAmount`) from the reclaim path — the only + error the internal tracker can produce, and only if tracked values are + negative. + +## The Two Concepts Side by Side + +| | Network base reserve | `EphemeralAccount` internal reserve | +|---|---|---| +| Defined by | Stellar protocol | `BASE_RESERVE_STROOPS` in `contracts/ephemeral_account/src/lib.rs` | +| Value | 0.5 XLM per ledger entry (network-wide) | 1 XLM (1,000,000,000 stroops), hardcoded | +| Enforced by | The network, at submission | Nobody — it is read-modify-write bookkeeping in contract storage | +| Failure surface | `op_low_reserve`, `op_underfunded`, `tx_insufficient_balance`, fee failures | At most `InvalidAmount` (contract error code 4); normally just `0` reclaimed | +| Fixed after deploy? | Moves only when the network changes the base reserve | Hardcoded at compile time; not read from `ReserveContract` (which is not deployed — see `testnet/registry/reserve-contract-status.md`) | +| Inspecting it | Horizon `accounts/{id}` balances and subentries | `get_reserve_remaining`, `get_reserve_available`, `is_reserve_reclaimed`, `get_last_reserve_event`, `get_reserve_reclaim_event_count` | +| What to do | Fund the account | Nothing is broken; it is expected lifecycle output | + +> `testnet/registry/contract-status.md` documents the wiring gap explicitly: +> `ReserveContract` is not deployed on testnet, and `EphemeralAccount` uses +> its own hardcoded constant instead of reading from it. Do not expect any +> `ReserveContract` error in a testnet failure — there is no such call path. + +## Prerequisites + +- A funded testnet identity for queries (`stellar keys generate --global + testnet-investigator`) +- Contract IDs from `testnet/registry/contract-status.md` +- Access to Horizon (classic balances) and Soroban RPC (contract calls) + +## Diagnosis + +### Step 1: Classify the error layer first + +Decode the failure before reading any balance: + +```bash +stellar transaction get --network testnet --output json +``` + +| What the result shows | Layer | Go to | +|---|---|---| +| Operation code `op_low_reserve` / `op_underfunded` / `op_line_full`, or `tx_insufficient_balance` | Network | Step 2 | +| `Error(Contract, #4)` from a `reclaim_reserve`/`sweep`/`expire` call | Internal tracker | Step 4 | +| `Error(Contract, #N)` with any other N | Not a reserve problem at all | `testnet/examples/error-handling.md` | +| Soroban resource-fee rejection at submission | Network (rent/fee) | Step 3 | + +The network never mentions "reserve" in a contract error, and the internal +tracker never produces a classic operation code. The layer is unambiguous once +you look at the result XDR. + +### Step 2: Network-level — inspect the account's real balances + +```bash +curl -fsS "https://horizon-testnet.stellar.org/accounts/" | + jq '{balances, subentry_count, signers}' +``` + +Each trustline, offer, signer, or data entry consumes one base reserve. An +account at exactly the minimum cannot accept an operation that adds another +entry — that is `op_low_reserve`. The fix is ordinary: send the account more +XLM (Friendbot for brand-new addresses, see `testnet/faucet/README.md`), or +remove the entry the account cannot afford. + +In Bridgelet flows the typical network-reserve situations are: + +| Flow | Reserve pressure | +|---|---| +| Funding a brand-new address for the first time | The payment must leave the destination able to hold its own ledger entry (≥ 0.5 XLM plus any entries created) | +| Recording multi-asset payments via Horizon transfers | Each new trustline/asset entry on the counterparty adds reserve the payer must cover | +| Deploying/initializing contract instances | Source pays rent as a resource fee; underfunded source → fee rejection, not `op_low_reserve` | + +### Step 3: Network-level — fee/rent failures on Soroban calls + +A Soroban transaction's resource fee includes ledger rent for the contract +instance and storage it touches. An underfunded source fails at submission +with a fee-class rejection even though every classic balance looks "fine" for +a pure payment. Fund the submitting identity and resubmit; nothing in the +contract state is implicated. + +### Step 4: Internal tracker — read the tracked values + +```bash +stellar contract invoke \ + --simulate-only \ + --id "$EPHEMERAL_ACCOUNT_ID" \ + --network "$NETWORK" \ + --source "$READER" \ + -- \ + get_reserve_remaining + +stellar contract invoke \ + --simulate-only \ + --id "$EPHEMERAL_ACCOUNT_ID" \ + --network "$NETWORK" \ + --source "$READER" \ + -- \ + get_reserve_available + +stellar contract invoke \ + --simulate-only \ + --id "$EPHEMERAL_ACCOUNT_ID" \ + --network "$NETWORK" \ + --source "$READER" \ + -- \ + is_reserve_reclaimed +``` + +Expected lifecycle (all of it normal): + +1. `initialize()` calls `init_reserve_tracking` with 1 XLM. +2. Each successful `sweep` / `sweep_claim` / `expire` / `recover` path calls + `reclaim_reserve_to`, which transfers `min(reserve_available, + reserve_remaining)` to the destination and emits a `reserve` event + (`ReserveReclaimed { destination, amount, sweep_id, fully_reclaimed, + remaining_reserve }` — topic `reserve`, see + `testnet/registry/event-topics.md`). +3. Once fully reclaimed, `is_reserve_reclaimed` is `true`, later reclaims + transfer 0 and emit `fully_reclaimed: true, amount: 0`, and + `reclaim_reserve()` is a safe, repeatable no-op. + +A `0` reclaim or a fully-reclaimed flag is therefore **not an error**. It +means the lifecycle already completed. `reclaim_reserve()` on an account that +is still `Active` or `PaymentReceived` fails with `InvalidStatus` (12) — also +by design; there is nothing to reclaim until a terminal state. + +### Step 5: Cross-check the event trail when the numbers look wrong + +```bash +stellar contract events \ + --contract-id "$EPHEMERAL_ACCOUNT_ID" \ + --start-ledger "$START_LEDGER" \ + --filter '{"topics": [["reserve"]]}' \ + --network testnet +``` + +The last `reserve` event's `remaining_reserve` must equal the current +`get_reserve_remaining` reading. A mismatch means you are reading a different +account ID, or a stale deployment after a testnet reset — route to +`contract-not-found-error.md`, not this runbook. + +## Common Causes + +| Cause | Which reserve | Fix | +|---|---|---| +| Payment would push destination below 0.5 XLM + entries | Network | Send more XLM or reduce entries | +| Source lacks balance for payment + its own entries | Network (`op_underfunded`) | Fund the source | +| Soroban rent/resource fee exceeds source balance | Network | Fund the submitting identity | +| Expecting `reserve` reclaim to equal the *network* reserve | Internal | It equals the hardcoded 1 XLM tracking constant; they are unrelated numbers | +| Seeing `amount: 0` reclaim events and filing a bug | Internal | Normal once fully reclaimed; check `is_reserve_reclaimed` | +| `ReserveContract`-related expectations | Neither | `ReserveContract` is not deployed and not wired; see `testnet/registry/reserve-contract-status.md` | + +## Resolution + +- **Network reserve errors**: fund the implicated account so it clears + `base_reserve × entries`, remove an unaffordable entry, or split the + operation. Then re-check with a Horizon account read before resubmitting. +- **Internal tracker observations**: no on-chain action is required. Record + the `reserve` events if you are building an audit trail; the tracked values + are already consistent by construction. +- **Never** attempt to "fix" a network `op_low_reserve` by calling + `reclaim_reserve()` — different layer, no effect on network balances beyond + what a normal transfer would do, and the account must be in a terminal + state for that call to succeed at all. + +## Prevention Checklist + +- [ ] Dashboards label the two reserves distinctly (e.g. "XLM balance vs. + network min" and "tracked internal reserve") +- [ ] Alerts on `op_low_reserve`/`op_underfunded` route here; alerts on + `reserve` events with `amount: 0` do not page anyone +- [ ] Multi-asset tests budget 0.5 XLM per expected ledger entry on both + sides before exercising `record_payment` +- [ ] Issue templates ask for the decoded result XDR, so the layer is stated + up front + +## Escalation + +Escalate per `testnet/runbooks/on-call-testnet.md` only if the network +rejection persists for an account whose Horizon balance is demonstrably above +the minimum reserve for its entry count, with the result XDR attached. + +## Related Documentation + +- `testnet/registry/event-topics.md` — the `reserve` event and + `ReserveReclaimed` payload +- `testnet/registry/reserve-contract-status.md` — why `ReserveContract` is + not a factor on testnet +- `testnet/registry/contract-status.md` — the hardcoded-reserve wiring note +- `testnet/examples/multi-asset-sweep.md` — multi-asset flows that stress + counterparty entries +- `testnet/examples/error-handling.md` — mapping every contract error code +- `testnet/faucet/README.md` — funding testnet accounts +- `testnet/runbooks/expired-account-recovery-check.md` — verifying where + funds actually are after the terminal state + +--- + +## Version History + +| Version | Date | Changes | +|---|---|---| +| 1.0 | 2026-10-15 | Initial insufficient-reserve runbook distinguishing network vs. internal reserve | diff --git a/testnet/runbooks/wrong-network-passphrase.md b/testnet/runbooks/wrong-network-passphrase.md new file mode 100644 index 0000000..fbd3ddb --- /dev/null +++ b/testnet/runbooks/wrong-network-passphrase.md @@ -0,0 +1,224 @@ +# Runbook: Transaction Signed With the Wrong Network Passphrase + +## Overview + +Every Stellar transaction signature covers the network it was signed for. The +network passphrase is mixed into the transaction hash as +`network_id = SHA256(passphrase)`, and each signature is computed over +`SHA256(network_id || envelope_xdr)`. A signature produced against one +passphrase is therefore cryptographically invalid on any other network — even +when the same key, account, and operation are used. + +This is one of the most common copy-paste mistakes when switching between +testnet and mainnet configs: a `.env` file, an SDK `networkPassphrase` +argument, or a `stellar network add` entry carries the wrong value, and every +submission from that environment fails. + +Testnet passphrase (must match exactly, including the spaces around `;`): + +```text +Test SDF Network ; September 2015 +``` + +Mainnet: + +```text +Public Global Stellar Network ; September 2015 +``` + +Source of truth: `testnet/docs/network-config.md`. + +## Symptom + +- Transactions are rejected at submission with **no contract error code**. +- Horizon returns HTTP 400 with result code `tx_bad_auth`. +- Soroban RPC `sendTransaction` returns status `ERROR` with an `errorResultXdr` + that decodes to `tx_bad_auth`; a later `getTransaction` shows `FAILED` with + the same code. +- Simulation succeeds, submission fails — consistently, for every transaction + from the affected environment. + +## Why It Is Not a Contract-Level Authorization Failure + +This is the distinction this runbook exists to make. A wrong-passphrase +failure and a contract auth failure look superficially similar ("my +authorization was rejected") but live at completely different layers: + +| | Wrong network passphrase | Contract-level auth failure | +|---|---|---| +| Failing layer | Network envelope signature verification | Contract code | +| Result code | `tx_bad_auth` (transaction-level) | `Error(Contract, #N)` or a host trap | +| Contract code executed? | **No** — rejected before any invocation runs | Yes | +| Simulation outcome | **Succeeds** (simulation does not verify signatures) | May fail with the contract error | +| Typical Bridgelet example | any call to any contract | `UnauthorizedDestination` (13), `AuthorizationFailed` (3), or an `ed25519_verify` trap in `execute_sweep` | + +Key consequences: + +- You will **never** see `Error(Contract, #N)` for this failure. If your + failure carries a contract error code, the passphrase was fine — go to the + runbook for that error instead (`failed-sweep-signature.md`, + `nonce-desync.md`, etc.). +- `UnauthorizedDestination` (controller code 13) and the `ed25519_verify` trap + in the sweep digest are **not** caused by a wrong network passphrase. The + sweep digest (`SHA256(destination ‖ nonce ‖ controller_id)` — see + `docs/SIGNATURE_FORMAT.md`) does not contain the passphrase at all; it is + the *outer transaction* signature that breaks. +- A contract ID from another network does not produce this error either — it + produces a "contract not found" style failure. See + `contract-not-found-error.md`. + +## Prerequisites + +- The exact transaction (or at least its environment config) that failed +- Access to the Soroban RPC and Horizon endpoints you submitted against +- A funded testnet identity for test submissions +- Network values from `testnet/docs/network-config.md` + +## Diagnosis + +### Step 1: Confirm the failure is transaction-level + +```bash +stellar transaction get --network testnet --output json +# or via RPC: +curl -fsS https://soroban-testnet.stellar.org \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","id":1,"method":"getTransaction", + "params":{"hash":""}}' +``` + +Decode `errorResultXdr`. If the transaction-level code is `tx_bad_auth` (or +`tx_insufficient_fee` variants never apply here), continue. If you see an +operation-level Soroban diagnostic event carrying `Error(Contract, #N)`, stop: +the passphrase is not your problem. + +### Step 2: Reproduce with simulation + +```bash +stellar contract invoke \ + --simulate-only \ + --id "$SWEEP_CONTROLLER_ID" \ + --network testnet \ + --source testnet-investigator \ + -- \ + get_nonce +``` + +If simulation succeeds while real submission of the equivalent signed +transaction fails with `tx_bad_auth`, that combination is the fingerprint of a +passphrase mismatch. Simulation skips signature verification, which is exactly +why it cannot catch this. + +### Step 3: Inspect the configured passphrase + +```bash +# What does the CLI think "testnet" is? +stellar network show testnet +``` + +For SDK/config-driven flows, print the value actually in use: + +```bash +echo "$STELLAR_NETWORK_PASSPHRASE" +cat .env 2>/dev/null | grep -i passphrase +``` + +Compare character-by-character with +`testnet/docs/network-config.md`. Common defects: + +| Defect | Example | +|---|---| +| Mainnet passphrase in a testnet config | `Public Global Stellar Network ; September 2015` | +| Missing space before `;` | `Test SDF Network; September 2015` | +| Missing space after `;` | `Test SDF Network ;September 2015` | +| Trailing whitespace/newline in a config value | `Test SDF Network ; September 2015 ` | +| Typo'd network name | `Test SDF Network ; September 2016` | + +The passphrase must match **exactly**. There is no prefix or fuzzy matching. + +### Step 4: Check whether the signing and sending environments differ + +A frequent shape of this bug is a CI matrix or a shared signing service where +one leg signs with mainnet config while another submits to the testnet RPC: + +- transaction built and signed with `--network mainnet` (or an SDK + `TransactionBuilder` with `Networks.PASSPHRASE`), then broadcast to + `https://soroban-testnet.stellar.org`; +- signer tool configured once with mainnet values while relayers run testnet; +- a locally re-signed envelope attached to a submission path that assumes the + original signature applies to the target network. + +Signatures are **not portable between networks**. There is no re-broadcast +fix; the transaction must be rebuilt and re-signed with the target network's +passphrase. + +### Step 5: Rule out the adjacent failure — source account missing on target +network + +If the same keypair was never funded on testnet, the network rejects with +`tx_no_account` rather than `tx_bad_auth`, because the source account does not +exist. That is a different fix (fund via Friendbot, see +`testnet/faucet/README.md`), not a passphrase problem. + +```bash +stellar keys fund --network testnet +``` + +## Common Causes + +| Cause | Where it hides | +|---|---| +| `.env` copied from a mainnet setup | `STELLAR_NETWORK_PASSPHRASE` | +| SDK hardcodes `Networks.PASSPHRASE` (mainnet) in a testnet build | transaction builder code | +| `stellar network add` entry with a subtly wrong passphrase | `~/.config/stellar/` CLI config | +| Signer runs in a different environment than the relayer | `tools/sweep-signer` config vs. relayer config | +| Whitespace/newline in a config file value | dotenv files, YAML/JSON configs | + +## Resolution + +1. Correct the passphrase to `Test SDF Network ; September 2015` everywhere it + is configured (CLI network entry, `.env`, SDK constant, signer config). +2. Rebuild and re-sign the transaction. Do not attempt to reuse the old + signature. +3. Submit and confirm: `getTransaction` returns `SUCCESS`, and the hash the + CLI printed matches what the network recorded. +4. Re-run the simulation-vs-submission check from Step 2 as a regression + check for the whole environment, not just the one transaction. + +## Prevention Checklist + +- [ ] Single source of truth for the passphrase — + `testnet/docs/network-config.md` — referenced, not copied, by each + environment +- [ ] CI asserts the configured passphrase equals the expected literal before + running +- [ ] Signer and relayer configs are generated from the same file +- [ ] A smoke test that simulates **and** submits one no-op read before any + change window (simulation alone proves nothing about signatures) +- [ ] Never log or cache signed envelopes across networks + +## Escalation + +If the passphrase is confirmed correct, the transaction was signed after any +config fix, and `tx_bad_auth` still appears for a funded account, capture the +`errorResultXdr`, the passphrase string (it is public), and the transaction +envelope, and escalate per `testnet/runbooks/on-call-testnet.md`. + +## Related Documentation + +- `testnet/docs/network-config.md` — canonical passphrase and endpoints +- `testnet/runbooks/contract-not-found-error.md` — the other direction of the + same config mix-up (IDs from the wrong network) +- `testnet/runbooks/failed-sweep-signature.md` — sweep-digest signature + failures (a different layer; produce traps, not `tx_bad_auth`) +- `testnet/faucet/README.md` — funding a testnet account (`tx_no_account`) +- `docs/SIGNATURE_FORMAT.md` — what the sweep digest does and does not + contain + +--- + +## Version History + +| Version | Date | Changes | +|---|---|---| +| 1.0 | 2026-10-15 | Initial wrong-network-passphrase runbook |