This guide describes the second implementation phase. It supersedes the local JSON-journal and static-policy limitations in the first remediation report. External broker qualification and native-platform evidence remain release gates.
The bundled backend is SQLite with WAL, FULL synchronous commits, indexed submissions, unique pending-account and intent-deduplication constraints, migration versioning, and bounded database workers. Use a local disk and private state directories. SQLite WAL is not a cross-host coordination service; do not place it on NFS/SMB. See SQLite's WAL requirements.
Clients sharing a broker credential must use the same durable state location and quota configuration. Local worker processes share quota and stream ownership through database transactions. Leases renew every five seconds and expire after thirty seconds; monotonically increasing epochs reject stale renewals. Lease loss cancels the client operation. A broker request already transmitted can still complete and remains subject to reconciliation.
For multiple hosts, the implemented RemoteBackend adapter and rustyschwab-authority service share execution storage, quotas, leases and risk reservations. The authority also supplies a broker gateway with verified operator actions and dispatch-time fencing. See shared authority deployment. The bundled authority is a single writer on local SQLite; it does not provide active-active storage or automatic failover. Schwab does not enforce the SDK's lease tokens or retract already-transmitted requests.
- Stop all old-version writers before migration. Do not run JSON-only and SQLite clients concurrently.
- Back up token/key files and the old journal.
OrderJournal::new("tokens.orders.json")openstokens.orders.store/state.sqlite3. Existing protected JSON is imported once in a transaction, preserving pending intent and consumed approval keys. The source file is retained.- Existing
state.sqlite3paths open directly. The state directory must be private (Unix 0700; protected inheritable owner/SYSTEM ACL on Windows). - Inspect status and readiness before enabling submissions.
SQLite writes run on a bounded blocking-worker pool. A submission runs on its own task once it is journaled, so a caller that cancels, disconnects or times out abandons only the answer: the intent still reaches its outcome. Each intent is armed in the journal immediately before its request leaves the host. A process that dies between journaling and arming leaves an unarmed intent, which provably reached no broker; reconciliation resolves it as rejected (audited never_dispatched, its tag and reservation freed) once it is ten minutes old, longer than any submission's own deadline. Arming only succeeds on an unresolved intent, so a dispatcher that wakes late cannot send what recovery resolved. An armed intent is never resolved without broker evidence.
Use rustyschwab-admin --state <journal-or-database> backup <new-private-directory>/state.sqlite3 for an online backup. Restore to a new location with restore <new-journal-path>, inspect it, and change the deployment configuration during a maintenance window. A restore marks every account baseline unreconciled (audited state_restored): the backup predates whatever reached the broker since, so trading waits for reconciliation, which also learns the order tags the broker now holds. Do not overwrite a live database or copy only a WAL database's main file. Backups contain account/order information and require the same access controls as the live state.
export --after <sequence> --limit 1000 emits ordered NDJSON audit pages. Store them in the organization's immutable audit sink and persist its acknowledged sequence. prune --before-ms <timestamp> --exported-through <acknowledged-sequence> removes eligible resolved submissions, preserving pending submissions, live order links, held risk reservations, consumed approval and used-order-tag tombstones, and audit history. Audit-history retention itself remains the responsibility of the external archival system; the local audit table is intentionally append-only. prune-history --before-ms <timestamp> removes terminal broker orders, their observed activity and released risk reservations last seen before the timestamp, keeping working orders, held reservations and every spent tag. Run both on a schedule: the risk check and reconciliation read working orders through indexes, but history still costs disk. Both prunes are audited.
Each audit row carries prev_hash (the previous row's hash; 64 zeros for the first row) and its own hash, a SHA-256 over a domain tag, prev_hash, sequence, at, kind, subject and payload, each prefixed by its length so no two rows share an encoding. The row is chained in the same write transaction that inserts it, so several processes on one database file share a single chain. Schema version 5 chained the rows that already existed, in sequence order, when it migrated the file; sequences are never reused, so a row removed from the end shows as a gap at the next event.
rustyschwab-admin --state <journal-or-database> verify-audit walks the chain, prints rows, the sequences range and the head hash, and exits non-zero naming the first sequence whose link or content does not match. A row that was edited, removed or reordered breaks the chain at or just after the change. Run it on a schedule and before restoring or archiving.
The chain shows that the history is internally consistent; it cannot show that the newest rows are still there, and anyone who can rewrite the whole table can rewrite the whole chain. Close that gap with a checkpoint: record head (with the last sequence) somewhere the database's writers cannot reach, such as the audit sink, a ticket or a signed log, and compare it with the chain's hash at that sequence later. Exported and delivered events carry prev_hash and hash, so a sink can also check that each batch continues from the last event it stored; delivery batches are still hashed as before.
Nothing in this release deletes audit rows (pruning leaves them, and the paper venue and approval revocations read them as state). If an operator ever removes the oldest rows after exporting them, the last removed row's sequence and hash must go into the single row of audit_anchor(id, sequence, hash) in the same transaction; verification then starts after the anchor, and the first remaining row must link to it. Removing rows without that anchor fails verification at the first remaining row.
Every SDK submission receives a generated unique tag if the caller does not supply one; the MCP server derives it from the intent digest instead, so a resubmitted identical intent is refused as a duplicate. A tag belongs to its account for as long as the broker may hold the order: an accepted or unresolved submission consumes it permanently, including after submission retention, while a definite rejection (a broker 4xx, a risk refusal, or a failure before transmission) releases it so the same request can be retried. A tag the broker is seen to hold on any order is spent too, so an intent that reached the broker after the last backup cannot be resubmitted under its tag from a restored state. Pending intents are resolved automatically only when exactly one returned broker order has both the tag and matching request fields. Absence, duplicate candidates and stripped tags remain unresolved; the one timeout-based resolution is the never-armed intent above, which provably was not sent.
SchwabClient::reconcile_account obtains positions/balances and a bounded recent order baseline. It persists order trees and deduplicates activity observations by order ID and content digest. Those digests are observation identifiers, not invented broker execution IDs. The broker order snapshot remains authoritative for filled and remaining quantity. Terminal broker states release associated local risk reservations. A full 3,000-order result triggers bounded time-window subdivision; a still-full minimum window or exhausted page/deadline budget fails closed.
The default history starts at 59 days. The recovery worker scans it in full on its first pass and every ten minutes; in between, each pass lists only orders entered since the previous complete pass (less a five-minute overlap). Known nonterminal orders the list did not return receive direct ID lookups regardless of age, and older ambiguous intents extend the scan.
A pass costs about two requests per account (the order list and the account), plus one per working order entered before the window. SCHWAB_RECOVERY_INTERVAL_MS (default 2000) sets the cadence, and account activity on the stream pulls the next pass forward, though never to less than half an interval after the last, so a busy session can double the pass rate. The cadence is bounded by the rate budget: at the default 2 req/s, one account at 2 s uses half of it, and startup warns when reconciliation alone would use more than half. It must also stay below the risk policy's max_snapshot_age_ms, since submissions against an older baseline are refused; set SCHWAB_BASELINE_MAX_AGE_MS (used by /readyz and get_server_status) to the same value. Startup refuses an interval at or above SCHWAB_BASELINE_MAX_AGE_MS or any account's max_snapshot_age_ms, warns above half of either, and warns when SCHWAB_BASELINE_MAX_AGE_MS exceeds an account's max_snapshot_age_ms (readiness would report ready while that account's orders are refused). For several accounts, raise both together rather than letting reconciliation starve reads and orders.
schwab_reconciliation_untagged_matches_total counts unresolved submissions for which the broker lists a matching order that lacks their tag. Reconciliation never resolves those on a guess, since an identical manual order would match too. Compare the order with the intent and record the decision with rustyschwab-admin and evidence. ReconciliationOptions configures lookback, page/target budgets and the overall deadline. Each history request spans at most 59 days. Missing targeted orders and incomplete scans invalidate readiness; absent tags can still require operator evidence. Older external orders never observed by the utility require an explicitly configured initial history horizon or authoritative external inventory. REST positions and orders are separate reads, not an atomic exchange snapshot. Applications requiring stronger consistency must use a custom reconciliation/risk service. Never infer exactly-once execution or a complete lifetime execution ledger from these observations.
Set SCHWAB_RECOVERY_ACCOUNTS to comma-separated account hashes to run background recovery. Stream recovery runs independently of delivery: once per stream generation it seeds a REST quote baseline for each Level One equity/option key and, when account activity is subscribed, reconciles the linked accounts under their own durable generation so the activity log can vouch for the gap. It does not gate submissions: the risk check reads the account baselines the recovery worker keeps, not the stream, and each snapshot's fresh and staleReason say whether a quote can be relied on. A quote baseline needs bid/ask and a broker timestamp not ahead of this host; an old timestamp is a quiet key, reported through the entry's age and lag. A key Schwab returns no quote for is reported as such rather than waited on. Other services retain their raw stream semantics; they do not acquire an invented REST recovery contract. Inspect restBaseline, source timestamps, field freshness and activity reconciliation state.
The stream is the Schwab login's: every principal's status reports the same one, a policy reload keeps it, and removing the principal that controls it stops it. Stream control (stream_connect, stream_disconnect, stream_subscribe, stream_unsubscribe, stream_set_service_fields) is audited as stream_control in that principal's journal.
For an unknown submission:
- Disable further submissions for the affected account; leave cancellation available.
- Run
rustyschwab-admin --state <state> reconcile <account-hash>with the broker credentials configured. This is read-only broker access. - If still unresolved, query the exact broker order/execution history and inspect the journal tag, payload and timestamps.
- Use
resolve <attempt> --evidence <specific broker evidence>only after a trusted operator has established the outcome. No MCP tool can resolve its own unknown outcome. - Verify account readiness and outstanding risk reservations before resuming.
For issuer-managed short-lived credentials, set SCHWAB_MCP_IDENTITY_FILE to:
{"issuer":"https://identity.example","audience":"https://rustyschwab-mcp.example/mcp","jwks_uri":"https://identity.example/keys","max_lifetime_seconds":900}The verifier accepts RS256 only and validates signature, key ID, issuer, audience, expiration, issued-at, optional not-before and maximum lifetime. JWKS is fetched over HTTPS with a bounded response and timeout: refreshed each minute, and at once (at most every ten seconds) when a token names an unknown key, so a rotation is picked up. A failed refresh keeps the last good keys for ten minutes, then fails closed. Fetches run one at a time, outside the key cache's lock. Optional fields bind tokens further: token_type (for example "at+jwt", RFC 9068, which refuses ID tokens), authorized_parties (the azp or client_id values accepted, so only named clients act for a user) and required_scopes (also advertised as scopes_supported in the protected-resource metadata). Map the token subject to identity_subject in each principal; issuer-only principals may omit token_sha256. Configure your authorization server to issue access tokens specifically for the MCP resource, following the MCP authorization specification. The application exposes protected-resource metadata; it is not an authorization server.
Static bearer credentials remain available but require token_expires_at (Unix seconds) in network policy. Policy files reload every two seconds. Removal, malformed replacement or revocation invalidates affected access conservatively; a reload retires every credential checked under the previous policy, and a client presents its token again. An open WebSocket connection is refused from its next request once its credential expires or the policy reloads; in-flight broker writes may already have executed and must be reconciled. Changing issuer configuration requires restart.
For approval, use the principal's audit database, not the SDK order database. Its legacy path is the configured token path first given extension orders.json, then extension <SHA256(principal-name)>.audit.json (for example, tokens.orders.<hash>.audit.json); that path migrates to its own .store/state.sqlite3. This preserves existing principal approval history across upgrades:
rustyschwab-admin --state <principal-audit> pending
rustyschwab-admin --state <principal-audit> show <digest>
rustyschwab-admin --state <principal-audit> approve desk <digest> --operator supervisor --expires-at <unix-seconds>
rustyschwab-admin --state <principal-audit> revoke <digest> --operator supervisor
rustyschwab-admin --state <principal-audit> digest place_order arguments.json
Approve what you can read. Every intent that needs approval is recorded when it is asked for (place_order, replace_order or check_order, on MCP or the authority gateway), with its arguments. pending lists those awaiting approval as orders (PLACE on account …1234: BUY 10 AAPL LIMIT @ 150.25 DAY NORMAL (tag mcp-…)), with the model's rationale shown apart as its words; show prints one. approve prints the order its digest binds and asks for confirmation (or --yes); it refuses an intent asked for by a different principal, and one the journal has never seen needs --yes after checking it with digest. A model that shows the user one order while asking for another's digest therefore cannot get the wrong order approved unseen. The authority's operator API offers the same as {"action":"pending","principal":"desk"}. A lapsed, unused approval can be granted again; a live, spent or revoked one cannot.
The kill switch halts submissions and then cancels every working order: rustyschwab-admin --state <sdk-state> kill --reason ... --operator ... [--account HASH ...] (broker credentials configured as for reconcile; all linked accounts when none are named), or {"action":"kill","reason":"..."} on the authority's operator API for the accounts it serves. It reports each order's outcome; clear the halt with resume once the book is safe. release-reservation <attempt> --operator ... --evidence ... frees the risk reserved for a resolved submission whose order is gone, audited as risk_released_by_operator.
Approvals last at most one hour, bind a principal and exact tool arguments, require a distinct operator identity string, and are consumed transactionally with dispatch intent. An approval is spent once an order reaches Schwab, whatever Schwab answers; a submission refused before anything was sent (halted, suspended, over a risk limit, rate-limited, unavailable, timed out before sending) gives it back, audited as approval_refunded, so the same intent can go once the refusal clears. A switch, halt or suspension known before dispatch never consumes it at all. A desk may be holding a submission open while you approve: place_order and replace_order accept wait_ms (at most 45 seconds), which polls this store and submits the moment the digest is granted, so approving promptly completes a trade rather than only unblocking the next retry. Waiting grants nothing — the approval is still yours — and a hold that ends unapproved dispatches nothing. The local CLI trusts filesystem access and the supplied operator attribution; deployments requiring verified two-person approval must wrap it with enterprise identity and audit controls. Durable revocation tombstones also override approval digests supplied in legacy static policy.
An agent that crashes, hangs or loses its connection leaves the orders it placed working with nobody watching. A rustyschwab-mcp principal that holds place_order or replace_order can carry dead_man_secs (30 to 86400; a shorter window would cancel a principal that is only thinking, and a longer one outlives a DAY order): after that long with no tool call by the principal, the server cancels the working orders that principal placed.
- Liveness. Every tool call by the principal resets its timer, including denied ones and
get_server_status, and a call still running is not silence. One watcher serves a principal across all its sessions, on stdio and on the network transports. A policy reload retires the watcher it replaces and the new one inherits the timer; a principal the reload removes is no longer watched once its old sessions are gone. The timer starts when the server starts, so working orders left by a previous run are cancelled if the agent never returns. - Across processes. Each call also touches a heartbeat file beside the principal's audit journal (
<journal>.alive). A watcher whose window has run out checks it first, so a principal served by several processes over the same state is silent only when all of them have stopped hearing from it, and the process it stopped using does not cancel orders it is still managing through another. - Stdio. A stdio client that disconnects or crashes ends the session but not the process:
rustyschwab-mcpstays until the switch fires for the silence that began with the principal's last call, then exits. If the principal comes back through another server first (the client relaunched), it leaves the watch to that one and exits without firing.SIGINTorSIGTERMstill stops it at once, so a client that kills its server on quit leaves the orders working. - Which orders. Only the principal's own, from its audit journal over the last 59 days: the venue order id a completed dispatch recorded, or, for a dispatch that recorded none (a lost acknowledgement, a process that died mid-dispatch), the order tag in the arguments it dispatched. These are matched against the working orders the venue lists on the accounts the principal traded and is granted; a filled entry's working children count as its own. Another principal's order on the same account, an order placed outside MCP, and an order already pending cancel are left alone.
- No other switch applies. The cancels go through the backend as the operator's own control, so they do not need
cancel_orderin the principal's tools orSCHWAB_MCP_ALLOW_CANCEL=1, and the SDK exempts cancels from the circuit breaker. An operator halt does not stop them either. - Once per silence. The switch fires once and re-arms on the principal's next call. A pass that cannot read the audit journal or the venue, or whose cancel is refused, is repeated up to three more times, ten seconds apart, within the same silence. A principal with nothing working fires nothing (a debug log only).
- Evidence. Each firing that found something is a
dead_man_cancelevent in the principal's audit journal (exported withrustyschwab-admin), with the window, every account and order id with its outcome (canceled, orfailedwith the code and message) and any read that failed; it is logged at warn and counted inschwab_mcp_dead_man_firings_total{outcome="canceled"|"failed"}. The principal'sget_server_statuscarriesdeadMan.windowSecsandsecondsLeft, so the agent knows to keep calling.
The switch lives in the rustyschwab-mcp process, so it cancels nothing while that process is down, and it does not cover orders a principal placed outside MCP. The authority gateway does not enforce the field and rejects a trader policy that sets it. For a book that must be flat regardless of who placed what, use the kill switch above.
The SDK accepts any implementation of RiskProvider. The bundled PortfolioRisk requires a fresh reconciled account and its positions; buys also need available funds (a margin account's availableFunds, or a cash account's cashAvailableForTrading). It reserves decimal quantity/notional with the symbol and side transactionally and retains reservations across cancellation/unknown outcomes; a definite rejection or an operator resolution releases them in the same transaction. It supports long equity, single-leg limit/stop-limit orders, and defined-risk option strategies (option legs on one underlying and one expiration, priced net), which count as buys of their contracts at the most they can lose at expiration (rustyschwab_types::trading::defined_risk); each leg's contract multiplier must be listed in the account's option_multipliers, and a working strategy at the broker counts the unfilled share of its risk. Other assets, conditional strategies, shorts and portfolio-margin rules require a deployment-owned risk provider.
Network MCP trading requires SCHWAB_RISK_POLICY_FILE and SCHWAB_RECOVERY_ACCOUNTS, in addition to principal permissions and SCHWAB_MCP_ALLOW_TRADING=1. Example risk file:
{"ACCOUNT_HASH":{"max_open_quantity":"10","max_open_notional":"1000","max_position_quantity":"20","max_snapshot_age_ms":5000,"max_daily_loss":"500"}}max_open_quantity and max_open_notional cap what all working buys could still add, and available funds bound their notional; max_position_quantity caps one symbol's position once its working buys fill. A sell may sell only the symbol's long position less its other working sells, and draws on no buy capacity, since closing what is held only reduces exposure. A replacement is judged without the order it replaces (Schwab swaps them atomically), so a protective sell of the whole position can be moved. The caps apply to held reservations plus externally placed open orders in the broker baseline. Positive local order links prevent double counting; partial external orders use validated remaining quantity. Unsupported or unbounded external orders fail closed under the built-in provider. It is not a complete portfolio margin model, nor can it observe activity beyond its history horizon or make separate REST reads atomic. Risk is revalidated after auth/quota waits, immediately before transmission. External providers must implement dispatch revalidation; fenced gateway dispatch also requires atomic fence/halt validation.
max_daily_loss is the optional daily-loss control. The loss is Schwab's start-of-day liquidation value less the current one, both read from the reconciled baseline, so it counts realized and unrealized losses alike and resets with Schwab's trading day. The day's deposits and withdrawals come out of the current value, so a deposit does not hide a loss: reconciliation reads the day's cash transfers from Schwab's transactions (eight transfer types, one request each) and reuses a reading for five minutes. Once it exceeds the cap, the provider records the operator halt (audited as operator_halt_changed by daily-loss-control, with the figures in the reason) and fails the reservation closed with Halted; each reconciliation also judges the baseline, so the halt normally trips within one recovery interval of the breach rather than at the next order. Clearing the halt does not reset the day: the next reconciliation or reservation trips it again while the loss still exceeds the cap. To trade on regardless, raise or remove the cap and restart. A baseline that carries no liquidation values, or whose transfers could not be read, cannot be measured; with a cap configured, reservations then fail closed as BaselineStale until a reconciliation reads them.
The reason names the account and the equity either side of the fall, so it is written for operators: the audit journal, rustyschwab-admin and an MCP principal granted every account read it in full, while a principal scoped to particular accounts is told only that a halt is active and that an operator must clear it. Halting by hand (rustyschwab-admin operator halt --reason ...) puts your words on the same path; keep them operational, and assume a model will read them when the principal is unscoped.
SCHWAB_HEALTH_BIND=127.0.0.1:9091 enables /livez and /readyz. Readiness checks token validity, circuit state, durable-store access, pending submissions and required account-baseline age. Trading readiness also requires a risk provider. Expose the loopback listener through an operations gateway when needed. No account identifiers or token material are returned by these probes.
Import ops/dashboard.json into Grafana and select your Prometheus datasource. Poll readiness to update aggregate readiness gauges. Use ops/alerts.yml with Prometheus and install the optional metrics recorder with SCHWAB_METRICS_BIND. Initial operating targets should be reviewed against measured broker behavior: no automatic ambiguous submission replay; no trading from an unreconciled required account; alarm on any unknown outcome; record p50/p95/p99 request latency, recovery duration, baseline age, stream loss and time spent unready. Define availability/error budgets for the broker dependency separately from the utility.
Metrics are opt-in. Set SCHWAB_METRICS_BIND=127.0.0.1:9090 and the MCP server starts a Prometheus scrape listener on that address. Unset, there is no listener and every metrics::counter!/histogram! call is a no-op: the SDK depends only on the metrics facade, so a library consumer can install its own recorder.
SCHWAB_METRICS_BIND=127.0.0.1:9090 cargo run -p rustyschwab-mcp --release
# in another shell
curl http://127.0.0.1:9090/metricsEvery metric, with its kind, labels and meaning, is listed in the module docs of rustyschwab::metrics and rustyschwab_mcp::metrics (cargo doc -p rustyschwab-mcp --open). Those tables and the CATALOG constants are the source of truth: a workspace test keeps them, every call site, ops/alerts.yml and ops/dashboard.json in line, and the server registers the help text with Prometheus.
Starter PromQL:
# 5xx error rate per operation
sum by (op) (rate(schwab_http_requests_total{status="5xx"}[5m]))
# p99 latency by op
histogram_quantile(0.99,
sum by (le, op) (rate(schwab_http_request_duration_seconds_bucket[5m])))
# Tool call error ratio
sum by (tool) (rate(schwab_mcp_tool_calls_total{outcome="error"}[5m]))
/ sum by (tool) (rate(schwab_mcp_tool_calls_total[5m]))
# Circuit breaker open (alert on > 0)
max_over_time(schwab_circuit_state[1m]) == 2
# Stream healthy (alert on 0 during market hours)
schwab_stream_connected
# Sustained token refresh failures
increase(schwab_token_refresh_total{outcome="failure"}[15m])
HTTP/WebSocket process shutdown handles SIGINT and, on Unix, SIGTERM. It permanently closes submission admission and allows up to thirty seconds for HTTP draining. Transport cancellation abandons only a write's answer; the write itself runs to its outcome unless the process exits first, in which case reconcile on restart. A principal's audit record left unresolved by a process exit is settled from the venue journal on that principal's next submission once it is two minutes old: with the venue's outcome when it has one, and, when nothing reached the venue, as such with its approval given back. For application-owned shutdown, stop admission first, preserve unresolved intents and reservations, disconnect streams, and let the process close its database connections. Never clear pending records to make a health probe pass. If disk is full, storage is corrupt, identity keys are unavailable or broker order pages are truncated, fail closed and investigate using the corresponding runbook above.
Prometheus (SCHWAB_METRICS_BIND) and the health probes above need no extra build. Export to an OpenTelemetry collector is opt-in at compile time: build rustyschwab-mcp with --features otel (it forwards to rustyschwab/otel), or call rustyschwab::telemetry::init_otel in your own application. Nothing from the OpenTelemetry crates is compiled without the feature, and a build that has it exports nothing until the environment names an endpoint, at which point rustyschwab-mcp and rustyschwab-authority behave exactly like a build without the feature.
The exporter speaks OTLP over HTTP/protobuf only (port 4318 on a stock collector), on the blocking reqwest client and the rustls stack the rest of the workspace uses. A different OTEL_EXPORTER_OTLP_PROTOCOL (grpc, http/json) stops the process at startup instead of posting to a port that would not understand it. Configuration is the standard environment:
| Variable | Effect |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
Collector base URL; /v1/traces and /v1/metrics are appended. Setting it (or a per-signal endpoint) turns export on. |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT |
Per-signal URL, used as given. |
OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TIMEOUT (and the _TRACES_ / _METRICS_ variants) |
Request headers, such as a collector API key, and the per-request timeout. |
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES |
The resource. Without either, the service is rustyschwab-mcp or rustyschwab-authority rather than unknown_service:<executable>. |
OTEL_SDK_DISABLED=true, OTEL_TRACES_EXPORTER=none, OTEL_METRICS_EXPORTER=none |
Switch export off while leaving the endpoint configured. |
OTEL_BSP_*, OTEL_METRIC_EXPORT_INTERVAL |
Span batching and the metrics push interval (60 s by default). |
RUST_LOG |
Filters the log output and the exported spans alike; a span below the active level is not exported. |
Treat OTEL_EXPORTER_OTLP_HEADERS as a secret like any other credential in the process environment.
What leaves the process is spans, plus metrics when a metrics endpoint is set, and nothing else. The server opens one mcp.tool span per tool call (field tool) and the SDK one schwab.request span per logical Schwab request (fields op and http.request.method, both fixed labels). Log events (warn!, debug!, ...) are not exported, even inside a span, because their fields carry error text, file paths, symbols and attempt ids; span fields never carry tool arguments, orders, account numbers, paths, query strings or tokens, and a new span must keep it that way. The metrics are the catalog in the repository README with the same names and labels, as OTLP instruments (duration histograms use second-scale buckets). With SCHWAB_METRICS_BIND also set, one recorder feeds both Prometheus and OTLP, so the two always agree.
Hold the returned guard until the program is done: dropping it exports whatever is still buffered and shuts the providers down, blocking for at most the exporter timeout. rustyschwab-mcp and rustyschwab-authority keep it for the life of main, so a clean exit (end of a stdio session, SIGINT or SIGTERM on the network transports) flushes. A process that is killed outright loses the last batch, up to five seconds of spans by default and up to a minute of metrics. rustyschwab-authority installed no log subscriber before; built with otel it logs to stderr as rustyschwab-mcp does (RUST_LOG, info by default).
To bring Prometheus-scraped metrics into a collector instead of pushing them (for example to keep rustyschwab-mcp built without otel, or to use a collector that you already point at your Prometheus targets), leave OTEL_EXPORTER_OTLP_METRICS_ENDPOINT unset and scrape the endpoint with the collector's prometheus receiver:
receivers:
prometheus:
config:
scrape_configs:
- job_name: rustyschwab-mcp
scrape_interval: 15s
static_configs:
- targets: ["127.0.0.1:9090"] # SCHWAB_METRICS_BIND
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318 # spans from OTEL_EXPORTER_OTLP_ENDPOINT
exporters:
otlphttp:
endpoint: https://otlp.example.com
service:
pipelines:
traces: {receivers: [otlp], exporters: [otlphttp]}
metrics: {receivers: [prometheus], exporters: [otlphttp]}Choose one path for metrics, push or scrape; running both ships every series twice.
The suite includes migration/backup/restore, cross-process locking, process kill during an uncommitted transition, tag matching, activity deduplication, stale generations, quota/fencing, approvals, risk reservations, JWT negative cases, policy reload and malformed-input/decimal properties.
Build enterprise-bench in release mode, then run tools/benchmark.py against a pinned local Schwabdev checkout. It uses the same synthetic loopback quote payload for both clients and reports latency percentiles and throughput. Run python3 tools/soak.py --seconds 60 --consumer-delay-ms 10 --output soak.json on Linux/macOS for forced-disconnect/overflow testing with sampled child RSS. The underlying enterprise-bench stream <seconds> <consumer-delay-ms> also runs without the RSS wrapper. A short local run is not proof of long-duration memory stability, broker throughput or production latency. Preserve fixture hash, reference commit, compiler, dependency lockfile and hardware/OS alongside results.
CI runs feature/platform/MSRV checks and short soak jobs. The release workflow creates binaries, a CycloneDX SBOM, checksums and GitHub build-provenance attestations; it uploads artifacts without publishing a release. Compatibility checking compares PRs with the base revision. Intentional breaking changes require an appropriate version bump and migration notes, not a bypass of the check. Verify provenance before deployment using the GitHub attestation verification workflow.
Still required externally: native Windows execution, live broker contract/entitlement qualification, gateway/identity integration, backup restoration on the production filesystem, storage exhaustion exercises and a sustained production-shaped soak. This document does not mark those gates passed.
See support and compatibility policy. Local qualification evidence is written to docs/audit/ (git-ignored; the same artifacts are uploaded by qualification.yml).
The manual/weekly qualification.yml workflow runs the full native suite, including storage faults and authority/recovery integrations, on Linux, macOS and Windows, and retains test transcripts even on failure. Separate fast/slow synthetic stream jobs offer 15 minutes, one hour, four hours or five hours per profile. These jobs become available after the workflow is committed to the hosted repository; adding the file does not constitute a hosted run.
The soak wrapper exits nonzero on early completion, no data, failure to reconnect, unexpected fast-consumer loss, missing slow-consumer overflow, missing RSS samples, peak RSS above 128 MiB or warmed growth above 32 MiB. Warmup defaults to thirty seconds. These are initial synthetic acceptance thresholds, configurable with --max-rss-mib, --max-rss-growth-mib and --warmup-seconds; they are not production capacity guarantees. Deliberately slow consumption is expected to lose data and require recovery. Synthetic credentials remain valid for the selected run, and token refresh is restricted to an unused loopback endpoint.
Storage qualification includes an SQLite page-limit injection producing SQLITE_FULL. It verifies that a failed transaction cannot clear a pending order. This checks database rollback semantics without filling the workstation disk; it does not close the actual production-filesystem exhaustion gate. Missing, corrupt and incomplete backup schemas are rejected before creating destination state. Concurrent backups cannot replace each other's output, and restore preserves source database bytes.
Before a live qualification run, identify the deployment filesystem, broker account entitlements, enterprise issuer, risk provider and any cross-host coordinator/gateway. Begin with authenticated read-only queries and recovery using an isolated qualification journal. Order placement, cancellation and replacement exercises require a concrete reviewed test protocol, account and limits. Preserve redacted evidence, not token files or private raw account snapshots, in the qualification record.
Operator halts now use halt_submissions / clear_operator_halt and are durable, and refuse submissions with Error::Halted. suspend_submissions / resume_submissions control only temporary recovery, and a suspended or draining client refuses submissions with Error::Suspended (MCP unavailable, retry later; authority HTTP 503). Canonical approval hashing sorts JSON object keys recursively; verify or reissue outstanding approvals when upgrading from the earlier key-order-sensitive candidate. Existing consumed tombstones remain retained.
The stream soak additionally enforces --max-delivery-gap-seconds (default ten seconds). A run that delivers briefly and then stalls cannot pass on total-message count alone. REST recovery cycles also have a deadline, with end-to-end tests for timeout, fresh baselines, account reconciliation and operator-halt preservation.
The soak wrapper also rejects RSS sampling gaps longer than five seconds (--max-sample-gap-seconds). Each ps call has a two-second deadline. Child failures, malformed result metrics and sampling/overall timeouts save failed JSON evidence with process status, elapsed time and a bounded stderr tail; the child is terminated when necessary. Missing build artifacts and invalid command arguments remain invocation errors. Diagnostic files may contain process stderr: review them before sharing. See docs/audit/deployment-qualification/README.md after a local run.
Execution databases are validated and migrated once per open (the first operation on a StateDatabase handle) and reject missing required tables, metadata or schema versions, or a schema from a newer build, at that point; later operations reuse pooled connections. Schema changes are ordered migrations recorded in schema_migrations, with meta.schema holding the highest applied version. Paper reopen additionally checks account presence and event-history continuity against the cursor. Do not treat these errors as instructions to create replacement tables: recover a verified complete backup or perform an explicitly reviewed migration. Missing state must not reset consumed tags, approval history, cash or event deduplication.