Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
677 lines (492 loc) · 24.2 KB

DEPLOYMENT.md

File metadata and controls

677 lines (492 loc) · 24.2 KB
Raw

NeuroWealth Backend Deployment Guide

Complete guide for deploying the NeuroWealth backend across all environments.

Table of Contents

Prerequisites

Requirement Notes
Node.js 20 Matches Dockerfile and CI
Docker For local development and container builds
Kubernetes 1.25+ For production (EKS, GKE, AKS)
PostgreSQL 14+ Managed database (RDS, Cloud SQL)
Container registry For pushing Docker images
Stellar Soroban RPC STELLAR_RPC_URL or comma-separated STELLAR_RPC_URLS for failover
TLS certificate cert-manager, cloud LB, or manual Secret for ingress
Secrets store External Secrets Operator, Sealed Secrets, or kubectl create secret

Local Development (Docker Compose)

Quick Start

# Clone the repository
git clone https://github.com/Neurowealth/Backend.git
cd Backend

# Copy environment variables
cp .env.example .env

# Start PostgreSQL with Docker Compose
docker-compose up -d

# Install dependencies
npm install

# Generate Prisma client
npx prisma generate

# Apply database migrations
npx prisma migrate deploy

# Start development server
npm run dev

Docker Compose Services

The docker-compose.yml runs:

  • PostgreSQL on port 5432
  • Application on port 3001 (optional)

Verifying Local Setup

# Check health endpoint
curl http://localhost:3001/health/live

# Check readiness (requires database connection)
curl http://localhost:3001/health/ready

Staging (Docker)

Build Docker Image

# Build the image
docker build -t neurowealth-backend:staging .

# Tag for your registry
docker tag neurowealth-backend:staging <registry>/neurowealth-backend:staging

# Push to registry
docker push <registry>/neurowealth-backend:staging

Deploy with Docker

# Run the container
docker run -d \
  --name neurowealth-backend-staging \
  -p 3001:3001 \
  --env-file /path/to/staging.env \
  <registry>/neurowealth-backend:staging

Staging Environment Variables

NODE_ENV=staging
STELLAR_NETWORK=testnet
STELLAR_RPC_URL=https://soroban-testnet.stellar.org
CORS_ORIGINS=https://staging.neurowealth.io
LOG_LEVEL=debug

Staging enforces the CORS allowlist exactly as production does, and refuses to start if the allowlist is empty or contains *. Only NODE_ENV=development and NODE_ENV=test permit arbitrary origins.

Production (Kubernetes)

Manifest Layout

All manifests live under deploy/k8s/:

File Purpose
namespace.yaml neurowealth namespace
configmap.yaml Non-secret environment (CORS, rate limits, RPC URLs, contract IDs)
secret.yaml.example Template only — copy values into a real Secret
serviceaccount.yaml Pod service account
deployment.yaml App Deployment with initContainer migration + probes
service.yaml ClusterIP on port 3001
ingress.yaml TLS termination (adjust host / ingress class)
migration-job.yaml Standalone pre-deploy migration Job
hpa.yaml HPA pinned to 1 replica (see scaling constraints)

Environment Matrix

Setting Staging Production
NODE_ENV staging production
STELLAR_NETWORK testnet mainnet
STELLAR_RPC_URL Testnet Soroban RPC Mainnet Soroban RPC
CORS_ORIGINS Staging frontend URL Production frontend URL
LOG_LEVEL debug info
replicas 1 1 (until worker split)
Secrets Staging Secret / external store Production Secret / external store

Build and Push Image

docker build -t <registry>/neurowealth-backend:<version> .
docker push <registry>/neurowealth-backend:<version>

Update the image: field in deployment.yaml and migration-job.yaml to your registry tag.

Migration Strategy

The default Dockerfile CMD runs prisma migrate deploy && node dist/index.js. In Kubernetes, the Deployment overrides the command to node dist/index.js only. Migrations run in the initContainer (or standalone Job) so a failed migration blocks the rollout instead of leaving a half-started pod serving traffic.

Rollout Procedure

1. Migrate

Option A — initContainer (default in deployment.yaml): migrations run automatically before each pod starts.

Option B — standalone Job (recommended for large migrations):

# Update image tag in migration-job.yaml, then:
kubectl apply -f deploy/k8s/migration-job.yaml
kubectl wait --for=condition=complete job/neurowealth-migrate -n neurowealth --timeout=300s

2. Deploy

kubectl apply -f deploy/k8s/namespace.yaml
kubectl apply -f deploy/k8s/serviceaccount.yaml
kubectl apply -f deploy/k8s/configmap.yaml
# secrets already created above
kubectl apply -f deploy/k8s/deployment.yaml
kubectl apply -f deploy/k8s/service.yaml
kubectl apply -f deploy/k8s/ingress.yaml

3. Verify Readiness

kubectl rollout status deployment/neurowealth-backend -n neurowealth

# Port-forward for local check:
kubectl port-forward svc/neurowealth-backend 3001:3001 -n neurowealth

curl -s http://localhost:3001/health/live
curl -s http://localhost:3001/health/ready

Readiness returns 200 only when database, event listener, and agent loop are all healthy. During rollout or shutdown it returns 503.

4. Smoke Test

curl -s -o /dev/null -w "%{http_code}" https://api.neurowealth.io/health

Scaling Guidance

Current Constraint: Single Active Consumer

The monolith starts three subsystems in every pod (src/index.ts):

  1. HTTP API
  2. Stellar event listener — polls Soroban RPC every 5 s, persists cursor to event_cursors
  3. Agent cron loop — hourly rebalance, snapshots, daily protocol scan

There is no leader election. Running multiple replicas will:

  • Duplicate event processing (mitigated by processed_events idempotency, but wastes RPC quota and risks race conditions)
  • Run duplicate cron jobs (rebalance checks, snapshots)

Recommendation: keep replicas: 1 until the architecture is split.

Future Scaling Path

  1. Add feature flags: ENABLE_EVENT_LISTENER, ENABLE_AGENT_LOOP
  2. Split deployments:
    • neurowealth-api — stateless HTTP, replicas: N, HPA enabled
    • neurowealth-worker — listener + agent, replicas: 1
  3. Optional: K8s Lease or Postgres advisory lock for worker leader election before scaling workers beyond 1

HPA

deploy/k8s/hpa.yaml is pinned to minReplicas: 1 / maxReplicas: 1. Re-enable scaling only after the worker/API split.

Environment Variables Reference

Copy .env.example as a checklist. Set every value via your secrets manager — never commit production secrets.

Required (app will not start without these)

Variable Notes
NODE_ENV Must be production
PORT Default 3001
DATABASE_URL PostgreSQL connection string
STELLAR_NETWORK mainnet, testnet, or futurenet
STELLAR_RPC_URL Soroban RPC endpoint for the chosen network
STELLAR_AGENT_SECRET_KEY 56-char Stellar secret (S…)
VAULT_CONTRACT_ID Deployed vault contract ID
USDC_TOKEN_ADDRESS USDC token contract on Stellar
ANTHROPIC_API_KEY Claude API key for the agent
JWT_SEED 64-hex secret for signing sessions — rotate every 90 days
WALLET_ENCRYPTION_KEY 64-hex (32 bytes) — openssl rand -hex 32
TWILIO_AUTH_TOKEN Required for WhatsApp webhook signature validation

Required in Production Only

Variable Notes
ADMIN_API_TOKEN Strong token (≥ 8 chars) for /api/admin/* — inject via secrets manager
CORS_ORIGINS or ALLOWED_ORIGINS Comma-separated frontend origins (e.g. https://app.example.com) — do not use *. Enforced in production and staging; a single-label subdomain wildcard such as https://*.example.com is permitted. The process refuses to start if the list is empty or a wildcard is present.
CORS_REQUIRE_ORIGIN true in production, false elsewhere. Reject requests carrying no Origin header (curl, k8s probes, server-to-server). Set false if browser clients omit the header.

Recommended

Variable Default Notes
LOG_LEVEL info Winston log level in production
RATE_LIMIT_* / AUTH_RATE_LIMIT_* see .env.example Tune per environment
INTERNAL_SERVICE_TOKEN — Service-to-service bypass for rate limits
TRUSTED_IPS — Comma-separated IPs that skip rate limits (probes, internal scrapers)
DLQ_ALERT_THRESHOLD 50 Alert when dead-letter queue exceeds this count

Reverse Proxy

Express trust proxy is set to 1 in src/index.ts so req.ip reflects the client behind a single load balancer. If you run behind CDN + LB (two hops), adjust that setting before deploy.

Full list and defaults: .env.example.

Health Probes

Probe Path Success Failure Use
Liveness GET /health/live 200 n/a Process is running; restart if unreachable
Readiness GET /health/ready 200 when DB, event listener, and agent loop are ready 503 during startup or shutdown Route traffic only to healthy instances

Additional endpoints:

  • GET /health — basic JSON status (also available via healthRouter)
  • GET /health/deep — authenticated (X-Internal-Token) dependency probe; used by the k8s readiness probe in deploy/k8s/deployment.yaml
  • GET /metrics — Prometheus scrape target

Liveness / Readiness Matrix

Liveness answers "is this process wedged?" and never touches dependencies, so a downstream outage can never trigger a restart loop. Readiness answers "should this pod receive traffic?" and is driven by GET /health/deep.

Each deep check runs in parallel under its own budget (HEALTH_CHECK_TIMEOUT_MS, default 1000). A hung dependency is reported as unhealthy with a timed out error once its budget expires, so total probe latency stays ~1s regardless of how many dependencies hang — well under the probe's timeoutSeconds: 5.

Dependency Criticality When it fails or hangs Liveness (/health/live) Readiness (/health/deep) Rationale
PostgreSQL (database) critical unhealthy 200 — no restart 503 — pod removed from rotation Pod cannot serve requests without its DB connection
Agent loop (agentLoop) critical stopped or no tick in 2× interval → unhealthy; recent error → degraded 200 — no restart 503 when unhealthy; 200 when degraded Per-pod process; another replica may be healthy
Stellar RPC (stellarRpc) non-critical unhealthy 200 — no restart 200, status: degraded Shared external service: failing readiness would drain every replica at once
Twilio (twilio) non-critical unhealthy 200 — no restart 200, status: degraded Shared external service; only messaging is affected

The deep probe response lists any non-healthy dependency in degradedDependencies (e.g. ["stellarRpc"]). Per-check error strings are redacted and truncated to 200 characters, so they never carry secrets. The policy lives in DEPENDENCY_POLICY in src/routes/health.ts.

Each check also exports Prometheus gauges:

  • health_dependency_status{dependency, criticality} — 0 healthy, 1 degraded, 2 unhealthy
  • health_dependency_latency_ms{dependency}

Alert on health_dependency_status{criticality="non-critical"} == 2 sustained for a few minutes: those failures do not affect readiness, so the probe alone will not surface them.

Kubernetes Example

livenessProbe:
  httpGet:
    path: /health/live
    port: 3001
  initialDelaySeconds: 10
  periodSeconds: 15
  timeoutSeconds: 5

# Readiness follows the Liveness / Readiness Matrix above.
# Keep timeoutSeconds > HEALTH_CHECK_TIMEOUT_MS so a hung dependency
# is reported as degraded/unhealthy rather than timing out the probe.
readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - 'wget --spider -q --header="X-Internal-Token: $INTERNAL_SERVICE_TOKEN" http://localhost:3001/health/deep'
  initialDelaySeconds: 15
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

AWS ALB Example

  • Health check path: /health/ready
  • Matcher: 200
  • Interval: 30 s
  • Unhealthy threshold: 3

During graceful shutdown (SIGTERM/SIGINT), readiness returns 503 with status: shutting_down so load balancers stop sending new requests before the process exits.

Secrets Management

Required Production Secrets

Secret Rotation Notes
JWT_SEED Every 90 days Invalidates active sessions; schedule maintenance window
WALLET_ENCRYPTION_KEY Coordinated migration Re-encrypt custodial_wallets rows before swapping the key; loss of key = unrecoverable wallets
STELLAR_AGENT_SECRET_KEY Rare Fund a new agent key and update contract permissions before swap
ADMIN_API_TOKEN On compromise Rotate immediately; update secrets store and redeploy
TWILIO_AUTH_TOKEN On compromise Update Twilio console and redeploy
DATABASE_URL Per provider policy Use least-privilege DB user; enable SSL

Secret Managers (Recommended)

Provider Best for Notes
AWS Secrets Manager AWS-hosted production Automatic rotation hooks; inject via ECS task secrets or Lambda env
HashiCorp Vault Multi-cloud / on-prem Dynamic secrets, audit trail; use AppRole or K8s auth
GitHub Actions secrets CI/CD and staging Store DATABASE_URL, JWT_SEED, etc. per environment; never log values

Creating Kubernetes Secrets

kubectl create namespace neurowealth

kubectl create secret generic neurowealth-secrets \
  --namespace=neurowealth \
  --from-literal=DATABASE_URL='postgresql://...' \
  --from-literal=JWT_SEED='...' \
  --from-literal=WALLET_ENCRYPTION_KEY='...' \
  --from-literal=STELLAR_AGENT_SECRET_KEY='...' \
  --from-literal=ANTHROPIC_API_KEY='...' \
  --from-literal=ADMIN_API_TOKEN='...' \
  --from-literal=TWILIO_AUTH_TOKEN='...'

Required keys match src/config/env.ts startup validation. Optional keys: TWILIO_ACCOUNT_SID, INTERNAL_SERVICE_TOKEN, SLACK_WEBHOOK_URL, PAGERDUTY_ROUTING_KEY.

Secrets Best Practices

Do:

  • Inject secrets at runtime from a secrets manager
  • Use separate credentials per environment (staging vs production)
  • Take a DB snapshot before running prisma migrate deploy

Do not:

  • Commit .env files or bake secrets into Docker image layers
  • Log secret values (Winston redacts in production but avoid passing secrets in error messages)

Database Migrations

The full process — change classification, validation, backup and downtime assessment, and the three rollback tiers — is documented in MIGRATIONS.md. This section covers the deployment mechanics only. Read MIGRATIONS.md before opening a migration PR.

Pre-Deployment Checklist

The authoritative checklist is in MIGRATIONS.md §6. The short form:

  • Review pending Prisma migrations (npx prisma migrate status)
  • Classify the change (A–D) and record its downtime impact in the PR
  • Confirm migration SQL is non-destructive or has a documented data backfill
  • Take a database backup/snapshot (provider console or pg_dump); record the snapshot identifier in the PR
  • Schedule during low traffic; notify on-call
  • Rehearse the rollback (bash scripts/rehearse-migration-rollback.sh)
  • Staging deploy passed CI (migration-smoke and Rollback rehearsal jobs green)

Applying Migrations

Option A — Docker Container (single-instance):

docker run -d \
  --name neurowealth-migrate \
  --env-file /path/to/production.env \
  <registry>/neurowealth-backend:<version> \
  npx prisma migrate deploy

Option B — Kubernetes initContainer (recommended):

initContainers:
  - name: migrate
    image: <registry>/neurowealth-backend:<version>
    command: ["npx", "prisma", "migrate", "deploy"]
    envFrom:
      - secretRef:
          name: neurowealth-backend-secrets

Option C — Standalone Job (large migrations):

kubectl apply -f deploy/k8s/migration-job.yaml
kubectl wait --for=condition=complete job/neurowealth-migrate -n neurowealth --timeout=300s

Option D — Safe script with smoke test:

DATABASE_URL=postgresql://... bash scripts/apply-migration.sh

Verifying Migration

# Check migration status
npx prisma migrate status

# Verify database tables
psql $DATABASE_URL -c "\dt event_cursors"
psql $DATABASE_URL -c "\dt processed_events"

Rollback Procedure

Application Rollback

# Roll back to previous ReplicaSet
kubectl rollout undo deployment/neurowealth-backend -n neurowealth

# Or pin a known-good image:
kubectl set image deployment/neurowealth-backend \
  api=<registry>/neurowealth-backend:<previous-version> \
  -n neurowealth

kubectl rollout status deployment/neurowealth-backend -n neurowealth

Database Rollback

Prisma migrations are forward-only — there is no built-in migrate down. Every migration in prisma/migrations/<name>/ ships a hand-written rollback.sql alongside its migration.sql, and CI enforces that every migration has one.

There are three rollback tiers, in order of preference. The full decision matrix is in MIGRATIONS.md §6.1.

Tier Use when Mechanism
1 — Application only The migration is correct; the code shipped with it is wrong Redeploy the previous image. Schema unchanged.
2 — rollback.sql The migration itself is the problem scripts/rollback-migration.sh <migration>
3 — Snapshot restore The migration lost or corrupted data Restore the pre-deploy snapshot, redeploy the previous image, reconcile 
# Tier 2 — reverse one migration, then confirm the app still works
DATABASE_URL=postgresql://... \
HEALTHCHECK_URL=http://localhost:3001/health/ready \
  bash scripts/rollback-migration.sh 20260901000000_add_agent_circuit_breaker

When to Rollback

Situation Action
Migration applied, app bug only Tier 1 — roll back the application image; DB unchanged
Bad constraint / failed backfill, no data loss yet Tier 2 — scripts/rollback-migration.sh <migration>; redeploy previous app
Bad migration with partial writes Tier 3 — restore snapshot; replay DLQ after fix; document manual reconciliation
Migration is documented as partially irreversible Tier 3 — its rollback.sql cannot restore the data it drops

Tier 3 — Snapshot Restore Steps

  1. Stop traffic to new instances (drain load balancer).
  2. Restore database from the pre-deploy backup/snapshot.
  3. Deploy the previous application image (matching the restored schema).
  4. Run npm run smoke against the restored DB.
  5. Re-enable traffic; post-mortem and fix-forward migration in a new release.

WARNING: never rely on migrate reset in staging or production — it drops the database. Do not use migrate dev against a shared environment either; it may reset the database to match the schema.

Monitoring and Observability

Metrics

  • Metrics: GET /metrics on port 3001 (Prometheus)
  • Request tracing: clients may send X-Request-ID or X-Correlation-ID; the server echoes X-Request-ID on every response and includes correlationId in structured logs
  • DLQ: monitor dead_letter_events count and event_cursors.lastProcessedLedger lag — see docs/OBSERVABILITY.md and docs/RUNBOOK.md

Monitoring Assets

Pre-built alert rules and Grafana dashboards live under deploy/monitoring/:

Path Purpose
deploy/monitoring/prometheus/alert-rules.yaml Prometheus alert rules (critical + warning)
deploy/monitoring/grafana/dashboards/system-overview.json System overview dashboard
deploy/monitoring/grafana/dashboards/agent-loop.json Agent loop health dashboard
deploy/monitoring/grafana/dashboards/dlq.json DLQ and cursor lag dashboard
deploy/monitoring/grafana/dashboards/latency.json HTTP, DB, and event latency dashboard
deploy/monitoring/grafana/provisioning/datasources.yaml Grafana datasource provisioning
deploy/monitoring/grafana/provisioning/dashboards.yaml Grafana dashboard provisioning

Prometheus Configuration

rule_files:
  - /etc/prometheus/rules/alert-rules.yaml

Grafana Setup

cp deploy/monitoring/grafana/provisioning/* /etc/grafana/provisioning/
cp deploy/monitoring/grafana/dashboards/*.json /etc/grafana/dashboards/

Grafana will auto-load the dashboards on next restart.

Logs and Metrics Commands

# Docker logs
docker logs neurowealth-backend --tail 200 -f

# Kubernetes logs
kubectl logs -l app=neurowealth-backend --tail=200 -f

# Prometheus metrics
curl -sS http://localhost:3001/metrics | head -50

Troubleshooting

Common Issues

Symptom Check
Pod CrashLoopBackOff kubectl logs deployment/neurowealth-backend -n neurowealth; verify all required secrets
Readiness 503 kubectl logs — DB connection, RPC URL, or background service startup failure
Migration initContainer failed kubectl logs <pod> -c migrate -n neurowealth
Events not processing SELECT * FROM event_cursors; — cursor lag; ensure only one replica runs the listener
Duplicate rebalances Confirm replicas: 1; check agent cron is not running on multiple pods

Health and Readiness Commands

# Liveness — should always return 200 once the process is up
curl -sS http://localhost:3001/health/live | jq .

# Readiness — 200 when all subsystems ready, 503 during startup/shutdown
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:3001/health/ready

# Detailed subsystem status
curl -sS http://localhost:3001/health/ready | jq .

Database and Migration Commands

# Check migration status
DATABASE_URL="postgresql://..." npx prisma migrate status

# Apply pending migrations (production-safe, non-destructive)
DATABASE_URL="postgresql://..." npx prisma migrate deploy

# Safe migration + smoke test
DATABASE_URL="postgresql://..." bash scripts/apply-migration.sh

# Verify DB connectivity from the app host
psql "$DATABASE_URL" -c "SELECT 1"

Common Startup Failures

# Missing or invalid env — app prints all validation errors at once
NODE_ENV=production node dist/index.js

# Verify required production vars are set (no values printed)
env | grep -E '^(NODE_ENV|DATABASE_URL|ADMIN_API_TOKEN|CORS_ORIGINS|JWT_SEED|WALLET_ENCRYPTION_KEY)='

# Stellar network mismatch warning
# Ensure STELLAR_NETWORK=mainnet only when NODE_ENV=production and keys are mainnet

Rate-Limit / Proxy Issues

# The app trusts one reverse-proxy hop (trust proxy = 1).
# If client IPs look wrong behind CDN + LB, update src/index.ts before redeploying.
curl -H "X-Forwarded-For: 203.0.113.1" http://localhost:3001/health/live

CI Validation

Manifests are validated in CI with kubeconform (see .github/workflows/k8s-validate.yml). Run locally:

kubeconform -summary deploy/k8s/*.yaml

Related Documentation

  • .env.example — full environment variable reference
  • readme.md — local development, auth flow, rate limiting
  • Dockerfile — image build stages and default CMD
  • scripts/apply-migration.sh — CI/CD migration gate with smoke test
  • docs/RUNBOOK.md — DLQ replay, cursor management
  • docs/OBSERVABILITY.md — alerting and metrics