[DanProtocol]

Closes #43

Summary

Background worker daemons (settlement, reservation cleanup, reconciliation) previously ran with no liveness signal — a stalled or deadlocked worker looked identical to a healthy idle one to the orchestrator. This PR adds heartbeat liveness telemetry and an HTTP health endpoint so Render can detect stalled workers instantly.

Implementation

• WorkerHealthTelemetryService (src/infrastructure/telemetry/worker-health-telemetry.service.ts): in-process telemetry singleton tracking:
  • worker lifecycle (startup / shutdown timestamps + state)
  • per-daemon last sweep timestamps and processed counts (settlement, reservation cleanup, payment reconciliation, reconciliation)
  • active jobs count
  • distributed-lock acquisition failures count (wired into DistributedLockService.acquire)
• WorkerLivenessServer (src/infrastructure/telemetry/worker-liveness.server.ts): dependency-free standalone HTTP server exposing GET /health:
  • 200 — healthy (running with at least one fresh sweep heartbeat; distinguishes a healthy idle worker from a stalled one)
  • 503 — stalled (no fresh sweep within WORKER_LIVENESS_MAX_SWEEP_AGE_MS), stopped, or not started
• Runner integration (src/worker.ts): lifecycle heartbeats on start/stop, per-daemon sweep heartbeat recording with per-daemon error isolation (one failing daemon no longer hides the others' heartbeats; errors aggregate to the existing WORKER_LOOP_ERROR audit path), liveness server start/stop, and port-conflict resilience.
• Config (src/config/index.ts): new validated env vars — WORKER_LIVENESS_ENABLED (default true), WORKER_LIVENESS_PORT (default 4001), WORKER_LIVENESS_MAX_SWEEP_AGE_MS (default 120000).

Security

• Liveness payload contains no secrets or credentials — only booleans, counters, timestamps, pid, and uptime. Covered by an explicit unit-test assertion (no secret/password/token/seed/apikey/private substrings, allowlisted top-level keys).

Testing

• New suite tests/workers/worker-health-telemetry.test.ts (18 tests) covering:
  • health state transitions: NOT_STARTED → STALLED → HEALTHY → STALLED (heartbeat staleness) → STOPPED
  • healthy idle worker (0-item sweeps) classified HEALTHY vs stalled worker 503
  • active jobs floor, lock-failure counting, secret-free payload
  • HTTP endpoint status codes: 200/503/404, method gating, concurrent probes
  • runner integration (sweep heartbeats, lifecycle, liveness server lifecycle)
• Full suite: 348 passed / 4 pre-existing failures in tests/settlements/worker-concurrency.test.ts caused by missing STELLAR_SETTLEMENT_SIGNER_SECRET_KEY in the local environment (verified identical on the unmodified base commit; CI supplies its own env). npm run type-check, npm run lint, and npm run build all pass.

Ops rollout note

Render worker service can point its health check at :4001/health once WORKER_LIVENESS_PORT is exposed (env vars documented in .env.example).