Powernode System Extension

This extension provides the substrate layer beneath the Powernode platform: PXE-to-production bare metal, a signed module supply chain, SDWAN, and fleet autonomy — the full path from a powered-off machine to a governed node in a running fleet. The Powernode platform is the control plane; this extension implements what it runs on. See the platform repo for the end-to-end demo and how the two fit together.

A self-contained extension for the Powernode platform that provides node lifecycle management, declarative module composition, on-node runtime, and a fleet autonomy layer. Each autonomy tick senses fleet state, gates every proposed action through intervention policy and approval, executes the approved remediations, and records the decisions as learnings. It does not rewrite its own skills: the trigger meant to evolve a skill from repeated learnings is refused on every tick (see M8 under Milestones complete).

This repository is mounted into the Powernode platform as a submodule at extensions/system/. It can be operated independently — the platform consumes it via the standard extension contract.

---

What this extension provides

Operator surface

• Node + instance lifecycle with a polymorphic Task model + AASM state machines covering pending → provisioning → running → stopped → terminated
• Declarative templates composed from versioned NodeModules with rsync-glob filter rules (mask, file_spec, package_spec, dependency_spec)
• Cloud provider abstractions (AWS, GCP, Azure, OpenStack, local QEMU) with per-region/zone catalogs
• Visual Template Composer in the React UI — drag modules onto a canvas, preview conflicts + footprint live, save with one click
• Fleet Dashboard — live event feed, correlation chains, drift queue, honeypot canary alerts, attribution feedback for failure analysis

On-node runtime

• Go agent (powernode-agent) — single static binary, ~20MB, replaces legacy bash scripts. Multi-cloud identity discovery (AWS/GCP/Azure/DigitalOcean/ libvirt fw-cfg), mTLS enrollment, OCI module pull, fs-verity verification, erofs+overlayfs union mount, heartbeat, task lease, cert rotation
• Multi-arch initramfs builder — produces six artifact families per arch (kernel+initramfs bundle, raw disk image, ISO, iPXE chainload, qcow2, bootc-compatible OCI) for both amd64 and arm64 in one CI run

Module supply chain

• Two-stage CI pipeline (Containerfile builder + erofs composer) that preserves the legacy rsync-glob composition layer while modernizing multistrap → mmdebstrap and mksquashfs → mkfs.erofs
• Cosign-signed OCI artifacts with Sigstore Fulcio (no long-lived signing keys; ephemeral OIDC-bound certs)
• Per-module trust policy (cosign_identity_regexp + cosign_issuer_regexp) pinning each module to its expected publisher

AI-driven autonomy

• 45 fleet sensors detecting silent instances, module drift, cert expiry, promotion readiness, config drift, SLO violations, honeypot canary access, external workload pressure (cross-extension stigmergic coordination), instance state drift, GitOps drift, package repository drift, project SLO breaches, disk-image publication failure streaks, SDWAN health (peer reachability, BGP session, VIP reachability, drift, credential expiry), and storage assignment drift
• 67 AI Skill executors spanning read-shape (concierge chat), fleet autonomy (drift remediation, CVE response, module composition, rolling upgrades), SDWAN topology composition + remediation, container runtime provisioning, package + module authoring, architecture catalog, federation, and platform deployment. See docs/SKILL_EXECUTORS.md for the catalog with descriptors and example I/O.
• FleetAutonomyService — gates every autonomous action through intervention policy + approval chain (auto_approve / notify_and_proceed / require_approval / blocked); the same operator approval queue every other extension's autonomy decisions flow through
• Learning loop — every confirmed/rejected operator decision feeds back into compound learnings that boost or downweight similar candidates next time
• Cross-domain stigmergic coordination — bidirectional pressure exchange with sibling extensions via the platform's signal bus
• Per-module consent budget — operators set a daily ceiling on autonomous decisions per module; exhausted budget forces require_approval regardless of policy

Observability

• System::FleetEvent persistent log (90-day routine retention, 365-day critical retention) with per-tick correlation IDs
• SystemFleetChannel ActionCable broadcast for live UI updates
• Compliance snapshot — audit-grade JSON document of every node, instance, module digest, certificate, CVE exposure, drift state, and autonomy decision

---

Requirements

To operate this extension you need a running Powernode platform installation. See the parent platform repo for installation instructions.

This extension contributes:

• Rails models under System:: and Sdwan::, services, and controllers across the operator API, the on-node API and the worker API
• A React/TypeScript frontend: page components, reusable components, custom hooks and API client services
• Worker jobs (12 scheduled): system_task_reaper, out_of_band_exec_reaper, system_fleet_reconcile, system_cve_feed, system_cve_responder_reconcile, system_fleet_event_retention, sdwan_flow_sample_retention, system_gitops_sync, system_cloud_sync, system_fulfillment_request_reconcile, system_package_repository_sync, system_platform_health_sweep
• Database migrations under server/db/migrate/
• Sidekiq cron schedule entries

This list gives no file counts, because hand-kept counts drift. For current figures, use the sources that are generated or checked against the code:

• Skill executors: docs/SKILL_EXECUTOR_CATALOG.md, generated by the platform's rails system:skills:generate_catalog
• MCP actions: the platform's generated catalog, docs/reference/auto/mcp-tools.md in the platform repo
• Fleet sensors: docs/FLEET_SENSORS.md, whose roster server/spec/docs/reference_counts_spec.rb checks against the code

The three figures this README does state (fleet sensors, skill executors and scheduled jobs) are checked the same way, by server/spec/docs/readme_counts_spec.rb.

---

Layout

extensions/system/
├── server/                 # Rails models, services, controllers, specs
├── frontend/               # React TypeScript surface
├── worker/                 # Sidekiq job classes
├── agent/                  # Go on-node agent (powernode-agent)
├── initramfs/              # Multi-arch boot artifact builder
├── templates/
│   └── module-repo/        # Canonical module-source layout
└── docs/
    ├── ARCHITECTURE.md     # Subsystem reference
    ├── tutorials/          # Numbered learning sequence (01-first-boot → 12-disk-image-ci)
    ├── runbooks/           # Step-by-step operator guides
    ├── history/            # Archived phase plans + acceptance reports
    └── …                   # Domain-specific reference (see Documentation)

---

Documentation

Reference

Doc	What it covers
docs/ARCHITECTURE.md	8 subsystems, threat model, state machines, API surfaces
docs/USE_CASE_MATRIX.md	What works / doesn't / what to expect for 10 NodeInstance container scenarios — READ FIRST when designing a deployment
docs/CONTAINER_RUNTIMES.md	Phase 1 Docker + Phase 2 K3s lifecycle + operator troubleshooting
docs/SKILL_EXECUTORS.md	Skill executor catalog (descriptors and example I/O)
docs/FLEET_SENSORS.md	Fleet sensors + intervention policy reference table
docs/DISK_IMAGE_CI.md	Webhook + CI worker + OCI artifact pipeline
docs/MCP_API_REFERENCE.md	All system_* / system_sdwan_* / kubernetes_* / docker_* MCP tool actions

Operator runbooks

See docs/runbooks/README.md for the full index with audience + prerequisites + runtime per runbook. Highlights:

Runbook	Goal
docs/runbooks/node-provisioning.md	Full Node + NodeInstance lifecycle (create → enroll → drain → decommission) with per-AASM-state error recovery
docs/runbooks/sdwan-network-setup.md	SDWAN end-to-end: networks, peers, VIPs, firewall, route policies, BGP, federation
docs/runbooks/module-authoring.md	Author + register + sign + publish a new NodeModule
docs/runbooks/cve-response.md	Full CVE response workflow with SBOM-aware matching
docs/runbooks/gitops-reconciliation.md	Operator workflow for GitOps reconciler: fleet.yaml, sync, proposal review, apply
docs/runbooks/acme-issuance.md	ACME DNS-01 cert lifecycle: provider setup, issue, renew, revoke, endpoint failover
docs/runbooks/acme-smoke.md	P2.5.7 acceptance smoke test (6 scenarios)
docs/runbooks/instance-pool-tuning.md	Pool sizing, reaping, draining, troubleshooting
docs/runbooks/multi-cluster-k3s.md	Multi-cluster K3s with target_cluster_id (an HA control plane is NOT IMPLEMENTED — see the runbook banner)
docs/runbooks/disk-image-ci.md	Disk image build + signing + publication operator workflow
docs/runbooks/federation-setup.md	Multi-region / multi-account federation peering
docs/runbooks/federation-troubleshooting.md	Diagnosing federation peering failures
docs/runbooks/vault-credential-restoration.md	DR runbook for credential restoration

Tutorials

docs/tutorials/ — numbered, dependency-aware learning sequence (beginner → advanced):

#	Tutorial	What it teaches
01	First boot (single-node QEMU)	Catalog seed, kernel + initrd, local QEMU, agent enrollment
02	Your first custom module	manifest.yaml, rsync globs, Containerfile, cosign-keyless
03	Container runtime — Docker	docker-engine module, mTLS handshake, SDWAN binding
04	Container runtime — K3s cluster	k3s-server/agent, VIP-backed api_endpoint, multi-node join
05	Multi-cluster K3s + SDWAN isolation	target_cluster_id, per-tenant SDWAN, trust boundary
06	Fleet-atomic module upgrade	rolling_module_upgrade (plan only), why the upgrade cannot be staged, the manual pointer-flip procedure
07	CVE response end-to-end	ExposureCalculator, CveResponseExecutor, remediation plan (the remediation step is NOT IMPLEMENTED)
08	Instance pools for bursty batch	InstancePool, atomic acquire, reaper auto-replenishment
09	Honeypot canaries	mark_canary, HoneypotAccessSensor, intervention policy
10	GitOps-managed fleet	fleet.yaml, sync cycle, proposal review
11	Multi-region federation	Spawn modes, P9.x data residency + WORM audit + schema negotiation
12	Disk image CI publication	DiskImageWebhook, CI worker, signed OCI artifact, retention

Start with docs/tutorials/INDEX.md for a Mermaid decision tree mapping your goal to a starting tutorial.

Subsystem-specific

• docs/SMOKE_TEST.md — platform-level smoke catalog (18 seeded scripts across 8 passes; covers boot, container runtimes, SDWAN, federation, ACME, storage, credentials, hardware/CI extras)
• docs/credential-restoration.md — Vault transit credential design
• docs/agent-peering.md — NodeInstance-as-Agent design
• docs/agent-internals.md — Go agent package-by-package reference (23 internal packages)
• docs/gitops.md — GitOps reconciler design
• docs/history/ — archived phase plans + acceptance reports
• initramfs/README.md — multi-arch boot artifact builder
• agent/README.md — Go agent build + 17 subcommands (architecture in docs/agent-internals.md)
• templates/module-repo/README.md — canonical module-source layout

---

Development

# Inside the Powernode platform working tree, where this extension is mounted
# as a submodule at extensions/system/

cd extensions/system

# Backend specs
cd server && bundle exec rspec

# Frontend type-check
cd ../frontend && npx tsc --noEmit

# Go agent tests
cd ../agent && go test ./...

When working on this extension, always commit inside extensions/system/ first, then update the submodule pointer in the parent platform repo. See CONTRIBUTING.md for the full submodule workflow.

---

License

MIT — see LICENSE. Code of Conduct: see CODE_OF_CONDUCT.md.

---

Community

• GitHub Discussions — nodealchemy/powernode-platform/discussions for questions, general inquiries, technical support, and commercial/enterprise inquiries
• GitHub issues — nodealchemy/powernode-system/issues for bugs + feature requests
• Security vulnerabilities — report a private advisory; see SECURITY.md
• Code of Conduct — see CODE_OF_CONDUCT.md
• X / Twitter — @nodealchemy for general updates and informal questions

---

Status

Active development. Spec coverage: 1,430 examples / 0 failures (as of 2026-05-04). The Golden Eclipse roadmap (Track A through Track F) is substantially complete on backend + autonomy axes; frontend operator surface covers M-FE-1 (Visual Composer) and M-FE-3 (Fleet Dashboard, with Boot Replay viewer in active sweep).

Recent shipments (May 2026)

• Slice 3 — first-class Sdwan::VirtualIp (primary + failover holders) backing the cluster api_endpoint. Failover between holders is real for ordinary peers; for the API VIP there is no second k3s-server holder to fail over to, because an HA control plane is NOT IMPLEMENTED (a second server bootstraps a separate cluster — see docs/runbooks/multi-cluster-k3s.md)
• Slice 7 — pre-warmed System::InstancePool with atomic acquisition
  • reaper auto-replenishment; cuts ephemeral provisioning latency from 5–10 min cold-boot to <30 s claim
• Slice 9 (a–f) — static subnet routing, first-class VIPs, iBGP/FRR, comprehensive frontend, observability/autonomy, route policies (JSONB statements compiled to FRR route-map + prefix-list/as-path-list)
• Slice 10 — config-variety dockerd daemon.json overrides via dependant module hierarchy (per-node + per-instance customization without rebuilding the base module)
• Phase 2 K3s — full container runtime stack: cluster provisioner, agent reconciler state machine, module catalog seed, multi-cluster join validation, and multi-cluster worker placement via target_cluster_id (IMP-a5f236e8cc56)
• Phase 1 Docker — managed Devops::DockerHost with InternalCaService TLS provisioning + cascade-FK decommission

Milestones complete

• M0 — Foundation contracts + legacy spec porting (BootstrapToken, NodeCertificate, ModuleArtifact, mTLS, AASM)
• M2 — Go agent v0 (~4,400 LOC across 9 packages including security)
• M3 — Multi-arch image builder (six artifact families × amd64/arm64)
• M4 — QEMU thin slice (LocalQemuProvider with Libvirt/Recorder/Disabled runner triplet, virtio-fw-cfg seed, 15-spec integration coverage)
• M5 — MCP CRUD surface (SystemFleetTool, ~25 actions, per-action permission gates)
• M6 — AI Skills catalog (8 executors)
• M7 — FleetAutonomyService (gate_action!, 8 sensors at M7 completion, DecisionEngine, approval chains). For today's sensor roster, see docs/FLEET_SENSORS.md.
• M8 — Compound learning extraction (LearningExtractor wired into the tick loop). Its auto-evolve trigger was deleted: it called the platform's auto_evolve_skill verb with no user and no agent, which the verb refuses for lack of ai.skills.update, so it never mutated a skill. Skill evolution goes through that verb's own gated door.

Active sweep (May 2026)

Per-account encryption key restoration via Vault transit, SBOM-aware CVE matching (M-D2-2), GitOps reconciliation (M-D2-3), NodeInstance-as-Agent peer registration (F-3), Boot Replay viewer (M-FE-3 completion), Module Marketplace skeleton (M-FE-2), AI Concierge chat (M-FE-4). For historical phase tracking see docs/history/TASKS.md.

---

Related

• Powernode platform — the parent platform that mounts this extension
• Cosign — module signing
• erofs — verified-mount lower layer (fs-verity-checked)
• oras — OCI artifact tooling