Skip to content

CONTAINER_RUNTIMES.md

Latest commit

 

History

History
438 lines (325 loc) · 25 KB

CONTAINER_RUNTIMES.md

File metadata and controls

438 lines (325 loc) · 25 KB
Raw

Container Runtimes — Operator Guide

Status: active

System extension support for managed Docker daemons (Phase 1) and K3s Kubernetes clusters (Phase 2). kubeadm modules are NOT IMPLEMENTED (server/db/seeds/k3s_modules.rb and the runtime handshake still name a "Phase 3" for them), and an HA control plane is NOT IMPLEMENTED with K3s HA parked — a second k3s-server bootstraps a separate cluster rather than joining the first (see runbooks/multi-cluster-k3s.md, Phase 4).

Architecture (one-paragraph summary)

A NodeInstance with the appropriate runtime module assigned (docker-engine, k3s-server, or k3s-agent) auto-bootstraps the daemon on the next agent heartbeat tick. The agent installs the binary, generates local TLS material (Docker) or captures k3s-generated state (K3s), and posts a runtime/handshake to the platform. The platform creates the corresponding Devops::DockerHost or Devops::KubernetesCluster row. All daemon API traffic flows over the SDWAN overlay /128 — no public daemon sockets.

Lifecycle Diagram (Phase 1 Docker)

sequenceDiagram
    actor Op as Operator
    participant Agent as powernode-agent
    participant Plat as Platform
    participant CA as InternalCaService

    Op->>Plat: 1. assign docker-engine<br/>module to Node
    Plat-->>Agent: heartbeat picks up<br/>new assignment
    Agent->>Agent: 2. install docker-ce<br/>generate Ed25519 server keypair
    Agent->>Plat: 3. POST /runtime/handshake<br/>runtime=docker phase=wants_cert<br/>csr_pem=...
    Plat->>CA: 4. sign CSR
    CA-->>Plat: signed cert + chain
    Plat->>Plat: create managed<br/>Devops::DockerHost<br/>bound to instance
    Plat-->>Agent: 5. cert returned
    Agent->>Agent: 6. write daemon.json with TLS<br/>+ listen on SDWAN /128<br/>systemctl start docker
    Agent->>Plat: 7. POST phase=ready<br/>version=25.0.3
    Plat->>Plat: 8. host status:<br/>pending → connected
    Op->>Plat: 9. docker_list_containers
    Plat->>Agent: over SDWAN /128 (mTLS)
    Agent-->>Op: container list
Loading

Lifecycle Diagram (Phase 2 K3s with VIP failover)

sequenceDiagram
    actor Op as Operator
    participant A1 as Agent<br/>(bootstrap server)
    participant Plat as Platform
    participant VIP as Sdwan::VirtualIp
    participant A2 as Agent<br/>(worker)

    Op->>Plat: 1. assign k3s-server module
    Plat-->>A1: heartbeat picks up
    A1->>A1: 2. apt install k3s,<br/>systemctl start k3s
    A1->>A1: 3. k3s generates kubeconfig<br/>+ tokens
    A1->>Plat: 4. POST phase=bootstrap
    Plat->>VIP: allocate api_endpoint VIP<br/>(slice 3)
    VIP-->>Plat: VIP /128 + holders=[A1]
    Plat->>Plat: 5. create Devops::KubernetesCluster<br/>+ bootstrap node row<br/>status=bootstrapping<br/>api_endpoint=VIP /128
    A1->>Plat: 6. POST phase=ready
    Plat->>Plat: cluster status=active

    Op->>Plat: 7. assign k3s-agent module
    Plat-->>A2: heartbeat picks up
    A2->>Plat: 8. POST phase=join_request
    Plat-->>A2: 9. api_endpoint VIP /128 + token
    A2->>A2: 10. write systemd drop-in<br/>K3S_URL + K3S_TOKEN<br/>systemctl start k3s-agent
    A2->>Plat: 11. POST phase=ready
    Plat->>Plat: node status=active

    Op->>Plat: 12. download kubeconfig

    note over VIP,A1: If bootstrap server drains:<br/>VIP migrates to next holder<br/>workers' K3S_URL keeps resolving
Loading

Multi-cluster routing via target_cluster_id — IMPLEMENTED (IMP-a5f236e8cc56)

Putting a worker in a chosen cluster works. Set target_cluster_id in the k3s-agent module assignment's config; the platform surfaces it back to that node on runtime/k3s_agent/config (only when it names a Devops::KubernetesCluster in the node's own account whose status isn't error), k3sd.AgentManager.TargetClusterID is refreshed from that endpoint each tick — via a NEW channel, k3sd.HTTPAgentConfigClient, separate from k3sd.ModulesAPI.AssignedModules(ctx) ([]string, error) (still module names only) — and is consumed in JoinRequest(ctx, m.TargetClusterID). If it's omitted, or names a cluster the platform can't back (foreign-account, or error status), the single-cluster fallback still applies: with exactly one non-error cluster the join resolves without a target; with more than one the platform refuses rather than guessing: AmbiguousClusterError, with system.k3s_ambiguous_cluster_join_refused emitted at severity high (kubernetes_cluster_provisioner_service.rb:329) and 409 returned. The worker never joins the wrong cluster in that case either; no node is produced at all.

Single-cluster accounts are unaffected either way — with exactly one non-error cluster the join is unambiguous and succeeds without a target.

The diagram below is now current behaviour, not just intent:

flowchart TB
    Op[Operator]
    subgraph A["Account"]
        T1[Template: cluster-a-worker<br/>module: k3s-agent<br/>config.target_cluster_id = C1]
        T2[Template: cluster-b-worker<br/>module: k3s-agent<br/>config.target_cluster_id = C2]
        subgraph C1["KubernetesCluster C1"]
            S1[k3s-server #1]
            W1[k3s-agent worker]
        end
        subgraph C2["KubernetesCluster C2"]
            S2[k3s-server #2]
            W2[k3s-agent worker]
        end
    end
    Op --> T1 --> W1
    Op --> T2 --> W2
    W1 -. "joins via VIP" .-> S1
    W2 -. "joins via VIP" .-> S2
Loading

Historical — corrected 2026-09-17. Before IMP-a5f236e8cc56, the agent had no producer for target_cluster_id and this section said so. Kept visible for an operator who read an earlier revision:

Withdrawn claim What is actually true
"In accounts with more than one cluster, k3s-agent module assignments must carry metadata.target_cluster_id so the agent joins the right cluster" Required by the platform, and impossible to supply from the agent. Setting it on the assignment changes nothing about the join.
"Without target_cluster_id, the agent silently joins the most-recently-created active cluster the API returns" There is no most-recently-created fallback for a multi-cluster account. resolve_membership_cluster! (kubernetes_cluster_provisioner_service.rb:351) auto-selects only when there is exactly one candidate; with more than one it raises AmbiguousClusterError and the join fails. "Candidate" is where.not(status: "error"), so a cluster in pending, bootstrapping, degraded or disconnected counts toward the ambiguity, not just an active one.
"Known gap (2026-06-03): there is no server-side validation guard that rejects an ambiguous join when target_cluster_id is omitted in an account with more than one cluster" That guard shipped. The omitted-ID case is precisely the one that is now guarded: resolve_membership_cluster! emits and raises rather than guessing. What is missing is the agent-side producer, not the server-side check.
"remove the metadata to fall back to join most recent active cluster" (Troubleshooting) Removing it is what triggers the refusal. There is no most-recent fallback once a second non-error cluster exists, and nothing on the assignment reaches the agent to remove.

Module Catalog

Module Variety Packages Purpose
docker-engine subscription docker-ce, docker-ce-cli, containerd.io, docker-buildx-plugin, docker-compose-plugin Docker Engine binary install
k3s-server subscription k3s (single binary; bundled containerd + runc) K3s control plane
k3s-agent subscription k3s (same binary; agent mode) K3s worker node

All three live in the Container Runtimes NodeModuleCategory (position=70, between Network Overlay at 60 and userland at 90+).

Provisioning a Docker Daemon

Prerequisites

  • NodeInstance must have at least one Sdwan::Peer with an assigned overlay /128. The daemon binds to that address only; provisioning errors with MissingSdwanPeerError if no peer is attached.
  • The instance's Node must have the docker-engine module assigned.

Operator Path A: assign module + wait

# 1. Assign the module via existing API (admin UI or MCP)
curl -X POST /api/v1/system/node_module_assignments \
  -H "Authorization: Bearer $JWT" \
  -d '{"node_id": "<node-uuid>", "node_module_name": "docker-engine"}'

# 2. Wait ~60s for agent reconcile loop. The managed host appears at:
curl /api/v1/system/managed_docker_hosts -H "Authorization: Bearer $JWT"

Operator Path B: MCP action (single call)

platform.system_provision_docker_runtime({
  node_instance_id: "0193cdef-..."
})
// → { success: true, host: { id, name, status, api_endpoint, ... } }

This path is what the System Concierge agent invokes when an operator chats "provision docker on instance X".

Verifying

# Via REST
curl /api/v1/system/managed_docker_hosts/<host-id> -H "Authorization: Bearer $JWT"

# Via MCP — list managed hosts (excludes external operator-registered hosts)
platform.system_list_managed_docker_hosts()

# Inspect containers running on the host (uses encrypted SDWAN overlay)
platform.docker_list_containers({ host_id: "<host-id>" })

Provisioning a K3s Cluster

Step 1: Bootstrap the cluster

Assign k3s-server to the first NodeInstance. The agent installs k3s, captures kubeconfig + tokens from /etc/rancher/k3s/k3s.yaml + /var/lib/rancher/k3s/server/node-token, and posts phase=bootstrap. Cluster appears within ~60s.

platform.kubernetes_list_clusters()
// → { clusters: [{ id, name, flavor: "k3s", status: "bootstrapping", ... }] }

Step 2: Add workers

Assign k3s-agent to additional NodeInstances. The agent posts phase=join_request, gets the cluster's api_endpoint + agent_token, writes a systemd drop-in at /etc/systemd/system/k3s-agent.service.d/override.conf with K3S_URL + K3S_TOKEN, then starts k3s-agent.service.

Step 3: Use kubectl

Download the kubeconfig from the platform:

platform.kubernetes_get_kubeconfig({ cluster_id: "<id>" })
// → { kubeconfig: "apiVersion: v1...", api_endpoint: "https://[fd00::]:6443" }

Or via UI: /app/devops/containers/kubernetes → cluster card → "kubeconfig" button.

# Save + use
echo "$KUBECONFIG_YAML" > ~/.kube/powernode-cluster.yaml
kubectl --kubeconfig ~/.kube/powernode-cluster.yaml get nodes

The kubectl traffic flows over the SDWAN overlay — operators must be on the same SDWAN network or have a federation route.

Decommissioning

Docker daemon

Three operator vectors:

// 1. MCP — destroys managed host row + Vault TLS material
platform.system_decommission_docker_runtime({ host_id: "<host-id>" })

// 2. Unassign the docker-engine module from the Node
//    Agent sees module gone, stops dockerd, posts phase=stopped
//    Platform marks host status=disconnected (but keeps row)

// 3. Terminate the NodeInstance — cascade nullifies node_instance_id
//    Host row goes orphan; operator can clean via MCP

K3s cluster

platform.kubernetes_decommission_cluster({ cluster_id: "<id>" })
// → cascade-deletes all member KubernetesNode rows; underlying
//   NodeInstances are NOT terminated. Operator must separately
//   unassign k3s-server / k3s-agent modules to fully clean up.

Troubleshooting

"MissingSdwanPeerError" during provision

The NodeInstance has no SDWAN peer with an assigned /128. Attach one via:

platform.system_sdwan_attach_peer({
  network_id: "<sdwan-net-id>",
  node_instance_id: "<instance-id>"
})

Daemon stays in pending status

The agent has provisioned the host row but hasn't yet reported phase=ready. Common causes:

  • docker-ce package install in progress (~30-60s on first run)
  • TLS cert hasn't been written yet
  • systemd unit fails to start (check via SSH: journalctl -u docker.service -n 50)

K3s cluster stays in bootstrapping status

The bootstrap node has installed k3s but the agent hasn't captured + posted state yet. Common causes:

  • K3s bootstrap takes ~30-60s; check /etc/rancher/k3s/k3s.yaml exists on the node
  • Server token file /var/lib/rancher/k3s/server/node-token not yet populated
  • The agent's heartbeat tick interval (default 30s) hasn't fired yet

kubernetes_get_kubeconfig returns 422 "not yet available"

encrypted_kubeconfig is blank because the cluster is still bootstrapping. Wait for cluster_status: active before retrieving.

Docker daemon TLS verification fails

Symptoms: platform.docker_list_containers returns x509: certificate signed by unknown authority or tls: bad certificate. Common causes:

  • Operator's local truststore isn't the issue — the platform manages mTLS internally; the API call from the platform to the daemon uses Vault-stored client certs.
  • Server cert was rotated but agent didn't pick it up: rotate Docker daemon TLS via the broader system.cert_rotate flow (there is no dedicated runtime_docker_tls_rotate action — it was removed in the 2026-05-19 audit; no executor backed it), then re-test after one heartbeat tick.
  • Server cert was minted by a different InternalCaService root than the platform now trusts (rare; only happens after CA replacement). Decommission via system_decommission_docker_runtime and re-provision; the new host gets a fresh cert from the current CA.
  • On the node itself: journalctl -u docker.service | grep -i tls shows the actual handshake error.

Docker daemon listens on the wrong address

Symptoms: system_provision_docker_runtime succeeds but daemon connections fail. The daemon.json hosts array should contain tcp://[<sdwan-/128>]:2376. If it doesn't:

  • Check the agent's reconciler state cache: cat /var/lib/powernode-agent/reconciler.json on the node.
  • Verify the SDWAN peer is up: wg show wg-pn on the node should show a recent latest handshake.
  • The Sdwan::Peer.host_address is the source of truth — confirm it via platform.system_sdwan_get_peer({ peer_id: '<peer-id>' }).

daemon.json overrides not applied (slice 10)

Slice 10 introduced config-variety dockerd modules with per-node + per-instance overrides. If overrides aren't being picked up:

  • Verify the override module is assigned to a higher-priority slot than the base docker-engine module — overrides require greater effective_priority per the dependant-modules pattern.
  • After assignment, the agent re-renders daemon.json on its next reconcile tick (~30 s). To force immediate: systemctl restart powernode-agent on the node.
  • Layer ordering visible in cat /etc/docker/daemon.json after reconcile — keys present in the override module win over the base.

K3s agent can't join cluster

Symptoms: agent posts phase=join_request but cluster fails to add it; agent log shows K3S_URL connection refused or bad token.

  • api_endpoint mismatch: K3s api_endpoint uses an SDWAN VIP (slice 3). If the worker isn't on the same SDWAN network as the bootstrap node, the VIP is unreachable. Confirm with system_sdwan_list_peers that both peers are on the same network.
  • Token mismatch (rare): platform regenerated the join token but the agent has a stale cache. Force re-fetch by removing the systemd drop-in /etc/systemd/system/k3s-agent.service.d/override.conf and restarting powernode-agent.
  • Multi-cluster refusal: with more than one non-error cluster in the account, a join carrying no usable target_cluster_id is refused, not misrouted — the handshake returns 409 (AmbiguousClusterError) and system.k3s_ambiguous_cluster_join_refused is emitted at severity high. Set target_cluster_id in the k3s-agent module assignment's config to the intended cluster's id (IMP-a5f236e8cc56) — it now reaches the agent — and confirm it resolves via kubernetes_get_cluster before retrying; allow one agent tick for the fetch to pick it up. See Multi-cluster routing for the full mechanism and the withdrawn instructions it replaced.

kubelet logs unavailable

To retrieve kubelet logs for a managed K3s node:

# Via SSH (node must be reachable on SDWAN /128)
journalctl -u k3s.service -n 200             # k3s-server
journalctl -u k3s-agent.service -n 200        # k3s-agent

There is no run-arbitrary-command-on-an-instance MCP action — SSH (above) is the primary path for ad-hoc log retrieval. To drive instance lifecycle through MCP instead, use system_provision_instance (and system_terminate_instance), follow the returned task with system_get_task (status, progress, error_message), and read the instance's FleetEvents with system_recent_signals (exact kind, e.g. system.instance_silent, or a correlation_id). Not the introspection verb recent_events: it reads agent execution events (Ai::ExecutionEvent) and never returns a FleetEvent.

CNI selection (Phase O4 — shipped)

K3s clusters today support two CNI plugins, auto-selected from the bootstrap NodeInstance's network_profile and overridable explicitly. The KubernetesClusterProvisionerService enforces the choice + raises CniProfileMismatchError when an operator-supplied override would be incompatible with the host profile.

network_profile Auto-default CNI Rationale
heavyweight (≥4 GB RAM, x86/Pi5/server-class) ovn_kubernetes Headroom for OVN-controller + OVN-K8s; supports encrypted pod-to-pod via OVN tunnels
lightweight (constrained / Pi-class) flannel Bundled with K3s; no OVN footprint
(anything else / unset) flannel (DEFAULT_CNI_PLUGIN) Conservative fallback

Explicit operator override (passed at bootstrap via the cni_plugin argument):

platform.system_provision_instance({
  node_id: "<bootstrap-node-id>",
  // ... other args ...
  // For K3s bootstrap, the provisioner reads cni_plugin from the
  // template/operator hint and validates against network_profile.
})

If the operator-supplied cni_plugin is ovn_kubernetes and the bootstrap node has network_profile: "lightweight", the service raises CniProfileMismatchError rather than silently degrading. Same check runs at join time: a KubernetesNode whose profile mismatches the cluster's CNI is refused with the same error class. The valid cni_plugin set is the union of the values in NETWORK_PROFILE_TO_CNI plus DEFAULT_CNI_PLUGIN (i.e., flannel, ovn_kubernetes).

Pod-to-pod encryption posture:

  • flannel (default for lightweight) — VXLAN over the host's primary NIC by default. Optional encryption via SDWAN overlay: when the cluster's Sdwan::Network has pod_subnet_prefix set, the provisioner stamps the cluster's bootstrap config with --flannel-iface=<the host's wg device> (wg-sdwan-<short_id>; handle form only when the network allocates no HostVrfAssignment), --flannel-backend=host-gw, and --cluster-cidr=<pod_subnet_prefix> so flannel runs in host-gw mode bound to the SDWAN WireGuard interface. Pod traffic between nodes then flows through the existing WireGuard tunnels via the AllowedIPs covering the SDWAN /64. No VXLAN encapsulation, no MTU computation, no nested-header fragmentation — the kernel's per-node /24 routes (installed by flannel host-gw from the K8s API) point at the other node's overlay IP, which WG already routes correctly.
  • ovn_kubernetes (default for heavyweight) — pod traffic flows over OVN tunnels with native encryption support; pairs cleanly with the SDWAN overlay for hub-to-hub paths. pod_subnet_prefix is ignored on ovn-K8s clusters (OVN owns its own pod-network layer); the provisioner emits a system.cluster_bootstrap.pod_subnet_prefix_ignored warning event when the field is set on an ovn-K8s path.

Routing pod traffic over SDWAN (pod_subnet_prefix — shipped 2026-05-19)

Operator workflow:

  1. Declare the pod CIDR on the SDWAN network at create time (or update an existing network — but see "Migration" note below):

    platform.system_sdwan_create_network({
  name: "tokyo-edge",
  pod_subnet_prefix: "10.42.0.0/16",   // RFC1918 pod CIDR — flannel default size
  // other fields per usual
})

    The platform validates that pod_subnet_prefix does NOT overlap (a) the SDWAN /64, (b) any peer's lan_subnets, (c) any VIP /128, (d) any other network's pod_subnet_prefix in the same account.

  2. Bootstrap the k3s cluster with cni_plugin: "flannel" on a NodeInstance attached to the network. The provisioner detects the network's pod_subnet_prefix, stamps the cluster's metadata["pod_cidr"] + metadata["sdwan_network_id"], and emits a SubnetAdvertisement(source: "pod_subnet") row.

  3. The agent receives the new flannel args via the bootstrap_config endpoint on its next heartbeat:

    • flannel_iface: "wg-sdwan-<short_id>" (resolved per host; handle form only with no HostVrfAssignment)
    • flannel_backend: "host-gw"
    • cluster_cidr: "<pod_subnet_prefix>"

    The agent appends --flannel-iface --flannel-backend=host-gw --cluster-cidr to its k3s install args. Subsequent k3s installs (idempotent on content_hash) carry the new posture.

Migration model for existing clusters: setting pod_subnet_prefix on a network that already has running flannel clusters triggers the agent's reconcile loop to re-install k3s with the new flags on next heartbeat. This causes a brief (~30–60 s per node) pod-network outage as k3s restarts. The operator's act of setting the field IS the explicit opt-in; the platform doesn't auto-migrate without operator action.

Routing topology trade-off:

Network routing_protocol Pod-traffic path
static (default — hub-and-spoke) Spoke A → Hub → Spoke B (hub-hairpinned)
ibgp (slice 9c) Spoke A → Spoke B directly (FRR-advertised /24s)

iBGP is the recommended posture for clusters with non-trivial pod-to-pod traffic; static-mode hub-hairpinning works but adds a round-trip per pod packet.

Immutability: once a Devops::KubernetesCluster references the network, pod_subnet_prefix cannot be changed (k3s pod CIDR is immutable post-bootstrap — same constraint as cni_plugin). Destroy + rebuild the cluster to migrate to a different pod CIDR.

See USE_CASE_MATRIX.md use cases 9 + 10 for the operator-facing summary; runbooks/multi-cluster-k3s.md for per-tenant pod_subnet_prefix isolation patterns; tutorials/04-k3s-cluster.md §"Pod traffic over SDWAN" for the live operator walkthrough.

Daemon can't pull from registry

Symptoms: docker pull fails with timeout or connection refused.

  • Most operators use a registry mirror co-located on the SDWAN. Configure via dependant module override: 
    # daemon-json-override module
hosts: ["tcp://[<sdwan-/128>]:2376", "unix:///var/run/docker.sock"]
registry-mirrors: ["https://registry.<sdwan-domain>"]
  • For pulls from registry.example.com (Powernode Gitea container registry), the agent injects credentials into ~/.docker/config.json automatically when the node has a valid Vault token.

Source Files

Backend (parent repo):

  • server/db/migrate/20260505000100_add_node_instance_to_devops_docker_hosts.rb
  • server/db/migrate/20260505000200_create_kubernetes_cluster_management_tables.rb
  • server/app/models/devops/docker_host.rb (managed/external state machine)
  • server/app/models/devops/kubernetes_cluster.rb, kubernetes_node.rb
  • server/app/services/ai/tools/docker_provisioning_tool.rb
  • server/app/services/ai/tools/kubernetes_cluster_tool.rb
  • server/app/services/ai/tools/kubernetes_provisioning_tool.rb
  • server/app/controllers/api/v1/devops/kubernetes/clusters_controller.rb

Backend (system extension):

  • extensions/system/server/app/services/system/docker_daemon_provisioner_service.rb
  • extensions/system/server/app/services/system/kubernetes_cluster_provisioner_service.rb
  • extensions/system/server/app/controllers/api/v1/system/node_api/runtime_controller.rb
  • extensions/system/server/db/seeds/docker_runtime_module.rb
  • extensions/system/server/db/seeds/k3s_modules.rb
  • extensions/system/server/db/seeds/smoke_test_docker_runtime.rb (live smoke)
  • extensions/system/server/db/seeds/smoke_test_k3s_runtime.rb (live smoke)

Agent (Go):

  • extensions/system/agent/internal/dockerd/ — handshake.go, manager.go, applier.go, shell_applier.go, modules.go (Phase 1)
  • extensions/system/agent/internal/k3sd/ — handshake.go, server_manager.go, agent_manager.go, applier.go, shell_applier.go (Phase 2)
  • extensions/system/agent/internal/runtime/service.go — chains both reconcilers into PostSend after sdwanMgr

Frontend:

  • frontend/src/features/devops/docker/pages/DockerHostsPage.tsx (Managed badge)
  • frontend/src/features/devops/kubernetes/ (full hub)
  • frontend/src/pages/app/devops/KubernetesHubPage.tsx

Related Docs

  • SKILL_EXECUTORS.md — docker_provision + provision_cluster skill executors
  • ARCHITECTURE.md — Container Runtimes subsystem entry
  • MCP_API_REFERENCE.md — operator-curated system_* / kubernetes_* / docker_* action subset (the parent platform's full machine-generated MCP_TOOL_CATALOG.md is gitignored — regenerate with cd server && bundle exec rails mcp:generate_tool_catalog)

Last verified: 2026-06-03 (rev 2)