Skip to content

04-k3s-cluster.md

Latest commit

 

History

History
458 lines (376 loc) · 21.3 KB

04-k3s-cluster.md

File metadata and controls

458 lines (376 loc) · 21.3 KB
Raw

Tutorial 04 — Container runtime — K3s cluster

Status: active

What you'll learn: Provision a K3s control plane + worker on Powernode NodeInstances. The platform auto-bootstraps k3s-server, allocates a SDWAN-backed api_endpoint VIP (slice 3 — survives bootstrap-node loss via automatic failover), and joins a worker to it. This tutorial builds a single cluster, which is the case that works: with exactly one non-error cluster in the account a worker joins without a target. Choosing among several is NOT IMPLEMENTED — see Step 5.

Time: ~25 min (with two NodeInstances)

Builds on: Tutorial 03 — you understand the runtime handshake pattern. K3s uses the same machinery with different phase names (bootstrap and join_request instead of wants_cert).

Sets you up for: Tutorial 05 — Multi-cluster K3s + SDWAN isolation.

What you're building

sequenceDiagram
    actor Op as Operator
    participant Plat as Platform
    participant A1 as Agent<br/>(bootstrap server)
    participant A2 as Agent<br/>(worker)
    participant VIP as Sdwan::VirtualIp

    Op->>Plat: assign k3s-server module<br/>to bootstrap NodeInstance
    Plat-->>A1: heartbeat picks up assignment
    A1->>A1: apt install k3s,<br/>systemctl start k3s
    A1->>A1: k3s generates kubeconfig<br/>+ tokens
    A1->>Plat: POST phase=bootstrap
    Plat->>VIP: allocate api_endpoint VIP<br/>(slice 3)
    VIP-->>Plat: VIP /128 + holders=[A1]
    Plat->>Plat: create Devops::KubernetesCluster<br/>api_endpoint=VIP /128
    A1->>Plat: POST phase=ready
    Plat-->>Plat: cluster status=active

    Op->>Plat: assign k3s-agent module<br/>to worker NodeInstance
    Plat-->>A2: heartbeat picks up assignment
    A2->>Plat: POST phase=join_request
    Plat-->>A2: api_endpoint=VIP /128 + token
    A2->>A2: write systemd drop-in<br/>K3S_URL + K3S_TOKEN
    A2->>A2: systemctl start k3s-agent
    A2->>Plat: POST phase=ready
    Plat-->>Plat: node=active in cluster

    Op->>Plat: kubernetes_get_kubeconfig
    Op->>VIP: kubectl over SDWAN<br/>(via VIP /128)
Loading

By the end you'll have a working K3s cluster with one control-plane + one worker, addressable via a VIP that survives control-plane failover.

Concept refresher

Phase 2 K3s uses the same runtime/handshake machinery as Phase 1 Docker but with two new phases:

  • bootstrap — k3s installed, kubeconfig + tokens generated; platform creates the KubernetesCluster row
  • join_request — worker agent asks for the api_endpoint + token to join

The api_endpoint is a slice 3 first-class Sdwan::VirtualIp with the bootstrap server as its primary holder. When the bootstrap server is lost or drained, the VIP automatically migrates to another k3s-server-bearing peer in the cluster, and worker K3S_URL keeps resolving — no kubectl / operator action required. (Single-server cluster: VIP failover has nowhere to go; for production HA you want ≥2 servers.)

target_cluster_id (set in the k3s-agent module assignment's config) now reaches the agent (IMP-a5f236e8cc56) — see Step 5 for the mechanism. This tutorial's account has exactly one cluster, so a worker's join carries an empty target either way and that's unambiguous — the join succeeds without needing it. In an account with more than one non-error cluster, an unconfigured (or unresolvable) target still gets the platform's refusal — it refuses rather than guessing: AmbiguousClusterError, with system.k3s_ambiguous_cluster_join_refused emitted at severity high, and no node is produced at all. A correctly-configured target_cluster_id avoids that refusal by naming the intended cluster.

Prerequisites

Requirement How
Tutorial 03 worked You understand runtime handshake + SDWAN /128 binding
≥2 NodeInstances available Provision via Tutorial 01 if needed
Both instances have SDWAN peers on the same network system_sdwan_attach_peer
k3s-server + k3s-agent modules promoted to live Default in Tutorial 01 catalog
Operator permission system.kubernetes_provision Default for admins

Step 1 — Provision the bootstrap server

If you don't already have a NodeInstance ready:

platform.system_create_node({
  hostname: "k3s-server-1",
  node_template_id: "<base-or-k3s-server-template>",
  ...
})
platform.system_provision_instance({ node_id: ... })
platform.system_sdwan_attach_peer({ network_id, node_instance_id })

Expected outcome: instance is running and has an SDWAN /128.

Step 2 — Assign the k3s-server module

platform.system_assign_module_to_template({
  template_id: "<template-id>",
  // The template-assignment verbs take the NodeModule UUID, not its name — system_list_modules returns { id, name }.
  module_id: "<k3s-server-module-id>"
})

Expected outcome: within ~60s, the agent picks up the new assignment and installs k3s. The bootstrap handshake completes in ~30–60s after that.

Watch via the cluster row, not an event poll — nothing emits a system.k3s.* handshake FleetEvent (the only system.k3s* kind in the platform is system.k3s_ambiguous_cluster_join_refused, and the fleet event reader system_recent_signals filters by one exact kind, with no prefix filter):

platform.kubernetes_list_clusters()
// → { clusters: [{ id: "cluster-<uuid>", status: "bootstrapping", api_endpoint: "https://[fd00:abcd:1::100]:6443", ... }] }
//   The row appears when the agent posts phase=bootstrap (VIP allocated,
//   api_endpoint set) and moves to `active` when it posts phase=ready.

Step 3 — Verify the cluster

platform.kubernetes_list_clusters()
// → { clusters: [{
//      id: "cluster-<uuid>",
//      name: "k3s-cluster-1",
//      flavor: "k3s",
//      status: "active",
//      api_endpoint: "https://[fd00:abcd:1::100]:6443",
//      node_count: 1
//    }] }

Expected outcome: status: active. The api_endpoint is the VIP-backed /128, not the bootstrap server's instance /128 — that's what gives you slice 3 failover.

Step 4 — Provision the worker

platform.system_create_node({
  hostname: "k3s-worker-1",
  node_template_id: "<worker-template>",
  ...
})
platform.system_provision_instance({ node_id: ... })
platform.system_sdwan_attach_peer({ network_id, node_instance_id })

Step 5 — Assign k3s-agent (single-cluster accounts)

This tutorial's account has exactly one cluster, so the worker joins it without a target:

platform.system_assign_module_to_template({
  template_id: "<worker-template>",
  module_id: "<k3s-agent-module-id>"
})

✅ Choosing a cluster with target_cluster_id is IMPLEMENTED (IMP-a5f236e8cc56)

This tutorial's account has exactly one cluster, so Step 5 above doesn't need it — but if you're following this tutorial as a template for a multi-cluster account, pass config: { target_cluster_id: "<cluster-id-from-step-3>" } on the system_assign_module_to_template call instead, and it now reaches the node: k3sd.AgentManager.TargetClusterID is declared and consumed (JoinRequest(ctx, m.TargetClusterID)), and is refreshed each tick — before Reconcile — from k3sd.HTTPAgentConfigClient, a NEW channel separate from k3sd.ModulesAPI.AssignedModules(ctx) ([]string, error) (which still hands the reconcilers module names only). The server half was already real: handle_join_request forwards params[:target_cluster_id].presence into join_request! (runtime_handshake_handlers.rb:164), so a value that arrived was always honoured — the gap was purely that no value ever arrived. See runbooks/multi-cluster-k3s.md, Phase 3 for the full worked multi-cluster procedure.

Withdrawn claim What is actually true
"metadata.target_cluster_id is the mandatory tag on k3s-agent assignments in multi-cluster accounts" Required by the platform, and impossible to supply from the agent. Setting it on the assignment changes nothing about the join.
"without it, the agent picks the first cluster it finds" There is no picks-the-first fallback for a multi-cluster account. resolve_membership_cluster! (kubernetes_cluster_provisioner_service.rb:351) auto-selects only when there is exactly one candidate; with more than one it raises AmbiguousClusterError and the join fails. "Candidate" is where.not(status: "error"), so a cluster in pending, bootstrapping, degraded or disconnected counts toward the ambiguity, not just an active one.
"most common cause is wrong target_cluster_id. The agent tries to join and gets a 404" (Troubleshooting) The agent never sends the field on phase=join_request, so it cannot be wrong — and no path here returns 404. AmbiguousClusterError maps to 409 and NoClusterAvailableError to 422 (runtime_handshake_handlers.rb:167-170). An operator debugging by status code was looking for the wrong response entirely.
"target_cluster_id is mandatory when more than one cluster exists in the account" No node is produced at all. The platform emits system.k3s_ambiguous_cluster_join_refused at severity high (kubernetes_cluster_provisioner_service.rb:329) and returns 409. Looking for a misplaced node will not find one. There is no operator-side workaround: add workers before a second cluster exists.

Expected outcome: within ~60s the agent picks up the assignment, requests the join token, and joins:

platform.kubernetes_list_nodes({ cluster_id: "cluster-<uuid>" })
// → { nodes: [{ role: "server", status: "active", ... },
//             { role: "agent",  status: "active", ... }] }   // the new worker

No system.k3s.handshake.* FleetEvent exists to poll for the join. The one join-related kind, system.k3s_ambiguous_cluster_join_refused, fires only when the join is REFUSED (see the banner above), so it is the thing to check when the worker never appears:

platform.system_recent_signals({ kind: "system.k3s_ambiguous_cluster_join_refused", limit: 5 })

Step 6 — Retrieve kubeconfig + use kubectl

platform.kubernetes_get_kubeconfig({ cluster_id: "cluster-<uuid>" })
// → { kubeconfig: "apiVersion: v1\nclusters:\n- cluster:\n    server: https://[fd00:abcd:1::100]:6443\n...",
//     api_endpoint: "https://[fd00:abcd:1::100]:6443" }

Save it locally (you need SDWAN access from your operator workstation — import a WireGuard config via system_sdwan_create_access_grant if you haven't already):

mkdir -p ~/.kube
platform.kubernetes_get_kubeconfig({ cluster_id }) | jq -r '.data.kubeconfig' > ~/.kube/k3s.yaml
chmod 600 ~/.kube/k3s.yaml

kubectl --kubeconfig ~/.kube/k3s.yaml get nodes
# → NAME           STATUS   ROLES                  AGE   VERSION
#   k3s-server-1   Ready    control-plane,master   5m    v1.30.5+k3s1
#   k3s-worker-1   Ready    <none>                 1m    v1.30.5+k3s1

Verification

Cluster status via MCP:

platform.kubernetes_list_nodes({ cluster_id: "cluster-<uuid>" })
// → { nodes: [
//      { instance_id, role: "control-plane", status: "ready", version: "v1.30.5+k3s1" },
//      { instance_id, role: "worker",        status: "ready", version: "v1.30.5+k3s1" }
//    ] }

Sanity check pod scheduling:

kubectl --kubeconfig ~/.kube/k3s.yaml run hello \
  --image nginx:1.27-alpine --port 80 --restart Never
kubectl --kubeconfig ~/.kube/k3s.yaml get pod hello -w
# → hello   0/1   ContainerCreating
# → hello   1/1   Running

VIP failover (advanced): this tutorial used to tell you to call system_drain_instance on the bootstrap server and watch the VIP migrate within ~10s. It does not. That verb writes two drain_* marker keys onto the instance config and emits a FleetEvent; it stops no service, transitions no state, and nothing — the SDWAN VIP allocator included — reads those markers. The instance stays running and keeps holding the VIP. See node-provisioning.md Phase 5 for the full contract.

To actually move a non-anycast VIP off its current holder, promote the standby explicitly with platform.system_sdwan_failover_virtual_ip (takes virtual_ip_id; approval-gated under system.sdwan_vip_failover), or stop the holder outright with platform.system_stop_instance. Read the current holder back with:

// system_sdwan_get_virtual_ip declares virtual_ip_id (required) and nothing
// else — there is no name lookup. Get the id from system_sdwan_list_virtual_ips.
platform.system_sdwan_get_virtual_ip({ virtual_ip_id: "<vip-id>" })
// → { virtual_ip: { holders: [...], ... } }

kubectl --kubeconfig ~/.kube/k3s.yaml get nodes
# Still works if the VIP /128 now resolves to a different host

(Single-server cluster: stopping the only server causes an outage; this is why production wants ≥2 servers.)

Cleanup

platform.kubernetes_decommission_cluster({ cluster_id: "cluster-<uuid>" })
// Cascade-deletes KubernetesNode rows; releases VIP; underlying NodeInstances NOT terminated

platform.system_terminate_instance({ instance_id: "<bootstrap-instance>" })
platform.system_terminate_instance({ instance_id: "<worker-instance>" })

Troubleshooting

Cluster stuck at status: bootstrapping — the bootstrap server's k3s isn't healthy. SSH to the instance and check systemctl status k3s and journalctl -u k3s.service. Common causes:

  • /dev/kvm-related: cgroup v2 needs a kernel with the right config (Ubuntu 24.04 is fine; some bare-metal kernels disable it)
  • DNS: K3s needs working DNS resolution to pull container images. Check resolvectl status inside the instance.

VIP not allocated — system.sdwan.virtual_ip.allocated event never fires. The cluster status falls back to using the bootstrap server's instance /128 as api_endpoint (no failover). Check the SDWAN Manager agent's recent activity — agent_introspect takes the agent's UUID, not a slug, so resolve it via list_agents first:

// 1. List agents and find the SDWAN Manager's UUID — agent_introspect resolves
//    its agent_id by UUID only (a slug like "sdwan_manager_agent" silently
//    resolves to nothing).
platform.list_agents()
// → { agents: [{ id: "<sdwan-manager-uuid>", name: "SDWAN Manager", ... }, ...] }

// 2. Introspect it by UUID
platform.agent_introspect({ agent_id: "<sdwan-manager-uuid>" })

Worker stuck at join_request — in a single-cluster account, check the worker's SDWAN reachability to the api_endpoint VIP first. In an account with more than one non-error cluster the join is refused, not misrouted: the handshake returns 409 (AmbiguousClusterError), and system.k3s_ambiguous_cluster_join_refused is emitted at severity high. A target_cluster_id naming a missing or error cluster returns 422 (NoClusterAvailableError). Both are mapped at runtime_handshake_handlers.rb:167-170; neither case returns a 404. Earlier revisions of this page blamed a wrong target_cluster_id and a 404 response — withdrawn in Step 5. Check:

platform.system_get_template({ template_id: "<template-id>" })
// → { template: { ..., modules: [{ id, name, variety }, ...] } }
// Confirm k3s-agent is among the template's modules, then confirm the account
// still has exactly one non-error cluster — a second one refuses every
// subsequent worker join (409), and produces no node.
platform.kubernetes_list_clusters()

kubectl from operator workstation hangs — your workstation isn't on the same SDWAN network. Either:

  • Add an access grant: platform.system_sdwan_create_access_grant({ network_id, ... })
  • Use a federation peer if the cluster is on a remote account (see Tutorial 11)

Pods can't reach each other across nodes — flannel CNI uses the host's primary NIC by default, not SDWAN. To switch to ovn-kubernetes CNI (encrypted pod traffic over OVN tunnels), bootstrap the cluster with the cni_plugin override:

platform.system_provision_instance({
  node_id: "<bootstrap-node-id>",
  // ... usual args ...
  // The provisioner reads cni_plugin from the template/operator hint
  // and validates against the bootstrap node's network_profile.
})

The provisioner auto-defaults the CNI from network_profile: heavyweight nodes (≥4 GB RAM) get ovn_kubernetes; lightweight nodes get flannel. Explicit overrides are validated — passing ovn_kubernetes on a lightweight node raises CniProfileMismatchError. See CONTAINER_RUNTIMES.md §"CNI selection (Phase O4)" for the full table.

Pod traffic over SDWAN (encrypted pod-to-pod)

For flannel clusters, you can route pod-to-pod traffic over the SDWAN WireGuard overlay instead of the host's primary NIC. This is opt-in per SDWAN network.

Set pod_subnet_prefix on the SDWAN network (must not overlap the SDWAN /64, peer LAN subnets, VIP CIDRs, or another network's pod_subnet_prefix). Create the network first — the /64 is server-allocated, so you don't pass a prefix — then stamp pod_subnet_prefix as a network attribute:

platform.system_sdwan_create_network({
  name: "k3s-prod-net"
  // No prefix input — the overlay /64 (cidr_64) is server-allocated.
})
// pod_subnet_prefix is a Sdwan::Network attribute (flannel pod CIDR, e.g.
// "10.42.0.0/16"); the k3s smoke seeds set it directly on the network row.
// For >2 hosts, run the network in iBGP mode (direct peer-to-peer):
//   system_sdwan_update_network_routing_mode({ network_id, routing_protocol: "ibgp" })

Bootstrap the cluster with cni_plugin: "flannel" on a NodeInstance peered to this network. The provisioner stamps the cluster's metadata with pod_cidr + sdwan_network_id; the agent then receives extra flannel args (--flannel-iface=<the host's wg device>, --flannel-backend=host-gw, --cluster-cidr=10.42.0.0/16) at install time. K3s starts flannel in host-gw mode, which installs per-node /24 routes pointing at each node's overlay IP. The kernel routes those overlay IPs through the existing WG tunnels via the SDWAN AllowedIPs.

Verify pod traffic on the WG interface (live smoke):

# On node A (kubectl gives you the WG iface name from --flannel-iface output)
kubectl get pods -A -o wide                          # find pods on node A + node B
# the device is per host: read it from the runtime bootstrap_config's
# flannel_iface, or `ip -o link | grep wg-sdwan-` on the node itself.
tcpdump -i <the host's wg device> 'host <pod-IP-node-B>'  # should show traffic during pod-to-pod test

Migration of an existing flannel cluster: setting pod_subnet_prefix on a network that already has running flannel clusters triggers the agent's reconcile loop to re-install k3s with the new flags on next heartbeat (~60 s). This causes a brief 30–60 s pod-network outage per node as k3s restarts. The operator's act of setting the field IS the explicit opt-in; if you can't tolerate the outage, drain workloads first or wait for a maintenance window.

Immutability: pod_subnet_prefix cannot change once any cluster references the network — k3s pod CIDR is immutable post-bootstrap. Destroy + rebuild the cluster to migrate to a different pod CIDR.

Agent binary requirement: the new --flannel-iface / --flannel-backend=host-gw / --cluster-cidr arguments are consumed by the on-node Go agent's k3sd package. Existing NodeInstances running an older agent binary will ignore the new fields (the JSON is zero-value-safe — unknown fields are skipped). For the feature to take effect on existing instances, the agent binary must be rebuilt + the initramfs republished + instances rebooted (or re-provisioned) so they pick up the new binary. New instances provisioned after the agent binary rebuild get the feature automatically. The agent binary is built via the system extension's CI; see extensions/system/agent/Makefile for the build flow and runbooks/disk-image-ci.md for the initramfs republish path.

ovn-Kubernetes ignores pod_subnet_prefix (its OVN layer owns the pod-network); setting the field on an ovn-K8s path emits a system.cluster_bootstrap.pod_subnet_prefix_ignored warning event but otherwise bootstraps normally.

What's next

  • Tutorial 05 — Multi-cluster K3s + SDWAN isolation — multiple clusters with per-tenant networks; target_cluster_id and cross-tenant trust boundaries.
  • runbooks/multi-cluster-k3s.md — multi-cluster operator workflows: VIP-backed api_endpoint, kubeconfig retrieval, cross-cluster patterns — and the banner stating that an HA control plane (Phase 4) is NOT IMPLEMENTED: a second k3s-server bootstraps a separate cluster, it does not join the first.
  • CONTAINER_RUNTIMES.md — Phase 2 K3s full lifecycle reference (kubeadm modules are not implemented, and an HA control plane is not implemented with K3s HA parked).
  • SMOKE_TEST.md Pass 2 — smoke_test_k3s_runtime.rb and smoke_test_ovn_k8s_cni.rb exercise the same handshake at the platform layer without a live K3s install.

Last verified: 2026-06-03