Disk Image CI/CD — Operator Guide

Status: active

End-to-end disk image build pipeline: NodePlatform → Gitea Actions → OCI ingest → publication. Uses the platform's GitOps + CI worker infrastructure with Cosign signing for supply-chain integrity.

Architecture (one-paragraph summary)

A System::NodePlatform carries a build_script that produces a disk image (kernel + initramfs + erofs module blob). The build runs on a self-hosted Gitea Actions runner (provisioned via provision_ci_worker) triggered by a webhook. After build, the runner pushes the artifact as an OCI blob (Cosign-signed via the platform's keyless identity), POSTs the webhook back to platform, which ingests via DiskImagePublicationProcessor. Ingest verifies the Cosign signature + SHA-256 over the pulled artifact (the CI pipeline produces erofs blobs, but the server does not perform a server-side erofs/fs-verity verification — see DiskImageOciIngestService). The resulting DiskImagePublication row links the OCI digest to the platform record + retention policy.

End-to-End Flow

sequenceDiagram
    actor Op as Operator
    participant Runner as Gitea Runner
    participant Plat as Platform
    participant Reg as OCI registry
    participant Ret as Retention service
    participant Agent as powernode-agent

    Op->>Runner: 1. trigger build<br/>(push tag OR dispatch_gitea_workflow)
    Runner->>Runner: 2. run build_script:<br/>apt-mirror, kernel,<br/>erofs blob, initramfs
    Runner->>Reg: oras push artifact<br/>cosign sign keyless
    Runner->>Plat: 3. POST webhook<br/>OCI digest + sha256<br/>HMAC-signed
    Plat->>Plat: 4. DiskImageWebhook<br/>validates signature
    Plat->>Reg: 5. DiskImagePublicationProcessor<br/>fetch manifest + cosign verify + sha256
    Reg-->>Plat: verified manifest
    Plat->>Plat: 6. create DiskImagePublication<br/>update NodePlatform.disk_image_oci_ref
    Ret->>Plat: 7. prune images beyond retention_count
    Op->>Plat: 8. provision instance from Template
    Plat->>Agent: deploy
    Agent->>Reg: fetch OCI artifact at boot
    Reg-->>Agent: kernel + initramfs + erofs blob
    Agent-->>Op: instance booted from custom image

Six artifact families × two architectures

The initramfs builder publishes six artifact families per architecture, each suited to a different deployment context. The Disk Image Manager agent tracks publications per (NodePlatform, artifact_family, architecture) triple.

flowchart LR
    subgraph Build["Build pipeline"]
        BS[build.sh<br/>--arch &lt;arch&gt;]
    end

    subgraph Families["6 artifact families"]
        F1[kernel + initramfs.cpio.zst<br/>iPXE / direct kernel boot]
        F2[raw disk image .img<br/>USB / SD card / dd]
        F3[ISO 9660 .iso<br/>DVD / IPMI virtual media]
        F4[iPXE chainload .ipxe<br/>network boot entry]
        F5[qcow2 image<br/>libvirt / QEMU pre-baked]
        F6[OCI image<br/>bootc-compatible]
    end

    subgraph Arches["Per-arch publication"]
        A1[amd64]
        A2[arm64]
    end

    BS --> F1
    BS --> F2
    BS --> F3
    BS --> F4
    BS --> F5
    BS --> F6
    F1 --> A1
    F1 --> A2
    F2 --> A1
    F2 --> A2
    F3 --> A1
    F3 --> A2
    F4 --> A1
    F4 --> A2
    F5 --> A1
    F5 --> A2
    F6 --> A1
    F6 --> A2

Building UEFI images (UKI) — the claim-by-ID fleet boot disk

The disk-image-amd64-uefi and disk-image-arm64-uefi variants produce the generic, flashable claim-by-ID boot disk (one image for the whole fleet; a per-device identity.cfg is dropped onto the BOOT partition). Both use one consistent model — a UKI (Unified Kernel Image): ukify fuses the kernel + initramfs + cmdline into a single EFI binary at the ESP's removable-media fallback path (/EFI/BOOT/BOOTX64.EFI on amd64, /EFI/BOOT/BOOTAA64.EFI on arm64). UEFI firmware boots it with zero config — no GRUB, no loader entries. Layout is a 2-partition GPT: ESP (FAT32, label BOOT) + persist (ext4, label persist), assembled loop-free (mtools + mkfs.ext4 -E offset + sgdisk).

Build-environment requirement. build.sh's kernel-initrd variant needs a real distro kernel (/boot/vmlinuz-* + /lib/modules/*) and dracut to build the generic (hostonly=no) initramfs; ukify + the systemd-boot stub (linuxx64.efi.stub / linuxaa64.efi.stub) then fuse the UKI. The CI runner is a container with neither, so each job installs them and builds in an architecture-matched environment:

Image	Build environment	Why
`build-amd64-uefi` job	native on the amd64 runner (`apt install linux-image-generic dracut-core`)	runner is amd64 → dracut runs without emulation
`disk-image-arm64-uefi` (in the `build` job)	QEMU-emulated arm64 container (`docker run --platform linux/arm64`; binfmt via setup-qemu)	no native arm64 runner, and dracut must run as arm64 to copy/resolve arm64 binaries

If dracut/kernel aren't present, build.sh emits a tiny placeholder kernel and the UKI assembler fails loud (missing/empty .../kernel) — never a silently non-bootable image.

Native arm64 runner (recommended reliability upgrade). The arm64 build emulates apt + dracut, which is slow (~30–60 min) and inherently less reliable than native. Registering a native arm64 Gitea runner and pointing the arm64 UEFI build at it (dropping the docker run --platform wrapper, building like amd64) is the highest-reliability path.

Setup: Initial CI Worker + Webhook

Step 1: Bootstrap the CI worker for an account

platform.bootstrap_disk_image_ci({
  owner: "<account>",                              // Gitea owner (account name)
  repo: "disk-images",                             // Gitea repo containing the build workflow
  label: "ubuntu-2404-amd64-builder",              // operator-chosen identifier; webhook + worker rows key off this
  platform_api_base: "https://platform.example.org", // optional; defaults to POWERNODE_PUBLIC_URL
  create_platform_read_token: true                 // mints a read-scoped JWT for the runner to call back
})
// → {
//     success: true,
//     webhook:   { id, label, action, url: "https://platform.example.org/api/v1/system/webhooks/disk_image/built/<webhook_id>" },
//     ci_worker: { id, name, action },
//     gitea_secrets_set: { "POWERNODE_DISK_IMAGE_WEBHOOK_SECRET": "ok", "POWERNODE_CI_WORKER_TOKEN": "ok", ... },
//     note: "..."
//   }

No secret is returned (IMP-fa6cf8ee1eb6 / IMP-27cc7dceb97b). An MCP tool result is truncated into ai_messages.processing_metadata and forwarded in full to the model provider, so it is not a private channel the way an HTTP response is. The webhook secret, the CI worker token and the optional platform read token are delivered out of band, straight into the repo's Gitea Actions secret store — gitea_secrets_set is the delivery receipt. That is the whole point of this action: the CI workflow reads them from ${{ secrets.* }} and no plaintext ever rides the chat transcript.

This action is idempotent on label (re-running rotates secrets + token without creating duplicates) and is a one-shot setup, not a build trigger. It creates:

A System::DiskImageWebhook row (per-pipeline; the URL above embeds its UUID)
A Worker row with role ci_worker (NOT a System::Task, and NOT a NodeInstance — the operator installs and registers a Gitea Actions runner themselves; the worker token is written into the repo secret POWERNODE_CI_WORKER_TOKEN, not returned)
Gitea repo Actions secrets (webhook secret + URL + optional read token + OCI registry creds if configured)

Triggering an actual build is a separate action (dispatch_gitea_workflow — see "Triggering a build" below). bootstrap_disk_image_ci does not have account_id, force, or ref/arches parameters; those framings in earlier doc revisions were aspirational.

Step 2: Provision the build webhook

When bootstrap_disk_image_ci provisions the webhook for you, you do not need this action — it already set the URL + secret in the repo's Actions secrets. To provision a webhook standalone (e.g., to attach a second build pipeline to the same account):

platform.provision_disk_image_webhook({
  label: "ubuntu-2404-arm64-builder"   // operator-chosen identifier
  // platform_api_base optional; defaults to POWERNODE_PUBLIC_URL
})
// → {
//     success: true,
//     webhook_id: "<uuid>",
//     label: "ubuntu-2404-arm64-builder",
//     webhook_url: "https://platform.example.org/api/v1/system/webhooks/disk_image/built/<webhook_id>",
//     secret_delivery: "not disclosed here — ...",
//     note: "..."
//   }

Webhooks are per-pipeline (the URL embeds the webhook UUID), not per-NodePlatform. The action does not accept node_platform_id, webhook_url, or shared_secret — the URL is built server-side from POWERNODE_PUBLIC_URL + the issued webhook id.

The secret is not returned on the MCP surface (IMP-fa6cf8ee1eb6): a tool result is persisted with the conversation and forwarded to the model provider. The webhook this action creates is therefore inert until you fetch a secret for it:

curl -X POST -H "Authorization: Bearer $OPERATOR_JWT" \
  https://platform.example.org/api/v1/system/disk_image_webhooks/<webhook_id>/rotate_secret

That endpoint needs system.disk_image_webhooks.rotate_secret (which the tool's own permission does not imply) and is approval-gated — if it answers pending, the plaintext is revealed once in the HTTP approval-decision response, so approve over the operator UI/API, not via approve_deferred_operation, which deliberately drops the reveal. Prefer bootstrap_disk_image_ci, which writes the secret straight into the repo's Gitea Actions secrets and needs no reveal at all.

Operator configures the webhook URL + secret in the build repo's CI workflow YAML so the runner can call back after a successful build.

Operator Workflow

Triggering a build

// Direct dispatch (params: owner, repo, workflow_file, ref, inputs)
platform.dispatch_gitea_workflow({
  owner: "<account>",                       // Gitea owner
  repo: "disk-images",                      // repo name
  workflow_file: "build-disk-image.yaml",   // the param is `workflow_file`, not `workflow`
  ref: "master",                            // branch/tag ref (required)
  inputs: { platform_slug: "ubuntu-2404-base" }
})

// Or via git push to the configured branch

Monitoring a build

// List recent runs
platform.list_gitea_workflow_runs({
  owner: "<account>",
  repo: "disk-images",
  workflow_file: "build-disk-image.yaml"    // optional filter
})

// Get a run + its jobs, then tail a specific job's logs by job_id
platform.get_gitea_workflow_run({ owner: "<account>", repo: "disk-images", run_id: "<run-id>" })
platform.get_gitea_job_logs({ owner: "<account>", repo: "disk-images", job_id: "<job-id>" })

Inspecting publications

# Via REST
curl /api/v1/system/disk_image_publications -H "Authorization: Bearer $JWT"

# Per-platform recent publications
curl "/api/v1/system/node_platforms/<id>/disk_image_publications" \
  -H "Authorization: Bearer $JWT"

Each publication serializes as (per System::DiskImagePublicationSerializer):

id — publication UUID
platform_id — owning NodePlatform (serialized as platform_id, sourced from the node_platform_id column)
status — one of queued, awaiting_upload, verifying, published, failed, retired, purged
active — boolean; whether this publication is the platform's current default
arch — amd64 / arm64
firmware_ref — UEFI/firmware artifact ref (nil for non-UEFI families)
git_sha / git_sha_short — source commit
oci_ref — fully-qualified registry path (e.g. registry.example.com/account/disk-images@sha256:...)
sha256 / sha256_short — artifact content digest
size_bytes — artifact size
attempt_count — number of ingest attempts (a publication may re-verify after a transient failure)
attestation_predicate / attestation_present / cosign_bundle_present — Cosign attestation surface for inline display
file_object_id / prior_file_object_id — current + previous artifact object (the prior link drives rollback-chain visualization)
webhook_id / webhook_label / triggered_by_worker_id — provenance of the ingest
verified_at — UTC timestamp when cosign + sha256 verification passed
published_at — UTC timestamp when the publication transitioned to published
retired_at — UTC timestamp when a newer publication superseded this one (nil while current)
purged_at — UTC timestamp when the artifact was hard-deleted past the grace window
error_message — populated when status = failed (e.g. cosign verify failure detail)

Fields not in the serialized row: built_at (use published_at), cosign_identity (verification result is in error_message on failure; the identity used is recorded elsewhere), sbom_url, version, signed_at, erofs_digest (there is no server-side erofs/fs-verity verification; the current pipeline verifies the cosign signature + SHA256 over the pulled artifact). SBOM ingest from the OCI registry is not yet wired — System::Sbom::CycloneDxParser exists but is only used by the modules SBOM webhook, not by disk-image publication; no SBOM package data is populated on a DiskImagePublication today. Doc revisions before 2026-05-19 listed several of these fields aspirationally.

Promoting a publication

The latest publication is auto-promoted to current for its NodePlatform when ingest succeeds. Promotion (and rollback) is a pointer column-flip on the NodePlatform: disk_image_oci_ref / disk_image_git_sha / disk_image_file_object_id are repointed at the target publication and the previously-published publication is transitioned to retired. To set a specific publication as the default (promote or roll back), pass its id:

platform.system_set_default_disk_image_publication({
  publication_id: "<publication-id>"   // must currently be in status "published"
})

To roll back, pass an earlier publication's id (it must still be published; a retired row's artifact is restored during the agent-driven rollback path — see DISK_IMAGE_MANAGER_AGENT.md). The next NodeInstance provisioned from a Template using this Platform will fetch the newly-pointed image. There is also a dedicated system_revert_disk_image action — it wraps the RollbackPublication executor, auto-selecting the previous publication (or pass publication_id for a specific target); system_set_default_disk_image_publication remains the explicit set-default action.

Retention Policy

NodePlatform.disk_image_retention_count (default: 3) controls how many publications are kept per platform. The DiskImageRetentionService (runs via Sidekiq cron) prunes older publications past the count, removing both the DB row + the OCI blob from the registry.

To change retention:

# Via API
curl -X PATCH /api/v1/system/node_platforms/<id> \
  -H "Authorization: Bearer $JWT" \
  -d '{"disk_image_retention_count": 5}'

Secret Rotation

Three secret types in this pipeline:

Cosign keyless identity — Gitea Actions OIDC; rotates per-run automatically. No operator action.

OCI registry credentials — used by Gitea runner to push artifacts. Stored as Gitea Actions secret. Rotate via:

platform.set_gitea_action_secret({
  owner: "<account>",
  repo: "<repo>",
  name: "OCI_REGISTRY_TOKEN",
  value: "<new-token>"
})

Webhook signing secret — HMAC-shared between platform + build script. Rotate via provision_disk_image_webhook (issues a new pair; operator updates the runner's env).

Troubleshooting

Build succeeds but publication doesn't appear

The webhook didn't reach the platform (firewall? wrong URL?) or HMAC signature mismatch. Check:

# List webhook deliveries (filter to recent or by webhook id):
curl "/api/v1/system/disk_image_webhooks?recent=true" -H "Authorization: Bearer $JWT"

# Or via MCP:
# platform.system_list_disk_image_webhooks({ recent: true })

If signature mismatched, rotate the webhook secret.

Cosign verification fails

DiskImagePublicationProcessor rejects ingests where Cosign verify fails. Likely causes:

Build runner used a different Cosign identity than the platform's cosign_identity_regexp config on NodePlatform
OCI artifact was tampered post-signing

Inspect:

curl /api/v1/system/disk_image_publications/<id> -H "Authorization: Bearer $JWT"
# Look for status="failed" + error_message containing the cosign failure detail.
# The publication's status enum is queued/awaiting_upload/verifying/published/failed/retired/purged
# (there is no separate cosign_verify_failed sub-status).
#
# The NodePlatform row also carries the last attempt:
#   disk_image_publication_status — overall publication state on the platform
#   disk_image_publication_error — last failure detail surfaced to operators

Runner stuck in pending

Gitea Actions runner provisioned but not online. Check:

CI worker provisioning is synchronous in the MCP tool (no System::Task row is created), so a "pending" task wouldn't appear in system_list_tasks. Instead check the Worker row directly via the operator API:

curl "/api/v1/workers?role=ci_worker" -H "Authorization: Bearer $JWT"

If the worker exists but the Gitea Actions runner side hasn't registered, the operator's manual runner install is the missing step (the platform issues the token; the operator must install + register the runner against it). Reprovisioning the bootstrap (rotates secrets + issues a fresh worker token) — force is not a parameter; just re-run with the same label:

platform.bootstrap_disk_image_ci({
  owner: "<account>",
  repo: "disk-images",
  label: "ubuntu-2404-amd64-builder"
})
// Idempotent on label — rotates secrets + token, returns fresh values.

Source Files

Models:

extensions/system/server/app/models/system/disk_image_webhook.rb
extensions/system/server/app/models/system/disk_image_publication.rb

Services:

extensions/system/server/app/services/system/disk_image_publication_processor.rb — webhook → ingest
extensions/system/server/app/services/system/disk_image_oci_ingest_service.rb — OCI manifest fetch + Cosign verify
extensions/system/server/app/services/system/disk_image_direct_upload_ingest_service.rb — fallback for non-CI uploads
extensions/system/server/app/services/system/disk_image_retention_service.rb — prune past retention count

Controllers:

extensions/system/server/app/controllers/api/v1/system/disk_image_publications_controller.rb
extensions/system/server/app/controllers/api/v1/system/disk_image_webhooks_controller.rb
extensions/system/server/app/controllers/api/v1/system/webhooks/disk_image_built_controller.rb — receives Gitea webhook
extensions/system/server/app/controllers/api/v1/system/worker_api/disk_image_publications_controller.rb — runner-facing

MCP tools:

server/app/services/ai/tools/disk_image_operator_tool.rb — bootstrap_disk_image_ci, provision_disk_image_webhook, provision_ci_worker
server/app/services/ai/tools/gitea_actions_tool.rb — secrets, workflow dispatch, run monitoring

Related Docs

../initramfs/README.md — multi-arch boot artifact build details
ARCHITECTURE.md — disk image pipeline subsystem
DISK_IMAGE_MANAGER_AGENT.md — the autonomy surface that promotes/rolls back these publications

Last verified: 2026-06-03

Navigation Menu

Repository navigation

DISK_IMAGE_CI.md

DISK_IMAGE_CI.md

Disk Image CI/CD — Operator Guide

Architecture (one-paragraph summary)

End-to-End Flow

Six artifact families × two architectures

Building UEFI images (UKI) — the claim-by-ID fleet boot disk

Setup: Initial CI Worker + Webhook

Step 1: Bootstrap the CI worker for an account

Step 2: Provision the build webhook

Operator Workflow

Triggering a build

Monitoring a build

Inspecting publications

Promoting a publication

Retention Policy

Secret Rotation

Troubleshooting

Build succeeds but publication doesn't appear

Cosign verification fails

Runner stuck in pending

Source Files

Related Docs

FilesExpand file tree

DISK_IMAGE_CI.md

Latest commit

History

DISK_IMAGE_CI.md

File metadata and controls

Disk Image CI/CD — Operator Guide

Architecture (one-paragraph summary)

End-to-End Flow

Six artifact families × two architectures

Building UEFI images (UKI) — the claim-by-ID fleet boot disk

Setup: Initial CI Worker + Webhook

Step 1: Bootstrap the CI worker for an account

Step 2: Provision the build webhook

Operator Workflow

Triggering a build

Monitoring a build

Inspecting publications

Promoting a publication

Retention Policy

Secret Rotation

Troubleshooting

Build succeeds but publication doesn't appear

Cosign verification fails

Runner stuck in pending

Source Files

Related Docs