Skip to content

Commit 2cf3634

Browse files
committed
feat: fetch a signed extension set without a delivery image
A converged host should not pull a large image layer to learn that nothing changed, and an image is the wrong carrier when the consumer is a pod without a container runtime. The agent can now materialize its staging from a published build: it downloads a signed SHA256SUMS inventory, checks an optional pinned digest, verifies the detached signature against a pinned key, and fetches only the files whose staged hash differs before the existing integrity, base-version and activation flow runs unchanged. A converged pass downloads the inventory and its signature, nothing else. CI publishes one release per build from the same raws artifact, with every staging file and the signed inventory. The producer signs the raws with the repository key when it exists and falls back to a run-local key otherwise, so the pipeline stays green before key custody is set up; the publish job skips with a notice until then. The key is named unhardened because its custody is a repository secret, not an HSM, and rotation into a hardened custodian removes the qualifier. Version retention must not live in the merge directory: systemd-sysext merges every raw, so the two instances systemd-sysupdate keeps would both merge and files dropped by the newer version would survive. The manifest therefore carries explicit REMOVE=<name> tombstones, which delete the raw, its unit and its enable symlink before the activation flow, and fetch prunes staging to the verified inventory so a build that drops an extension converges instead of wedging. Removing an extension is written down and signed, never inferred. Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>
1 parent ccafb2c commit 2cf3634

8 files changed

Lines changed: 1130 additions & 35 deletions

File tree

‎.github/workflows/sysext.yml‎

Lines changed: 83 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,15 +36,27 @@ jobs:
3636
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
3737
with:
3838
submodules: recursive
39-
- name: Generate the run-local signing CA
39+
- name: Provide the signing key
4040
shell: bash
41+
env:
42+
SYSEXT_UNHARDENED_SIGNING_KEY: ${{ secrets.SYSEXT_UNHARDENED_SIGNING_KEY }}
4143
run: |
4244
install -d -m 0700 .build/keys
43-
openssl req -x509 -newkey rsa:3072 -nodes -days 30 \
44-
-subj "/CN=SCI sysext CI run-local CA/" \
45-
-keyout .build/keys/sysext-verity.key \
46-
-out mkosi/verity.crt
47-
chmod 600 .build/keys/sysext-verity.key
45+
if [ -n "${SYSEXT_UNHARDENED_SIGNING_KEY:-}" ]; then
46+
printf '%s\n' "$SYSEXT_UNHARDENED_SIGNING_KEY" > .build/keys/sysext-verity.key
47+
chmod 600 .build/keys/sysext-verity.key
48+
openssl req -x509 -key .build/keys/sysext-verity.key -days 3650 \
49+
-subj "/CN=sci-sysext-unhardened" \
50+
-out mkosi/verity.crt
51+
echo "using the repository signing key"
52+
else
53+
openssl req -x509 -newkey rsa:3072 -nodes -days 1 \
54+
-subj "/CN=sci-sysext-run" \
55+
-keyout .build/keys/sysext-verity.key \
56+
-out mkosi/verity.crt
57+
chmod 600 .build/keys/sysext-verity.key
58+
echo "no repository signing key; using a run-local key"
59+
fi
4860
- name: Create the mkosi venv
4961
shell: bash
5062
run: |
@@ -125,3 +137,68 @@ jobs:
125137
uses: ./.github/workflows/test.yml
126138
with:
127139
sysext_artifact: sysext-raws
140+
publish:
141+
name: Publish the signed release
142+
needs: producer
143+
if: github.event_name != 'pull_request'
144+
runs-on: ubuntu-24.04
145+
permissions:
146+
contents: write
147+
steps:
148+
- name: Checkout
149+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
150+
with:
151+
submodules: recursive
152+
- name: Download the sysext raws artifact
153+
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
154+
with:
155+
name: sysext-raws
156+
path: .build/ci-sysext-hv/publish
157+
- name: Check the signing secret
158+
id: key
159+
shell: bash
160+
env:
161+
SYSEXT_UNHARDENED_SIGNING_KEY: ${{ secrets.SYSEXT_UNHARDENED_SIGNING_KEY }}
162+
run: |
163+
if [ -n "${SYSEXT_UNHARDENED_SIGNING_KEY:-}" ]; then
164+
echo "present=true" >> "$GITHUB_OUTPUT"
165+
else
166+
echo "present=false" >> "$GITHUB_OUTPUT"
167+
echo "::notice::no SYSEXT_UNHARDENED_SIGNING_KEY configured; skipping the release"
168+
fi
169+
- name: Regenerate the published inventory
170+
if: steps.key.outputs.present == 'true'
171+
shell: bash
172+
working-directory: .build/ci-sysext-hv/publish
173+
run: |
174+
find . -maxdepth 1 -type f ! -name SHA256SUMS ! -name 'SHA256SUMS.sig' -printf '%f\n' \
175+
| LC_ALL=C sort | xargs sha256sum > SHA256SUMS
176+
sha256sum -c -- SHA256SUMS
177+
- name: Sign the published inventory
178+
if: steps.key.outputs.present == 'true'
179+
shell: bash
180+
env:
181+
SYSEXT_UNHARDENED_SIGNING_KEY: ${{ secrets.SYSEXT_UNHARDENED_SIGNING_KEY }}
182+
run: |
183+
signing_key="$RUNNER_TEMP/sysext-signing.key"
184+
install -m 0600 /dev/null "$signing_key"
185+
trap 'rm -f "$signing_key"' EXIT
186+
printf '%s\n' "$SYSEXT_UNHARDENED_SIGNING_KEY" > "$signing_key"
187+
openssl dgst -sha256 -sign "$signing_key" \
188+
-out .build/ci-sysext-hv/publish/SHA256SUMS.sig \
189+
.build/ci-sysext-hv/publish/SHA256SUMS
190+
rm -f "$signing_key"
191+
- name: Publish the release
192+
if: steps.key.outputs.present == 'true'
193+
shell: bash
194+
env:
195+
GH_TOKEN: ${{ github.token }}
196+
run: |
197+
suite=$(sed -n 's/^Release=//p' mkosi/mkosi.conf | head -n 1)
198+
tag="${suite}-g${GITHUB_SHA:0:7}"
199+
gh release view "$tag" >/dev/null 2>&1 \
200+
|| gh release create "$tag" --target "$GITHUB_SHA" \
201+
--title "$tag" --notes "SCI sysext artifacts for $tag"
202+
gh release upload "$tag" .build/ci-sysext-hv/publish/* --clobber
203+
gh release view "$tag" --json assets --jq '.assets[].name' | grep -qx 'SHA256SUMS.sig'
204+
echo "published $tag: $(gh release view "$tag" --json url --jq .url)"

‎delivery/Containerfile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,11 @@ ENV HOST_ROOT=/host \
77
INTERVAL=60 \
88
NSENTER=1
99

10+
# Fetch mode downloads and verifies a signed build from the host network.
11+
RUN apt-get update \
12+
&& apt-get install -y --no-install-recommends curl ca-certificates openssl \
13+
&& rm -rf /var/lib/apt/lists/*
14+
1015
COPY . /opt/sci/
1116
COPY sci-delivery.sh /usr/local/bin/sci-delivery.sh
1217
RUN chmod 0755 /usr/local/bin/sci-delivery.sh

‎delivery/README.md‎

Lines changed: 109 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,11 @@ them into the host on every pass.
2323
`<HOST_ROOT>/var/lib/extensions/` (write `.tmp`, then `mv`);
2424
* a missing or content-different `<name>-activation.service` shim is copied
2525
to `<HOST_ROOT>/etc/systemd/system/` the same way;
26+
* a `REMOVE=<name>` directive makes the agent delete only what it
27+
names: `<HOST_ROOT>/var/lib/extensions/<name>.raw`,
28+
`<HOST_ROOT>/etc/systemd/system/<name>-activation.service` and its
29+
`multi-user.target.wants` enable symlink, before the entries reconcile,
30+
and never restarts that extension's units;
2631
* when anything changed, exactly one sequence runs: `systemctl
2732
daemon-reload` → `systemd-sysext refresh` → `systemctl reload
2833
dbus.service` → `systemctl restart <units>` for each extension whose raw
@@ -95,7 +100,9 @@ The manifest grammar is v2: `<name> <restart-unit>...` is an entry, `#`
95100
comments and empty lines are ignored, and a line matching
96101
`^[A-Z][A-Z0-9_]*=<value>$` is a **directive**, not an entry. Unknown
97102
directives are a validation failure (fail closed, so a typo cannot leave the
98-
gate unset). Exactly one directive exists, and it is required:
103+
gate unset). Two directives exist: `TARGET_BASE`, which is required,
104+
and `REMOVE`, the tombstone (see "The delivery manifest"). The gate
105+
reads:
99106

100107
```
101108
TARGET_BASE=2150
@@ -186,6 +193,23 @@ you deliver, and the agent never deploys a raw the manifest does not list
186193
(extra `SHA256SUMS` lines are ignored, so a trimmed manifest still
187194
verifies).
188195

196+
A tombstone is a directive, not an entry:
197+
198+
```
199+
REMOVE=sci-51-virt-qemu
200+
```
201+
202+
For each `REMOVE` name the agent deletes
203+
`<HOST_ROOT>/var/lib/extensions/<name>.raw`,
204+
`<HOST_ROOT>/etc/systemd/system/<name>-activation.service` and its
205+
`multi-user.target.wants` enable symlink, then runs the normal activation
206+
flow. Removal is idempotent: a name that is
207+
already absent is not a change. A name that is also an entry, a
208+
duplicate `REMOVE` name, or a name that does not match
209+
`[A-Za-z0-9][A-Za-z0-9._-]*` fails validation, so a tombstone cannot
210+
unmerge content the same manifest deploys, and it cannot reach outside
211+
the two deployment trees. Trimming an entry deletes nothing.
212+
189213
The `sci-51-virt-qemu` entry carries no restart units: the qemu driver
190214
plugs into the already-running `libvirtd`, and that service is restarted
191215
by `sci-40-libvirt`'s entry — this extension ships no units of its own.
@@ -199,11 +223,92 @@ bash delivery/build-image.sh
199223
`build-image.sh` stages the entries of the manifest and fails before
200224
docker runs when one of them has no raw.
201225

226+
## Remote delivery (fetch mode)
227+
228+
`FETCH_URL` switches the agent from image mode to fetch mode: it then
229+
downloads one build from that base URL, proves it, and reconciles it. A
230+
fetch pass does not trust the staging directory it finds.
231+
232+
| Variable | Meaning |
233+
|---|---|
234+
| `FETCH_URL` | base URL of one build (unset selects image mode) |
235+
| `FETCH_PUBKEY` | PEM public key for `SHA256SUMS.sig` (required) |
236+
| `FETCH_SUMS_DIGEST` | pinned sha256 of `SHA256SUMS` (optional) |
237+
| `FETCH_DIR` | writable fetch root (default `/var/lib/sci-sysext`) |
238+
239+
`STAGING_DIR` defaults to `$FETCH_DIR/staging` in fetch mode when the
240+
caller does not set it. Image mode keeps `/opt/sci`.
241+
242+
The release carries one flat file set: `extensions.manifest`, the raws,
243+
the shims, the run certificate, the agent, `SHA256SUMS` and
244+
`SHA256SUMS.sig`. The `publish` job of `.github/workflows/sysext.yml`
245+
derives it from the `sysext-raws` artifact and tags it
246+
`<Release>-g<short sha>`, for example `2150.9.0-g1a2b3c4`. One release
247+
per build, and no second product.
248+
249+
The verification order (all before any host mutation):
250+
251+
1. download `SHA256SUMS` and `SHA256SUMS.sig`.
252+
2. compare the sums digest with `FETCH_SUMS_DIGEST` when it is set.
253+
3. check the detached signature with `openssl dgst -sha256 -verify`.
254+
4. parse the sums as the inventory: `<sha> <name>`, bare filenames
255+
only.
256+
5. download only the files whose staged hash differs.
257+
6. prove the complete staged set with `sha256sum -c`.
258+
7. prune staged files that the inventory no longer lists, so a dropped
259+
extension converges.
260+
261+
Then the normal pass runs: manifest validation, staged-raw integrity,
262+
base-version gate, deployment and activation.
263+
264+
**Convergence.** When the staged set already matches the inventory, a
265+
pass downloads exactly two files, `SHA256SUMS` and `SHA256SUMS.sig`, and
266+
changes nothing. No host-mutating command runs. A rebuild therefore
267+
costs the inventory download per interval plus the files that changed.
268+
269+
**Failure semantics.** Every fetch failure is fatal. The pass exits
270+
non-zero with one `[sci-delivery] error: fetch: ...` line and writes
271+
nothing to the host. It makes no `systemctl` or `systemd-sysext` call,
272+
so a host that lost its network keeps the set it has. The agent refuses
273+
an unreachable URL, a missing or unreadable `FETCH_PUBKEY`, a malformed
274+
`FETCH_SUMS_DIGEST`, a digest mismatch, a bad signature, an invalid or
275+
duplicated inventory line, a name that would traverse, a downloaded
276+
file that does not match its line, and an incomplete staged set.
277+
278+
**Prerequisite.** dm-verity checks the raws against the certificate at
279+
`/usr/lib/verity.d/verity.crt` in the base image. `/usr` is read-only at
280+
runtime, and no delivery path can install that anchor. The merge
281+
refuses a release signed by a key that the base image does not carry,
282+
whatever the sums check says. `FETCH_PUBKEY` is a second, separate
283+
trust decision: it covers the inventory, not the raws.
284+
285+
The public key is `sysext-unhardened-signing.pem`. The key custody is
286+
not hardened, and an operator is expected to rotate the key into a
287+
hardened custodian before the `unhardened` qualifier disappears.
288+
289+
Chart values:
290+
291+
```yaml
292+
sciSysext:
293+
url: https://github.com/<org>/<repo>/releases/download/2150.9.0-g1a2b3c4
294+
unhardenedSigningKey: |
295+
-----BEGIN PUBLIC KEY-----
296+
...
297+
-----END PUBLIC KEY-----
298+
```
299+
300+
`sciSysext.url` becomes `FETCH_URL`. The agent reads `FETCH_PUBKEY` as a
301+
file path, so the chart must place the `sciSysext.unhardenedSigningKey`
302+
body in `sysext-unhardened-signing.pem` and name that file in
303+
`FETCH_PUBKEY`. The key is public, so it can travel as a chart value. The
304+
private key never does.
305+
202306
## Safety properties
203307

204-
* **Never deletes host content**: unlisted host extensions in
205-
`/var/lib/extensions` are left alone, and the agent only writes manifest
206-
entries. The single file it removes is its own
308+
* **Deletes only what a `REMOVE` directive names**: unlisted host
309+
extensions in `/var/lib/extensions` are left alone, and the agent only
310+
writes manifest entries. A `REMOVE` name loses its raw and its shim,
311+
and nothing else. The only other file it removes is its own
207312
`.sci-delivery-pending` marker.
208313
* **Idempotent**: unchanged files are not rewritten (no mtime change) and
209314
an unchanged pass makes zero systemctl/systemd-sysext calls.

0 commit comments

Comments
 (0)