Skip to content

Commit 07e8bb4

Browse files
committed
ci: prove the sysext artifacts on GitHub
Local-only proof dies with the machine that ran it. The sysext workflow runs the quick suite on every change and, on push, builds all extensions, runs the full suite with the VM boot test and publishes one artifact: the raws staging, plain files with every verity-signed raw, SHA256SUMS, the shims, the manifest, the run certificate and the agent. A deployment can take the EROFS files without a container runtime. The integration job consumes the raws the way a consumer would: it restores the run certificate from the staging, bootstraps the pinned toolchain, rebuilds the delivery image from the verified staging, boots two Garden Linux hypervisors on the runner and proves the sysext refresh and the QEMU and Cloud-Hypervisor live migrations against the produced artifact. The consumer workflow stays callable standalone, so any prior run's raws can be re-tested by run ID. The adapter tests pin every workflow contract structurally, on the parsed workflow instead of its step text, so CI and tests cannot drift apart. The hosted runners need explicit accommodations: mkosi's unprivileged sandbox needs the AppArmor user-namespace restriction lifted, qemu-system-x86 provides the virtual qemu-kvm package, /dev/kvm must be accessible for the VM boot test, the finished qemu scenario's VM is stopped before the cloud-hypervisor stage because its nested I/O starves the staging copy, and the debug collection includes the HV serial consoles, the only record of a frozen guest's final kernel output. Signed-off-by: Fabian Wiesel <fabian.wiesel@sap.com>
1 parent 9588a63 commit 07e8bb4

13 files changed

Lines changed: 1425 additions & 17 deletions

File tree

‎.github/actions/test/integration/build/action.yml‎

Lines changed: 42 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,30 @@ name: build
22
description: Build Bootable Virtual Machine Image
33
inputs:
44
image_tag:
5-
description: "Image tag to use for the build"
6-
required: true
5+
description: "Image tag to use for the build (legacy mode)"
6+
required: false
7+
default: ""
8+
mode:
9+
description: "legacy: build the Garden Linux HV image; sysext: only prepare the Ubuntu guest"
10+
required: false
11+
default: "legacy"
12+
sysext_key:
13+
description: "Private test key prepared by prepare-sysext-hv.sh (sysext mode); its .pub is reused for the Ubuntu guest"
14+
required: false
15+
default: ""
716
runs:
817
using: "composite"
918
steps:
19+
- name: Validate the test mode
20+
if: ${{ inputs.mode != 'legacy' && inputs.mode != 'sysext' }}
21+
shell: bash
22+
run: |
23+
echo "unsupported mode '${{ inputs.mode }}' (expected legacy or sysext)" >&2
24+
exit 1
25+
1026
# TODO https://github.com/actions/cache
1127
- name: Set ENV
28+
if: ${{ inputs.mode != 'sysext' }}
1229
shell: bash
1330
env:
1431
REPO: ghcr.io/gardenlinux/gardenlinux-sci
@@ -22,6 +39,7 @@ runs:
2239
echo "DISK_SIZE=$DISK_SIZE" >> $GITHUB_ENV
2340
2441
- name: Pull ORAS artifacts
42+
if: ${{ inputs.mode != 'sysext' }}
2543
shell: bash
2644
run: |
2745
LAYERS=$(oras manifest fetch "$REPO:$TAG" | jq -r '.layers[]')
@@ -32,11 +50,13 @@ runs:
3250
oras blob fetch "$REPO@$TAR_SHA" -o "$RUNNER_TEMP/$TAG.tar" # wasteful, we only systemd-boot
3351
3452
- name: Extract systemd-boot
53+
if: ${{ inputs.mode != 'sysext' }}
3554
shell: bash
3655
run: |
3756
tar -xf "$RUNNER_TEMP/$TAG.tar" --no-same-owner --strip-components=5 -C "$RUNNER_TEMP" usr/lib/systemd/boot/efi/systemd-bootx64.efi
3857
3958
- name: Set up a raw disk image
59+
if: ${{ inputs.mode != 'sysext' }}
4060
shell: bash
4161
run: |
4262
ESP_END="${ESP_SIZE}GiB"
@@ -62,6 +82,7 @@ runs:
6282
sudo mkfs.ext4 "$VAR_PART" -L VAR
6383
6484
- name: Populate the ESP partition
85+
if: ${{ inputs.mode != 'sysext' }}
6586
shell: bash
6687
run: |
6788
mkdir -p "$RUNNER_TEMP/esp"
@@ -88,25 +109,39 @@ runs:
88109
sudo losetup -d "$LOOPDEV"
89110
90111
- name: Convert to QCOW2
112+
if: ${{ inputs.mode != 'sysext' }}
91113
shell: bash
92114
run: |
93115
qemu-img convert -O qcow2 "$DISK_IMG" "/opt/${TAG}.qcow2"
94116
95-
- name: Generate ignition
117+
- name: Generate test SSH key and password
96118
shell: bash
119+
env:
120+
SYSEXT_KEY: ${{ inputs.sysext_key }}
97121
run: |
98-
ssh-keygen -t ed25519 -f "/opt/ssh_host_ed25519_key" -N "" -C "user" -q
99-
chmod 600 /opt/ssh_host_ed25519_key
122+
if [ -n "$SYSEXT_KEY" ]; then
123+
# One key for the HVs and the Ubuntu guests.
124+
install -m 600 "$SYSEXT_KEY" /opt/ssh_host_ed25519_key
125+
install -m 644 "$SYSEXT_KEY.pub" /opt/ssh_host_ed25519_key.pub
126+
else
127+
ssh-keygen -t ed25519 -f "/opt/ssh_host_ed25519_key" -N "" -C "user" -q
128+
chmod 600 /opt/ssh_host_ed25519_key
129+
fi
100130
SSH_KEY=$(cat /opt/ssh_host_ed25519_key.pub)
131+
PASSWORD=$(openssl passwd "password")
101132
echo "SSH_KEY=$SSH_KEY" >> $GITHUB_ENV
133+
echo "PASSWORD=$PASSWORD" >> $GITHUB_ENV
134+
135+
- name: Generate ignition
136+
if: ${{ inputs.mode != 'sysext' }}
137+
shell: bash
138+
run: |
102139
cp ./.github/actions/test/integration/build/dev-user-butane.yaml ./.github/actions/test/integration/build/dev-user-butane-hv1.yaml
103140
yq '.storage.files += load("./.github/actions/test/integration/build/root-hints.yaml")' ./.github/actions/test/integration/build/dev-user-butane.yaml > ./.github/actions/test/integration/build/dev-user-butane-hv2.yaml
104141
105142
sed -i "s|SSH_KEY_GOES_HERE|$SSH_KEY|g" ./.github/actions/test/integration/build/dev-user-butane-hv1.yaml
106143
sed -i "s|SSH_KEY_GOES_HERE|$SSH_KEY|g" ./.github/actions/test/integration/build/dev-user-butane-hv2.yaml
107144
108-
PASSWORD=$(openssl passwd "password")
109-
echo "PASSWORD=$PASSWORD" >> $GITHUB_ENV
110145
sed -i "s|PASSWORD_GOES_HERE|$PASSWORD|g" ./.github/actions/test/integration/build/dev-user-butane-hv1.yaml
111146
sed -i "s|PASSWORD_GOES_HERE|$PASSWORD|g" ./.github/actions/test/integration/build/dev-user-butane-hv2.yaml
112147
sed -i "s|TAG_GOES_HERE|${{ inputs.image_tag }}|g" ./.github/actions/test/integration/build/dev-user-butane-hv1.yaml

‎.github/actions/test/integration/collect-logs/action.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,11 @@ runs:
1313
sudo cp /var/log/libvirt/qemu/HV1.log "libvirt-HV1.log" || true
1414
sudo cp /var/log/libvirt/qemu/HV2.log "libvirt-HV2.log" || true
1515
16+
# The HV serial consoles: the only record of a frozen guest's
17+
# final kernel output (the journal goes down with the guest).
18+
sudo cp /var/log/HV1.log "serial-HV1.log" || true
19+
sudo cp /var/log/HV2.log "serial-HV2.log" || true
20+
1621
- name: Obtain logs from hypervisors
1722
shell: bash
1823
run: |

‎.github/actions/test/integration/dependencies/action.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ runs:
1010
sudo apt-get install -y \
1111
parted \
1212
squashfs-tools \
13-
qemu-kvm \
13+
qemu-system-x86 \
1414
genisoimage \
1515
libvirt-daemon-system \
1616
libvirt-clients \
@@ -26,6 +26,12 @@ runs:
2626
echo 'security_driver = "none"' | sudo tee -a /etc/libvirt/qemu.conf
2727
sudo systemctl restart libvirtd
2828
29+
# The libvirt qemu user must reach /dev/kvm for the hypervisor VMs.
30+
- name: Make /dev/kvm accessible
31+
shell: bash
32+
run: |
33+
if [ -e /dev/kvm ]; then sudo chmod 666 /dev/kvm; fi
34+
2935
- name: oras
3036
uses: oras-project/setup-oras@1d808f7d7f6995cc68b7bf507bfe5c5446e1dc9d # v2
3137
with:

‎.github/actions/test/integration/setup/action.yml‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,27 @@
11
name: setup
22
description: Set up Hypervisor and VM Environment for Integration Tests
3+
inputs:
4+
mode:
5+
description: "legacy: Garden Linux image + ignition; sysext: published base + delivery container"
6+
required: false
7+
default: "legacy"
8+
hv_dir:
9+
description: "Output of prepare-sysext-hv.sh --prepare (sysext mode)"
10+
required: false
11+
default: ""
312
runs:
413
using: "composite"
514
steps:
15+
- name: Validate the test mode
16+
if: ${{ inputs.mode != 'legacy' && inputs.mode != 'sysext' }}
17+
shell: bash
18+
run: |
19+
echo "unsupported mode '${{ inputs.mode }}' (expected legacy or sysext)" >&2
20+
exit 1
21+
22+
# ---- legacy path ------------------------------------------------------
623
- name: Configure the default libvirt network
24+
if: ${{ inputs.mode != 'sysext' }}
725
shell: bash
826
run: |
927
sudo virsh net-destroy default
@@ -13,12 +31,14 @@ runs:
1331
sudo virsh net-autostart default
1432
1533
- name: Copy ignition file
34+
if: ${{ inputs.mode != 'sysext' }}
1635
shell: bash
1736
run: |
1837
sudo cp "/opt/${TAG}-hv1.ign" /var/lib/libvirt/images/HV1.ign
1938
sudo cp "/opt/${TAG}-hv2.ign" /var/lib/libvirt/images/HV2.ign
2039
2140
- name: Create HyperVisor 1
41+
if: ${{ inputs.mode != 'sysext' }}
2242
shell: bash
2343
run: |
2444
cp ./.github/actions/test/integration/setup/hv.xml ./.github/actions/test/integration/setup/hv1.xml
@@ -30,6 +50,7 @@ runs:
3050
sudo virsh start HV1
3151
3252
- name: Create HyperVisor 2
53+
if: ${{ inputs.mode != 'sysext' }}
3354
shell: bash
3455
run: |
3556
cp ./.github/actions/test/integration/setup/hv.xml ./.github/actions/test/integration/setup/hv2.xml
@@ -41,6 +62,7 @@ runs:
4162
sudo virsh start HV2
4263
4364
- name: Configure SSH and libvirt on HyperVisors
65+
if: ${{ inputs.mode != 'sysext' }}
4466
shell: bash
4567
run: |
4668
KEY=/opt/ssh_host_ed25519_key
@@ -110,3 +132,72 @@ runs:
110132
111133
wait_for_ssh HV1
112134
wait_for_ssh HV2
135+
136+
# ---- sysext path ------------------------------------------------------
137+
- name: Boot the sysext HyperVisors and run the delivery container
138+
if: ${{ inputs.mode == 'sysext' }}
139+
shell: bash
140+
env:
141+
HV_DIR: ${{ inputs.hv_dir }}
142+
run: |
143+
set -Eeuo pipefail
144+
[ -n "$HV_DIR" ] && [ -f "$HV_DIR/metadata.env" ] || { echo "hv_dir with metadata.env is required in sysext mode" >&2; exit 1; }
145+
source "$HV_DIR/metadata.env"
146+
KEY=$HV_DIR/hv-test-key
147+
SSH="-i $KEY -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
148+
for f in "$KEY" "$HV_KERNEL" "$HV_ARTIFACT_TAR" "$HV_DIR/sci-delivery.ref" "$HV_DIR/HV1.xml" "$HV_DIR/HV2.xml" "$HV_DIR/disks/HV1.qcow2" "$HV_DIR/disks/HV2.qcow2"; do
149+
[ -s "$f" ] || { echo "missing prepared artifact: $f" >&2; exit 1; }
150+
done
151+
[ -n "$HV_ARTIFACT_REF" ] || { echo "empty delivery image reference" >&2; exit 1; }
152+
sudo virsh net-info default >/dev/null || { echo "the libvirt default network is not active; sysext mode does not create it" >&2; exit 1; }
153+
# The runner home ships as 750; qemu must traverse it to reach the disks.
154+
sudo chmod o+x "$HOME"
155+
add_host() { sudo virsh net-dumpxml default | grep -q "mac='$1'" || sudo virsh net-update default add ip-dhcp-host "<host mac='$1' ip='$2' name='$3'/>" --live --config; }
156+
add_host 52:54:00:12:34:56 192.168.122.2 hv1
157+
add_host 52:54:00:65:43:21 192.168.122.3 hv2
158+
for spec in "HV1 192.168.122.2" "HV2 192.168.122.3"; do
159+
set -- $spec
160+
sudo virsh dominfo "$1" >/dev/null 2>&1 && { echo "refusing to replace a pre-existing domain: $1" >&2; exit 1; }
161+
sudo virsh define "$HV_DIR/$1.xml"
162+
sudo virsh start "$1" || { namei -l "$HV_DIR/disks/$1.qcow2" >&2 || true; exit 1; }
163+
for ((t=1; t<=60; t++)); do
164+
ssh $SSH "root@$2" true 2>/dev/null && break
165+
[ "$t" -lt 60 ] || { echo "timeout waiting for $1 to respond to SSH" >&2; sudo cat "/var/log/$1.log" || true; exit 1; }
166+
sleep 5
167+
done
168+
scp -q $SSH "$KEY" "root@$2:/opt/ssh_host_ed25519_key"
169+
scp -q $SSH "$HV_ARTIFACT_TAR" "root@$2:/opt/sci-delivery.tar"
170+
scp -q $SSH "$HV_DIR/sci-delivery.ref" "root@$2:/opt/sci-delivery.ref"
171+
scp -q $SSH ./.github/actions/test/integration/setup/virbr-vm.xml "root@$2:/opt/virbr-vm.xml"
172+
ssh $SSH "root@$2" bash -s <<'EOF'
173+
set -Eeuo pipefail
174+
chmod 700 /root/.ssh
175+
chmod 600 /opt/ssh_host_ed25519_key
176+
systemctl is-active containerd.service >/dev/null || systemctl start containerd.service
177+
REF=$(head -n 1 /opt/sci-delivery.ref)
178+
test -n "$REF"
179+
ctr -n k8s.io images import /opt/sci-delivery.tar
180+
rm -f /opt/sci-delivery.tar
181+
ctr -n k8s.io container rm -f sci-delivery >/dev/null 2>&1 || true
182+
# hostIPC is required: systemd-sysext waits on a dm-verity udev cookie
183+
# posted by the host udevd, so join PID 1's IPC namespace.
184+
ctr -n k8s.io run --rm --privileged --with-ns pid:/proc/1/ns/pid \
185+
--with-ns ipc:/proc/1/ns/ipc \
186+
--env HOST_ROOT=/host --env INTERVAL=0 --env NSENTER=1 \
187+
--mount type=bind,src=/,dst=/host,options=rbind:rw \
188+
"$REF" sci-delivery < /dev/null
189+
systemd-sysext status
190+
systemctl is-active libvirtd.service
191+
mkdir -p /var/log/libvirt/ch
192+
printf 'Host 192.168.122.*\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n IdentityFile /opt/ssh_host_ed25519_key\n IdentitiesOnly yes\n User root\n' > /root/.ssh/config
193+
chmod 600 /root/.ssh/config /opt/ssh_host_ed25519_key
194+
virsh pool-destroy default || true
195+
virsh pool-undefine default || true
196+
virsh pool-define-as default dir --target /var/lib/libvirt/images
197+
virsh pool-build default; virsh pool-start default; virsh pool-autostart default
198+
virsh net-destroy default || true
199+
virsh net-undefine default || true
200+
virsh net-define /opt/virbr-vm.xml
201+
virsh net-start default; virsh net-autostart default
202+
EOF
203+
done
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
2+
<name>HV_NAME_GOES_HERE</name>
3+
<memory unit='GiB'>6</memory>
4+
<vcpu>4</vcpu>
5+
<os>
6+
<type arch='x86_64'>hvm</type>
7+
<kernel>KERNEL_GOES_HERE</kernel>
8+
<cmdline>root=PARTUUID=ROOT_UUID_GOES_HERE console=ttyS0 systemd.gpt_auto=0 audit=0 rw</cmdline>
9+
<boot dev='hd' />
10+
</os>
11+
<features>
12+
<acpi />
13+
</features>
14+
<cpu mode='host-passthrough' check='none' migratable='on' />
15+
<devices>
16+
<interface type='network'>
17+
<mac address='MAC_GOES_HERE' />
18+
<source network='NETWORK_GOES_HERE' />
19+
<model type='virtio' />
20+
<!-- Keep the libvirt-assigned NIC out of slot 0x02: the NVMe
21+
device arrives through qemu:commandline and QEMU claims the
22+
first free PCI slot (0x02) for it. -->
23+
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' />
24+
</interface>
25+
<memballoon model='virtio'>
26+
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' />
27+
</memballoon>
28+
<console type='pty'>
29+
<target type='serial' port='0' />
30+
</console>
31+
<serial type='file'>
32+
<source path='/var/log/HV_NAME_GOES_HERE.log' />
33+
<target port='0' />
34+
</serial>
35+
</devices>
36+
<qemu:commandline>
37+
<qemu:arg value='-blockdev' />
38+
<qemu:arg value='driver=qcow2,node-name=sciroot,file.driver=file,file.filename=DISK_PREFIX_GOES_HERE/HV_NAME_GOES_HERE.qcow2' />
39+
<qemu:arg value='-device' />
40+
<qemu:arg value='nvme,serial=HV_NAME_GOES_HERE,drive=sciroot,bootindex=1' />
41+
</qemu:commandline>
42+
</domain>

‎.github/actions/test/integration/test/cloudhypervisor/action.yml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ runs:
1212
IP_HV1="192.168.122.2"
1313
IP_HV2="192.168.122.3"
1414
15+
# The qemu scenario's migrated VM still runs on HV2. Its nested I/O
16+
# starves the staging copy: the destination HV froze ~2h into the
17+
# ubuntu.raw transfer while idle HV1 takes the same copy in seconds.
18+
ssh $SSH_OPTS "$USER@$IP_HV2" 'virsh destroy VM-QEMU >/dev/null 2>&1 || true; virsh undefine VM-QEMU >/dev/null 2>&1 || true' || true
19+
1520
scp $SSH_OPTS ./.github/actions/test/integration/test/cloudhypervisor/vm.xml "$USER@$IP_HV1:/opt/vm-chv.xml"
1621
scp $SSH_OPTS /opt/ubuntu.raw "$USER@$IP_HV1:/var/lib/libvirt/images/ubuntu.raw"
1722
scp $SSH_OPTS /opt/ubuntu-cloud-init-ds.iso "$USER@$IP_HV1:/var/lib/libvirt/images/ubuntu-cloud-init-ds.iso"
@@ -82,7 +87,7 @@ runs:
8287
exit 1
8388
fi
8489
echo "Waiting for VM to respond to SSH... (\$i/\$MAX_ITER)"
85-
sleep 10
90+
sleep 60
8691
done
8792
EOF
8893
@@ -154,6 +159,6 @@ runs:
154159
exit 1
155160
fi
156161
echo "Waiting for VM to respond to SSH... (\$i/\$MAX_ITER)"
157-
sleep 10
162+
sleep 60
158163
done
159164
EOF

‎.github/actions/test/integration/test/cloudhypervisor/vm.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<domain type='kvm'>
22
<name>VM-CHV</name>
33
<memory unit='GiB'>2</memory>
4-
<vcpu placement='static'>1</vcpu>
4+
<vcpu placement='static'>2</vcpu>
55
<os>
66
<type arch='x86_64' machine='pc'>hvm</type>
77
<!-- see https://github.com/cloud-hypervisor/rust-hypervisor-firmware/issues/382 -->

‎.github/actions/test/integration/test/qemu/action.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@ runs:
1717
scp $SSH_OPTS /opt/ubuntu-cloud-init-ds.iso "$USER@$IP_HV1:/var/lib/libvirt/images/ubuntu-cloud-init-ds.iso"
1818
1919
# the cloud-init data is not migrated from source to destination, but needed for the VM to run at the destination
20+
# the disk itself must already exist at the destination, because --copy-storage-all does not create it
2021
scp $SSH_OPTS /opt/ubuntu-cloud-init-ds.iso "$USER@$IP_HV2:/var/lib/libvirt/images/ubuntu-cloud-init-ds.iso"
22+
scp $SSH_OPTS /opt/ubuntu.qcow2 "$USER@$IP_HV2:/var/lib/libvirt/images/ubuntu.qcow2"
2123
2224
- name: Start VM on HyperVisor 1
2325
shell: bash
@@ -80,7 +82,7 @@ runs:
8082
exit 1
8183
fi
8284
echo "Waiting for VM to respond to SSH... (\$i/\$MAX_ITER)"
83-
sleep 10
85+
sleep 60
8486
done
8587
EOF
8688
@@ -152,6 +154,6 @@ runs:
152154
exit 1
153155
fi
154156
echo "Waiting for VM to respond to SSH... (\$i/\$MAX_ITER)"
155-
sleep 10
157+
sleep 60
156158
done
157159
EOF

0 commit comments

Comments
 (0)