Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,35 @@ help: ## Display this help.

##@ Development

.PHONY: dev-vm-up dev-vm-provider dev-vm-down

dev-vm-up: ## Create the dev VM if needed and start it.
@if $(LIMA) list --quiet $(DEV_VM_NAME) 2>/dev/null | grep -Fxq "$(DEV_VM_NAME)"; then \
echo "$(DEV_VM_NAME) already exists"; \
else \
$(LIMA) create --yes --name=$(DEV_VM_NAME) \
--set='.portForwards[0].hostSocket = "$(DEV_VM_SOCKET)"' \
$(DEV_VM_TEMPLATE); \
fi
$(LIMA) start --yes $(DEV_VM_NAME)
@echo "Provider address: unix://$(DEV_VM_SOCKET)"

dev-vm-provider: dev-vm-up ## Start the provider inside the dev VM.
$(LIMA) shell $(DEV_VM_NAME) -- sudo -n -iu chp-dev bash -lc 'cd "$(CURDIR)" && exec go run ./cmd/cloud-hypervisor-provider \
--address /run/chp/iri-machinebroker.sock \
--provider-root-dir /var/lib/chp \
--cloud-hypervisor-firmware-path /usr/local/bin/hypervisor-fw \
--network-interface-plugin-name isolated \
--machine-class t1,2,1073741824'

dev-vm-down: ## Delete the dev VM.
-$(LIMA) delete --yes --force $(DEV_VM_NAME)

LIMA ?= limactl
DEV_VM_NAME ?= chp-dev
DEV_VM_TEMPLATE ?= dev-vm/chp-dev.yaml
DEV_VM_SOCKET ?= $(CURDIR)/iri-machinebroker.sock

.PHONY: manifests
FILE="config/machinepoollet-broker/broker-rbac/role.yaml"
manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
Expand Down
16 changes: 10 additions & 6 deletions config/samples/machine.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
metadata:
annotations:
foo: bar
labels:
bar: baz
app: local-test
spec:
class: small
image:
image: ghcr.io/ironcore-dev/ironcore-image/gardenlinux:latest
power: 0
class: t1
volumes:
- name: root
device: oda
local_disk:
size_bytes: 5368709120 # 5 GiB
image:
image: ghcr.io/ironcore-dev/gardenlinux/gardener:2150.12.0-kvm
7 changes: 5 additions & 2 deletions dev-vm/chp-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@ provision:
set -Eeuo pipefail

GO_VERSION="1.26.2"
IRONCORE_VERSION="v0.6.1"

case "$(uname -m)" in
x86_64) GO_ARCH="amd64" ;;
Expand Down Expand Up @@ -303,7 +304,7 @@ provision:
# it is available on PATH for local testing.
GOBIN=/usr/local/bin GOCACHE=/tmp/gocache GOPATH=/tmp/gopath \
/usr/local/go/bin/go install \
github.com/ironcore-dev/ironcore/irictl-machine/cmd/irictl-machine@main
github.com/ironcore-dev/ironcore/irictl-machine/cmd/irictl-machine@${IRONCORE_VERSION}
rm -rf /tmp/gocache /tmp/gopath
- mode: data
path: /opt/cloud-hypervisor-prepare.sh
Expand Down Expand Up @@ -589,7 +590,9 @@ networks:

# Port forwarding rules. Forwarding between ports 22 and ssh.localPort cannot be overridden.
# Rules are checked sequentially until the first one matches.
# portForwards:
portForwards:
- guestSocket: /run/chp/iri-machinebroker.sock
hostSocket: iri-machinebroker.sock
# - guestPort: 443
# hostIP: "0.0.0.0" # overrides the default value "127.0.0.1"; allows privileged port forwarding
# # default: hostPort: 443 (same as guestPort)
Expand Down
78 changes: 0 additions & 78 deletions dev-vm/install-chp.sh

This file was deleted.

112 changes: 46 additions & 66 deletions docs/development/dev_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,79 +7,70 @@ with nested virtualization enabled.

- [Lima](https://lima-vm.io/) v2.0.0+
- macOS with Apple Silicon (M3 or later for nested virtualization)
- Go (version matching `go.mod`)

## 1. Create the Dev VM
## 1. Start the Dev VM

The dev VM definition is located at `dev-vm/chp-dev.yaml`. It provisions an Ubuntu VM with:

- A `chp` system user/group (UID/GID 65532) that runs cloud-hypervisor
- A `chp-dev` user (your dev user inside the VM)
- Two cloud-hypervisor instances (`vm1`, `vm2`) managed via systemd
- Nested virtualization enabled for KVM access
- A `chp` system user/group (UID/GID 65532) for the QMP monitor and shared storage
- A `chp-dev` user (your dev user inside the VM) with access to `chp` and `kvm`
- Cloud Hypervisor instances started on demand by the provider

```bash
limactl create --name chp-dev dev-vm/chp-dev.yaml
limactl start chp-dev
make dev-vm-up
```

## 2. Shell into the VM
Use `make dev-vm-down` to delete the instance. After changing the Lima
template, run `make dev-vm-down && make dev-vm-up` so the instance picks up the
new configuration.

## 2. Run the Provider

```bash
limactl shell chp-dev
make dev-vm-provider
```

Your host home directory is mounted read-only at `~` inside the VM.
The provider uses `/var/lib/chp` as its root directory, shared between
`chp-dev` and `chp`. The target starts it inside the VM using the repository
mounted from the host and refreshes the `chp`/`kvm` group membership for the
process. It remains attached to the terminal; stop it with `Ctrl-C`.

## 3. Clone / Access the Repository
### Access the provider socket from macOS

Since the host home directory is mounted, you can access the repository directly.
For a writable copy:
Lima forwards the provider's Unix socket to the host automatically. Once the
provider is running, use the `unix://` URI form required by gRPC:

```bash
cd ~
git clone https://github.com/ironcore-dev/cloud-hypervisor-provider.git
cd cloud-hypervisor-provider
irictl-machine --address "unix://$(pwd)/iri-machinebroker.sock" get machine
```

## 4. Verify Cloud-Hypervisor is Running

The VM provisioning automatically starts two cloud-hypervisor instances via systemd:
The `irictl-machine` client must be compatible with the provider. Alternatively,
use the client installed inside the VM:

```bash
# Check service status
systemctl status cloud-hypervisor@vm1
systemctl status cloud-hypervisor@vm2

# Verify sockets exist
ls -la /run/chp/ch/
limactl shell chp-dev -- irictl-machine \
--address unix:///run/chp/iri-machinebroker.sock get machine
```

You should see `vm1.sock` and `vm2.sock` owned by `chp:chp`.
`make dev-vm-up` prints the resolved `unix://` address after starting the VM.
If the provider listens on a different address, update the `guestSocket` entry
in `dev-vm/chp-dev.yaml` and recreate the VM.

## 5. Verify KVM Access
## 3. Shell into the VM (optional)

```bash
# Check KVM is available
ls -la /dev/kvm

# Your user should be in the chp group
id
# Expected: groups=... chp ...
limactl shell chp-dev
```

## 6. Run the Provider

The provider needs to use `/var/lib/chp` as its root directory (shared between `chp-dev` and `chp`):
The host home directory is mounted read-only at the same absolute path inside
the VM, while the guest user's `~` is a separate writable home directory. The
repository is therefore available without cloning it:

```bash
go run ./cmd/cloud-hypervisor-provider \
--provider-root-dir /var/lib/chp \
--cloud-hypervisor-sockets-path /run/chp/ch \
--cloud-hypervisor-firmware-path /usr/local/bin/hypervisor-fw
cd /Users/<host-user>/path/to/cloud-hypervisor-provider
```

## 7. Run Tests
## 4. Run Tests

### Unit Tests

Expand All @@ -91,48 +82,37 @@ make test

### Integration Tests

Integration tests require the cloud-hypervisor instances to be running and accessible:
From inside the VM, integration tests start Cloud Hypervisor on demand and
require KVM plus the prepared firmware:

```bash
export CH_SOCKET_DIR=/run/chp/ch/
export CH_FIRMWARE_PATH=/usr/local/bin/hypervisor-fw
make integration-tests
```

## Architecture Overview

```
Host (macOS)
└── Lima VM (chp-dev)
├── chp-dev user ── runs the provider process
│ └── writes disk images to /var/lib/chp (0775, chp:chp)
├── chp user ────── runs cloud-hypervisor via systemd
│ ├── /run/chp/ch/vm1.sock
│ ├── /run/chp/ch/vm2.sock
│ └── reads disk images from /var/lib/chp
└── /dev/kvm ────── chp user has access via kvm group
```

## Troubleshooting

### "Failed to ping cloud-hypervisor socket"

The provider can't connect to the sockets. Check permissions:
The provider cannot connect to a per-machine socket. Check the provider root
and the effective groups used by the provider:

```bash
ls -la /run/chp/ch/*.sock
sudo -iu chp-dev id
ls -la /var/lib/chp
```

Sockets need group read/write for `chp-dev`. Fix with `sudo chmod g+rw /run/chp/ch/*.sock`.
Per-machine Cloud Hypervisor sockets are created by the provider below
`/var/lib/chp/machines` and are not pre-created by systemd.

### "Cannot open disk path" / "Permission denied (os error 13)"

Cloud-hypervisor (`chp` user) can't read disk images created by the provider (`chp-dev` user).
Ensure:
The QMP service (`chp` user) and provider (`chp-dev` user) must be able to
access the provider data directory. Ensure:

1. The provider uses `--provider-root-dir /var/lib/chp` (not the home directory)
2. `/var/lib/chp` has permissions `0775` and is owned by `chp:chp`
3. `chp-dev` is in the `chp` group (`id` should show `chp` in groups)
3. `chp-dev` is in the `chp` group (`sudo -iu chp-dev id` should show `chp`)

### "Error manipulating firmware file" / "No such file or directory"

Expand All @@ -154,7 +134,7 @@ sudo systemctl start cloud-hypervisor-prepare.service
Ensure nested virtualization is enabled (requires Apple M3+):

```bash
grep -c vmx /proc/cpuinfo 2>/dev/null || grep -c svm /proc/cpuinfo 2>/dev/null
ls -l /dev/kvm
```

If 0, nested virtualization is not available on your hardware.
If `/dev/kvm` is missing, nested virtualization is not available to the VM.
Loading