(14) n-vm - #1800
Draft
daniel-noland wants to merge 81 commits into
Draft
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
|
daniel-noland
force-pushed
the
pr/daniel-noland/n-vm
branch
from
September 3, 2026 04:15
812d59f to
1f4fdcf
Compare
daniel-noland
added a commit
that referenced
this pull request
Sep 3, 2026
The guest-booting tests do not skip without `testroot`/`vmroot` -- they panic: `n_vm::container::scratch_root_resolve` reports "could not find testroot/vmroot in the working directory or any parent, and N_VM_TEST_ROOT/N_VM_VM_ROOT are not set". So the roots are a precondition of any run that includes them, and the recipe that starts the run is the only place that can guarantee it. The harness gave `(setup-roots)` to `test` alone. Four recipes in this file invoke nextest, and the other three had nothing: - `coverage` had the `N_VM_TEST_ROOT` exports but no dependency, so its guard never fired. - `coverage-archive` had neither -- and it is the one CI actually runs: `ci::coverage` calls it, not `coverage`. It builds the archive in nix and then runs the tests on the runner, so it needs exactly what `test` needs. - `test-each` had neither, and its per-package archives include `dataplane-n-vm`'s suite. Measured on #1800: `check/debug` and `coverage/debug` reported an identical 28 failures, which is what a missing precondition looks like when three of four entry points are missing it and the fourth is the one you fixed. `ci.just` is the map worth keeping in mind here: `check-test`, `sanitize`, `shuttle`, `loom` and `cross-test` all funnel into `test`, so they inherit this; `coverage` and `test-each` do not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
daniel-noland
force-pushed
the
pr/daniel-noland/n-vm
branch
4 times, most recently
from
September 3, 2026 20:32
8ad9be1 to
74001d3
Compare
daniel-noland
added a commit
that referenced
this pull request
Sep 6, 2026
The guest-booting tests do not skip without `testroot`/`vmroot` -- they panic: `n_vm::container::scratch_root_resolve` reports "could not find testroot/vmroot in the working directory or any parent, and N_VM_TEST_ROOT/N_VM_VM_ROOT are not set". So the roots are a precondition of any run that includes them, and the recipe that starts the run is the only place that can guarantee it. The harness gave `(setup-roots)` to `test` alone. Four recipes in this file invoke nextest, and the other three had nothing: - `coverage` had the `N_VM_TEST_ROOT` exports but no dependency, so its guard never fired. - `coverage-archive` had neither -- and it is the one CI actually runs: `ci::coverage` calls it, not `coverage`. It builds the archive in nix and then runs the tests on the runner, so it needs exactly what `test` needs. - `test-each` had neither, and its per-package archives include `dataplane-n-vm`'s suite. Measured on #1800: `check/debug` and `coverage/debug` reported an identical 28 failures, which is what a missing precondition looks like when three of four entry points are missing it and the fourth is the one you fixed. `ci.just` is the map worth keeping in mind here: `check-test`, `sanitize`, `shuttle`, `loom` and `cross-test` all funnel into `test`, so they inherit this; `coverage` and `test-each` do not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
daniel-noland
force-pushed
the
pr/daniel-noland/n-vm
branch
from
September 6, 2026 04:40
9a27e11 to
c4f1428
Compare
daniel-noland
changed the base branch from
main
to
pr/daniel-noland/stats-collector
September 6, 2026 04:40
daniel-noland
changed the title
Absorb the `n-vm`, `n-it`, `n-vm-macros`, and `n-vm-protocol` crates from
the external `testn` repository into the workspace, replacing the
`n-vm = { git = "...testn", tag = "v0.0.10" }` dependency with path
dependencies, and add the supporting nix infrastructure for running tests
inside a QEMU/cloud-hypervisor guest.
This is a squash of the `n-vm-again` line, rebased onto current main. The
branch had forked before several subsystems landed upstream; commits that
are already merged (lifecycle, dpdk `test_support`, fixed-size, lookup,
match-action) or superseded on main (the acl/dpdk line) are dropped, as are
changes unrelated to the VM harness (cascade, the ckb dev-shell package,
npins bumps). Their stale copies of `Cargo.toml`, `default.nix`, the
justfile, and `dpdk-sys/build.rs` are likewise discarded in favour of the
authoritative versions on main -- only the VM test infrastructure and its
direct consumers are carried forward.
Contents:
- `n-vm` / `n-vm-protocol`: QEMU and cloud-hypervisor backends, dynamic
vsock allocation, hugepage and NIC-model configuration, scratch-only
container mode with nix-store bind mounts. `Arch` is an explicit
dimension threaded through the QEMU arg builders, so architecture-
divergent config is testable for every ISA on any build host.
- `n-vm-macros`: the `#[in_vm]` attribute and companion attributes
(`#[network]`, guest/hypervisor config), with 16 compile-fail trybuild
fixtures covering backend/NIC/hugepage validation.
- `n-it`: in-guest init system (PID 1) with mount-table-driven teardown.
- `nix`: `linux-fancy` kernel built from config fragments (VFIO, IOMMU,
virtio, e1000/e1000e), `testroot`/`vmroot` derivations, `merge-config.nix`.
- `hardware` / `dpdk-sys`: e1000/e1000e NIC binding and `rte_net_e1000` PMD
linkage; `driver()` returns `Ok(None)` for devices with no bound driver
rather than erroring, which is normal when the kernel driver was never
loaded or was already unbound.
- `mgmt`: re-enable `test_sample_config` under `#[in_vm]`.
- `justfile`: build `testroot`/`vmroot` in `setup-roots` and export
`N_VM_TEST_ROOT`/`N_VM_VM_ROOT` for `#[in_vm]` tests.
aarch64 cross support:
- Cross-arch `#[in_vm]` runs end to end: the foreign test binary runs under
user-mode QEMU inside the container (mirroring `scripts/test-runner.sh`)
rather than relying on a host `binfmt_misc` handler.
- aarch64's vIOMMU is QEMU's `virt` SMMUv3, lowered as a `-machine
iommu=smmuv3` option (not a `-device` like x86 intel-iommu) and auto-probed
from the device tree. `VIommuLowering.device` is an `Option` so a
machine-option-only IOMMU is expressible; `CONFIG_ARM_SMMU_V3` is added to
the guest kernel.
- The test QEMU is built headless (nixpkgs' `nixosTestRunner` profile). The
VMs always run `-nographic`, so the GUI display backends were dead weight
that dragged gtk4/gtk3/cairo/pango/vte/libepoxy/SDL into every test root --
invisible on the native cache-hit path, but loud on the cross path where
`qemu-system-aarch64` is built from source.
- cloud-hypervisor sets `mergeable = false`: it rejects `mergeable` together
with `shared`, and `shared` is required for virtiofs.
Correctness fixes folded in from the branch:
- `#[in_vm]` without `#[test]`/`#[tokio::test]` is now a compile error. It
previously compiled to an ordinary function that libtest never collected,
so the test silently never ran -- the worst failure mode for a harness.
- `#[should_panic]` with `#[in_vm]` is rejected. The generated function runs
under libtest at all three dispatch tiers, and a panic is absorbed at
whichever tier produces it, so the semantics are incoherent.
- `TestResult::parse` requires whitespace after `WIRE_PREFIX`. It previously
stripped the prefix with no boundary check, so `n-it-resultpass` parsed as
a `pass` verdict; for a parser whose contract is "garbled verdict means
failure", a spurious pass is the dangerous direction.
- The container cleanup guard stays armed until removal actually succeeds.
`into_result` previously disarmed both safety nets before
`collect_and_cleanup`, which can return early on inspect failure -- leaking
the container on exactly the error path the guard exists to cover.
- `n-it` treats `ECHILD` from `waitpid(-1, WNOHANG)` as the normal end of
reaping rather than logging an unexpected-errno warning every shutdown.
The `hardware/tests/dpdk_in_vm.rs` integration test (virtio-net/e1000/
e1000e) is intentionally NOT carried forward: it was written against the
pre-rewrite dpdk device API (`StartedDev` queue handles, `RxOffloadConfig`,
public `Headers` fields) and needs a port to the current API. It is
preserved on the `backup/n-vm-again` branch for a clean revival.
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 569fe84a1da7338d3a34267b1a722a00cd54675a)
…attrs `#[in_vm]` required a companion `#[test]` / `#[tokio::test]` below it, which was a trap rather than a feature. An attribute macro only receives the attributes written *below* it, so a `#[test]` placed above `#[in_vm]` was invisible to the macro and tripped its "missing test attribute" check -- with a diagnostic that read as nonsense to anyone looking at a `#[test]` on the line above. Both pre-existing call sites in this repo (mgmt's reconcile_fuzz and hardware's scan tests) were written in that broken order. Replace it with `#[n_vm::test]`, which *is* the test attribute: - `#[test]` is always injected, so there is no companion attribute to write, omit, or misorder. The `missing_test_attr` check and fixture are gone. - The guest runtime comes from the signature: a `fn` runs its body directly, an `async fn` runs on a tokio runtime. Flavor moved from `#[tokio::test]` into this attribute's own arguments -- `#[n_vm::test(multi_thread, worker_threads = 4)]`. Most of this already existed: the macro already consumed `#[tokio::test]`, parsed its flavor, injected `#[test]`, and stripped `async`; it just did so only when the user supplied that second attribute. - A stray `#[test]` or `#[tokio::test]` is now a hard error with the migration, rather than being silently dropped. Dropping `#[tokio::test]` quietly would silently change which scheduler the test ran on. - `worker_threads` without `multi_thread` is rejected: a current-thread runtime has no worker pool to size, so accepting it would misreport what the test runs on. Route the remaining attributes by tier. Previously every leftover attribute was emitted onto the generated dispatch function, so it ran at *all three* tiers. That is wrong for anything with side effects: `#[wrap(with_caps( [CAP_NET_ADMIN]))]` exists precisely because the body needs privileges only the guest has, and running it on the unprivileged host tier failed with EPERM before a VM was ever started. Harness-level attributes (`#[cfg]`, `#[ignore]`, doc comments) stay on the dispatch function where libtest and rustdoc can see them; everything else moves onto an inner function that only the guest branch calls. `#[cfg_attr]` is deliberately body-level -- it most often expands to a body wrapper, and the macro cannot know what it expands to. Also fixes a guest kernel gap found while validating this: the allnoconfig base left `CONFIG_INOTIFY_USER` off, and no fragment asked for it, so anything using a file watcher died in the guest with a bare "Failed to init inotify" -- which is what stopped mgmt's `test_sample_config` (EAL itself came up fine). `reconcile_fuzz` is marked `#[ignore]` rather than fixed: with attribute routing its `with_caps` now correctly runs as root in the guest, but bolero resolves its corpus work dir from the test's `file!()` path via the manifest dir, and the workspace source tree is not mounted in the VM. Running it on the host is not an alternative -- needing CAP_NET_ADMIN is why it wants a VM. Exposing sources to the guest is a design question, not a quick fix. Verified: 211 unit tests, 17 compile-fail fixtures, and all 13 in-VM integration tests pass (both async flavors among them, which is what exercises the inner-function routing for `async` bodies). Signed-off-by: Daniel Noland <daniel@githedgehog.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 041dc76279d85ec7022a4e0ec39b0087eb045e98)
`bolero::check!()` aborted with "could not resolve target work dir" inside
the VM, which is why `reconcile_fuzz` was ignored. The cause is not the
fuzzing engine and not specific to coverage-guided runs: `check!()` records
`file!()` and `CARGO_MANIFEST_DIR` at compile time and later resolves them to
locate a corpus directory. In a guest whose working directory is `/` and
which cannot see the source tree, neither resolves.
This is on the plain `cargo test` path, not just under a fuzzer: bolero only
skips corpus discovery under `--cfg fuzzing_random`, which nothing here sets,
so every bolero test took the branch that needs a resolvable work dir.
Mount the workspace root at a fixed `/workspace` in the guest and make it the
test process's working directory. Cargo records `file!()` relative to the
*workspace* root (e.g. `mgmt/tests/reconcile.rs`), so the relative path then
canonicalizes against the mount and bolero resolves
`<pkg>/tests/__fuzz__/<name>` as it does on the host.
Matching the host's absolute workspace path in the guest would also work --
bolero falls back to walking `CARGO_MANIFEST_DIR`'s ancestors -- but that
path varies per developer and per CI runner, so it cannot be baked into the
`vmroot` derivation that has to pre-create the mount point. A fixed path
plus a working directory needs nothing user-specific.
The workspace root is found by walking up from the working directory for a
`Cargo.toml` declaring `[workspace]`, overridable via `N_VM_WORKSPACE`. The
walk is necessary because cargo runs tests from the *package* root, not the
workspace root -- which is also why `canonicalize()` fails on the host and
bolero falls through to its ancestor search there.
Mounted read-only for now. virtiofsd serves the root share with
`--readonly`, so the guest cannot write regardless of the bind mount's flags,
and claiming otherwise in the mount would be a lie:
PROBE cwd = "/workspace"
PROBE mount: none /workspace virtiofs ro,relatime 0 0
PROBE write FAILED: Read-only file system (os error 30)
That is enough to read an existing corpus and to generate fresh inputs, but
newly-found crashes cannot persist back to the host tree. Note that
`--announce-submounts` already makes /workspace its own mount in the guest,
so relaxing this later needs the virtiofsd flag and a remount, not a second
virtiofsd instance.
Also make the guest-body wrapper conditional. Wrapping the body in an inner
function is only needed when there are body-level attributes to route, and
the wrapper is observable: bolero derives its on-disk corpus directory from
`type_name` of a probe declared at the `check!()` site, so wrapping
unconditionally would bake `__n_vm_guest_body` into that path for every test
and churn it whenever this macro's internals are renamed. Tests without body
attributes now expand as before.
`reconcile_fuzz` is un-ignored and fuzzes in the guest, with `with_caps`
raising CAP_NET_ADMIN as root there rather than failing on the host:
running main process from /workspace
test reconcile_fuzz ... run time: 2.09s | iterations/s: 2.39 | rng inputs: 5
| exit reason: max duration (2s) exceeded
Verified: 211 unit tests, 17 compile-fail fixtures, all 13 in-VM integration
tests, and reconcile_fuzz.
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e7a90bf6e037e8d041597bddf1d91d13d3180c71)
…tory
A fuzz target needs somewhere to save generated inputs and crash artifacts,
but making the shared workspace writable would defeat much of the reason to
run the test in a VM at all. The premise of these tests is that they drive
our code into misbehaving against a real kernel; a target that gets far
enough to issue an unintended `unlink` must not be able to reach the
developer's source tree.
So the writable window is exactly one directory wide, and it is scoped by
the *server* rather than by the guest:
- The root share keeps `--readonly`, so its daemon cannot write anywhere at
all no matter what the guest does with its mount flags.
- A second daemon serves only the corpus directory, so there is nothing else
for it to reach even though it can write.
- `n-it` overmounts the corpus directory (announced as its own submount) with
that second share.
Guest-side mount flags alone would not be sufficient here: `mount -o
remount,rw` is available to root in the guest, and root in the guest is
exactly what a fuzz target runs as.
Opting in is explicit, via a `#[corpus]` companion attribute, so that "this
test can write to the source tree" is visible at the call site and reviewable
rather than ambient. Without it a test sees no writable filesystem backed by
the host at all.
Granularity is the `__fuzz__` directory rather than the per-test
subdirectory beneath it. The per-test name comes from bolero's own
`fuzz_dir()` derivation -- which strips `test_`/`fuzz_` affixes and is
computed from the call site's `type_name`, the very thing this macro's
body wrapper perturbs -- so depending on it would couple the mount layout to
bolero internals. A `__fuzz__` directory exists only to hold corpora, so it
is already a tight enough boundary.
The host tier creates the directory as the invoking user, since the guest
sees the workspace read-only and cannot create it, and virtiofsd's existing
`--translate-uid=squash-host` keeps anything written inside it owned by the
developer rather than by root.
Both halves of the boundary are asserted by
`corpus_is_writable_and_rest_of_workspace_is_not`, which writes inside the
corpus directory and then requires `ReadOnlyFilesystem` for two paths outside
it. Verified end to end:
virtiofs virtio3: discovered new tag: corpus
writable corpus share requested at /workspace/mgmt/tests/__fuzz__
corpus share mounted read-write at /workspace/mgmt/tests/__fuzz__
$ ls -ld mgmt/tests/__fuzz__
drwxr-xr-x 2 dnoland users ...
Verified: 211 unit tests, 17 compile-fail fixtures, 14 in-VM integration
tests, and reconcile_fuzz (now `#[corpus]`).
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 283dfe90575b8b5ebc1bb50a6e095427317be8b0)
`Arch::kernel_image_path` hardcoded `/bzImage` and `/Image`. That is correct only while there is exactly one kernel per architecture, and it stops being true as soon as we want to run the same test against both a kernel built from our own config fragments and a distro's production kernel -- which is the point of the exercise: the minimal config documents what we depend on, and the production kernel checks that claim against what we actually ship on. So which kernels exist becomes a fact about the nix build rather than a constant in Rust. nix emits `testroot/n-vm-manifest.json` alongside a `kernels/<profile>/` tree, and the container tier resolves the image path from it. The staging this implies is not new: `just setup-roots` already materializes `testroot`/`vmroot` from nix, and `ScratchRoots::resolve` already finds them. Extending that is deliberate -- cargo must never invoke nix, because nix does not handle the recursion well. Artifacts are built first, tests only ever read them, and a missing manifest is a hard error naming the fix rather than a silent fallback to a stale path. No new mount code was needed: `build_scratch_mounts` already bind-mounts every first-level `testroot` entry at the container root, so `kernels/` lands at `/kernels`. The resolved path is threaded through `TestVmParams` rather than looked up in the backends, for the same reason `arch` already is (see development/code/avoid-global-reasoning.md, "Code should be modular"): it keeps the argument lowering a pure function of its inputs, so both backends stay testable for any kernel on any host with no manifest on disk. The manifest carries `boot`, `config`, `initramfs`, and `modules` fields that nothing reads yet. They are the shape a modular foreign kernel needs -- one whose virtiofs is `=m`, and so cannot mount the root that holds its own modules without an initramfs first -- and emitting them now avoids a schema migration later. Only the two `host_pages = "4k"` integration tests are runnable on a host with no hugepages reserved; both pass, on both backends. The rest fail identically before and after this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 612e552f3f88a319bbf2e857687e8709fb454d53) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A test can require a kernel feature its kernel does not have. Today that surfaces as whatever the missing feature breaks -- a socket option returning `ENOPROTOOPT`, a `tc` filter that will not attach, a mount that fails for no stated reason -- deep inside a test body, in a VM, with no hint that the kernel is the cause. This records the config so the question can be asked directly, and adds `KernelConfig` to read it. The three Kconfig states are kept apart rather than collapsed to a boolean: `y` and `m` both satisfy a requirement, but `m` additionally implies a module to load, and that distinction is the whole reason a foreign kernel is harder than one we build. The config sources differ by kind and that is fine, because the manifest normalizes the result: a kernel we build has `linux-fancy.configfile` already merged and dependency-resolved, while a foreign kernel's config has to be recovered from the image with `extract-ikconfig`. Both land at `kernels/<profile>/config`, so nothing downstream cares which it was. Nothing consumes this yet. It is the prerequisite for checking a test's declared requirements against its kernel, which matters most for the case this is all aimed at: a skipped test reading "requires CONFIG_NET_CLS_FLOWER, kernel has it n" is a finding about the kernel we ship on, where the same test failing obscurely is just noise. Note for later: our own kernel sets neither CONFIG_IKCONFIG nor CONFIG_MODULES, and has zero `=m` symbols -- the static-linking premise the original design leaned on. Exercising an initramfs boot against a kernel we control means turning CONFIG_MODULES on first. Parsing tolerates unrecognized lines rather than rejecting them; a `.config` is generated, not hand-written, and failing closed on a banner comment would reject a file that is almost certainly fine. The real config is 1955 entries: 879 tristates, 54 numbers, 13 strings, 3 hex, and 1006 explicit `# CONFIG_X is not set`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit e040a7f2768f67b0f0b685b3d18983e91792c910) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Groundwork for retiring the #[hypervisor]/#[guest]/#[network] companion attributes in favour of a `const VmConfig` written at the call site. The attributes are stringly-typed, so they re-introduce invalid states the enums in this module already make unrepresentable, and then hand-check for them -- `hugepage_count` with `hugepage_size = "none"` is a compile-fail test only because the two are independent strings. Adds `VmConfig::DEFAULT` so struct update syntax can express an override, and `check`/`assert_valid` as `const fn` so a contradiction is a build error rather than a launch failure two tiers later. `Default` now delegates to `DEFAULT` rather than being derived: a derived impl consults each field's own `Default` independently, and nothing would notice if the two answers diverged. One `const fn check` serves both the compile-time assertion, which can only panic with a literal, and the runtime path, which can afford to format the actual numbers in. Otherwise each would need its own copy of the conditions and they would drift; a test pins them to the same verdict. `assert_valid_for` additionally rejects a NIC model the pinned hypervisor cannot emulate. This is worth calling out because it was expected to regress: the config is a const the macro cannot read, so that check looked like it had to become a runtime skip. It does not -- the *backend* is one of the macro's own arguments, so combining the two in a `const fn` keeps it a compile error. `RequestedBackend::resolve` gains `needs_qemu`, and the unpinned case now improves rather than degrades: a test that asks for an emulated Intel NIC without naming a backend is not a contradiction, it is a test that wants QEMU, so it gets QEMU -- with KVM intact when the guest arch matches the host. Only an explicit cloud-hypervisor pin conflicts, and that is the case `assert_valid_for` catches at compile time. No behaviour change: 195 unit tests pass, and both runnable integration tests still boot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 6cfa6e7e5abd448f89fc80b11a16e82e253ea222) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Retires #[hypervisor], #[guest], and #[network] in favour of a `const VmConfig` named by `config = PATH`. The motivation is not brevity. An attribute is the one place an editor cannot help: no completion on field names, no hover, no go-to-definition, and errors land on the attribute rather than the value. A `const` is ordinary Rust in an ordinary position, so all of that works -- and a testing framework is touched often enough for that to matter daily. It also removes hand-written checks the type system already had covered. `hugepage_count` alongside `hugepage_size = "none"` needed a dedicated compile-fail test only because the two were independent strings; `GuestHugePageConfig::None` has no count to set. Six compile-fail cases existed solely to police attribute parsing and are gone with it, along with ~240 lines of the macro. The NIC/backend check survives as a compile error, which is worth spelling out because it was expected not to. The macro cannot read a config named by path -- but the *backend* is one of its own arguments, so the two meet in `VmConfig::assert_valid_for`, a `const fn` rustc evaluates. An unpinned test that asks for an emulated Intel NIC now simply gets QEMU, since that is what it evidently wants; only an explicit cloud-hypervisor pin is a contradiction. That assertion is emitted *beside* the test function rather than inside it. Inside, it silently did nothing: `#[test]` items are stripped in a non-test build, and the assertion went with them -- so it could never be covered by a compile-fail case, which is exactly how this was caught. At module scope it is checked in every build. Only `#[cfg]` carries over to it; `#[ignore]` and doc comments are meaningless on a const. A leftover companion attribute is now a hard error rather than being ignored. Nothing consumes those attributes any more, so left alone the VM would quietly boot with the default configuration while the test appeared to ask for something else -- the worst outcome for a migration. Shared configs in the integration suite also make the duplication visible: IOMMU_VM and HOST_4K_VM are each used by a cloud-hypervisor test and a QEMU test that previously restated the same settings. 195 unit tests, 13 compile-fail cases, both runnable integration tests still boot, clippy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit a97b41f0642d98f99630ba4a7c1327a364de3042) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A test can need a kernel symbol its kernel does not have. Until now that surfaced as whatever the absence broke -- an ioctl returning ENOTTY, a filter that would not attach -- deep in the test body, several tiers from the cause. `VmConfig::kernel_features` states the dependency, and the container tier checks it against `kernels/<profile>/config` before booting. Declaring it also makes the dependency a fact rather than an accident. A test that needs NET_CLS_FLOWER and does not say so passes only because the fragment list happens to enable it, and keeps passing until someone trims the list. The catalog carries both the Kconfig symbol and the module name, because they are used by different consumers and are not mechanically related -- CONFIG_NET_CLS_FLOWER builds cls_flower.ko. The symbol answers "does this kernel have it"; the module name answers "what must be loaded first", which is what a modular foreign kernel will need. `builtin_only` is kept distinct from `modular` so a symbol found at `=m` that cannot be modular is recognisable as a bug in the table rather than yielding a module name that does not exist. It is a module of consts rather than an enum so a consuming project can declare its own features without editing this crate, while still getting a compile error for a typo and completion on the catalog. An unmet requirement is a hard failure, not a skip: every kernel that exists today is one we build from our own fragments, so a missing symbol means the fragment list is wrong. When a profile can name a kernel we did not build, an unmet requirement there is a finding about that kernel rather than a bug in our config, and should become a skip. Doing this immediately found one: `mlx5-sriov.config` requests 17 MLX5 symbols and **none of them reach the built kernel**. CONFIG_MLX5_CORE depends on NET_VENDOR_MELLANOX, which the allnoconfig base leaves at `n` and no fragment enables -- and unlike `select`, a `depends on` is never auto-satisfied. merge_config.sh warns about exactly this and nothing reads the warnings. The fragment is left alone here; this commit only makes the gap visible. 202 unit tests, 13 compile-fail cases, 3 runnable integration tests including one that declares real requirements and boots. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 9670a2180d35c06c3dfba777f00d832ac8e57474) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`mlx5-sriov.config` requested 17 MLX5 symbols and not one of them reached
the built kernel. `CONFIG_MLX5_CORE` sits inside an `if
NET_VENDOR_MELLANOX` block, and that gate is `default y` -- which is
exactly what makes it easy to miss, because allnoconfig overrides defaults
to n and a `depends on`, unlike a `select`, is never auto-satisfied.
`MLX5_CLS_ACT` additionally needs NET_TC_SKB_EXT ("TC recirculation
support"), also off under allnoconfig. That one selects SKB_EXTENSIONS
itself, so a single line covers it.
Everything else the chain needs was already present: PCI, INET,
NETDEVICES, ETHERNET, NET_SWITCHDEV, BRIDGE, MACSEC, NET_CLS_ACT,
PSAMPLE, VFIO_PCI_CORE, INFINIBAND, VDPA. Two near-misses resolve
themselves -- PTP_1588_CLOCK is off but PTP_1588_CLOCK_OPTIONAL is
`default y if PTP_1588_CLOCK=n`, and MLX5_TC_SAMPLE accepts PSAMPLE at
either y or n. Checked against the Kconfig in the 6.18.20 tree we
actually build, since these chains move between releases.
The failure was silent by construction: merge_config.sh warns for each
dropped symbol, the build succeeds, and the result is a kernel with no
mlx5 support. The fragment's author knew the hazard -- there are already
comments explaining the VFIO_CONTAINER, IOMMU_SUPPORT and VDPA gates for
the same reason -- the vendor gate was just missed. Comments here follow
that precedent.
Found by the kernel-feature check added in the previous commit, on its
first real use.
All 45 symbols the fragment requests now land, all 17 MLX5 among them.
Existing tests still boot on the rebuilt kernel; a probe requiring
MLX5_CORE, VFIO_PCI and NET_CLS_FLOWER now passes where it previously
failed the pre-boot check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit f85581577c0a00eb62c0a56604e95d75f62eda14)
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Groundwork for running one test under several environments. A profile becomes a (kernel, hypervisor) pair rather than just a kernel, and the manifest now carries two of them -- `cloud_hypervisor` and `qemu` -- sharing the single kernel we build. The hypervisor belongs to the environment, not to the test. That `n-vm/tests/integration.rs` disagrees is visible in the file: `test_which_runs_in_vm_with_iommu` and `..._with_qemu_iommu` are the same test written twice, as are the two `vm_boots_with_standard_host_pages` variants. They now at least share a named config; making the hypervisor a profile is what will let them collapse to one test each. Profile names are Rust identifiers because each is destined to become a module in the generated test tree (`some_test::qemu`), so nextest can select a single environment with a filter. Sharing one kernel between profiles is why the artifact directory is named for the kernel (`kernels/union/`) rather than for the profile: several profiles reference the same image, and installing it once keeps the manifest honest about that. An unrecognised hypervisor is rejected rather than defaulted. Defaulting would run the test somewhere other than where the profile said, which is worse than not running it. Nothing selects a profile by name yet -- the macro still routes the backend through `RequestedBackend`. 205 unit tests, 3 runnable integration tests, clippy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 31cc82d516cf13bff4978a80423523474b445645) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`N_VM_PROFILE=<name>` points a whole invocation at one environment, overriding the manifest's `default`. A run mode rather than a per-test setting, which is deliberate: it needs no compile-time knowledge of which profiles exist, so it works today while the question of how a proc macro learns the profile list is still open. A test that resolved to a different hypervisor than the selected profile runs on is skipped, not failed -- "this test does not suit this environment" is a fact about the pairing, not a defect in either. That is what makes selection mean something: without it, choosing a profile would change the kernel while leaving the test on whatever backend it asked for. The check lives in the host tier because that is the last place a skip can still be expressed; inside the container the only outcomes left are pass and fail. It reads the manifest from `testroot` on the host, while the container tier reads the same file from its own mount point. A manifest that cannot be read is deliberately *not* treated as a mismatch. The container tier reports that failure with a better message, and skipping here would disguise a broken `testroot` as a routine environment mismatch. An unknown profile name is a hard error listing the alternatives, because a typo would otherwise run the default environment while appearing to select another. Verified both ways: under the default profile the QEMU test skips and the cloud-hypervisor one boots; under `N_VM_PROFILE=qemu` the reverse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 197558ca2400c2759cbc1d1be9b1076716f80062) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Step 4 of the profile plan needs a kernel whose virtiofs is a *module*, so
the initramfs boot path can be developed against a config we control rather
than against a distro's artifacts -- where a failure could equally be a bad
fetch, a bad extraction, or a bad boot.
Our default kernel has zero `=m` symbols and no CONFIG_MODULES at all: the
static-linking simplification the original design leaned on. Flatcar
4593.2.4 has CONFIG_VIRTIO_FS=m and CONFIG_FUSE_FS=m, which is the
bootstrap deadlock -- mounting the workspace needs virtiofs, virtiofs is a
module, and the module tree lives on the workspace. `modular.config`
reproduces exactly that shape.
`linux-fancy` becomes `mkLinuxFancy { extraFragments }`, a first step
toward the parameterised kernel builder the profile design calls for.
Extra fragments merge last -- after the arch fragments and after
disable.config -- because modular.config's whole job is turning `=y` into
`=m`, which only works if it wins.
The trap worth recording: the first build succeeded and produced a kernel
with **no modules installed anywhere**, despite CONFIG_MODULES=y. nixpkgs
decides at eval time whether a kernel is modular, and that decision creates
the `modules` output and the `modules_install` step
(`kernel/build.nix:81,151`). It reads the configfile to find out, but only
when that is a literal path or `allowImportFromDerivation` is set. Ours is
a derivation, so nixpkgs saw an empty config and concluded "not modular" --
another silent success producing a wrong artifact, like the mlx5 fragment.
Answered by reading our own fragments, which are paths, so no IFD is
involved. IFD would force the config derivation to build during
evaluation and is unavailable under restricted eval; a hand-set flag would
be a second source of truth able to drift from the fragment describing it.
vsock stays built in for now. Flatcar has it modular and eventually this
profile should too, but keeping it static means a failure in module
loading can still report itself over the result channel rather than
producing a silent VM.
Modules are left uncompressed and unsigned: Flatcar ships `.ko.xz`, and
the plan is for nix to decompress boot-critical modules when building the
cpio so the guest never needs a decompressor. Keeping ours plain keeps
that difference out of the way while the boot path is brought up.
Produces fuse.ko and virtiofs.ko with `modules.dep` recording
`virtiofs.ko: fuse.ko` -- the dependency the cpio's load order is derived
from. The static kernel is untouched: same store path, no rebuild, 205
unit tests and the integration suite unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 50ab228c76139f85208847e90dbb205aeb477245)
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Resolves the boot-critical module closure and packs it with a pre-init into a cpio. Needed only when the root filesystem transport is `=m`: mounting the workspace needs virtiofs, virtiofs is a module, and the module tree lives on the workspace. The initramfs is the only way out, because the kernel unpacks it itself, from memory, before any driver loads. The closure and load order are resolved here, by the real `modprobe` against the real module tree, rather than in the guest. We know the answer at build time, so the pre-init reads an ordered list and calls `finit_module` down it -- no `modules.dep` parsing, no dependency resolution, no uevent handling in the VM. Verified: the emitted order is `fuse.ko` then `virtiofs.ko`, dependency first. Modules are decompressed on the way in. A distro tree ships `.ko.xz`, and `finit_module` cannot read that without CONFIG_MODULE_DECOMPRESS, which is not something we can rely on for someone else's kernel. Doing it here means the guest never needs a decompressor. The compressor is chosen from what the *target* kernel can decompress, read from its config at build time (so no import-from-derivation). This is not assumable: ours has CONFIG_RD_GZIP=n and CONFIG_RD_ZSTD=y, while Flatcar ships a `.cpio.gz`. Guessing would produce a panic with no useful message. The kernel sniffs the format from magic bytes, so the filename carries no information and is just `initramfs`. The cpio also carries `/newroot`, because the boot path pivots. An earlier draft of the design had it *not* pivot -- `vmroot` is only 30 KB of symlinks and mount points, so the initramfs could simply be the root. That reasoning missed the binding constraint: there are two virtiofs shares, `root` and `corpus`, and `/nix/store`, `/workspace` and `/test-bin` live *inside* the root share rather than being shares of their own. Not pivoting would mean the pre-init hardcoding a list of vmroot's subdirectories to bind into place -- a worse coupling than the syscall it avoids. `mk-initramfs` is exported but not yet wired to a profile: the pre-init does not exist yet. Verified by building against a stand-in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 43d46b161b07945814b915da9fb3608608f36920) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
PID 1 inside the initramfs, for guest kernels whose root filesystem
transport is a module. Four steps: load the modules named in
/modules.load, mount the virtiofs root share at /newroot, pivot onto it,
exec /bin/n-it.
Separate from n-it rather than a mode of it, because n-it is dynamically
linked against /nix/store -- which is only reachable *after* virtiofs is
mounted, so it cannot be the thing that mounts virtiofs. Splitting leaves
this binary with a dependency surface of syscalls alone, and leaves n-it
unconstrained, which matters because that is where the interesting logic
lives.
Static linking works against glibc with `-C target-feature=+crt-static`
plus glibc's `static` output on the link path; the result is `static-pie
linked` with no interpreter. musl was the fallback and is not needed --
which is fortunate, since the toolchain ships rust-std only for
x86_64-unknown-linux-gnu and wasm32-wasip1, so a musl target would have
meant changing the toolchain too.
Two details that would each have produced a VM that boots and does
nothing:
argv is forwarded to n-it. The kernel hands everything after `--` on its
command line to init as arguments, and n-it reads them
(`n-it/src/child.rs:66`) to learn which test binary to run and which test
to select. Dropping them boots a VM that runs no test.
devtmpfs is mounted here rather than relied upon. The kernel calls
`devtmpfs_mount` from `prepare_namespace`, which is skipped when a cpio
supplies the root -- so on this path /dev starts empty, /dev/console does
not exist, and the kernel's own "unable to open an initial console" leaves
this process with no stdio. Without mounting it first, every subsequent
failure would be invisible on the console.
`pivot_root(".", ".")` rather than a separate put_old directory: it needs
no writable directory under the new root, which is mounted read-only.
No dependency resolution, no modules.dep parsing, no uevent handling --
the nix build already resolved the closure and its order with the real
modprobe, and this reads the answer.
Not yet wired to a profile or booted; that needs the manifest entry and
the container-tier cmdline for an initramfs profile.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit e9a00221df1dc923c26a430126c6e757f604b9e9)
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Builds the pre-init statically, packs it into the modular kernel's initramfs, and installs both alongside the module tree so the manifest can describe a profile that boots through an initramfs. The static link needs two things and neither is a default. `+crt-static` asks for it; glibc's static archives then have to be found, and they live in a *separate output* that is not part of the sysroot -- so without the library path the link fails on `-lc`. Applied with `overrideAttrs` rather than through `workspace-builder`, because `args` is merged with `//`, which would replace the whole `env` attrset rather than adding to it and silently discard the sysroot and toolchain settings every other crate depends on. Artifacts are now keyed by *kernel* rather than by profile, because three profiles share two kernels: the image is installed once and referenced by each. `boot` moves with it -- it follows from whether a kernel can reach its own root, not from which hypervisor a profile pairs it with, and putting it on the profile would let two profiles sharing a kernel disagree about something they cannot disagree about. `initramfs` and `modules` are emitted only for kernels that have them, so the direct-boot profiles are byte-identical to before. The per-profile compressor earned itself immediately: the modular kernel has `CONFIG_RD_ZSTD=y` and every other `CONFIG_RD_*` unset, so zstd is not a preference but the only format it can decompress. A hardcoded gzip -- which Flatcar's own `.cpio.gz` would have suggested -- would panic with no useful message. Verified in the built artifact: the cpio contains `init`, `modules.load`, `fuse.ko`, `virtiofs.ko` and `newroot`; `init` is `static-pie linked` and `ldd` reports `statically linked`; the load order is fuse before virtiofs. Not yet booted. The container tier still passes no `-initrd` and builds a cmdline with `root=`/`init=`, which an initramfs boot ignores -- that is the next step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 36245639c29bcdd58f3a8efaba749b739a233092) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Both were crate constants, so the only VM anyone could ask for was the one this crate happened to want. They are the two numbers that decide how many of these VMs a host can run at once, which is what makes them worth setting per test rather than per crate: a fuzz campaign wants more memory than the default gigabyte, and most tests want fewer than six vCPUs so that more of them fit. The four topology constants are gone. Both hypervisors reject a topology whose levels do not multiply to the vCPU count, so the arrangement is now derived once from the count and used by both. At the default of 6 it reproduces the old 1 socket x 3 dies x 1 core x 2 threads exactly, which `the_default_vcpu_count_lowers_to_the_topology_it_always_did` pins. `memory_mib` and `guest_hugepages` are independent to write and not independent in effect; `check` now catches a VM too small for its own reservation, which previously booted without hugepages and said so only on the console. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit df299f83e4ba44c18406c7dec33ca9d0b0f02a1b) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Failover, ECMP and a bond losing a member all need more links than the two that were written out by hand, and a test that never touches the network was paying for those two anyway -- each is a TAP device, a virtio device and a queue pair set up before the guest runs. The interfaces are now derived from the count. `MAX_FABRIC_NICS` is where the derivation runs out of addresses, not a policy: the MAC's last octet and the link-local address are both the index, and a collision does not fail -- the guest brings both links up and forwards out of whichever answered, which is the symptom a failover test would then be explaining. A fabric link does not reach a cloud-hypervisor guest at all. They sit on PCI segment 1 and that guest enumerates nothing there, with or without the vIOMMU; moving them to segment 0 makes all of them appear. This predates the count being a lever -- the two default links were equally invisible -- so the integration test pins QEMU and says why, and the defect is left reported rather than worked around. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 211d58f1e3897729f433cf573d4d1f5ae1931646) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The choice of kernel was a whole-run lever only. `N_VM_PROFILE` points a suite that has no opinion at a different environment, which is what it is for, but nothing that depends on a *modular* kernel could say so: the union kernel builds everything in, so a test of loading a module -- or of failing to load one -- had no way to ask for flatcar and would have been swept back onto a kernel with no modules by the next sweep. A declared profile therefore outranks the environment, matching a pinned `RequestedBackend`: naming one is saying what the test is for. A profile is a (kernel, hypervisor) pair, so it can contradict a pinned backend, and that is now a failure rather than the skip `resolve` would produce. Skipping is right when the environment chose the profile -- the run asked for something this test cannot use -- and wrong when the test wrote both halves itself, because a skip is reported as a pass and the contradiction would never be seen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 9c1a7f2433333d65c384e974f8d03eb59fede4f5) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
libfuzzer's default `-rss_limit_mb` is 2048, twice the default guest, so it was never the limit that fired: the guest kernel ran out first and killed the engine. An engine killed from outside writes no artifact, so the input that grew the heap is gone -- the finding a memory bug is supposed to produce is exactly what the failure destroys. `None` rather than a floor when nothing is left, because libfuzzer reads `-rss_limit_mb=0` as "no limit"; saturating would turn a VM too small to fuzz into one whose engine is unbounded. A fuzz target that lands there is rejected by `check`, so that arm is only reachable for a config that never declared itself one. Appended rather than substituted, so `just fuzz -E='-rss_limit_mb=...'` still wins -- libfuzzer takes the last occurrence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit dcbe76f4aa9fddc1a1837b3c7443fcea56623050) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The startup sequence this dataplane runs identifies network devices and unbinds them from the kernel. The failure worth catching is it picking the wrong one -- by ordinal, or by whatever `/sys` listed first -- because on this system that takes the management link away and the host needs a physical reboot to come back. A VM where every NIC is the same device cannot see that: there is no wrong one to pick. So the model moves from the VM to the interface. `nic_model` still names the management link, and is what a `Uniform` fabric is made of; `fabric_nic_models` names one link per model instead. They are one field, so a count and a list cannot disagree. `first_qemu_only_nic` replaces the `nic_model.requires_qemu()` reads. An emulated model needs QEMU wherever in the machine it appears, and it returns the device rather than a bool so the error can name the one that forced the backend. E1000/E1000E join the kernel-feature table because their absence is quiet: a kernel without the driver still gets the device and binds nothing, which reads as "the model never reached the guest". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 2a57e1dbd9a7c4e5629e739845c1c8277155938f) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
An argument records what was asked for. A hypervisor that declined it, or accepted it and built something else, leaves every unit test green -- so these read `/proc/meminfo`, `/proc/cpuinfo`, `/proc/version` and `/sys/class/net` in the guest instead. The QEMU-requiring ones name `kernel_profiles::QEMU` rather than pinning `RequestedBackend::Qemu`. A pinned backend the run's profile does not offer resolves to a skip, and a skip is reported as a pass: written that way, four of these asserted nothing at all and said `ok`. Naming the profile is also what the declaration means -- the test is for that machine. The mixed-model test reads the bound driver rather than the MAC. Every MAC is derived the same way whatever the model is, so a VM that presented one device three times would still show three distinct addresses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 35a32a118ee6c2fad1b7b09ea3fc0264dfec4da6) Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`#[n_vm::test]` answers `cargo bolero list` on the host through bolero's own `TargetLocation`, which needs `bolero/std` in the graph rather than the default feature set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`just fmt --check` -- what CI runs as `ci::check-fmt` -- rejects `n-vm/src/bin/n-vm-reap.rs` and `n-vm/src/cloud_hypervisor/events.rs`. `rustfmt.toml` is byte-identical to `main`'s, so this is authoring-time drift against a different rustfmt, not a config disagreement, and it would have failed wherever these commits landed. Whitespace only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`just nixfmt`, which CI runs as part of `lint`, rejects the file the n-vm derivations were added to. Whitespace only: two `inherit (...)` lists that now exceed the line budget, one `RUSTFLAGS` concatenation, and a stray blank line. No evaluation change -- the `testroot`, `vmroot` and `devroot` derivation hashes are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`rust-no-direct-std-sync-import` reports 9 findings across `n-vm` and `n-it`, and the fix it asks for is the one change these crates must not make. The harness depends on no dataplane crate -- that is what makes returning it to `githedgehog/testn` a code move rather than a decoupling project -- and taking `dataplane-concurrency` would be exactly that coupling. It would also buy nothing. The facade exists so `loom`/`shuttle` can swap the lock backend under the model checkers; none of this code is under either, and none of it is on the dataplane's packet path. It is host-side test infrastructure that spawns hypervisors. `n-it/src/vsock_writer.rs` already carries a comment reaching the same conclusion by hand. Break-tested: with the exclusion in place, adding `use std::sync::Arc;` to `nat/src/lib.rs` still fails the scan, so the rule is narrowed rather than disabled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`cargo deny check` fails on RUSTSEC-2023-0081: safemem is unmaintained, archived by its author in 2019. It reaches the workspace only as safemem <- base64 0.7 <- cloud-hypervisor-client 0.3 <- n-vm, and n-vm is a dev-dependency everywhere it is used, so this never reaches a shipped artifact. The advisory is "unmaintained" rather than a vulnerability, and cargo-deny reports no safe upgrade. cloud-hypervisor-client 0.6 does drop base64 0.7, so the durable fix is that bump -- but it is a port rather than a version change: 0.6 removes `PlatformConfig::iommu_address_width` and moves `console_config::Mode`, both of which n-vm uses to configure the guest vIOMMU. Doing that inside an import commit series would mean changing behaviour the rest of the series asserts, so it is left as follow-up and the advisory ignored with the reason recorded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`origin/main` depends on n-vm as `git+https://github.com/githedgehog/ testn.git?tag=v0.0.10`, and `cargoVendorDir` pins its hash so crane fetches it as a fixed-output derivation. The absorption replaced that dependency with path deps, so `Cargo.lock` no longer names testn and the pin is dead. Left in place it is worse than clutter: the comment above `outputHashes` records that a key which stops matching only *warns*, so a stale entry is exactly the kind of thing that survives unnoticed and misleads the next person to read the list for what the workspace actually fetches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The guest-booting tests do not skip without `testroot`/`vmroot` -- they panic: `n_vm::container::scratch_root_resolve` reports "could not find testroot/vmroot in the working directory or any parent, and N_VM_TEST_ROOT/N_VM_VM_ROOT are not set". So the roots are a precondition of any run that includes them, and the recipe that starts the run is the only place that can guarantee it. The harness gave `(setup-roots)` to `test` alone. Four recipes in this file invoke nextest, and the other three had nothing: - `coverage` had the `N_VM_TEST_ROOT` exports but no dependency, so its guard never fired. - `coverage-archive` had neither -- and it is the one CI actually runs: `ci::coverage` calls it, not `coverage`. It builds the archive in nix and then runs the tests on the runner, so it needs exactly what `test` needs. - `test-each` had neither, and its per-package archives include `dataplane-n-vm`'s suite. Measured on #1800: `check/debug` and `coverage/debug` reported an identical 28 failures, which is what a missing precondition looks like when three of four entry points are missing it and the fourth is the one you fixed. `ci.just` is the map worth keeping in mind here: `check-test`, `sanitize`, `shuttle`, `loom` and `cross-test` all funnel into `test`, so they inherit this; `coverage` and `test-each` do not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`n-vm-macros`' 49 compile-fail fixtures fail in CI with
error: no matching package named `proc-macro2` found
location searched: crates.io index
note: offline mode (via `--offline`)
and pass on every developer machine. trybuild builds each fixture by
writing a scratch cargo project under `target/tests/trybuild` and shelling
out to `cargo` with an unconditional `--offline` (trybuild's own
`cargo.rs`). That resolve reads `$CARGO_HOME/registry`, not the nix vendor
directory the workspace itself was built from -- and a CI job that builds
entirely through nix never populates it, while a developer's is warm from
ordinary use.
`cargo fetch --locked` fills it, in all four recipes that run nextest.
Every download is checksum-verified against `Cargo.lock`, so this pins
exactly what the nix build already pinned, and on a warm cache it resolves
locally and does nothing.
The alternative -- ignoring the test under CI -- would drop the only
coverage the macro's diagnostics have, which is the coverage most worth
having for a proc macro whose failure mode is a confusing error message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A bind mount's `source` is resolved by the Docker daemon, in the daemon's
mount namespace. Its `target` is not. The two agree when the daemon runs on
the same host and stop agreeing when the tests run inside a container that
talks to a daemon outside it -- which is how the dataplane's CI runners are
built. Measured on one: `/nix/store` holds 7386 entries as this process sees
it and 0 as the daemon does, while `/home/runner/_work` is the same block
device at the same path on both sides.
The failure that produces is not the one you would expect, because every
mount here sets `create_mountpoint`. A source the daemon cannot find is
therefore *created*, as an empty directory, rather than reported. So
`/vm.root` mounted empty and read-only, and the first mount beneath it died
making its own mount point:
error mounting ".../dataplane" to rootfs at "/vm.root/workspace":
... mkdirat .../vm.root/workspace: read-only file system
naming `workspace` -- a directory the real `vmroot` derivation pre-creates
precisely so that this cannot happen. All 28 guest-booting tests failed this
way, and the error pointed at the mount that could not be made rather than
at the one that had silently gone wrong.
`N_VM_HOST_SHARE_DIR` names a directory both sides can see. When it is set,
`/nix/store/...` sources are rewritten to `<share>/nix/store/...` and the
forwarded environment is written under `<share>/tmp` instead of
`std::env::temp_dir`; targets are untouched, so rpaths still resolve in the
guest. Unset -- a daemon on this host, which is every developer machine --
nothing is rewritten and the mounts are exactly what they were.
The environment directory is the same bug and the worse one: a
container-local `/tmp` is no more visible than `/nix`, and the daemon would
have answered by creating an empty directory, so the guest would come up
with no environment at all. The doc comment on `write_forwarded_env` already
says that is a loss nothing downstream can report.
`host_share_dir` canonicalises, because the result reaches the Docker API as
an opaque absolute path: a `..` this process resolves without noticing would
not be resolved there.
Verified against a real daemon, booting real guests: the whole `n-vm`
integration suite passes with the share configured exactly as CI configures
it (a non-canonical path beside the checkout) -- 22 guests booted and the
5 QEMU-profile skips that `N_VM_SKIP_LOG` reports with or without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`build_scratch_mounts` mounts each first-level `testroot` entry at `/<name>`,
and one of them -- `n-vm-manifest.json` -- is a symlink into the kernel
image's store path. Mounted as-is, the daemon is handed a source whose target
is an absolute `/nix/store` path that means nothing in its namespace, so the
stat fails, `create_mountpoint` decides the source is missing, and the mkdir
that follows hits the symlink that is plainly there:
error while creating mount source path
'.../nix/store/...-dataplane-test-root/n-vm-manifest.json':
mkdir ...: file exists
Canonicalising first hands over the store path the link resolves to, which
the share export already contains. Unchanged where no share is configured: a
read-only bind of a symlink and of its target are the same mount.
The share is now threaded into `build_mounts`/`build_scratch_mounts` rather
than read from the environment inside them. That is what the other half of
this commit is about: `N_VM_HOST_SHARE_DIR` is process-wide state that
changes every mount source, so `scratch_mounts_include_nix_store` and
`scratch_mounts_include_vm_root` -- which assert a source -- passed here and
failed in CI, where the variable is set for the whole job. A test cannot
assert a source correctly while the answer depends on ambient environment.
Both now say which mode they mean, and a new test pins the property that
actually has to hold in either: sources move under the share, targets never
move, and the count is the same.
Verified with `N_VM_HOST_SHARE_DIR` set, which is how CI runs and is the
coverage gap that let both of these through: 295 n-vm unit tests, 60
n-vm-protocol unit tests, and the 27-test integration suite all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`vm_boots_with_host_hugepages` is the last test failing on the runners:
× hugepage pool unavailable
╰─▶ the 1073741824-byte hugepage pool has 0 free page(s); this VM needs 1
It asks for a 1 GiB *host* page because that is what makes the guest's memory
physically contiguous, which is the only thing DPDK-through-an-IOMMU can tell
apart. Nothing else asks -- `HostPageSize`'s docs record that the default
deliberately leaves this pool alone, after a version that did not made ten of
eighteen tests contend for a page they had no use for.
The runner cannot reserve it: measured on one, `CapEff` is zero, so writing
`/sys/kernel/mm/hugepages` is not available to it. It can ask the host's
daemon for a privileged container that can, and the hugepage sysfs is not
namespaced, so the reservation lands on the machine and every job on it sees
the result. Verified against a local daemon that a privileged container's
write does reach host sysfs.
Only ever raises the count. Lowering it would take pages away from whatever
else is running on a shared machine, and four is a floor rather than this
job's private allocation.
Never fails the job. A 1 GiB reservation is a request: the kernel has to find
that many physically contiguous gigabytes and on a long-lived machine it may
not. If it comes up short, the test that needs a page reports it precisely,
which is a better place to read it than a setup step -- so this warns and
carries on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The reservation reached the host and the kernel declined it: "asked for 4
1 GiB hugepages, got 0 (was 0)". That is the expected answer rather than a
malfunction -- a gigabyte page needs a physically contiguous gigabyte, and a
machine that has been up a while rarely has one, which is why the kernel
documentation reserves them at boot.
Compaction is the one cheap thing worth trying first: it migrates movable
pages to free contiguous runs, takes a few seconds, and disturbs nothing else
on the machine. Dropping caches would probably help more and is not worth it
-- it takes the page cache from every other job on a shared runner.
When that still is not enough the warning now names the durable fix, a boot
parameter on the runner host, rather than leaving the reader to infer it:
default_hugepagesz=1G hugepagesz=1G hugepages=4
Still never fails the job. `vm_boots_with_host_hugepages` reports the missing
page precisely, and that is the right place to read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`vm_boots_with_host_hugepages` is the last test failing on the runners, and it is not going to pass there soon: a 1 GiB page has to come from a physically contiguous gigabyte, and these machines cannot produce one. Measured over two runs -- a privileged container does reach the host's `nr_hugepages`, and the kernel answers 0 both before and after compaction. Reserving them wants `default_hugepagesz=1G hugepagesz=1G hugepages=4` on the host's command line, and a reboot. So the 1 GiB pair is `#[ignore]`d and a 2 MiB pair added beside it. `#[ignore]` rather than n-vm's own skip, deliberately. A skip is the right answer to a genuine mismatch between a test and the machine it was handed -- "cloud-hypervisor cannot emulate aarch64" is permanent and true everywhere. This is not that: the machine could run it, given a boot parameter. An ignore says "not yet", stays visible in the run summary, and comes back with `--ignored` on a host that has the pages. What the 2 MiB version does not cover is narrow and worth naming: only DPDK driving a device through an IOMMU can tell a contiguous gigabyte from 512 contiguous megabytes. Everything between here and there -- asking for hugepage backing at all, `memfd` with `MFD_HUGE_*`, the pool accounting, both VMMs' plumbing -- is the same code, and that is what stays covered. `ci::reserve-hugepages` now reserves 2 MiB pages, 1024 of them: a 1024 MiB guest backed by 2 MiB pages needs 512, and `check` and `coverage` can be on one machine at once. Verified locally: `vm_boots_with_host_hugepages_2m` boots a guest, the 1 GiB pair reports as ignored with its reason, and `N_VM_SKIP_LOG` confirms `..._2m_on_qemu` skips under the default profile exactly as the 1 GiB one it replaces did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
…uest honestly Two things the runners found once the tests stopped failing for other reasons. Neither had been seen before, because these crates have never been through this workspace's docs build or its coverage job. **33 rustdoc errors across `n-it` and `n-vm`.** `check-docs` builds with `-D warnings`, and the imported crates carry three kinds of broken link: - links into crates they do not depend on (`n_vm::run_in_vm` from `n-it`, which depends on `n-vm-protocol` and not on `n-vm`), - links from *public* documentation to *private* items -- `run_test_in_vm`'s summary steps through five private methods, and a reader of the public docs cannot follow any of them, - links whose target was simply missing a path: `HypervisorVerdict`, `kernel_profiles`, `KvmNotAccessible`, `ENV_PROFILE`. Fixed by kind rather than silenced: a real path where the item is reachable, a plain code span where it is not. A code span still names the thing; a link that cannot resolve is worse than no link. **A coverage guest needs more than 60 seconds.** `a_vm_boots_the_kernel_profile_it_named` overran the KVM allowance at 96s in the coverage job, and how it reported is the point: not "slow" but "no parseable test verdict from guest", because the timeout shoots the VM and the verdict dies with it. Coverage is not a small tax on a guest -- the whole suite went from 76s to 155s on the same runners. `N_VM_OVERHEAD_SCALE` multiplies the allowance, and the coverage job sets it to 3. Unset, unparseable, non-finite or non-positive all mean 1.0, so an ordinary run is untouched and a mistyped multiplier is not the reason a suite fails to run. A run-time knob rather than a `cfg`, for two reasons. The one this crate already argues for itself, at `ENV_VIRTIOFS_CACHE`: rebuilding `n-vm` to change a timeout also changes the binary under test. And the one that matters for where this crate is going -- a `cfg` would have to name dataplane's `instrumented`, which means nothing to another consumer. (`cfg(instrumented)` was tried first and `just clippy` rejected it as an unexpected condition name, which is its own argument: the flag that registers it is this workspace's, not n-vm's.) Verified with the gates that actually gate: `just clippy` and `just docs`, both through nix, and the integration suite green with the scale unset and at 3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`N_VM_OVERHEAD_SCALE` was set on the host and read in the container, and a container's environment is the explicit list in `build_config` and nothing else. So the coverage job showed the knob working and not working at once: its unit tests saw 180s, while the guest they launched was still shot at 60s and reported "no parseable test verdict from guest". Forwarded now, beside `N_VM_ENGINE_TIME_LIMIT`, which is in that list for exactly this reason -- the tier that consumes it never sees the invocation that chose it. The same fix to the tests, and it is the second time this shape has bitten: `vm_overhead_allowance` and `vm_test_timeout` read process-wide state, so `an_ordinary_test_gets_what_it_always_got` and `declared_work_is_added_to_the_allowance` passed here and failed in CI, where the variable is set for the whole job -- exactly what `N_VM_HOST_SHARE_DIR` did to the mount-source assertions. The scale is now a parameter (`vm_test_timeout_with`), the assertions name the scale they mean, and a new test pins that the multiplier multiplies and that 1.0 changes nothing. `the_overhead_scale_is_carried_into_the_container` covers the forwarding itself, which is the part no unit test could have caught before: it is a property of the container's environment, not of any function's return value. Verified with `N_VM_OVERHEAD_SCALE=3` set for the whole process, which is how CI runs it: 297 unit tests and the 27-test integration suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The 2 MiB reservation got zero on one runner -- "asked for 1024 2048kB hugepages, got 0 (was 0), even after compaction" -- and the recipe threw away the only thing that could explain it. `priv()` sent stderr to /dev/null, so a refused write and an unmet allocation looked identical, and neither is actionable from a count. Now the write's stderr is reported when it comes up short, along with `MemFree`/`MemAvailable`/`HugePages_*` and the pools the kernel actually has. Those tell the two cases apart: a write the daemon would not perform, versus memory that was not there. Also drops the explicit `-v /sys:/sys`. `--privileged` already gives a read-write sysfs, and sysfs is not namespaced, so that is the host's -- verified against a local daemon by writing the existing value back. The extra bind only added a mount whose mode depends on how the daemon is configured, which is one more thing that can differ between the runner where this worked and the one where it did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Raising the VM's own timeout uncovered a second deadline behind it. The coverage job's guest now survives to 98s instead of being shot at 60s, shuts down, and then all four vsock channels miss a fixed 5s drain window -- because an instrumented guest writes its coverage profile on the way out. Cutting that short is not a cosmetic loss. The verdict arrives on one of those channels, so a guest that passed is reported as "no parseable test verdict from guest", which reads as a broken harness rather than a slow one. Same knob, same reason: `DRAIN_TIMEOUT_BASE` times `N_VM_OVERHEAD_SCALE`. A test pins that the two move together, since raising only one of them is precisely the mistake this fixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
…n CI
Reserving hugepages from CI worked mechanically and cost more than it bought.
A privileged container does reach the host's `nr_hugepages` -- the daemon is
on bare metal and hugepage sysfs is not namespaced -- but:
- 1 GiB pages need a physically contiguous gigabyte the runners could not
find. Zero allocated, before and after compaction. That wants
`default_hugepagesz=1G hugepagesz=1G hugepages=4` at boot.
- Pinning 2 MiB pages took memory away from everything else on a shared
runner. Measured across four coverage runs: the suite went from 76s to
155-198s, and `a_vm_boots_the_kernel_profile_it_named` from **3.06s
passing** to 96-139s failing, its guest producing nothing on any channel.
Nothing else changed between the last good run and the first bad one.
That last point also corrects the reasoning behind `N_VM_OVERHEAD_SCALE`.
The 76s -> 155s slowdown was attributed to coverage instrumentation when it
was in fact this reservation; run 6 did the same instrumented suite in 76s.
The knob is kept -- it guards a kill deadline, where headroom is cheap -- but
its comment now says what is actually true.
So the tests are gated rather than deleted or skipped. They are valid tests
of the path that matters, and they pass on a host that reserves pages:
RUSTFLAGS='--cfg=host_hugepage_tests' cargo test -p dataplane-n-vm --test integration
Not a skip, because n-vm's skips are for mismatches that are permanent and
true everywhere -- "cloud-hypervisor cannot emulate aarch64". A host without
a pool could have one; that is an arrangement CI has not made, not a
property of the machine.
`check-cfg` is declared in `n-vm/Cargo.toml` rather than the workspace's
RUSTFLAGS, so it travels with the crate: n-vm opts into no workspace lints,
and that is what keeps returning it to `githedgehog/testn` a code move rather
than a decoupling project.
Verified: ignored by default, and all four run and pass under the cfg on a
host with pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
TEMPORARY, to be dropped before this PR is ready. On a workstation, every capability the Docker container tier provides is available to an unprivileged process through a user namespace -- measured, not assumed: `unshare -Urnm` succeeds, `CapEff` inside is the full set, a tap device can be created, `/nix/store` can be bind-mounted, `pivot_root` succeeds, and all four VMM devices are 0666 and openable. If that holds on the runners too, the whole class of bug this PR has been fighting -- a daemon in another mount namespace reinterpreting our paths -- stops existing rather than being worked around. Two things only the runner can answer. Whether a *nested* user namespace can be created from inside an unprivileged container, which depends on its seccomp profile and on `max_user_namespaces`. And whether `/dev/kvm`, `/dev/vhost-vsock`, `/dev/vhost-net` and `/dev/net/tun` are present *here*: Docker maps them into the n-vm container today, and without Docker the VMM would run in this container instead and would need them at this level. If they are absent, the approach needs a runner-spec change regardless, which is worth knowing before designing around it. `continue-on-error`, and the body was run locally first, so this reports rather than gates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Reverts 40c06a9. The probe answered its two questions; leaving a diagnostic step in the check job costs every future run for nothing. Both answers were no, and independently so: unshare -Ur: NO -- write failed /proc/self/uid_map: Operation not permitted /dev/kvm crw-rw---- root:kvm open rw: NO /dev/vhost-vsock crw-rw---- root:kvm open rw: NO /dev/vhost-net crw-rw---- root:kvm open rw: NO /dev/net/tun crw-rw-rw- root:root open rw: yes The first is narrower than it looks: `unshare(CLONE_NEWUSER)` itself succeeds, so namespaces are not disabled -- only the identity mapping is refused. The device result is the more fundamental of the two, and it explains why the container tier exists on a runner at all. Docker is what gives an unprivileged CI process access to `/dev/kvm`, by running the VMM as root in a container it hands the devices to. A namespace grants capabilities; it does not grant group membership. So collapsing the tiers is a runner-spec change rather than a code change, and not one to start until the runners can support it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
TEMPORARY, to be dropped once the answer is recorded. The unprivileged single-line mapping needs no capability -- it works on a workstation whose `CapEff` is also zero -- so the runners are actively refusing it, and the remedy differs entirely by cause. This gathers the fields that tell those causes apart, so the question put to whoever administers these machines can be a specific one rather than "please enable user namespaces". What each answer means: - `uid_map` other than `0 0 4294967295` -- the runner is already inside a user namespace, and a nested unprivileged mapping is restricted from there. Remedy is subuid/subgid ranges plus `newuidmap`, not a sysctl. - `Seccomp: 2` -- a filter is active and may be refusing the call. - `apparmor_restrict_unprivileged_userns: 1` -- Ubuntu 23.10+ mediation; a sysctl or a profile granting `userns create`. - `newuidmap` present with `cap_setuid` plus a subuid range -- the rootless path is already available and n-vm would use it instead of `unshare -r`. It also reports whether the runner user is in `kvm`, which is the other, independent blocker, so both can go in one request. `unshare -U` and `-Ur` are run separately: creation and mapping are distinct refusals, and only the pair distinguishes them. An earlier version of this probe tried to split the two writes that `-r` performs, and was dropped for failing on a machine where the thing it was diagnosing works -- a process inside a namespace with no mapping yet is nobody and cannot write even its own procfs files. `continue-on-error`, and the body was run locally first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Reverts 6107021. The cause is identified and recorded below; a diagnostic step in the check job costs every future run for nothing. uid=1001 user=runner groups=runner sudo users docker in kvm group: NO uid_map: 0 0 4294967295 <- not inside a user namespace Seccomp: 0 <- no filter apparmor profile: unconfined user/max_user_namespaces: 770638 kernel/unprivileged_userns_clone: 1 kernel/apparmor_restrict_unprivileged_userns: 1 <- the one restriction newuidmap: not installed /etc/subuid: runner:231072:65536 <- ranges already provisioned unshare -U (create only): ok unshare -Ur (create+map): FAILED -- write failed /proc/self/uid_map: EPERM Every other gate is permissive, and the two theories that would have implied a harder fix are both excluded: the runner is not already inside a user namespace, and no seccomp filter is active. `apparmor_restrict_unprivileged_ userns=1` is Ubuntu 23.10+ mediation, and creation-succeeds-mapping-refused is its signature. Two remedies, and the second is the smaller ask: relax the sysctl, or install the `uidmap` package. `/etc/subuid` and `/etc/subgid` already carry a range for `runner`, so somebody provisioned for rootless tooling and only the `newuidmap`/`newgidmap` helpers are missing; those carry `cap_setuid` and write the map through the privileged path rather than the restricted one. Also worth recording: `runner` is in `sudo` and `docker`, but not `kvm`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
daniel-noland
force-pushed
the
pr/daniel-noland/stats-collector
branch
from
September 6, 2026 06:15
a942fd1 to
c2d6db2
Compare
daniel-noland
force-pushed
the
pr/daniel-noland/n-vm
branch
from
September 6, 2026 06:15
c4f1428 to
2504cf2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Absorbs the in-VM test harness —
n-vm,n-it,n-preinit,n-vm-macros,n-vm-protocol, plus the guest kernel build — fromgithedgehog/testninto theworkspace, replacing the
n-vm = { git = testn, tag = v0.0.10 }dependencymainalready carries with path dependencies, and renaming the attribute#[n_vm::in_vm]→#[n_vm::test].Draft,
dont-merge. Opened now so the payload is reviewable and CI has anopinion, not because it is ready to land.
Why this is a sibling of the stack rather than its tip
These 60 commits were authored on top of the fuzzing campaign and have been
sitting at the top of the 16-PR chain (#1796 … #1774), where they block on every
PR below them. Nothing in the chain depends on n-vm — the chain was deliberately
ordered so that everything n-vm-independent sits below it — so the dependency
only ever ran one way, and it turns out it barely runs at all.
Measured with
git merge-tree --merge-base=origin/mainagainst all 13 chainbranches:
Cargo.toml, one hunkThat hunk is this PR's
command-fdsline sitting directly above thecriterionentry #1759 edits.
justfile,default.nix,nix/overlays/dataplane-dev.nix,.config/nextest.toml,mgmt/src/tests/mgmt.rsandCargo.lockallauto-merge. So the two lines can land in either order.
What is actually here
114 files, +22.3k/−140 — but only 11 files and +1026/−140 outside the five new
crates and
nix/pkgs/:Cargo.toml/Cargo.lock/.gitignore— workspace wiring.config/nextest.toml— the serialisedvmtest group and its package listdefault.nix(+590) —guest-kernels, kernel manifest, initramfs,testroot,vmroot,n-preinit-static, qemu selectionnix/overlays/dataplane-dev.nix(+119, purely additive) —mkLinuxFancyandthe
linux-fancy/linux-fancy-modularguest kernelsjustfile—setup-rootsgainstestroot/vmroot; areaprecipein_vm→testattribute renameThe five crates depend on no dataplane crate and opt into no workspace lints.
testrootis cloud-hypervisor + virtiofsd + qemu + kernel;vmrootis then-itbinary +n-preinit-static+ guest mount points. Neither containsdataplane code, which is what makes the eventual move back out to
testntractable.
Verification
cargo check --workspace --all-targets— cleancargo clippy --workspace --all-targets— cleanjust fmt --check— cleannix-instantiate default.nix -A testroot -A vmroot -A devroot— all evaluaten-vm,n-it,n-preinit,n-vm-macros,n-vm-protocol,nix/pkgs/{linux,flatcar,ubuntu}are byte-identical to the pre-rebase block tipDeviations from the original block, and CI fixes
The first CI run failed 29 tests,
lint, and DCO. Fixed on top of the import:just coveragenow depends on(setup-roots).just testalready did;coveragehad only theN_VM_TEST_ROOT/N_VM_VM_ROOTexports, so its guardnever fired. Without the roots the tests do not skip —
n_vm::container::scratch_root_resolvepanics — so both jobs reported thesame 28 failures.
cargo fetch --lockedbefore the nextest archive run.n-vm-macros'49 compile-fail fixtures fail in CI only: trybuild shells out to
cargo --offlinefor a scratch project, which reads$CARGO_HOME/registryrather than the nix vendor dir, and a nix-only CI job never populates it.
n-vm/n-itexempted fromrust-no-direct-std-sync-import. The fixthat rule asks for — taking
dataplane-concurrency— is the one couplingthese crates must not have if they are to go back to
testn. Break-tested:the rule still fires for
nat/src/lib.rs.safemem, unmaintained) ignored. Reaches us only viabase64 0.7 <- cloud-hypervisor-client 0.3 <- n-vm, dev-dependency only.cloud-hypervisor-client0.6 drops it but also dropsPlatformConfig::iommu_address_width, which n-vm uses — a port, not a bump.nixfmtondefault.nix,rustfmton twon-vmfiles. Whitespace; thenix reformat leaves every derivation hash unchanged.
testnvendor hash removed fromdefault.nix'soutputHashes—
Cargo.lockno longer names it, and a stale key only warns.The
fuzz_cleanjustfile lever is still dropped fromfeat(n-vm): give a fuzz target somewhere to put what it finds: it readsfuzz_corpus_root, whichexists only on the campaign line. The commit's real payload — the
n-itmountlayout and
CorpusPolicy— is intact.fuzz_cleangoes back once both lineshave landed.
Open question: do the lab runners boot guests?
No workflow names
n-vm, butci::check-testreachesjust test, which nowbuilds
testroot/vmrootand runs 28 guest-booting tests. Building the rootsis ~1050 derivations on a cold cachix. Whether the tests then pass depends on
the runners having
/dev/kvm, docker, and a 1GiB hugepage pool — which thisrun is the first to find out. If they do not, the fallback is to give the
vmnextest group a CI-only filter rather than to drop the roots.Supersedes
n-vm-again, CONFLICTING since 2026-06-30)origin/pr/daniel-noland/n-vm-tail(pushed, never had a PR)🤖 Generated with Claude Code