Skip to content

fix(xcat-core): the Ubuntu Genesis image is built from the build host kernel - #7835

Open
dhilst wants to merge 10 commits into
xcat2:masterfrom
VersatusHPC:fix/ubuntu-genesis-native-sbuild
Open

dhilst wants to merge 10 commits into
xcat2:masterfrom
VersatusHPC:fix/ubuntu-genesis-native-sbuild

Conversation

@dhilst

@dhilst dhilst commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

dracut copies the kernel, the modules and every command out of the root it runs in. The Ubuntu
Genesis deb was built once on the build host, so every Ubuntu release received the build host
kernel, and the pipelines converted the EL rpm with alien instead. builddebs.pl gains
--genesis, --genesis-only and --genesis-dist, and builds the deb once per codename inside
that codename's sbuild chroot. builddeb-genesis-base takes --expect-codename and stops in the
wrong root. genesis_log_errors fails the build on an error line a zero exit status hides.

…t kernel

builddebs.pl carries no Genesis step. The Ubuntu Genesis deb is produced by
xCAT-genesis-builder/builddeb-genesis-base, which nothing calls, and that script
reads /lib/modules and /boot of the host it runs on. One build therefore gives
every Ubuntu release the kernel and the modules of the build host.

The script also has no payload gate. dracut reports a command it cannot install
with a FAILED: line and exits 0, so the build that shipped the image with no
dhclient reported success. Its build root omits isc-dhcp-client, ifenslave and
util-linux-extra.

genesis_deb_per_codename.t asserts the chroot each codename builds in, the log
lines that mean a build failed although it exited 0, and the refusal of the
builder to run in a root of another release. genesis_payload_verification.t
drives verify-genesis-payload against payload trees with a known hole.
genesis_ubuntu_build_root.t reads the mandatory commands back from the dracut
module and asserts the build root supplies each one.

All three fail on this commit.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
… kernel

dracut copies the kernel, the kernel modules and every command out of the root
it runs in. The Ubuntu Genesis deb was built once, on the build host, so every
Ubuntu release got the build host's kernel. Nothing called the builder at all:
the pipelines converted the EL rpm with alien instead.

builddebs.pl gains --genesis, --genesis-only and --genesis-dist. The Genesis
deb is now built once per codename inside that codename's <codename>-<arch>-sbuild
chroot, the chroots xcat-dep's sbuild-all.pl already creates on the Ubuntu build
host. builddeb-genesis-base takes --expect-codename and stops when the root it
woke up in is a different release, so a build on the build host cannot produce a
codename's image.

builddebs.pl reads the build log through XCAT::BuildUtils::genesis_log_errors and
fails the build on FAILED:, a package apt cannot find and four more lines that a
zero exit status hides. The extracted payload goes through verify-genesis-payload
with the command list read back from the dracut module, plus the DHCP client, the
97xcat hooks and a /lib/modules that holds this chroot's kernel and no other.

The build root gains isc-dhcp-client, ifenslave and util-linux-extra, which
supply dhclient, ifenslave and hwclock.

genesis_deb_per_codename.t, genesis_payload_verification.t and
genesis_ubuntu_build_root.t fail without this change.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
…mage needs

The first native build stopped on the payload lint. The Ubuntu dracut module
installs ping, nc, nslookup, sfdisk, mkfs.btrfs, usb.ids, poweroff, reboot and
shutdown unconditionally, and no package in the build root supplies any of them.
dracut reported each one with a FAILED: line and exited 0, which is how the image
that shipped without dhclient was packaged. Add iputils-ping, netcat-openbsd,
fdisk, btrfs-progs, hwdata and systemd-sysv.

Two names change between releases. util-linux-extra appeared in 23.04, so the
jammy build stopped with "E: Unable to locate package util-linux-extra" where
hwclock is still in util-linux; bind9-dnsutils replaced dnsutils in 22.04.
add_first_available installs the first name apt knows and stops the build when a
release carries none of them.

dch reads debian/control from the working directory, not from the file it writes,
so the build directory has to be the working directory before it runs.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
…t boot

builddebs.pl publishes every Architecture:all deb into every release, because
every other xcat-core deb is the same file for all of them. The Genesis image is
not: it carries the kernel of the root that built it. All three images therefore
landed in all three suites, and apt serves the newest, which belongs to another
release.

deb_belongs_to_dist reads the codename back from the version and keeps an image
out of any other suite. A deb with no codename in its version is unaffected.

The jammy build also stopped on /usr/share/terminfo/l/linux and v/vt100:
ncurses-base installs terminfo under /lib, and the module asks for the
/usr/share copies, which come from ncurses-term. noble happened to have it.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
…ectory

The sbuild chroots bind-mount /var/lib/sbuild/build on /build, so every session
of every chroot sees the same directory. Staging the builder there let two builds
running at once overwrite each other's copy: one of them read a half-written
script and stopped with "-get: command not found". The output of an earlier run
stayed in it too, so a build collected another codename's deb and left 1.9 GB on
the build host.

Stage under /xcat-genesis-build instead. It lives in the session overlay and goes
away with the session.

26.04 also moved the backward-compatibility zone names out of tzdata into
tzdata-legacy, and the dracut module names 106 of them. Install it where apt has
it.

The three images (jammy 180 MB, noble 328 MB, resolute 432 MB) build with no
error in any log.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
…nd its comments over-explain

genesis_ubuntu_build_root.t called BAIL_OUT at four places where an
extraction stopped matching. prove stops every remaining file on a
bail-out, so one stale regex in this file hides the results of the tests
that would have run after it. die is just as loud and costs only this
file.

The branch also states one fact in four places. That dracut copies the
kernel out of the root it runs in appears in the builddebs.pl header, in
its Genesis section, in BuildUtils.pm and in the builder, each time with
the incident that produced it. The chroot stage directory is explained
twice, once at its declaration and again at its only use. Each fact now
stands where the reader meets it, without the bug report around it.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The header of genesis_payload_verification.t said the fixtures reproduce
"the three holes the released legacy Genesis image shipped with", while
verify-genesis-payload's own header counted four. A count of past
incidents is not what the fixtures are; each leaves out one thing the
image needs. The wording now matches release/2.19-rc1, which carries the
same file.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
…eads is missing

genesis_ubuntu_build_root.t called plan skip_all when xCAT-genesis-builder/builddeb-genesis-base was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.

Die instead, which is what makentp_ntp_deps.t already does for setupntp.

With xCAT-genesis-builder/builddeb-genesis-base moved aside the file now exits 2 and prints "builddeb-genesis-base not found";
before this change it exited 0 and printed "1..0 # SKIP builddeb-genesis-base not found". With the file
present the test passes either way.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>

@viniciusferrao viniciusferrao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Daniel, some details here:

  1. Could we limit deb_belongs_to_dist() to the Genesis packages?
    It currently treats any trailing ~... in a package version as
    a codename. With --release 1~rc1, ordinary xCAT packages are
    silently excluded from every suite, even without --genesis.
    A non-Genesis prerelease case would help cover this.

  2. The default Genesis plan includes focal, but the package requires
    debhelper-compat (= 13) and a stock Focal chroot provides compat
    12. Could we either arrange the newer build dependency there or
    leave Focal out of the Genesis defaults? Otherwise a plain
    --genesis run stops on its first release.

…y suite

deb_belongs_to_dist reads any trailing ~word in a package version as the
codename the deb was built for. Debian uses ~ for a prerelease, and --release
takes whatever the caller gives it, so `--release 1~rc1` puts one in every
package name:

  xcat-client_2.19.0-1~rc1_all.deb   noble=0 focal=0

The deb is then published into no suite at all, with --genesis nowhere in the
command, and the run still reports how many packages it published.

Only the Genesis image is built per codename, so only it can be excluded by
one. The new cases assert a prerelease version reaches every suite, that a
Genesis SCRIPTS deb is not treated as the image, and that the image itself is
still confined to its own suite when its version carries both.

The second half asserts genesis_dists(), which does not exist yet: a plain
--genesis run takes the release list the rest of the build uses, and focal is
on it. focal ships debhelper 12.10 -- measured on the focal management node --
against the package's debhelper-compat (= 13), so sbuild stops on the build
dependencies and the run ends on its first release.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
… suite

deb_belongs_to_dist read any trailing ~word in a package version as the
codename the deb was built for. Debian uses ~ for a prerelease and --release
takes whatever the caller gives it, so `--release 1~rc1` left every package
excluded from every suite, with --genesis nowhere in the command and the run
still reporting what it published. Only xcat-genesis-base carries a codename,
because only it carries a kernel, so only it is asked.

genesis_dists() names the releases a Genesis image can be built on. A plain
--genesis run took the release list the rest of the build uses, and focal is on
it: focal ships debhelper 12.10 and the package declares debhelper-compat
(= 13), so sbuild stopped on the build dependencies before dracut ran and the
run ended on its first release. The default plan now leaves focal out and says
so. A release named with --genesis-dist is still built as asked.

genesis_deb_per_codename.t covers both. Applying the codename rule to every
package again fails five assertions; a genesis_dists that filters nothing, or
an image that loses its own rule, fails two each.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
@dhilst dhilst modified the milestones: 2.19, 2.19.1 Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants